Sifting the Evidence
The importance of an organising structure
Systematically grading the evidence allows us to give more weight to better quality. Academic evidence becomes more prominent as you move towards the apex.
The point of using the framework is not to say that evidence at level 1 is invalid, although once you start relying on anecdotes, personal experience or gut feeling, evidence is subjected to more bias.
7
6
5
4
3
2
1
Evidence Level 7×
Systematic Reviews: Considered by academics to be the platinum standard
Advantages/Claims
Pull together in a systematic and objective way all the best quality evidence relevant to the study in question.
Disadvantages/Counter–Claims
Critics (Wilde, 2017; Gifford, 2016) argue that systematic reviews can be problematic because they can effectively deny the impact of bias and encourage an inappropriate subordination of other forms of evidence.
Evidence Level 6×
Meta-analysis: Considered by academics to be the gold standard
Advantages/Claims
A way of combining results of multiple studies to provde a better overall picture of the links between variables.
Disadvantages/Counter–Claims
Can only be as good in terms of the overall quality of evidence they provide as the quality of each study they include – ‘the garbage-in-garbage-out’ principle.
Evidence Level 5×
Randomised Controlled Trials
Advantages/Claims
Claim to include reliable (quantitative) before-and-after measurements, compare control and intervention groups to see how these groups fare with and without the treatment.
As well as testing for statistical significance (the likelihood an observed result happens by chance) they measure effect size – how big an impact occurs from the intervention or cause under consideration.
Disadvantages/Counter–Claims
Can be expensive and time-consuming.
Evidence Level 4×
Longitudinal Studies
Advantages/Claims
Claim provide a good level of evidence by measuring the impact of interventions over time.
Disadvantages/Counter–Claims
Lack of randomisation more susceptible to risk of selection bias – for example, people with a positive attitude to change being more likely to participate in a trial.
Evidence Level 3×
Cross-Sectional Studies: observe differences between groups at one point in time
Advantages/Claims
Relatively quick to conduct and captures multiple variables at a specific point in time.
Disadvantages/Counter–Claims
Relationships between cause and effect are limited and potentially spurious – for example, a correlation between motivation and performance could be due to teams rated as high-performing feeling better about themselves as a result, rather than high motivation contributing to performance.
Evidence Level 2×
Commercial Non Peer-Reviewed
Advantages/Claims
Pull together in a systematic and objective way all the best quality evidence relevant to the study in question.
Disadvantages/Counter–Claims
Plentiful supply but susceptible to accusations of bias and lack of exposure to external or objective scrutiny.
As in any field of practice the claims commercial organisations make about their products and services can be over-emphasised.
Evidence Level 1×
Expert Opinion, Anecdotes
Advantages/Claims
Can be very well referenced and insightful.
Disadvantages/Counter–Claims
Very large quantity of expert opinions and case studies that some would argue is the weakest form of evidence. What experts or observers think or believe may be interesting but are more naturally likely to be biased and rooted in vested interests.
What is important is what the evidence itself tells us not people’s opinions or experiences.