Evaluate the validity, reliability and accuracy of scientific evidence presented in claims, considering the hierarchy of evidence in medical research

What this dot point is asking

NESA wants you to evaluate scientific evidence according to the hierarchy of evidence, identify when a claim is overstated relative to the evidence, and apply the framework to media reporting of science. This is among the most heavily tested topics in Investigating Science.

The answer

Not all evidence is equal. The hierarchy of evidence ranks study designs by their ability to establish reliable claims about cause and effect.

The hierarchy of evidence

From strongest to weakest:

1. Systematic reviews and meta-analyses.

Statistical synthesis of all available studies on a question, with explicit inclusion criteria and quality assessment. Cochrane reviews and NHMRC clinical guidelines are the gold standard. A meta-analysis can detect small effects invisible in single studies and quantify how reliable the combined evidence is.

2. Randomised controlled trials (RCTs).

Participants are randomly assigned to treatment or control groups, ideally blinded and double-blinded. Randomisation balances confounders across groups, so any difference in outcome can be attributed to the treatment. Modern RCTs are pre-registered (protocol locked before data collection).

3. Cohort studies.

A group of people followed over time, with exposures recorded prospectively. Useful for studying long-term outcomes. Cannot establish causation alone because confounders may be unequally distributed between exposure groups. Australian examples: the 45 and Up Study (260,000 NSW participants), the Australian Longitudinal Study on Women's Health.

4. Case-control studies.

Compare people with a disease to similar people without it, retrospectively asking about exposures. Cheap and fast but vulnerable to recall bias and selection bias.

5. Cross-sectional studies and surveys.

Snapshot of a population at one moment. Useful for prevalence and association but cannot establish temporal sequence (which came first, the exposure or the outcome).

6. Case reports and case series.

A single patient or small group. Hypothesis-generating only. Useful for novel diseases (the first reports of HIV in 1981 were case series).

7. Expert opinion and anecdote.

The lowest level. Important for context and clinical experience, but a single doctor's clinical impression is not evidence of an effect.

What each level rules out

A claim about cause and effect requires study designs that can rule out:

• Confounders. Variables associated with both exposure and outcome.
• Reverse causation. The outcome causing the exposure, not vice versa.
• Selection bias. Sampling that does not represent the target population.
• Chance. Random variation producing apparent effects.
• Publication bias. Tendency to publish positive results.

Applying the hierarchy to a media claim

When a news headline claims "X causes Y":

1. What study is cited? A single study? A meta-analysis?
2. What study type? RCT? Cohort? Case report?
3. What is the sample size? Hundreds of thousands give confidence; dozens do not.
4. Is the effect large? Effect sizes (relative risk, odds ratio) matter as much as significance.
5. Has it been replicated? Single-study claims are provisional.
6. Who funded it? Conflict of interest can shape conclusions.

NHMRC and the Australian context

The National Health and Medical Research Council uses the hierarchy of evidence to set Australian clinical guidelines. NHMRC publishes evidence grading:

• Level I. Systematic review of RCTs.
• Level II. Properly designed RCT.
• Level III. Pseudo-randomised or comparative studies.
• Level IV. Case series.

Each guideline cites the evidence level supporting it. Practitioners are expected to weigh recommendations accordingly.

Common situations the hierarchy clarifies

A new "miracle drug" reported in the news

Usually a Phase II trial or even pre-clinical animal data. Promising but provisional. Treatment in patients requires Phase III RCTs and regulatory approval (TGA in Australia, FDA in the US).

Diet and cancer risk claims

Usually based on observational cohort studies. Associations are real but confounders are common. Strong dietary recommendations need RCTs (rare for diet because of compliance challenges) or very consistent observational evidence with biological mechanism (e.g. processed meat and bowel cancer).

Vitamin and supplement claims

Industry-funded short trials may show effects. Independent meta-analyses (e.g. of vitamin C and the common cold) often show no clinical benefit.

When evidence is uncertain

Even the highest levels of evidence can be uncertain. RCTs may be too small, too short or conducted on a non-representative population. Meta-analyses depend on the quality of included studies. Honest scientists report uncertainty alongside best estimates.

The right response to uncertain evidence is not to claim certainty in the opposite direction but to communicate the uncertainty honestly. This was a major lesson of COVID-19 public communication.