Communicate scientific understanding using suitable language and terminology, including the role of peer review and replication in confirming scientific findings

What this dot point is asking

NESA wants you to explain peer review and replication as the two main mechanisms by which scientific claims become established knowledge, and to recognise that science is self-correcting through these processes. This dot point underpins almost every "evaluate" question in Modules 6, 7 and 8.

The answer

A finding by one researcher is provisional. Scientific knowledge emerges only when claims survive critical scrutiny. Two formal processes do this work.

Peer review

Before publication in a reputable scientific journal, a manuscript is sent to two to four independent experts in the field (the "peers"). They evaluate the work and recommend acceptance, revision or rejection.

What reviewers check.

1. Originality and significance. Does the work advance the field?
2. Methodology. Are the variables, controls, sample size and statistical methods appropriate?
3. Data quality. Are the data plausible and well-presented?
4. Conclusions. Do the conclusions follow from the data without overreach?
5. Replicability. Are methods described in enough detail for others to repeat the work?
6. Ethics. Have appropriate approvals been obtained?

Strengths. Peer review filters out obvious errors, methodological flaws and overstated claims before they enter the scientific record. It is the central credibility-signal in modern science.

Limitations. Peer review is imperfect. Reviewers can miss fraud (Wakefield), share field-wide biases, or be too slow for fast-moving science. It does not detect data fabrication unless the data are visibly impossible.

Replication

Independent researchers repeat the original experiment, ideally in different labs with different operators and equipment, to test whether the result holds.

Types of replication.

• Direct replication. Same protocol, same conditions, different team.
• Conceptual replication. Same hypothesis tested with different methods.
• Systematic review and meta-analysis. Combining results from multiple studies to estimate the true effect.

What replication contributes. It distinguishes robust findings from one-off results due to chance, undisclosed bias or methodological flaws.

The 2010s reproducibility crisis

In 2015 the Open Science Collaboration attempted to replicate 100 published psychology studies. Only 36 per cent produced effects in the same direction at statistical significance. Similar concerns affect biomedical research: Amgen replicated only 6 of 53 "landmark" cancer studies in 2012, and Bayer replicated only 25 per cent of preclinical findings in 2011.

Causes. Publication bias (only positive results published), p-hacking (running analyses until something looks significant), small sample sizes, undisclosed conflicts of interest, and outright fraud.

Reforms.

• Pre-registration. Researchers register hypotheses, methods and analyses before collecting data.
• Open data. Raw data made publicly available for re-analysis.
• Registered reports. Journals accept papers based on the methodology before results are known.
• Higher statistical standards. Larger sample sizes, more stringent significance thresholds.

Australian context

The National Health and Medical Research Council (NHMRC) funds replication studies and enforces research integrity through the Australian Code for the Responsible Conduct of Research. The Australian Research Council (ARC) similarly enforces standards for non-medical research. Both require open data where appropriate.

Self-correcting nature of science

Science differs from belief systems in that claims are provisional and can be withdrawn. Retraction Watch maintains a public database of retracted papers. The Wakefield MMR paper, the South Korean stem-cell papers by Hwang Woo-suk and the social-priming literature in psychology are landmark retractions that show the system at work, even if slowly.

Examples in context

Example 1. Australian Synchrotron beamtime allocation. Researchers seeking time on the Australian Synchrotron's macromolecular crystallography beamline at Clayton submit two-page proposals that are peer-reviewed by an international panel before beamtime is allocated. The panel scores proposals on scientific merit, technical feasibility and replication value, with allocations published quarterly. The pre-experiment peer review is a quality gate before any data is collected, complementing the post-experiment peer review that occurs when results are submitted to journals. Australian Synchrotron data is increasingly required to be deposited in open structural databases (the Protein Data Bank, the Cambridge Structural Database), enabling independent re-analysis and contributing to reproducibility in structural biology.

Example 2. Australian Stroke Trials replication. The Royal Melbourne Hospital led an international consortium replicating the EXTEND-IA clot-retrieval trial after its 2015 publication. Each centre used the same imaging protocol, eligibility criteria and primary outcome (90-day modified Rankin Scale). Three independent replications (DAWN, DEFUSE 3, EXTEND-IA TNK) confirmed the original 30 per cent absolute improvement in functional outcomes. The replication chain prompted rapid revision of Australian Stroke Foundation clinical guidelines and the inclusion of endovascular clot retrieval on the Medicare Benefits Schedule. The case shows replication working as designed: an initial finding survives independent testing and becomes the basis of clinical policy within five years.

Try this

Q1. Outline three things a peer reviewer is asked to evaluate when assessing a manuscript. [3 marks]

• Cue. Methodology appropriateness; whether conclusions follow from data without overreach; sufficient methodological detail for replication.

Q2. The 2015 Open Science Collaboration replicated 100 psychology studies and found that only 36 per cent of original effects replicated at statistical significance. Discuss what this implies about peer review and what reforms followed. [4 marks]

• Cue. Peer review insufficient on its own; reforms include pre-registration, registered reports, open data and stricter sample-size requirements.

Q3. A new Australian paper claims a 40 per cent reduction in dementia risk from a dietary intervention. (a) State two things to check before treating the claim as established. (b) Explain why pre-registration matters. (c) Identify one example of a high-profile retraction. [2+2+2 marks]

• Cue. (a) Peer-reviewed journal, independent replication. (b) Locks the hypothesis and analysis plan, preventing p-hacking. (c) Wakefield MMR retraction (2010); Hwang Woo-suk stem-cell fraud (2006).