Quick questions on Peer review and reproducibility: HSC Investigating Science Module 5

Before publication in a reputable scientific journal, a manuscript is sent to two to four independent experts in the field (the "peers"). They evaluate the work and recommend acceptance, revision or rejection.

Independent researchers repeat the original experiment, ideally in different labs with different operators and equipment, to test whether the result holds.

In 2015 the Open Science Collaboration attempted to replicate 100 published psychology studies. Only 36 per cent produced effects in the same direction at statistical significance. Similar concerns affect biomedical research: Amgen replicated only 6 of 53 "landmark" cancer studies in 2012, and Bayer replicated only 25 per cent of preclinical findings in 2011.

The National Health and Medical Research Council (NHMRC) funds replication studies and enforces research integrity through the Australian Code for the Responsible Conduct of Research. The Australian Research Council (ARC) similarly enforces standards for non-medical research. Both require open data where appropriate.

Science differs from belief systems in that claims are provisional and can be withdrawn. Retraction Watch maintains a public database of retracted papers. The Wakefield MMR paper, the South Korean stem-cell papers by Hwang Woo-suk and the social-priming literature in psychology are landmark retractions that show the system at work, even if slowly.

Peer review filters out obvious errors, methodological flaws and overstated claims before they enter the scientific record. It is the central credibility-signal in modern science.

Peer review is imperfect. Reviewers can miss fraud (Wakefield), share field-wide biases, or be too slow for fast-moving science. It does not detect data fabrication unless the data are visibly impossible.

It distinguishes robust findings from one-off results due to chance, undisclosed bias or methodological flaws.

Publication bias (only positive results published), p-hacking (running analyses until something looks significant), small sample sizes, undisclosed conflicts of interest, and outright fraud.

Wakefield, Hwang Woo-suk and others passed peer review. Replication caught the failures.

Multiple failed replications across teams are needed before consensus shifts.

Systematic combination of multiple studies often gives the most reliable estimate. :::