Quick questions on Peer review and reproducibility: HSC Investigating Science Module 5

Before publication in a reputable scientific journal, a manuscript is sent to two to four independent experts in the field (the "peers"). They evaluate the work and recommend acceptance, revision or rejection.

The National Health and Medical Research Council (NHMRC) funds replication studies and enforces research integrity through the Australian Code for the Responsible Conduct of Research. The Australian Research Council (ARC) similarly enforces standards for non-medical research. Both require open data where appropriate.

Science differs from belief systems in that claims are provisional and can be withdrawn. Retraction Watch maintains a public database of retracted papers. The Wakefield MMR paper, the South Korean stem-cell papers by Hwang Woo-suk and the social-priming literature in psychology are landmark retractions that show the system at work, even if slowly.

Peer review filters out obvious errors, methodological flaws and overstated claims before they enter the scientific record. It is the central credibility-signal in modern science.

Peer review is imperfect. Reviewers can miss fraud (Wakefield), share field-wide biases, or be too slow for fast-moving science. It does not detect data fabrication unless the data are visibly impossible.

Publication bias (only positive results published), p-hacking (running analyses until something looks significant), small sample sizes, undisclosed conflicts of interest, and outright fraud.

Outline three things a peer reviewer is asked to evaluate when assessing a manuscript. [3 marks]

The 2015 Open Science Collaboration replicated 100 psychology studies and found that only 36 per cent of original effects replicated at statistical significance. Discuss what this implies about peer review and what reforms followed. [4 marks]

A new Australian paper claims a 40 per cent reduction in dementia risk from a dietary intervention. (a) State two things to check before treating the claim as established. (b) Explain why pre-registration matters.