The peer-review process for scientific publication has taken some flak lately. At the end of January, the Cochrane Library (Jefferson, T.O. et al., issue 1, 2003; Update Software, Oxford, UK) published a 'meta-review' of the studies the authors could find on the biomedical peer-review process. (The Cochrane Collaboration, initially established through the United Kingdom's National Health Service, publishes systematic reviews of the randomized clinical trials literature in its online Cochrane Library.) The authors concluded that there is "little empirical evidence to support the use of editorial peer review as a mechanism to ensure quality of biomedical research, despite its widespread use and costs." This report led to press coverage implying that the highly vaunted peer-review system has not proven to be efficacious. But is the criticism justified?
The Cochrane review evaluated published studies that analyzed various aspects of peer review. The 21 studies that met their criteria were comparative trials measuring the effects of altering a parameter of the peer-review process. However, few of these 21 studies actually tried to measure whether peer review improves the scientific quality of a paper or helps in selection of the 'right' papers for a particular journal. Those four studies that did were structured with checklists for referees or had independent groups of referees or readers assess in blinded fashions the pre- and post-review manuscripts. The authors of these studies were uniformly positive in their conclusions about the effects of peer review. Peer review was thought to have improved from about 60% to 85% of the papers studied. Thus, rumors suggesting the demise of peer review are, indeed, premature: although more studies may be justified, there is little indication that peer review is not a valuable aspect of the biomedical publication process.
So why the bad press? Part of this is a matter of misdefinition and unrealistic expectations. Good peer review can often prevent the publication of wishful thinking translated into poorly controlled experiments, but journal editors do not expect peer review to ferret out cleverly concealed, deliberate deceptions. A peer reviewer can only evaluate what the authors chose to include in the manuscript. This contrasts with the expectation in the popular press that peer review is a process by which fraudulent data is detected before publication (although that sometimes happens).
The goals of peer review are both lofty and mundane. It is the responsibility of journals to administer an effective review system. Peer review, as currently practiced by high impact biomedical journals such as Nature Immunology, is designed to select technically valid research of significant interest to, in our case, the general immunology community. Referees are expected to identify flaws, suggest improvements and assess novelty (providing citations of papers that detract from the impact of the paper under review). If the manuscript is deemed important enough to be published in a high visibility journal, referees ensure that it is internally consistent, thereby ferreting out spurious conclusions or clumsy frauds.
That poor science sometimes gets published in peer-reviewed journals demonstrates the vagaries of an all-too-human process that is highly dependent on the skills of its practitioners. In spite of an extensive biomedical peer-review system, over-generalized conclusions slip through and grand theories are often built upon skeletal supports. The negative spin that the press placed on the Cochrane Library meta-review plays into the hands of a public already made skeptical by medical advice that is here today, gone tomorrow. However, the final conclusion of that study is not that peer review is useless, but that better data is needed to verify that peer review is 'doing its job'. The data collected so far and discussed in the Cochrane analysis already indicate that the system improves publications.
One problem with manuscript selection is the inherent tension between referees and authors. Referees wish for only the most solid science to be published, yet when they 'switch hats' to that of author, they desire quick publication of their novel ideas and approaches. Authors of papers that blow against the prevailing winds bear a far greater burden of proof than normally expected in publishing their challenge to the current paradigm. Veering too far in one direction or the other leads to complaints either that peer review isn't stringent enough, or that it is stifling the freshest research. It is the job of the editors to try to avoid both extremes.
A final, albeit anecdotal, word on the value of peer review. Nature Immunology employs professional editors who were previously both authors and reviewers themselves. Papers that are serious contenders for one of the limited spots in Nature Immunology are sent on to in-depth peer review. We are continually impressed with peer review's positive impact on almost every paper we publish. Even papers that are misunderstood by reviewers are usually rewritten and improved before resubmission. Mistakes are made, but peer review, through conscientious effort on the part of referees, helps to protect the literature, promote good science and select the best. Until a truly viable alternative is provided, we wouldn't have it any other way.
