The subject of peer review, its shortcomings and ways to solve them, are a perennial source of discussion among scientists and editors. Virtually all scientists have anecdotes of how their papers have been rejected due to what they perceive as biased or vindictive refereeing. Meanwhile high profile examples of scientific misconduct raise questions as to whether peer review is even capable of filtering out faulty or fraudulent data.

Such a view is borne out by a recent paper published online in BioMedCentral's Medical Research Methodology by Emili García-Berthou and Carles Alcaraz (www.biomedcentral.com/1471-2288/4/13, 2004). These authors made a comprehensive study of errors in statistical tests published in the peer-reviewed literature. Specifically they looked at P-values reported in four volumes of Nature and two volumes of the British Medical Journal published in 2001. They estimated that around a third of the published papers showed inconsistent or incorrect rounding of the last digit.

This might not seem too serious, but García-Berthou and Alcaraz went on to recalculate all the P-values for which there was sufficient published data to do so. They found that 11–12% were incorrect, although <5% of these led to the opposite interpretation of the data (a significant result appearing non-significant, or vice versa). In the face of this inability to ensure a high level of accuracy in the details of scientific papers, they suggest that “authors of research papers (including systematic reviews) should make the raw data freely available on the Internet and journals should implement and stimulate this practice”, a practice that Nature Structural & Molecular Biology is happy to encourage for example by requesting deposition of structure factors in the Protein Data Bank

A more generally expressed concern is that getting published depends not on how good your work is, but on who you know. Peter Lawrence, who clearly laid out many of these concerns in a commentary in Nature last year (Nature 422, 259–261, 2003), considers the root cause of these biases to be the “cult of the journal”, the judging of work not by what has been done but by where it has been published.

Others suggest that peer review itself is inherently open to abuse as it allows for the exercise of power without responsibility. Referees hide behind a cloak of anonymity while editors are faceless creatures with the motto 'never apologize, never explain' tattooed on their hearts—a caricature we hope our readers do not feel accurately portrays the editorial staff of this journal at least.

An oft proposed solution is some form of 'open' peer review in which referees are made more accountable by relinquishing their anonymity. Only a few weeks ago Nature published in its Correspondence pages a letter (Van Meir, E.G. Nature 429, 803, 2004) calling for the publication of reviewers comments, anonymous or otherwise, and online dialogs between authors and reviewers to accompany papers, practices currently only adopted by a small number of mainly specialist journals such as Neoplasia. Although it is superficially appealing to make the peer review process more transparent, it is unclear how such approaches would improve the actual quality of decisions. Indeed, the anonymity of referees has the desirable effect of concentrating authors' attentions on what criticisms are being made rather than who is making them.

Despite its shortcomings scientists have recently been challenged to make the strengths of peer review better known to the general public. In June this year a report (www.senseaboutscience.org.uk/peerreview/index.htm) from a panel of academics, who had spent more than a year discussing how to help the public evaluate confusing and conflicting scientific claims, concluded that peer review is a much more reliable guide to whether findings are plausible than who conducted the work or how it was funded.

Tracey Brown, director of Sense About Science, a London-based watchdog group that published the report, feels that many poorly substantiated 'scientific' scare stories, for example the dangers of cell phone radiation, GM crops or the MMR (measles mumps rubella) vaccine, would have been strangled at birth if journalists and politicians could better appreciate the difference between work that has appeared in a rigorously peer-reviewed journal and unsubstantiated claims made and disseminated through the media. She summarizes the report's conclusions as “proposing a simple cultural shift towards wider knowledge of the peer-review process, so that all people with an interest in scientific issues start asking tough questions about the information that is put before them”.

This will be no easy task. A recent poll of more than 1,000 members of the general public carried out by the market-research company MORI on behalf of the UK's Science Media Centre and Nature (www.sciencemediacentre.org/press_releases/02-02-04_peerreview.htm) found that 71% had not the slightest idea what peer review actually was. There is some comfort to be drawn, however, as the vast majority of those polled felt that researchers should ensure that their work can be replicated by other scientists or at least scrutinized by them before publication. Even when possible risks to human health were uncovered, only 9% of those polled felt that these should be released straight to the media.

Winston Churchill once said that “democracy is the worst form of government, except all those other forms that have been tried from time to time”; perhaps peer review should be regarded in a similar light.