Sir

Your News Feature (Nature 419, 772–776; 2002) raises important questions about the reliability of peer review, but falls back on the justification often used by editors to shield themselves from widespread dissatisfaction with the system as currently practised: “If it ain't broke, don't try to fix it”.

We believe it may never have been working in the first place.

Perhaps peer review, in its current form, cannot be expected to detect fraud. But can we even rely on it to improve the chances that what is published is the best science, communicated as accurately as possible, and that what remains unpublished is dispensable?

Various studies, mostly in biomedical journals, have reported only modest author satisfaction (at best) with the review process, irrespective of the quality of the review. Papers that eventually became very highly cited were often rejected by the journal of first choice. Peer review is costly, biased, can be inefficient, does not always identify important work, and can allow publication of articles with serious deficiencies or omissions.

Rather than falling back on the churchillian cliché quoted in your feature that peer review is the worst system in the world except for all the others, members of the research community should cooperate to answer several questions.

We need to know whether peer review (in whatever form) is more effective than alternatives. Does it identify submissions of higher quality than do other selection methods, or chance, or no selection? Does peer review significantly improve the clarity, transparency, accuracy and usefulness of published papers compared with the submitted versions?

If peer review in its current, descriptive form is ineffective or less than effective, we should experiment with more analytical forms of assessment. For example, the quality of a new study could be assessed in the context of a pre-existing systematic review of studies on the topic. Such a population approach may make it easier to assess the contribution of an individual new study. At the same time, assessment should be standardized and specific for different experimental designs, and peer reviewers should be trained to use a single, structured-assessment instrument.

Ultimately, it is the larger population of readers (rather than a possibly biased sample of referees) who should decide whether the changes made during review substantially improve the document as a record of a peer's contribution to science. New systems should be tried that involve readers in the review process either after 'traditional' review (as in post-publication commentary, rapid replies and the like) or by developing a 'definitive' text by consensus before publication. Language experts have been investigating readers' reactions to texts for many years; it is time for editors and publishers in the 'harder' sciences to use their methods to extract useful experimental data from these reactions.