Quality and value: Models of quality control for scientific research

Nature (2006) | doi:10.1038/nature05031

A look at alternatives to the peer review system suggests it might be the best bet.

Peer review is used as a quality-control mechanism for biomedical literature1. Journals with a formal peer-review system are generally higher profile than those without, and peer review is often a pre-condition for journal indexing in biomedical databases such as PubMed. Variants of the current system or complete alternatives have scarcely been explored, as yet. Two years ago, I and my colleagues carried out an exploratory review of complete and partial alternatives to the current editorial peer-review system, both in and outside biomedical sciences. Here I update and summarise our findings.

The current system has three salient aspects:

  • quality assurance through expert opinions
  • managing competition for publication space
  • the scholarly task of improving scientific knowledge

According to our classification, complete alternatives are those systems that radically depart from all three aspects of editorial peer review. Partial alternatives are those that depart from no more than two aspects of current practice.

We performed a comprehensive layered search, selecting eight methodologically well-defined non-biomedical disciplines (agriculture, astronomy, archaeology, decision-making sciences, law, economics, mathematics, physics). We looked at 31 journals and asked their editors whether they were aware of alternative models of quality control in their disciplines. The answer was disappointing, as they all pursued what seems to be a universal model, whereby every published paper is reviewed, openly or anonymously, by a limited number of 'expert' reviewers.

We did not investigate means of access or content dissemination (such as electronic print servers) because we do not consider these as alternative models of peer review, but rather alternative methods of disseminating science whose quality is assured by application of the current model. Perhaps our arbitrary choice of disciplines and sources, coupled with our limited survey may have missed significant alternatives. Readers of Nature may be aware of alternatives, and we would welcome debate (make a comment on this article)

In biomedicine we identified one complete alternative (no peer review) and three partial alternatives: quantitative indicators (to our knowledge not implemented); pre-publication peer review on the Internet (at the Medical Journal of Australia); and the Cochrane collaboration editorial model (at the Cochrane Library). Given the ethical implications of a lack of a quality-control system we did not investigate its abolition further.

Quantitative indicators

We rejected methods based on impact measures, as they concern researchers' interest in the work of a particular author or authors, rather than the quality of the work itself. Because these measures are of published (or publicly available) work, their use as publication criteria could be disastrous: they would introduce a bias against high-quality work by new and unpublished authors, as well as assuming that future work by an author will be as significant as past work.

Evidence of the effectiveness of indicators as quality-control mechanisms is sparse: we identified only one study (of five major journals) that found a correlation between the number of submissions to a journal and its impact factor; and even then the findings were inconclusive2.

Pre-publication peer review on the Internet

Open peer review consists of making reviewers' comments public and/or inviting post-publication comments from readers, which are then published. Such a system was tested at the Medical Journal of Australia, where 60 (81%) authors and 150 (92%) reviewers agreed to take part in a study using website access for pre-publication peer review3. Although informal feedback from readers was positive, only 29% of comments led authors to change their manuscripts.

Cochrane collaboration model

The Cochrane editorial model is a collaborative effort to assess the effects of healthcare interventions by publishing and updating systematic reviews in the Cochrane database. This model does not aim to select the best items to publish in a space-limited journal. Editors and authors work together to produce and publish an agreed protocol and then develop it into a review. The editorial process includes pre- and post-publication peer review of protocols and full reviews, a commitment to update the review no more than every two years and a training course for both authors and peer reviewers. The process is completely open; anyone contributing to the review is acknowledged and often editors are also authors and reviewers and vice versa.

The salient characteristic of competition for space is replaced by collaboration, and by reliance on electronic publication, which virtually removes space constraints. We found some unpublished evidence of its effectiveness in an inventory prepared by the Canadian Cochrane Centre that lists examples of the practical impact of Cochrane reviews. By itself, however, this evidence does not seem sufficient to recommend adoption of the system for all biomedical literature.

Maybe there are no meaningful alternatives to the current system, or no one has thought of developing them. Although not perfect, the current model may be the best possible. Should we explore partial alternatives? This would mean that competition would have to give way to collaboration. Such a move would have major implications for science and the publications industry, but we think the goal is worthwhile: improving science.

REFERENCES

1. Jefferson, T., Rudin, M., Brodney, S. & Davidoff, F. Cochrane Database Methodol. Rev.1 , doi:10.1002/14651858.MR000016.pub2 (2006).

2. Joseph, K. S. Br. Med. J. 326, 283 (2003).

3. Bingham, C. M., Higgins, G., Coleman, R. & Van Der Weyden, M. B. Lancet 352, 441�445 (1998).

RELATED LINKS:

PubMed

The Cochrane

Medical Journal of Australia

Tom Jefferson is a medical epedemiologist at the Cochrane Vaccines Field in Rome, Italy. He is an active Cochrane reviewer and one of the editors of the Cochrane Acute Respiratory Infections Group. He co-edited Peer Review in Health Sciences (BMJ, 2000). He acknowledges the help of Marica Ferri, Melanie Rudin, Frank Davidoff and Miranda Cumpston in writing this piece.

Visit our peer-to-peer blog to read and post comments about this article.

Extra navigation

.
ADVERTISEMENT