Little evidence to support the use of editorial peer review to ensure quality of published research

Abstract

Data sources

Studies were identified using the following sources: CINAHL (Cumulative Index to nursing and Allied Health Literature), Cochrane Methodology Register, Dissertation Abstracts, Embase, Evidence-based Medicine Reviews, American College of Physicians Journal Club, Medline, PsycINFO and PubMed.

Study selection

Prospective or retrospective comparative studies were included that had two or more comparison groups, generated by random or other appropriate methods, and that reported original research, regardless of publication status.

Data extraction and synthesis

Because of the variety of study designs, unit of randomisation and outcome measures, a descriptive review was carried out.

Results

Twenty-eight studies were included. There was no clear-cut evidence of effect of the well-researched practice of reviewer and/ or author concealment on the outcome of the quality assessment process (nine studies). Checklists and other standardisation media have some evidence to support their use (two studies). There is no evidence that referees’ training has any effect on the quality of the outcome (one study). Different methods of communicating with reviewers and means of dissemination do not appear to have an effect on quality (three studies). On the basis of one study, little can be said about the ability of the peer-review process to detect bias against unconventional drugs. Validity of peer review was tested by only one small study in a specialist area. Editorial peer review appears to make papers more readable and improve the general quality of reporting (two studies), but the evidence for this has very limited generalisability.

Conclusions

At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research. The methodological problems in studying peer review are many and complex. At present, the absence of evidence on efficacy and effectiveness cannot be interpreted as evidence of their absence. A large, well-funded programme of research on the effects of editorial peer review should be launched urgently.

Commentary

Peer review, a process whereby journal editors rely on the views of independent (external) content experts in assisting decision-making on submitted manuscripts is an integral part of a wide range of journals. It is a process that has been in use for at least 200 years. Peer review is considered to raise the quality of the end product, a fact supported by surveys that indicate that clinicians give greater credence to findings published in peer-reviewed journals.¹

The fact is, however, that errors and omissions still occur in peer-reviewed publications, and studies on the reporting of reviews² and randomised controlled trials^3,4 in journals have confirmed this. If there is poor-quality reporting of reviews and trials, this raises a question over the quality of the initial research, and why the peer review process does not address some of these reporting deficiencies.

This Cochrane review looked at peer review and elements within that process. The authors outlined their gold standard: the process able to produce studies that are important, appropriate to the publication medium, useful, original, methodologically sound, ethical, complete and accurate. They also outline ideal measures in which true outcomes are used as indicators of quality compared with pragmatic outcomes which they summarise in a table in the review.

Not unsurprisingly, the variety of outcome measures and study designs made it inappropriate to conduct a meta-analysis so a descriptive summary is provided and the results are summarised well in the abstract.

Supporters of peer review will be disappointed that this review fails to provide evidence that it ensures the quality of reporting scientific research. Because of concerns about the quality of reporting, however, a number of groups have developed checklists for reporting a range of study designs (Table 1, next page). There is evidence that this has had an impact on the quality of reporting clinical trials, at least.⁵ These checklists can assist both authors and peer reviewers to improve the quality of publications and, together with the additional research into the peer review process called for by the authors here, should lead to improvements in the quality of scientific publications in the future.

Table 1 Checklists for reporting studies

Practice point

Editors should bear in mind the lack of convincing evidence of the impact of peer review when making editorial decisions. Readers should also be aware of this when reading publications in peer-reviewed journals.