Anonymous peer review, despite the criticisms often leveled against it, is used in more or less the same form by the great majority of scientific journals. The British Medical Journal (BMJ), however, has recently taken the bold step of abolishing referee anonymity, and now requires all referees to identify themselves to the authors. The editor, Richard Smith, justifies this move primarily on ethical grounds, arguing1 that "a court with an unidentified judge makes us think immediately of totalitarian states and the world of Franz Kafka". Many other journals, including Nature Neuroscience, will await the results of this experiment with interest. Yet, whatever the results, there are a number of reasons to think that open review may not be the best solution for all journals.
Few would deny that peer review, as currently practiced, has its drawbacks. There have been a number of studies on the effectiveness of peer review, mainly in the clinical literature (see http://www.wame.org for details and references), and some have found evidence of systematic biases among referees; one study, for instance, reported that that US referees were more positive than non-US referees toward papers from US authors2, and another found evidence of bias against female applicants in grant review3. Even if similar biases have not been demonstrated in basic science journals, it would seem complacent to deny the possibility that they might exist. Moreover, it is understandable that some authors are uncomfortable with a system in which their identities are known to the referees while the latter remain anonymous. Authors may feel themselves defenseless against what they see as the arbitrary behavior of referees who cannot be held accountable by the authors for unfair comments.
We believe, however, that some of the arguments against anonymous review are misplaced, at least as they pertain to scientific journals. Although the review process is often compared with a court trial, the analogy is inappropriate. In contrast to the law, journals are part of a pluralistic system in which authors themselves choose the standards by which they wish to be judged. This is not to deny that publication in prestigious journals is important for career advancement, but the power wielded by even the most influential journals is nowhere near absolute. The ultimate source of a journal's influence lies with the credibility of its editorial process, and its prestige derives largely from the quality of the papers it accepts for publication. In this sense, journals have only as much power as the scientific community chooses to grant them.
The primary role of the review process is, or should be, to help the editors decide which papers to publish. Filtering information is an important function of any journal, but this is particularly true for a journal such as Nature Neuroscience that aspires to attract a broad readership to its papers. Therefore, we look to our referees not only to identify technical flaws, but also to advise us about a paper's novelty, significance and likely interest to our readers. Referees should bear in mind that we receive many times more papers than we can publish, and that for every paper that is accepted, another—invisible to them—must be rejected to make space for it. We also ask referees to advise us whether a paper that is not yet acceptable is nevertheless potentially important, and if so, how it could be improved. In some cases, this may be simply a matter of rewriting the paper to make it clearer; in others it may mean requesting many additional experiments from the authors. It is also widely felt that the review process should help the authors of rejected papers to revise the paper for resubmission elsewhere. However, although improving papers is undoubtedly an additional benefit of the review system, we do not consider this to be its primary purpose.
Given this background, what are the arguments for opening up the review process so that authors know their referees' identities? Advocates of open review argue that openness will force referees to think more carefully about the scientific issues and to write more thoughtful reviews; it may also help to expose possible conflicts of interest in some cases. A few referees always sign their reviews as a matter of principle, and many more do so in specific cases, for instance when they wish to discuss the results directly with the authors, when they feel obliged to disclose a possible conflict, or when they believe that their identities will be obvious in any case. These are all legitimate reasons for openness, but the published literature gives little support to the idea that a general policy of disclosure improves the overall quality of reviews, as the BMJ acknowledges in its editorial1. The main argument for more openness is ethical, that it is fundamentally unfair for authors to be exposed to the judgment of someone acting behind the screen of anonymity.
Yet it is important to remember that decisions are not made by referees, but rather by editors. The editors take responsibility for their decisions, and they are accountable for the quality of the advice on which those decisions are based. Most of the abuses that an open review system is intended to prevent—hostile comments, unsubstantiated criticisms, excessive delay of competitors' manuscripts—can also be prevented by a careful editor.
We believe that there are several strong arguments against open review. For one thing, it may lead to serious problems in finding appropriate referees. The BMJ claims that, since it opened up its peer-review process, only a small percentage (about 2%) of referees have refused to review because of the change in editorial policy. However, an informal poll of some of our referees suggests that this might not be the case for basic science journals. Many of the people we contacted said they would refuse to review certain papers if their names were revealed. It might be especially difficult to find referees for authors who hold positions of power and influence, or for those who are considered quarrelsome or vindictive by their peers. In particular, younger, less-established scientists (who are reported4 to be among the best reviewers) would be reluctant to reveal themselves, for fear of retaliation from their more powerful colleagues. Even if they did review papers, it might be hard for them to be fully honest, knowing that the person they are reviewing may be evaluating their grants and recommending them for tenure. Anonymous review remains an important corrective for such unequal power relationships.
The opportunities for nepotism will also be increased by an open review system. It seems to be widely believed that more prominent authors receive preferential treatment in the review process, and although editors can try to minimize this tendency it may be impossible to eliminate altogether. It is very common, for instance, to receive reports that begin along the lines, "This is an excellent study from one of the leading groups in the field...", and although a case can be made that the authors' previous track record is relevant when judging their latest work, the editor must not allow this to become a dominant factor. In an open system, however, it seems almost inevitable that the opportunities to reciprocate favors over time will lead to referees placing more rather than less weight on an author's identity.
The biggest problem with open review, at least from an editor's point of view, is that it is likely to lead to more bland, even timid, reviews. Referees will be more likely to restrict their comments to technical concerns that are easily defended, rather than advising on necessarily more subjective issues such as conceptual novelty and general interest. Several referees commented that a policy of forced openness would cause reviews to resemble letters of recommendation, which are often so inflated as to be useless. The only way for the editor to get an honest opinion might then be to call the referee and get comments 'off the record', defeating the point of peer review altogether. Peter Strick, editor-in-chief of the Journal of Neurophysiology, notes that when the journal tried encouraging voluntary open review a decade ago, the editors quickly realized that this system promoted more problems than it solved, including bland and cautious reviews. The journal also experienced an occasional breakdown of the peer-review process, in which authors and referees bypassed the editors completely in negotiating how a paper should be revised.
If complete openness is not the answer, how can the review process be improved? One solution that is occasionally proposed is the opposite—a completely closed system in which referees (and perhaps even editors) are blind to the identities of the authors. However, this seems most unlikely to work; self-identifying clues are often an essential part of a manuscript, and if the referees have enough knowledge of the authors to hold any prejudice (whether positive or negative), they are also likely to be able to guess the authors' identities. We believe that the present framework—authors identified to referees who remain anonymous—is the only workable one, and that any efforts to improve the system should focus on how anonymous referees can be helped to do a better job.
Reviewing well requires a substantial commitment of time and energy, and it is generally admitted that being a good referee does not lead to any tangible rewards with respect to career advancement. Why are people willing to expend so much unrecognized effort? For some, it is as simple as civic duty and a feeling that they owe their colleagues the same type of treatment that they would wish for their own manuscripts. Some are motivated by loyalty to the journal or to a particular editor. Many referees enjoy having early access to new and interesting papers in their field. This is of course a slippery slope, given that access to privileged information can easily lead to its abuse. Other motives are more obviously problematic, for instance using refereeing as a way of blocking the dissemination of ideas opposed to one's own, or currying favor with editors who will be making decisions about one's own papers in the future. Ultimately, the peer review system will stand or fall on the availability of good citizens whose judgment and ethics can be trusted. However, like many other tasks critical for the success of science, reviewing skills are acquired haphazardly, usually from a limited number of mentors whose own approach may or may not be ideal.
We are exploring ways to improve our peer review system. Soon we plan to include on our web site a new Guide to Referees, which we hope will be helpful to referees, authors and readers alike in explaining how we reach our editorial decisions and what types of advice we find most useful. Over the next few months, we also plan to ask some of our authors—both rejected and accepted—to rate the performance of their referees. We hope this information will be useful for several purposes: it will help us in selecting referees, it will allow us to provide feedback for any interested referees on the perceived quality of their reports, and it will also allow us to recognize and thank those referees whose reports are felt to be the most useful. We welcome suggestions ( firstname.lastname@example.org) as to how we can improve our review process, and best represent the interests of authors referees and readers.
Smith, R. BMJ 318, 4–5 ( 1999).
Link, A. M. JAMA 280, 246–247 ( 1998).
Wenneras, C. & Wold, A. Nature 387, 341–343 (1997).
Evans, A. T., McNutt, R. A., Fletcher, S. W. & Fletcher, R. H. J. Gen. Intern. Med. 8, 422– 428 (1993).
About this article
American Journal of Neuroradiology (2017)
Frontiers in Neuroscience (2015)