Anonymous peer review, despite the criticisms often leveled against it,
is used in more or less the same form by the great majority of scientific
journals. The British Medical Journal (BMJ), however, has recently
taken the bold step of abolishing referee anonymity, and now requires all
referees to identify themselves to the authors. The editor, Richard Smith,
justifies this move primarily on ethical grounds, arguing1 that
"a court with an unidentified judge makes us think immediately of totalitarian
states and the world of Franz Kafka". Many other journals, including Nature
Neuroscience, will await the results of this experiment with interest.
Yet, whatever the results, there are a number of reasons to think that open
review may not be the best solution for all journals.
Few would deny that peer review, as currently practiced, has its drawbacks.
There have been a number of studies on the effectiveness of peer review, mainly
in the clinical literature (see http://www.wame.org for details
and references), and some have found evidence of systematic biases among referees;
one study, for instance, reported that that US referees were more positive
than non-US referees toward papers from US authors2, and another
found evidence of bias against female applicants in grant review3.
Even if similar biases have not been demonstrated in basic science journals,
it would seem complacent to deny the possibility that they might exist. Moreover,
it is understandable that some authors are uncomfortable with a system in
which their identities are known to the referees while the latter remain anonymous.
Authors may feel themselves defenseless against what they see as the arbitrary
behavior of referees who cannot be held accountable by the authors for unfair
comments.
We believe, however, that some of the arguments against anonymous review
are misplaced, at least as they pertain to scientific journals. Although the
review process is often compared with a court trial, the analogy is inappropriate.
In contrast to the law, journals are part of a pluralistic system in which
authors themselves choose the standards by which they wish to be judged. This
is not to deny that publication in prestigious journals is important for career
advancement, but the power wielded by even the most influential journals is
nowhere near absolute. The ultimate source of a journal's influence lies with
the credibility of its editorial process, and its prestige derives largely
from the quality of the papers it accepts for publication. In this sense,
journals have only as much power as the scientific community chooses to grant
them.
The primary role of the review process is, or should be, to help the editors
decide which papers to publish. Filtering information is an important function
of any journal, but this is particularly true for a journal such as Nature
Neuroscience that aspires to attract a broad readership to its papers.
Therefore, we look to our referees not only to identify technical flaws, but
also to advise us about a paper's novelty, significance and likely interest
to our readers. Referees should bear in mind that we receive many times more
papers than we can publish, and that for every paper that is accepted, another—invisible
to them—must be rejected to make space for it. We also ask referees
to advise us whether a paper that is not yet acceptable is nevertheless potentially
important, and if so, how it could be improved. In some cases, this may be
simply a matter of rewriting the paper to make it clearer; in others it may
mean requesting many additional experiments from the authors. It is also widely
felt that the review process should help the authors of rejected papers to
revise the paper for resubmission elsewhere. However, although improving papers
is undoubtedly an additional benefit of the review system, we do not consider
this to be its primary purpose.
Given this background, what are the arguments for opening up the review
process so that authors know their referees' identities? Advocates of open
review argue that openness will force referees to think more carefully about
the scientific issues and to write more thoughtful reviews; it may also help
to expose possible conflicts of interest in some cases. A few referees always
sign their reviews as a matter of principle, and many more do so in specific
cases, for instance when they wish to discuss the results directly with the
authors, when they feel obliged to disclose a possible conflict, or when they
believe that their identities will be obvious in any case. These are all legitimate
reasons for openness, but the published literature gives little support to
the idea that a general policy of disclosure improves the overall quality
of reviews, as the BMJ acknowledges in its editorial1.
The main argument for more openness is ethical, that it is fundamentally unfair
for authors to be exposed to the judgment of someone acting behind the screen
of anonymity.
Yet it is important to remember that decisions are not made by referees,
but rather by editors. The editors take responsibility for their decisions,
and they are accountable for the quality of the advice on which those decisions
are based. Most of the abuses that an open review system is intended to prevent—hostile
comments, unsubstantiated criticisms, excessive delay of competitors' manuscripts—can
also be prevented by a careful editor.
We believe that there are several strong arguments against open review.
For one thing, it may lead to serious problems in finding appropriate referees.
The BMJ claims that, since it opened up its peer-review process, only
a small percentage (about 2%) of referees have refused to review because of
the change in editorial policy. However, an informal poll of some of our referees
suggests that this might not be the case for basic science journals. Many
of the people we contacted said they would refuse to review certain papers
if their names were revealed. It might be especially difficult to find referees
for authors who hold positions of power and influence, or for those who are
considered quarrelsome or vindictive by their peers. In particular, younger,
less-established scientists (who are reported4 to be among the
best reviewers) would be reluctant to reveal themselves, for fear of retaliation
from their more powerful colleagues. Even if they did review papers, it might
be hard for them to be fully honest, knowing that the person they are reviewing
may be evaluating their grants and recommending them for tenure. Anonymous
review remains an important corrective for such unequal power relationships.
The opportunities for nepotism will also be increased by an open review
system. It seems to be widely believed that more prominent authors receive
preferential treatment in the review process, and although editors can try
to minimize this tendency it may be impossible to eliminate altogether. It
is very common, for instance, to receive reports that begin along the lines,
"This is an excellent study from one of the leading groups in the field...",
and although a case can be made that the authors' previous track record is
relevant when judging their latest work, the editor must not allow this to
become a dominant factor. In an open system, however, it seems almost inevitable
that the opportunities to reciprocate favors over time will lead to referees
placing more rather than less weight on an author's identity.
The biggest problem with open review, at least from an editor's point of
view, is that it is likely to lead to more bland, even timid, reviews. Referees
will be more likely to restrict their comments to technical concerns that
are easily defended, rather than advising on necessarily more subjective issues
such as conceptual novelty and general interest. Several referees commented
that a policy of forced openness would cause reviews to resemble letters of
recommendation, which are often so inflated as to be useless. The only way
for the editor to get an honest opinion might then be to call the referee
and get comments 'off the record', defeating the point of peer review altogether.
Peter Strick, editor-in-chief of the Journal of Neurophysiology, notes
that when the journal tried encouraging voluntary open review a decade ago,
the editors quickly realized that this system promoted more problems than
it solved, including bland and cautious reviews. The journal also experienced
an occasional breakdown of the peer-review process, in which authors and referees
bypassed the editors completely in negotiating how a paper should be revised.
If complete openness is not the answer, how can the review process be improved?
One solution that is occasionally proposed is the opposite—a completely
closed system in which referees (and perhaps even editors) are blind to the
identities of the authors. However, this seems most unlikely to work; self-identifying
clues are often an essential part of a manuscript, and if the referees have
enough knowledge of the authors to hold any prejudice (whether positive or
negative), they are also likely to be able to guess the authors' identities.
We believe that the present framework—authors identified to referees
who remain anonymous—is the only workable one, and that any efforts
to improve the system should focus on how anonymous referees can be helped
to do a better job.
Reviewing well requires a substantial commitment of time and energy, and
it is generally admitted that being a good referee does not lead to any tangible
rewards with respect to career advancement. Why are people willing to expend
so much unrecognized effort? For some, it is as simple as civic duty and a
feeling that they owe their colleagues the same type of treatment that they
would wish for their own manuscripts. Some are motivated by loyalty to the
journal or to a particular editor. Many referees enjoy having early access
to new and interesting papers in their field. This is of course a slippery
slope, given that access to privileged information can easily lead to its
abuse. Other motives are more obviously problematic, for instance using refereeing
as a way of blocking the dissemination of ideas opposed to one's own, or currying
favor with editors who will be making decisions about one's own papers in
the future. Ultimately, the peer review system will stand or fall on the availability
of good citizens whose judgment and ethics can be trusted. However, like many
other tasks critical for the success of science, reviewing skills are acquired
haphazardly, usually from a limited number of mentors whose own approach may
or may not be ideal.
We are exploring ways to improve our peer review system. Soon we plan to
include on our web site a new Guide to Referees, which we hope will be helpful to referees, authors
and readers alike in explaining how we reach our editorial decisions and what
types of advice we find most useful. Over the next few months, we also plan
to ask some of our authors—both rejected and accepted—to rate
the performance of their referees. We hope this information will be useful
for several purposes: it will help us in selecting referees, it will allow
us to provide feedback for any interested referees on the perceived quality
of their reports, and it will also allow us to recognize and thank those referees
whose reports are felt to be the most useful. We welcome suggestions (
neurosci@natureny.com) as to how we can improve our review process,
and best represent the interests of authors referees and readers.
