|
 |
|
EMBO reports 4, 8, 734–736 (2003)
doi:10.1038/sj.embor.embor909
Cover the eyes of Lady Justice
An appeal for double-blind peer reviewing
Johannes A. Schmid
|
|
|
|
Johannes A. Schmid is at the
University of Vienna, Austria.
e-mail: johannes.schmid@univie.ac.at
|
|
|
|
Blind Lady Justice, who dates back to the ancient Greek goddess Themis,
has become the symbol for fair and equal justice. No matter how she is painted,
drawn or otherwise depicted, her eyes are always covered so that she cannot be
influenced by the person being judged and thus does not fall prey to prejudice
or corruption. Indeed, the idea of a judicial system that is 'blind' to
anything other than facts is an important pillar of democratic societies that
aim for fair and equal justice for all their citizens.
The scientific community is a society that should aim for the same high
standards of justice to establish equal chances for everybody—at least in
those aspects that concern the judgement of scientific contributions from
individuals or research groups. In some ways, the publication system in science
reflects the judiciary system in democracy, with the reviewers being the
judges. Although they do not assess crimes against society, reviewers evaluate
scientific achievements. Their verdicts are often important for the future
lives and careers of the people being judged, so there is an evident need for a
fair reviewing system. But the reality clearly does not match this demand,
which puts an undue burden particularly on the younger and not yet established
scientists who still need to build their reputation by publishing in
high-impact journals. It is therefore necessary to take a
|
|
|
|
In some ways, the publication system in science
reflects the judiciary system in democracy, with the reviewers being the
judges
|
|
|
|
look at the shortcomings of the current system and to reconsider
alternatives. In particular, I propose the establishment of a double-blind
reviewing system, in which the author of a publication and its reviewer are not
aware of each other's identity.
As discussed in detail in a recent commentary by Peter Lawrence (2003), there is tremendous pressure in many areas
of research to publish in high-impact journals. This pressure is created by the
fact that both positions and funding are very limited, and this in turn puts a
heavy burden on many scientists, particularly younger ones. Because the
selection criteria for scientists or projects are based mainly on a scientist's
impact factor score, many postdoctoral fellows and younger group leaders need
publications in high-impact journals to advance their scientific careers. This
is not likely to change soon, as there is, at present, no other way to evaluate
the quality of a person's work—at least in a semi-objective way.
Some of the limitations that cause this publication pressure are
inherent in the way in which science is organized. Why, for instance, is it
necessary to keep the majority of younger scientists in an uncertain financial
situation by giving them only temporary contracts, whereas most other sectors
offer permanent contracts? This does not imply that scientists should have a
lifetime guarantee of employment, but providing contracts without a time limit
would certainly reduce some of the pressure that leads to unwanted side-effects
in our culture of scientific publishing. There are other restrictions that are
certainly not that easy to overcome: cutting-edge research is becoming more and
more expensive, and both public institutions and private companies that perform
research are forced to restrict or focus their budgets to fulfil the
obligations and demands of citizens and shareholders, respectively. Thus, even
established scientists are competing for limited funding possibilities. This
problem is further aggravated by the fact that a large number of graduate
students and postdoctoral fellows compete for a disproportionately small number
of available positions. All these factors contribute to the pressure that is
placed on scientists. And at the end of the day, scientists are judged
primarily on their publication records, because there is no other basis for
comparison.
|
|
|
|
And at the end of the day, scientists are judged
primarily on their publication records, because there is no other basis for
comparison
|
|
|
|
Of course, scientific publishing is an important part of
science—it spreads knowledge and expertise, making it available to a
broad spectrum of people. It also prevents important findings and conclusions
from being kept secret and motivates scientists to obtain as much information
as possible from their research and to critically review their own data. In
this respect, scientific publishing adds the important aspects of information
transfer and quality control to research that may otherwise only have been
performed for the mere satisfaction of an individual's curiosity.
But what is the reality of our scientific publication system, how is it
related to the pressure that scientists have to bear and why does it seem to be
unfair in some respects? One important point is that almost all reviewers are
also authors and referees on committees that make decisions about funding or
positions. When we compare this situation to the organization of democracy,
with its separation of powers, this would mean that reviewers are the
executive, legislative and judiciary authorities all in one—something
that modern societies have separated to prevent conflicts of interest. But in
our complex world of research, it would be practically impossible to separate
these powers, mainly because it is absolutely necessary to do research to be
able to judge it.
Another problem—and this is the crucial point—is that
reviewers know the names of authors and their institutions, which is in stark
contrast to the idea of Blind Lady Justice, who is not able to see the person
she must judge. Thus, reviewers are even more prone to conflicts of interest,
which arise from their joint function as authors and referees. The overall
effect is that the politics of publication have often become more important
than the quality of the research itself. Certainly, many reviewers try not to
profit from this situation, but it is evident that it is difficult to judge the
manuscript of a good friend—or a competitor—in an objective way
when the reviewer knows the identity of the author. Sometimes this situation
leads to mutual favours in reviewing or to the emergence of scientific lobbies
that tend to keep other scientists, newcomers or even new ideas out of their
elitist circle. It also leads to the establishment of 'popes' or 'aristocrats'
in the scientific community who have a substantial influence in their field,
with the power to define dogmas and to dictate to the community what is
scientifically fashionable. It is evident that a reviewer cannot easily give a
negative report on a manuscript from such an influential scientist, even if the
data presented is neither good nor solid.
On the other end of the scientific spectrum, young or as yet unknown
scientists have considerable difficulty in getting their manuscripts published
in high-impact journals—even if the results are excellent—because
the publication capacity of many journals is primarily occupied by the 'science
aristocrats' and their friends. This is further aggravated by the importance of
the reputation of the institution with which the authors are affiliated. The
worst situation that you can have in research is to be a young, unknown
scientist in an unknown institute. You might be a genius, but nobody would
notice it. Many ambitious scientists, having realized this situation, try to
get a position at a famous research institution and work with a famous guru
with the aim of optimizing their publication score as fast as
possible—sometimes publishing their results before their data are
sufficiently solid or confirmed.
|
|
|
|
...reviewers are the executive, legislative and
judiciary authorities all in one—something that modern societies have
separated to prevent conflicts of interest
|
|
|
|
Obviously, this can lead to misinterpretations, oversimplifications and
the propagation of immature or even false concepts and conclusions. It is a
fact that the results of many projects are published in such a way as to
optimize the sum of the impact factors rather than the efficiency and quality
of the information transfer. As a result, a large number of rather low-quality
manuscripts is published, a phenomenon that will eventually constrict the
'information channels' in science. This problem has become evident in many
research fields of late, so that it is actually impossible to read and digest
the vast bulk of literature that is published. At the same time, a significant
part of the information is immature or even questionable.
What might be a solution to this problem, which is caused, at least in
part, by the current publication and reviewing system in science? One
possibility would be to make the reviewing process completely transparent, as
suggested in a comment last year (Gura, 2002),
which means that both authors and reviewers are aware of each other. Although
this may bring more straightforwardness into the way in which results and
publications are discussed and judged, it is questionable whether established
scientists would still be willing to review others' work if they might face
potential conflicts with the authors.
The other alternative—namely to set up a double-blind reviewing
process, in which the reviewer does not know the identities of the
authors—might be more interesting and promising. Analogous to Blind Lady
Justice, the reviewer would not be influenced by the identity of the
author—something that might have an
|
|
|
|
The worst situation that you can have in research is
to be a young, unknown scientist in an unknown institute. You might be a
genius, but nobody would notice it
|
|
|
|
enormous
impact on our current scientific culture. As most journals are now installing
online-submission systems, it would be an easy task for the editors to remove
the authors' names and institutions. Of course, authors would not be allowed to
give away their identity by referring to their own work, for example, by using
phrases such as "as we have shown previously" followed by the
citation of their publication. This requirement could be clearly stated in the
instructions for authors and reviewers, and it should be made clear that any
violation of this rule would lead to an immediate rejection of the
manuscript.
What might be the effect of such a system? Undoubtedly, many big players
in science could no longer rely merely on their good name and reputation, as
their manuscripts would be judged according to the quality of research and the
innovative ideas they contain. This might downgrade some science aristocrats,
whose work is not—or is no longer—proportional to their reputation
and influence. More importantly, however, young or as yet unknown scientists
would have a much better chance of publishing their work in high-ranking
journals. This would help to raise their impact factor score and certainly make
it easier for them to get funding for future projects. In principle, such a
system would mix up much of the 'social ranking' in science, but in a fair and
honest way. Although elitist circles and lobbies will probably always exist, a
double-blind reviewing system would make it more difficult for such lobbies to
influence or dictate the direction of a whole research field. It would also
prevent individual scientists or lobbies from handing down their influence to
their students and followers independently of their real scientific skills, a
phenomenon of science aristocracy that is frequently observed these days.
It is obvious that such a change in our publication culture will not
occur overnight, because most of the important journals would have to switch to
a double-blind reviewing system at the same time. This is unlikely to happen,
simply because some of the high-ranking journals profit from the current
situation. In addition, well-established scientists will not put pressure on
journals to change their reviewing process.
However, one approach to starting this process and establishing this new
reviewing system might be to generate a section in a journal that contains only
articles that are evaluated by double-blind peer review. Thus, a journal's
editors would be able to test whether this reviewing system is accepted by the
scientific community—both by authors and readers—as indicated by
submissions to that specific section and the number of citations of
double-blind peer-reviewed articles. Indeed, publishing in such a section might
become a new quality criterion for 'publishing by fair means'—even for
well-established scientists.
If this is met with broad acceptance, the journal could then switch all
of its manuscripts to the new system. This might even strengthen its position
in the competitive field of scientific publishing, given that the journal would
gain the reputation of its articles being unbiased and unaffected by science
politics. This could kick-start a trend and may convince other editors to
switch their journals to a double-blind reviewing system. If journal editors or
well-established scientists on the editorial board resist such a change in
their reviewing system, the scientific community should demand it.
Certainly, there will be no revolution in scientific publishing, but we
should call for an evolution. It is time to establish new means of judging
scientific contributions in a fair and equal way—something that is
important for maintaining the health of the overall 'organism' of science.
|
|
References
Gura, T. ( 2002) Peer review, unmasked. Nature, 416, 258–260. | Article | PubMed | ChemPort |
Lawrence, P.A. ( 2003) The politics of publication. Nature, 422, 259–261. | Article | PubMed | ChemPort |
|
|
top  |
|
|
|
