COMMENT

Publish peer reviews

Jessica K. Polka and colleagues call on journals to sign a pledge to make reviewers’ anonymous comments part of the official scientific record.
Jessica K. Polka is executive director of ASAPbio in San Francisco, California, USA.
Contact

Search for this author in:

Robert Kiley is head of open research at the Wellcome Trust in London.

Search for this author in:

Boyana Konforti is director of scientific strategy and development at the Howard Hughes Medical Institute in Chevy Chase, Maryland, USA.

Search for this author in:

Bodo Stern is chief development and strategy officer at the Howard Hughes Medical Institute in Chevy Chase, Maryland, USA.

Search for this author in:

Ronald D. Vale is president of ASAPbio in San Francisco, California, USA; a professor of cellular and molecular pharmacology at the University of California, San Francisco; and an HHMI investigator.
Contact

Search for this author in:

Illustration by David Parkins

Long shrouded in secrecy, the contents of peer review are coming into the open. In the past decade, outlets such as eLife, F1000Research, Royal Society Open Science, Annals of Anatomy, Nature Communications, PeerJ and EMBO Press have begun to publish referee reports. Publishers including Copernicus, BMJ and BMC (the latter is owned by Springer Nature) have been doing so for even longer (see ‘Revealing peer review’). Last year, the organizers of Peer Review Week embraced the topic in a broader discussion of transparency.

We are representatives of two biomedical funders — the UK Wellcome Trust and the Howard Hughes Medical Institute (HHMI) in Chevy Chase, Maryland — and ASAPbio, a non-profit organization that encourages innovation in life-sciences publishing. We are convinced that publishing referee reports would better inform authors and readers, improve review practices and boost trust in science. Right now, less than 3% of scientific journals allow peer reviews to be published (see go.nature.com/2weh6vn).

To increase these numbers, our organizations held a meeting in February this year of around 90 invitees from the life sciences, predominantly from North America and Europe. Scientific authors, reviewers and readers participated, along with journal editors and leaders of granting agencies. We took care to include conservative voices, but the nature of the meeting attracted people ready for change. The ideas in this article were honed at that event, with later assistance from HHMI president Erin O’Shea; molecular biologist Needhi Bhalla at the University of California, Santa Cruz; Kenneth Gibbs, director of postgraduate training at the US National Institute of General Medical Sciences; and researcher Tony Ross-Hellauer at Know-Center in Graz, Austria.

Attendees agreed that the current lack of transparency around peer review does not serve science, and several journals committed to publishing reviews (although not necessarily reviewers’ identities) and author rebuttals. Here, we invite more journals to take up the cause. It is time for transparency to become the norm.

Revealing peer review

Journal editors have long consulted referees to select and improve papers. The focus has shifted to sharing them.

1750s: The UK Royal Society establishes a committee to vote on what is published in its journal, Philosophical Transactions.

1890s: UK scientific societies debate and abandon the adoption of a standardized referee system to curb “veritable sewage thrown into the pure stream of science”.

1940s–1960s: Formal peer review comes to be considered the linchpin of science. Science, Nature and the Journal of the American Medical Association take up the practice.

1970s: The term peer review becomes widely used.

1989: Inaugural Peer Review Congress organized to evaluate the process. It is held every four years.

1999–2003: The BMJ decides to disclose reviewers’ names after assessing effects in a randomized trial. The publisher BMC begins publishing signed reviewer reports. Atmospheric Chemistry and Physics promotes open discussion of submissions.

2006–16: Several journals and platforms start publishing reviewer comments. They include Biology Direct (2006), The EMBO Journal (2009), eLife (2011), F1000 Research (2012), PeerJ (2013) and Nature Communications (2016).

Defining review

The term ‘open review’ has many interpretations. ‘Open identities’ means disclosing reviewers’ names; ‘open reports’ (also called transparent reviews or published peer review) means publishing the content of reviews. Journals might offer one or the other, neither or both1.

In a 2016 survey2, 59% of 3,062 respondents were in favour of open reports. Only 31% favoured open identities, which they feared could cause reviewers to weaken their criticisms or could lead to retaliation from authors. Here, we advocate for open reports as the default and for open identities to be optional, not mandatory.

The vast majority of scientists think that peer review is essential for vetting research papers3. The process gives authors constructive feedback, offers editors insight and assures readers of the trustworthiness of research. Generally, however, only editors, authors and (sometimes) reviewers see referee reports. That enables several forms of abuse: referees might be superficial, rude or biased; authors might respond inadequately to reasonable criticism; editors might not hold authors or reviewers to account; and predatory publishers will charge fees without providing quality review.

Many benefits would accrue from publishing peer reviews (see ‘Potential benefits of published review’). The scientific community would learn from reviewers’ and editors’ insights. Social scientists could collect data (for example, on biases among reviewers or the efficiency of error identification by reviewers) that might improve the process. Early-career researchers could learn by example. And the public would not be asked to place its faith in hidden assessments.

Potential benefits of published review

Encourages good-quality, constructive comments. The expectation that reviews will be published will encourage editors and reviewers to hold them to a high standard.

Preserves useful scholarship. Peer reviews contain arguments and ideas that can reveal how thinking in a field evolves. This material should be preserved and made available to others.

Builds trust. Readers have a right to understand the level of scrutiny that a paper has undergone.

Makes journal decisions more transparent. Editors must integrate information from diverse sources, including reviewers, to make their decisions. Published peer review provides a window on the process.

Creates a pathway for crediting reviewing. Reviewers can point (even privately) to their work as evidence of scholarly activity for grants and promotions.

Provides a resource for training. Reports can show people how to (and how not to) assess a paper.

Bolsters systemic study of peer review. Published reports and rebuttals enable more research on best practices, leading to improvements in the system as a whole.

Studies of published peer reviews are small and often also involve open identities or other innovations, making effects hard to ascertain. Nonetheless, evidence so far suggests that the scientific community finds published reports valuable. At The EMBO Journal, peer-review files receive about 10% of the hits the papers themselves do4. A pilot by the publisher Elsevier found that one-third of its website visitors accessed peer-review reports, and several editors said they used published reports as instructive examples for inexperienced reviewers (see go.nature.com/2oujfgv). Editors at the European Journal of Neuroscience, which launched transparent review at the end of 2016, report that referees are writing better reviews and returning them more promptly (see go.nature.com/2oxgtyf).

Barriers, perceived and real

So why is the practice still rare? There are several reasons — some inertial, some conceptual.

Some disciplines are more keen than others. Nature Communications found that, given a choice, authors (and reviewers) of more than 70% of its evolution and ecology submissions opted for published reports. The figure was less than 50% for submissions in atomic, particle and theoretical physics5.

One concern is that, even if public reviews are anonymous, they might make reviewers reluctant to accept assignments or to criticize freely, because authors could resent criticism and retaliate against their presumed reviewers. The BMJ found that publishing peer-review reports with reviewers’ names did not change the quality of the peer reviews, suggesting that reviewers were not intimidated6. What is more, authors read unsigned reviewers’ reports during standard review anyway.

A bigger concern is that published reviews might be used unfairly in subsequent evaluation of the authors for grants, jobs, awards or promotions. There are few data about whether and how authors’ ethnicity, gender, country of origin or institution affect the evaluations of papers. Yet there is evidence of bias in scientific publishing. Women, for example, are less likely to be first or last authors in high-profile journals, and are less likely to be asked for peer reviews7,8 (see also go.nature.com/2pzyvcw). And workplace evaluations of female professionals also show gender bias (see go.nature.com/2ppat2k). So the concern is that individuals from under-represented minorities could receive biased reviews. Assessors for funding, hiring and promotions could pay more attention to negative comments when authors are from under-represented groups or less-prestigious institutions. Some fields are also more critical or competitive, which might skew reviews.

Making referee reports open could allow more-effective research into how competition and bias affect the process. Meanwhile, anyone participating in open peer review — or evaluating it after the fact — should be aware of this potential for unfairness.

Another risk is the ‘weaponization’ of reviewer reports. Opponents of certain types of research (for example, on genetically modified organisms, climate change and vaccines) could take critical remarks in peer reviews out of context or mischaracterize disagreements to undermine public trust in the paper, the field or science as a whole. Queries to eLife, The BMJ and EMBO Press about this problem revealed only one, mild example (see go.nature.com/2piygkb). But weaponization could be a greater concern for journals that publish work that is more likely to be politicized.

One precaution would be to add a disclaimer explaining the peer-review process and its role in scientific discussion. Opening up materials and establishing dialogues with journalists, politicians and the public is an opportunity to build trust and enhance understanding of the scientific process.

Published peer-review reports could also place editorial decisions under greater scrutiny and perhaps make editors more timid about overriding critical reviews (see go.nature.com/2bid8ag). Equally, published reports could boost appreciation for the role of editors in synthesizing and prioritizing diverse reviewer opinions. Editorial judgements have huge influence on our perceptions of scientific quality, and as such are valuable works of scholarship. Publishing decision letters, along with author responses, could contextualize and rebut criticism, correct misunderstandings and provide information that does not make it into the final paper.

Finally, there are pragmatic concerns. Editors report that manually posting peer-review materials can take approximately 25 minutes per manuscript. This is obviously much less than the time spent coordinating, conducting and assessing reviews, but is still significant. Most publishing platforms are not set up to display, organize or assign digital object identifiers (DOIs) to reviewer reports and related materials, and making changes to such systems can be onerous. Still, we expect that journals could streamline these tasks and, potentially, build in transparent costs for dealing with extra work. Editors will have to learn to handle reviews containing inappropriate material, such as libellous comments or unpublished results, and to become comfortable with making some correspondence public. Many have already done so.

Moving forward

We think that the value of published review reports to referees, authors, the public and editors far outweighs the risks and toil. In an ideal world, all published papers would be accompanied by the contents of their peer-review reports. For now, we recommend that the practice is encouraged while the scientific community assesses whether and how author characteristics, such as ethnicity and country of origin, influence reviewer feedback. Any structural barriers to equality must be eliminated.

Ideally, reviewer reports will be easy to find, and will be organized and archived intuitively with related materials (such as rebuttals and manuscript versions). Publishers should provide ways to recognize reviewers for their contributions. Technological innovation could reduce administrative burdens, tackle information overload and provide additional links and context for readers. Vendors of manuscript-management platforms should develop workflows that optimize and automate the process of publishing peer reviews, reducing burdens on journal staff, authors and reviewers. Indexing services, such as PubMed, should find ways to prominently link peer reviews to the original paper. Appropriate infrastructure is already being built: CrossRef began assigning DOIs for peer reviews in late 2017, and reviewer reports (with or without reviewer identities) can be archived in PubMed Central and Europe PubMed Central.

For robust systems to develop, however, published reviews must become more common. Even if today’s implementations are less than ideal, they will drive demand and pave the way for better iterations.

More than 20 editors and publishers representing more than 100 journals have already signed an open letter to show that they have begun to publish peer reviews, with or without reviewers’ identities, or that they plan to. We invite others to join. (Nature Communications has signed this pledge. Other Nature journals are considering doing so.)

Scientists can stimulate this change by requesting that the journals for which they write, review and edit are open about their peer-review process. Funders and academic societies could also help to shift attitudes, particularly if they implement or pilot published peer reviews in their society journals.

Science moves forward through criticism and disagreement. Exposing this inherent process, although uncomfortable for some, is a healthy step for science.

Nature 560, 545-547 (2018)

doi: 10.1038/d41586-018-06032-w
Nature Briefing

Sign up for the daily Nature Briefing email newsletter

Stay up to date with what matters in science and why, handpicked from Nature and other publications worldwide.

Sign Up

Updates & Corrections

  • Correction 13 September 2018: An earlier version of the timeline in this Comment stated that peer review began to be published at The EMBO Journal in 2010.

References

  1. 1.

    Ross-Hellauer, T. F1000Research 6, 588 (2017).

  2. 2.

    Ross-Hellauer, T., Deppe, A. & Schmidt, B. PLoS ONE 12, e0189311 (2017).

  3. 3.

    Publishing Research Consortium. PRC Peer Review Survey 2015 (Mark Ware Consulting, 2016).

  4. 4.

    Pulverer, B. EMBO J. 29, 3891–3892 (2010).

  5. 5.

    Nature Commun. 7, 13626 (2016).

  6. 6.

    van Rooyen, S., Delamothe, T. & Evans, S. J. W. Br. Med. J. 341, c5729 (2010).

  7. 7.

    Tregenza, T. Trends Ecol. Evol. 17, 349–350 (2002).

  8. 8.

    Shen, Y. A., Webster, J. M., Shoda, Y. & Fine, I. Preprint at bioRxiv https://doi.org/10.1101/275362 (2018).

Download references