Bernd Pulverer reflects on his experience at The EMBO Journal of publishing referees' reports, authors' responses and editors' comments alongside papers, as other EMBO publications adopt the same policy.
Two years ago at The EMBO Journal we added transparency to peer review. We invited authors to allow inclusion of 'peer-review process files' alongside their published papers. Almost all have agreed. Now, more than 400 primary papers published in the journal showcase details of the editorial process: referee comments from every round of revision, editorial decision letters, the authors' response, as well as a detailed timeline of submission, decisions, revisions and publication1,2 (see http://go.nature.com/nbus3f for an example of an EMBO J. process file).
In our view, these augmented papers are testament to the fact that carefully administered peer review works — works well, in fact. We were initially concerned that some authors and referees might be discouraged from contributing to the journal and so, until now, have made the files relatively hard to find. But, given the positive response from the community, we are this month extending the policy to all four European Molecular Biology Organization (EMBO) scientific publications — The EMBO Journal, EMBO Reports, Molecular Systems Biology and EMBO Molecular Medicine — and making the process files much more visible online.
The perennial concerns voiced about peer review and decisions made by professional editors — as opposed to part-time academic editors — stimulated us to think about how we might improve the process at EMBO. As a first step, we did a detailed annual analysis of where manuscripts rejected at our journal were eventually published, a summary of which we now publish annually (see http://go.nature.com/4y7fwp). This supported our sense that editorial decisions are generally informed and fair. For example, only 1% of manuscripts rejected in 2008 ended up in journals with an impact factor two points or more above that of The EMBO Journal; and only 9% have a citation rate higher than the average paper in the journal.
Our second thought was that a huge amount of effort goes into peer review — effort that remains largely invisible. Many an editor and referee will attest to how much the process can improve a published paper — painful as it may be to go through. Referees can be the best writers of published analyses of single papers, such as Science's Perspectives and Nature's News & Views. So why hide all their incisive, constructive comments, which can remain pertinent even after revision and publication?
An obvious solution was to publish our anonymous referee reports. It would showcase the quality and thoughtfulness of the majority of reports. And it would add interesting points about suggested further experiments, alternative interpretations and, sometimes, limitations.
Another appeal of this path was that peer review is rarely formally taught, yet so much depends on it. We hoped that the peer-review process files might serve as a teaching tool. Finally, a clear potential benefit was to fortify the peer-review process. Referees might feel compelled to take extra care when writing their report, as the report would be published, albeit anonymously.
It was immediately apparent that, for completeness, we'd have to post all referee reports on a paper, followed by the author response. In the spirit of transparency and accountability, and with the hope of addressing grumblings about professional editors, we decided to add editorial decision letters. We'd only correct simple typos in the reports, but we'd allow removal of data that were provided solely to address a referee's point, as they might be required for future publications.
The policy kicked off in January 2009 (ref. 1). We invite authors to opt out of the system at any stage, and referees are made aware at invitation that their comments will be posted in case of acceptance. In September this year we decided to discourage 'confidential comments for the editor' by referees, which are commonplace at many biological sciences journals2. Legitimate confidential comments are allowed — for example, notes about bio-security or conflicts of interest. But we want to move away from anything that gives rejected authors the sense that something went on behind the scenes that led to their rejection.
At the time, Biology Direct and a number of BioMed Central journals already included published reports and author responses. Nevertheless, as with any change to a long-established system, there were significant risks. Would we discourage trusted referees? Would they fear that their identities might be revealed, and would they write less incisive or less critical reports as a result? Would authors resent the airing of — in the words of one referee — the 'dirty washing' leading up to acceptance? What if reports were rude or even defamatory? Would divergent referee reports lend ammunition to those that believe the system is failing? More pragmatically, would producing the files increase our editorial costs significantly, and would this additional step slow down the publication process in a field in which every day can count?
The experience has been overwhelmingly positive. The number of submissions to the journal remains steady and just 5.3% of authors have opted out, few of them citing philosophical objections to the policy2. The objectors cite a reluctance to add to the already excessive literature or a perception that an otherwise excellent piece of work can be marred by prominent comments on small mistakes or limitations.
The rate of acceptance of invitations to review a paper has remained the same, and very few invited referees decline explicitly because of the policy. In one case, a referee who had failed to read to the end of his invitation letter in the first round did decline to re-review the revised manuscript, but agreed to post the first set of comments. Nor have we seen a significant change in the quality of referees' reports or authors' responses — for better or for worse. Several referees have acknowledged that they spend more time on phrasing their reports now and this is certainly true for my own two-finger-typed decision letters! And we estimate that each file takes around 1½ hours for our administrators and editors to produce.
Many of the process files include divergent referee opinions, but we feel that the reader is well aware that journals invite a referee panel with complementary expertise and thus different vantage points. It hasn't happened yet, but if we felt a referee's report was too aggressive, we'd go back to the reviewer to suggest a rethink, noting the possible publication of the comment.
So does anyone actually notice the process files? The numbers show that the access rate is about one-tenth that of the main paper and that almost all peer-review process files have been viewed. Access to the files correlates with access to the whole article. The most viewed files are those of the papers that most excited the editors and reviewers — not of controversial or borderline papers (see graph). We haven't been collecting data on how long readers spend looking at the files, but plan to. And now we've made the files much more visible and open access, we expect an uptick in access. Meanwhile, other journals, including the European Journal of Cell Biology have been taking note and are implementing similar enhancements.
One crucial limitation of the policy is, of course, that we do not release reports on manuscripts that end up being rejected. It goes without saying that these are often the more interesting cases to consider. However, a workable way to redress this shortcoming has eluded us. A partial solution we're pursuing instead follows the example of the Neuroscience Peer Review Consortium. This cooperative of 37 journals has agreed to share referee reports if an author desires. 'Review recycling' is in our view an important way to address a key bottleneck in the publishing process.
To mix metaphors, we feel we have pried open the black box of peer review with this initiative — and shown that it is not Pandora's box. Now, like many others interested in optimizing the scientific publishing and grant-review processes, we are considering several other enhancements to traditional 'single-blinded' peer review. We remain convinced that a high level of quality assessment is essential to filter and validate the increasingly vast and diverse literature.
Many journals now allow post-publication commenting, often curated and usually signed. Despite the ubiquity of social media, commenting on scientific reports has not yet reached a level at which it could give peer review a run for its money. Inspired by the physical and computational sciences, where pre-publication commenting is commonplace, Nature ran an interesting trial a few years ago in which authors were invited to open up their manuscripts to pre-publication scrutiny during a 'traditional' peer-review process. Around 70 authors participated and the editors carefully compared the input received in this and a peer-review-alone approach. In no instance did commenting add significant value3,4 (see http://go.nature.com/n67mfk for the report).
“Online comments, even on high-profile papers, remain sparse”
Nevertheless, Nature opened all its published content for readers' online comments in March 2010. Comments, even on high-profile papers, remain sparse, however, even in journals such as PLoS ONE that specifically set out to supplement their assessment process with comments. Everyone is busy, and few may wish to risk outing themselves as critics without tangible benefit.
If peer review benefits from anonymity, why not also mask the author's identity ('double-blinded' review)? We remain interested in this possibility, but fail to see how to implement it without adding delays or requesting anonymized manuscripts for initial peer review (removal of author names does not suffice to anonymize a manuscript from one's peers). Conversely, why not add accountability by asking referees to sign their reports? The British Medical Journal, among others, has bravely pursued this path, and its editors claim that neither their referee pool nor their reports have changed5. In our view, the stakes often remain too high for this in the competitive world of biological research. Can a rookie investigator really be expected to write a critical report on a manuscript submitted by an eminent colleague who may well review their next grant? Can an author who has been asked to revise a paper significantly be relied on not to persuade the referee to back down?
Last month we started to encourage referees to comment on each other's reports, where they feel this would aid the editorial decision. Comments are only expected in cases in which a referee has taken a particularly extreme line or made a mistake, or if a referee wants to underline an essential point made by a colleague that they had missed. In line with some other journals, we have also implemented another change: we now explicitly prompt reviewers to declare the common practice of delegating peer review to others in the lab. We request that reports are vetted by the invited referee and that co-referees are named. We regard this as an essential component of good mentorship.
Most successful scientists spend a good fraction of their time reviewing papers. Yet, there is little tangible individual credit derived from the anonymous and voluntary contribution to this cornerstone of the research system. Thankfully, the remarkable culture of willingness to help colleagues and journals through peer review remains healthy, despite ever-increasing publication rates. Nevertheless, we are keenly pursing means to allow funding agencies and tenure committees to take this essential activity into account, and we welcome suggestions and collaborations on this and other possible enhancements. Peer review is the most remarkable manifestation of a collaborative spirit of science and needs to be nurtured and fortified where necessary.
Rørth, P. EMBO J. 28, 1–3 (2009).
Pulverer, B. EMBO J. 29 (in the press).
Editorial Nature 444, 971–972 (2006).
Editorial Nature Cell Biol. 9, 1 (2007).
Smith, R. Br. Med. J. 318, 4 (1999).
About this article
Views on the peer review system of biomedical journals: an online survey of academics from high-ranking universities
BMC Medical Research Methodology (2013)
Nature Materials (2011)