Would the publication of anonymous referee reports and editorial decision letters of published papers benefit the scientific debate? Results from a trial seem to suggest this.
Transparency is one of the fundamental guiding principles in science: each experiment, each theoretical derivation needs to be documented and verifiable. Transparency promotes and drives scientific progress and ensures that scientific results are accountable to the highest standards.
At the same time, transparency in science is not always possible. For example, most corporate research is confidential, and in that sense doesn't have an active part in the scientific discussion. Another example is the peer review process, whether it is the review of funding proposals or scientific publishing. In the review of funding proposals the original ideas of scientists are protected until these researchers have the opportunity to implement their ideas. In scientific publishing, similar protection applies until a paper is published.
Once published, research is of course subject to scrutiny and verification by the scientific community. And often the issues debated then mirror those raised during the peer review process. Indeed, discussions during peer review can be intense. Referee reports can be several pages long, and author rebuttal letters even longer. We have had cases of rebuttal letters exceeding 20 pages — much longer than the paper in question. Unfortunately, only a few are privy to these discussions — authors, referees and editors. The question therefore is whether the scientific community would benefit from knowing retrospectively what has been discussed during peer review, at least for those papers that do get published.
This is the aim of a trial at The EMBO Journal, which is published by Nature Publishing Group1. Since 2009, the journal has published a Peer Review File as part of the supplementary online information of papers taking part in the trial. The file provides, amongst other data, the decision letters from editors to authors, which include the anonymized referee reports, as well as the rebuttal letters from the authors.
A Commentary in Nature by the head of scientific publication at the European Molecular Biology Organization (EMBO), Bernd Pulverer, describes the journal's experience with the trial2. The trial seems to be very successful. Almost 95% of authors have agreed to take part, and only a few reviewers declined to review papers under the condition of seeing their reports published. As for the interest in the published files, download rates are about 10% of that of the corresponding papers. Interestingly, it is not the review files of controversial papers that get downloaded most, but those of what the editors consider the best papers.
The successful experience at EMBO is in contrast to a number of other attempts to tweak peer review. An example is open peer review. There, a paper is peer reviewed publicly and anyone can leave comments on submitted papers. Editors then take those comments into account when deciding on a paper. A few years ago, Nature conducted a trial of open peer review. However, given the low uptake by other researchers as well as the generally low technical quality of the comments made, this trial was eventually abandoned3. Another example is double-blind peer review, where the identity of authors is withheld. However, experiments, for example at the American Physical Society, have concluded that it is difficult in practice to really conceal authors' identity from their colleagues.
In contrast to that, the reason for the more successful experience in the publication of the peer review files could be the minimal influence it has on the conventional process, while delivering the benefit of enhanced transparency. For example, in the EMBO model the editor's contributions would be more visible. This increased scrutiny of our efforts certainly could benefit science. After all, we editors have to take responsibility for the decisions that we take.
A further benefit would be that the extent to which a paper has been improved during the peer review process would become more obvious. This improvement can be quite substantial and the input provided by reviewers will be more identifiable and recognizable. Similarly, the relationship between reviewers and editors may also be influenced as reviewers will be aware that their comments could get published. On the other hand, there might still be parts of the editorial process that would not necessarily be apparent. In many instances informal communication between authors and editors as well as reviewers takes place. Nevertheless, all information relevant to the discussion would of course be contained in the official correspondence.
Scientific results are scrutinized at every stage, from the design of experiments or the derivation of theoretical findings to the discussions after publication. The issues raised during the peer review process constitute an important contribution to this debate. Therefore, the knowledge about these discussions has a value beyond their immediate purpose, which is a journal's decision whether to publish a paper or not.
Indeed, the benefits of the transparent review process seem to have convinced EMBO to extend this scheme to their other journals. Beyond these we are not aware of other trials of this scheme, particularly in the physical sciences. Of course, researchers from other fields may feel differently about this proposed transparency, but the benefits of this approach by far seem to outweigh the drawbacks. What is your opinion? We'd love to hear it.
About this article
Nature Materials (2013)