Peer review is a key element of scientific publishing. Here we discuss what constitutes the ideal referee report.
In recent years, ways to share and discuss science have expanded through increased interest in pre-publication deposition of manuscripts into preprint servers, post-publication commenting alongside the published paper, and online communities focusing on 'journal clubs' and post-publication manuscript assessment. Nevertheless, peer review remains the cornerstone for evaluating research papers before their formal public dissemination.
Peer review aims to subject a research study to critique by other experts in that scientific field, with a view to identifying shortcomings and strengthening the manuscript before publication — as such, ensuring that the process is constructive and timely is essential. However, no process is fool-proof, and peer review can be a source of frustration for not only the authors whose work is being subjected to scrutiny and the reviewers whose time and efforts are continuously being called on by research journals, but also the editors who are the guardians of the process, tasked with directing it and ensuring it is of high quality.
What are the core elements of successful reviewing? Good referees are experts in their fields who strive to review studies objectively and fairly. They provide a justified assessment of the potential significance of the work in light of the published literature and a thorough evaluation of its technical merits, as well as the logical consistency and quality of presentation. Importantly, reviewers should identify critical weaknesses, suggest experiments to bolster support of the central claims, and comment on the robustness, validity and reliability of data. Feedback on the adequate description of methodology, the appropriateness of experimental design and required controls, the validity of data analyses and statistics, and the quality and reproducibility of data is central in assessing the strength of a manuscript. To aid referees in this complex task, we include in the manuscript files the 'Reporting Summary' and 'Editorial Policy Checklist', documents for authors to fill out that summarize key elements of methodology, experimental design, statistics and reproducibility (Nat. Cell Biol. 19, 741; 2017).
Suggesting improvements for a study and identifying its limitations are at the heart of peer review. When deciding what revisions are needed, the good referee focuses on requests for experiments that provide further support for the main claims and rule out alternative hypotheses, strengthen causal links and explore regulatory mechanisms, and assess functional outcomes central to the study. But this should be within reason — although key gaps and logical avenues for further investigation should be noted, it is important to refrain from raising requests that are peripheral to the central message or beyond the conceptual scope of the work; such requests are routinely overruled by Nature Cell Biology editors.
To maintain consistency and avoid shifting expectations in subsequent rounds, when a referee agrees to review a new submission, they also make a commitment to review subsequent revised versions of the manuscript. This, of course, is additional work, but when evaluating a revision we ask referees to restrict comments to whether the points raised in the previous round were addressed successfully, and to the strength of the new data added in revision. However, critical shortcomings should be raised even if they were inadvertently missed in the previous rounds of review.
Refereeing is an involved, time-consuming endeavour. A careful balance is required: offering referees sufficient time to provide a thoughtful assessment, while also ensuring a timely review process for the authors. We appreciate that researchers' heavy schedules sometimes require extensions of reviewing deadlines; however, for the peer review to run smoothly it is important to honour reviewing commitments by submitting comments within the agreed timeframe. As editors, to ensure that referee time is not taken up unnecessarily, we vet revised manuscripts editorially and consult only with those experts whose feedback is necessary, rather than re-engaging the full referee panel automatically. We also aim to avoid constant calls on the same experts for multiple different manuscripts, by continuing to recruit new experts to refresh our referee pools.
A key aspect of peer review is that of mutual trust — authors entrust their unpublished work to a journal, whose editors then entrust it to a selected few expert referees for their remarks. Therefore, all requests to review and subsequent communications about the manuscript should remain strictly confidential. Equally, reviewers remain anonymous during and after the peer review process, unless of course they opt to reveal their identity by signing their report. Although co-reviewing with senior members of the laboratory is permitted within the same confidentiality confines, the reviewer originally engaged by the journal must notify the editor of the co-reviewer's identity, and, to ensure that confidentiality is not jeopardized, should not involve external independent investigators without first consulting the editor.
Nature Cell Biology editors discuss and critically assess referee comments in detail to reach an informed and fair decision. Within this process, we may consult further with referees — for example, to obtain necessary clarifications, or comments on points raised by another reviewer. After reaching our decision on the manuscript, we inform our referees of the outcome of the peer review and send them the full set of anonymous referee reports for an insight into how other experts evaluated the work and what drove our editorial decision.
The Nature journals have for many years offered our referees the option of obtaining a record of their yearly reviewing activity, which researchers can download from their journal account. However, we are acutely aware that such reports do not capture the hard work of our reviewers, the value that their contributions add to our publications, nor our appreciation for their commitment and efforts. We take this opportunity to thank all our reviewers, and welcome our readers' comments on how to improve our process for referees, and how to provide recognition for their essential contribution to scientific publishing.