What makes the ideal referee report?
There is no simple answer to this question, and it may depend on one's role in the peer-review process. An author probably hopes for a report that is returned quickly and is positive, or at least constructively critical. An editor will also appreciate a quick response, but will be most happy with a report that provides insightful comments that help to inform a decision. The perfect report for a reviewer is more difficult to define; perhaps it is one that is not misinterpreted, and ultimately improves a manuscript.
The peer-review process at Nature Chemistry is not a matter of counting votes1. Reports simply stating that a manuscript should be accepted or rejected, without providing justification for that opinion, are rarely useful. It is unlikely that two brief reports that say a manuscript is 'great' and recommend publication without any compelling reasons to back up these statements will outweigh a thoughtful and well-supported report that highlights lots of technical flaws or other problems. Similarly, a comprehensive and well-reasoned positive report will carry more weight than two brief negative reports that provide little evidence to support their point of view.
We do provide guidelines for referees on our website2, but based on our experience of the process — taken together with feedback from our authors and referees — we suggest the following guidelines that try to satisfy the needs of everyone involved.
Reports should begin with a short summary of the work in question, but nothing more than two or three sentences are necessary. This serves to focus the review, clearly stating how the work is viewed by the referee and highlighting any problematic differences between how the authors and readers will interpret the claims made. It can also help to avoid the oft-raised criticism 'This reviewer has clearly not read my paper', which we are certain is rarely (if ever) the case.
Reviewers may have been selected to cover different aspects of a piece of work, particularly in the case of interdisciplinary research. It will help to state upfront (or at least inform the editor) if there are parts of a paper that you don't feel comfortable evaluating. That said, reviewers are still welcome to provide opinion on areas outside their own expertise, and such information can be valuable to the editor in judging the appeal of the work to a general audience. If, for example, you are an experimentalist being asked to review a theoretical paper, then comments on how the work fits into the field and how it will influence future experiments could be the most valuable input you can provide. Authors should keep in mind when reading reports that an individual reviewer may have been chosen to represent a particular point of view.
The summary should be followed with a brief discussion of what relevant related research has gone before, from both the authors of the current work and from others, in an attempt to define the advance that has been reported. What have we learned in this paper? How and why does that change future directions in this research area? These comments are most valuable when backed by references: the comment 'this has been done before' is not helpful unless we know where it was published and by whom.
Following on from this, a summary of both the merits and problems of the research can be quite useful in order to focus on the most pertinent issues. This can be far more instructive than simply writing a report with a particular outcome (accept/reject/revise) in mind or aiming to justify a definitive recommendation.
“A good report should clearly distinguish between the claims made and their importance versus the evidence presented in support of those claims.”
A good report should clearly distinguish between the claims made and their importance versus the evidence presented in support of those claims. The data in a manuscript — whether experimental or theoretical — should fully support the conclusions. This brings us to another issue frequently raised in criticisms of peer review: the need for additional work. No-one sets out to be the dreaded 'third reviewer'3, that infamous being who can never be convinced of the merits of a piece of work. Although requests for additional experiments are regarded by some as a 'tyranny'4, such requests are often quite reasonable. And they can ultimately be the component that elevates a paper from just technically sound to truly ground-breaking.
To make it clear just what a reviewer's motives are for recommending extra work, it may be useful to divide suggestions for additional experiments into different groups. First, and most important, are those that are considered necessary to support the specific claims of the paper and are likely to be required for publication anywhere; this could include omitted control experiments or requests for complete characterization. Second are those that may allow for broader conclusions to be drawn and improve the appeal to a general audience — including, for example, extending the scope of a described reaction or showing that a conclusion is valid for other systems. Third are those experiments that are not essential, but might provide interesting avenues for future studies.
Ground-breaking research frequently generates new questions as well as answering old ones. There may be two (or more) equally valid interpretations of a data set, and referees should not necessarily insist that authors confirm one in particular unless there is an obvious solution that is technically feasible on a reasonable timescale. It may be more appropriate to request further explanation that highlights such problems. The data themselves and the discussion that results are likely to be valuable to the community in shaping future understanding.
What is not usually necessary is a list pointing out every error in spelling and grammar: proofreading is not the role of referees. That said, the writing should be as clear as possible, so reviewers are encouraged to point out areas where they feel language to be too specialized for the general reader and that could be improved without detriment to the scientific content of an article.
Many of the reports we receive are very good and easily fulfil the criteria discussed above. We realize that reviewing places a heavy burden on a researcher's time, so we are extremely grateful to those reviewers (already numbering in the thousands) without whom Nature Chemistry could not function.
Nature Chem. 2, 1 (2010).
About this article
Nature Reviews Chemistry (2019)