Referees provide an invaluable service in advancing science. We offer some food for thought about how to effectively peer review methods papers.
As a referee, you know that judging someone else's work is not easy and takes substantial time and care. As an author, you have likely seen that the quality of referee reports can vary wildly. As editors, one of our essential roles is to ensure that every paper we consider for publication undergoes a peer review process that is as thorough and fair as possible. For the next time that a review request from Nature Methods lands in your inbox, we offer a few pointers for constructively reviewing a methods paper.
First, ask yourself whether you will be able to provide a fair and timely review. If you have agreed to review, make every effort to be on time and contact us if you will be delayed or will not be able to submit your report. Remember that agreeing to review a paper also implies a commitment to answer follow-up questions from the editor and often to review a revised version later on.
Take a close look at the author list. If you cannot be objective because of present or past collaborations, or because you're a close competitor working on the same problem, decline to review. If you're unsure about a potential conflict, ask the editor. If you do not think that the paper fits your expertise, explain your limitations. The editor very likely has a specific reason for contacting you; for example, we often invite target 'end users' to help us evaluate the potential interest in a new method or the validity of the results. Still, it helps us to know if there are technical aspects you cannot evaluate. Suggestions of alternate experts are extremely helpful.
Upon accepting a request to review a paper, we hope that you will carefully read our email containing instructions for reviewing work submitted to Nature Methods. This letter also contains details of our requirements for different formats (whether an Article or Analysis, Resource or Perspective) and may have additional editorial queries specific to the paper. Our “For Referees” site also provides general information about writing an informative review. We appreciate if you tell us right away if you run into fixable problems, such as trouble accessing data or running a software tool, so we can ensure that you have all you need for a constructive review.
It should go without saying that the review process is confidential. We do, however, encourage principle investigators to involve members of their lab in the review process. Those working at the bench are often very qualified to evaluate technical aspects of methods, and it is important for training the next generation of referees. But please tell us who assisted you, to give credit where it is due.
We are interested in our referees' opinions about novelty and potential impact. Does the method address a real technical challenge in the field? Do the authors appropriately place it in context with the existing literature; are any important citations missing? Will it lead to new biological insights? Will it be useful to a broad community? Will it have applications beyond those shown in the paper?
We want to know whether the method is technically sound. Did the authors design and perform their experiments appropriately, in keeping with field standards, and include suitable controls and statistics? Did they appropriately validate the method using gold-standard data or other benchmarks? Did they sufficiently compare method performance to previous approaches? Are the conclusions supported by the data? Are there any serious limitations? Do they include sufficient technical description to allow a reader to reproduce the method? Ideally, a referee will look through not only the main paper, but also the methods section, supplementary information, reporting checklists, and, in some cases, even externally deposited data. For papers reporting new software, we request that referees perform a reasonable test of the tool and report any issues regarding usability, documentation and performance.
Both authors and editors welcome thorough but succinct reviews. Referees should make suggestions for addressing central aspects of the work that are lacking, not propose a wish list of nonessential experiments. You also need not highlight spelling or grammatical errors; these will be fixed prior to publication. Of course, we do want to know if poor writing obscures what the authors actually did. Reviews that are polite are by far the most effective; comments that are overly harsh or attack authors' scholarliness could cause authors (and editors) to question whether a referee is unbiased.
The anonymity of peer review means that referees are rarely recognized and celebrated for their crucial role in advancing science. Some initiatives, such as Publons, which recently began a partnership with Springer Nature, or a Nature pilot project to print the names of referees that participated in a paper's review, aim to rectify this. In looking forward to broader adoption of such formal mechanisms, we also say a hearty thank you to our referees; we are enormously grateful for your hard work.
About this article
Cite this article
The good referee. Nat Methods 15, 91 (2018). https://doi.org/10.1038/nmeth.4603