Nature Methods publishes less than 10% of the manuscripts it receives. This percentage reflects a choice of favoring quality and significance over quantity, to serve an interdisciplinary audience. The selection of papers presenting methodological developments that are likely to be influential relies, in the Nature tradition, on stringent peer review. Now, you may ask—and some of you have asked—how are methods peer reviewed?
Nature Methods' Articles report novel methods permitting experiments or measurements that were not possible before. Its Brief Communications describe new tools or improvements to existing methods, but only if they are significant enough to open new experimental doors. In both cases, you are entitled to ask, how can referees, short of trying their hands at a new method, evaluate its merit?
Our first line of action, of course, is to put the onus on the authors to include in the manuscript a thorough characterization of their new method's performance and sufficient validation data to demonstrate that it represents a real advantage over available approaches.
A team of full-time editors acts as a first filter by evaluating each manuscript for novelty, interest for our readers and chances to compete in peer review. At this stage 75 to 80% of manuscripts are rejected without further review—a decision based on editorial and scientific criteria, sometimes after consultation with experts. Each manuscript receives the undivided attention of at least one editor and is fully discussed among the editorial team. Yet the process avoids futile cycles of review for manuscripts that are better suited for a more specialized journal, saving time for authors and reviewers.
In a second step, editors select typically three reviewers, experts in the technologies at the core of the new method. Often, we also include one expert we call the 'end user': someone who because of their research interest could adopt the new method. This combination of expertise allows us to evaluate the manuscript from various angles, including technical merit and significance for potential users. The downside is that reviewers often disagree.
But the primary role of the reviewers is to provide information on which editors can base their decisions. We ask the reviewers to comment on the strength of the evidence provided, the level of innovation and the relevance for addressing important biological questions. Other questions (see our guide to referees at http://www.nature.com/nmeth/referees/index.html) aim to determine whether the new method is appropriately compared to available techniques, and whether the provided technical details are sufficient for the method to be applied by an independent party. Finally, it is helpful when the reviewers offer suggestions on how to strengthen the manuscript.
Armed with such information, the editors are in a reasonable position to make decisions, which are never a mere counting of votes. Each argument is weighed against the others, decisions are discussed between the editors, and sometimes we consult again with the referees. There have been cases in which the enthusiasm of the end user—or conversely their criticism—has prevailed over the level of technical innovation perceived by the other reviewers. In other cases, the 'technology referees' have pointed to a technical flaw that has overcome the end user's enthusiasm.
Another important parameter in the review of methods papers is the referee's level of involvement with the technology in question. In some cases, technical details likely to influence the assay performance may only be brought up by someone who has a current and extensive practical experience of the technology, such as a seasoned postdoc who uses the technology day in and day out. In fact, some of the best, most rigorous and informative reviews at Nature Methods are collaborative efforts between a principal investigator and a senior member of their laboratory.
Such collaboration is beneficial for the review quality as long as the principal investigator remains involved, to contribute a wide experience of the field and to provide guidance. After all, learning how to conduct peer review should be part of a researcher's education. But editors wish to know who participated in each review. It is only fair to give credit where credit is due, and it helps us discover the next generation of reviewers. Making editors familiar with young researchers' review styles is giving these scientists a step ahead in becoming independent reviewers and in having their own voice in the peer-review system—ultimately shaping the quality of the science they will read.
Although we do not claim it is perfect, we believe in this method of reviewing methods papers. As always, we welcome your comments. Importantly, let us take this opportunity to say how extremely grateful we are to the referees who spend considerable amount of time and effort to provide good reviews. No matter the method, the peers are the key to quality.