To improve your own papers, learn how to evaluate other scientists' work.
Before she had even defended her doctoral thesis, Brazilian student Rita Santos began to receive requests for her expert opinion. Her work on beak development in octopus larvae — along with her knowledge, care and keen judgement — had left an impression on scientists in the field and early on in her career, she was invited to become a peer reviewer.
Matthias Starck, a zoologist at the Ludwig Maximilian University of Munich in Germany and editor-in-chief of the Journal of Morphology, sent an invitation to Santos after receiving a recommendation from her supervisor. “I was a bit hesitant at first,” he says, “but the reports she turns in are just superbly thoughtful and well written.”
Peer review is the backbone of modern science, and academic researchers are expected to participate in the endeavour. Although time consuming, delving deeply into someone else's paper can benefit a scientist's own work. The process allows peer reviewers to read about research before it is generally known and to gain insight into how other scientists write manuscripts and present data. “I've learned a lot about science and the process of publishing it,” says Santos, who studies marine ecosystems at the Alfred Wegener Institute of Polar and Marine Sciences in Bremerhaven, Germany. “And you learn how to be critical without being impolite or discouraging to others.”
Whether or not they plan to pursue an academic career, junior researchers should get involved in peer review, says Sarah Blackford, a career adviser with the Society for Experimental Biology in London. “Not only will it help you to hone your power of judgement,” she says, “but it is also a great way to broaden your knowledge and demonstrate transferable skills for offering an authoritative view to your peers”.
Young scientists typically get their start as reviewers through supervisors or lab leaders, who may be overburdened or need to turn to junior team members who are familiar with specific methods or technology. Graduate students generally are not recognized for their ability to conduct independent peer review unless, like Santos, they are already establishing an academic reputation by publishing first-author papers. But they can gain experience by helping their supervisors or senior colleagues to prepare reviews.
“If I am too busy, or a manuscript is a little outside my field, there is nearly always an opportunity to propose postdocs and other early-career researchers who have expertise in the area requested,” says Ros Gleadow, a plant physiologist at Monash University in Melbourne, Australia. “They might then get invited by the journal to conduct the review.”
Even if they aren't invited, another natural first step is to review a paper jointly with seasoned colleagues or under their mentorship, says Emma Ganley, co-editor-in-chief of the journal PLoS Biology. Senior scientists might be better placed to judge a finding's weight and general significance, but junior researchers are often more up to date on methods and technology — proficiencies that any journal editor will appreciate.
“Young reviewers are extremely good in raising technical issues such as those related to microscopy or molecular techniques,” says Bernd Pulverer, head of scientific publications at the European Molecular Biology Organization (EMBO) in Heidelberg, Germany. Editors of EMBO journals encourage senior reviewers to involve trusted early-career lab members in peer reviews, provided that they have done experimentation in the relevant field. Their background experience will help them to carry out the key components of peer review: they must be able to assess whether work is new to the field and original enough to deserve publication — and by the journal in question.
They need to be able to evaluate the quality of data, look for potential inconsistencies and ascertain whether the methods and experiments are appropriate. If they see flaws or holes, they will be expected to suggest that the authors do more analysis or more experiments. And if they think that a paper is incomprehensible or biased (or plain tripe), they are obliged to tell journal editors just that.
Reviewers will assess whether a study is conceptually valid or technically sound, if its arguments are coherent and if claims and conclusions are sufficiently backed up by the data. Many journals will also ask whether the results challenge or confirm established concepts, and if they significantly advance the field at hand.
“Don't try to dictate to us what we should be publishing, but do provide strong arguments and detailed justifications of any statements you make,” says Karl Ziemelis, chief physical-sciences editor at Nature. “Just saying that this or that isn't a big deal in your field is much too vague. We would like to know why you think so, and how you came to that conclusion.”
The starting gate
New reviewers may be uncertain of what they are expected to produce and how overtly critical they should be. “I knew I was to assess the scientific strengths and weaknesses of the manuscript,” says Santos. “But I wasn't quite sure, at first, how deeply I should go into things like length, structure and language.” If they are at all confused, they should consult a seasoned reviewer, or contact the journal editor who commissioned the review, advises Ziemelis. They should also tell the editor if they feel that they might lack competence — or the time — to do a proper review. If the field in question is too distant from their own niche, they may need to decline to review a manuscript, or suggest someone who is more appropriate.
boxed-text“Do tell editors if you are happy to comment on one aspect of a paper but not on another,” says Pulverer. Journal editors also appreciate it when a researcher recommends colleagues who might be better placed to evaluate a paper or any specific aspects of the science. If a peer reviewer brings in a student or technical specialist to help out, those people should be named as contributing reviewers.
Similarly, Ziemelis says, researchers should tell the journal editor if they think that they are too closely affiliated with an author to judge the science neutrally. Any conflict of interest — personal, financial or owing to direct competition — renders a scientist unsuitable as a reviewer. It is always better to over-declare than to under-declare, says Irene Hames, an independent publishing consultant in York, UK, and former director of an international organization called the Committee on Publication Ethics.
Novice reviewers should also find out whether the journal offers 'double-blind' peer review, in which authors can request that their names and affiliations be withheld. A reviewer will need to decide whether she or he is comfortable reviewing the work of an anonymous author. Conversely, in the case of 'open' peer review, the author's and reviewer's identities are disclosed. But this model offers new reviewers the chance to look at what others have written and how authors have responded to comments.
If a junior researcher is contacted by a journal that they have never heard of, they should be cautious. An invitation from what might be a new or relatively unknown small journal isn't necessarily a reason to decline, but journals with questionable peer-review and publishing standards are increasing in number. If a journal says that it is open access, researchers should check whether a journal is listed on the Directory of Open Access Journals or the Open Access Scholarly Publishers Association. They should look for recognized experts on a journal's editorial board, and contact them to verify credentials and peer-review standards.
The write up
The review itself involves several steps (see 'Become a peer-review legend'). The first is to plan enough time and to stay in close contact with editors. There is no one-size-fits-all estimate for how long it takes to write a good review, but scientists should expect to spend at least eight hours and up to several weeks, say veteran reviewers.
Sloppy work or unresponsiveness might prompt editors to drop a reviewer — which could mean losing the respect of peers and colleagues and diminishing the chance of being added to editorial boards. It could also taint a researcher's reputation with editors of journals in which they may want in future to publish their own work.
Think how you would like a constructive review to be written if you were the author.
After an initial general read of the manuscript, novice reviewers should wait a full day or so before getting into the technical details and starting to draft a properly phrased review, says structural biologist Stephen Curry of Imperial College London. “Sit back and think how you would like a constructive review to be written if you were the author of the paper,” he says. Snarkiness or scorn should not be present. “Derisiveness, aggressiveness or rivalry have absolutely no place in a review,” Curry adds.
The document should start with a short, cohesive summary of the paper, says Pulverer, followed by comment on experimental design and the validity of controls. A key point of any review of biological work, he says, is whether the data and their interpretation support the reported findings. “We'd like reviewers to outline precisely what extra tests they think are needed and why,” he says. Reviewers should also make clear whether more experiments are essential or merely desirable.
The specific technical and editorial advice that reviewers are expected to provide depends largely on the subject area and the scope of a journal. Validating a twist in string theory or cosmology calls for a different approach than reviewing the results of an astronomical observation, geological fieldwork or clinical trials. If asked to assess theoretical work, a reviewer should focus on equations and their interpretation.
Most studies will require reviewers to examine observational and experimental data contained in supplementary material (or external repositories) and their representation in graphs and figures.
Reviewers should check guidelines for authors and reviewers carefully to be sure that they properly understand a journal's scope, how novel and 'big' any science must be to get published there, and whether referee reports and the authors' responses will be published online.
If the latter is the case, as it is for the EMBO journals, scientists should look at other reviews and authors' responses. This is a good way for novice reviewers to get a sense of the appropriate length and structure expected, and of the journal's overall review process, says Hames. If such information is ambiguous or unavailable, they should ask the journal for specifics.
Assessing the work of others nurtures critical thinking in ways that few other ventures can match. But at the end of the day, says Alaa Ibrahim, an astrophysicist with the American University in Cairo, it is good for authors to have others dissect their submitted work. “The worst thing,” he says, “is that your science gets published just to be proven faulty or wrong soon after.”
Related links in Nature Research
Related external links
About this article
Cite this article
Schiermeier, Q. Peer review: Close inspection. Nature 533, 279–281 (2016). https://doi.org/10.1038/nj7602-279a