There's a time to be critical

Journal name:
Nature
Volume:
473,
Page:
253
Date published:
DOI:
doi:10.1038/473253a
Published online

An accusation that referees are too demanding and editors too supine demands a response. Authors, editors and referees all have lessons to learn.

Last week one of our editors received the following from a referee of a paper currently under assessment:

“I guess the issue with this kind of paper is that there are an almost limitless number of changes/additions that could be made, especially considering the complexity of the data presented here. I suspect that this paper might run into a few reviewer 'issues' as it covers so much ground. In my review I have tried to be cognisant of your 27 April Nature article ('End the wasteful tyranny of reviewer experiments') and as such give this a 'yes' vote pending revisions.”

In the same week, we received a note from another reviewer to the effect that the “tyranny of reviewer experiments” had significantly increased the impact of the claims made in a manuscript he assessed, and he hoped that the authors would agree that the further work was worth the effort.

Clearly, some targets of the Nature article have taken note of it. In brief, that column, by Hidde Ploegh at the Massachusetts Institute of Technology in Cambridge, argued that referees too often ask for more experiments, and that editors too passively tend to pursue such requests (see Nature 472, 391; 2011).

But for the paper mentioned above, the question of whether further work is required is still open until the editor decides otherwise. Our editors must ask themselves: would further work lift the paper over a threshold of robustness or significance that justifies publication in Nature, or is it already sufficient? And have other referees differing views about this?

In resolving these questions, the editor will discuss the paper with colleagues and also with the referees.

“Referees generally put in very substantial amounts of labour on behalf of their fellow scientists.”

The accusation that editors are too passive was not specifically directed at Nature, but we take it seriously. We could too easily discount it on several grounds. Surveys of our published authors, as well as general surveys of scientists conducted independently, overwhelmingly support the view that papers have gained in their passage through peer review. Critics do not realize how much discussion and critical assessment underpins our editorial decisions. And without question, the ever-increasing pressure to publish is far too often leading authors to submit papers that would gain substantially in scientific significance with some further work.

It is important also to acknowledge that our referees generally put in very substantial amounts of labour on behalf of their fellow scientists, and make constructive suggestions that ensure that some of the extraordinary claims that Nature publishes are backed by the necessary evidence.

Nevertheless, a more reflective response is also required.

At Nature and at the Nature research journals, our teams of staff editors are expected to make their own conclusive judgements about a paper's position below or above their journal's threshold, and will often overrule referees' expectations in this respect in either direction. For example, we may decide that even if a paper lacks a new insight into mechanism, it represents a sufficient resource in the novelty of its data or technique to make a significant impact on the discipline. Conversely, we may decide that an additional piece of work would greatly increase a paper's range or depth of impact, and make that a condition of publication — we hope to the ultimate benefit of the community and the authors themselves (see Nature 463, 850; 2010).

But our editors do not necessarily have the expertise to judge whether, for example, an application of a novel technique or reagent has been adequately validated. Authors are free to challenge a request for more work in these circumstances, and an editor may seek technical advice from another expert to resolve the matter.

Spurred by this discussion, we looked back at recent decisions. We soon found several cases in which, with technical guidance where necessary, we overruled a referee's request for additional work — for example, when the editor felt that, contrary to a referee's assertion, the gain in robustness would not be sufficient to justify the effort and delay.

What lessons can be learnt, therefore? By authors: in the interests of robustness and genuine impact, resist the pressure to publish prematurely. By editors everywhere: don't be supine in the face of referees' requests.

And above all, by referees: please don't ignore any impulse to demand more, but be self-critical too.

Comments

  1. Report this comment #21786

    Xavier True said:

    "But our editors do not necessarily have the expertise to judge ..."

    Thank you for your honesty!!! Isn't that amazing, editor with no expertise judging and making decisions on scientific articles requiring the best expertise to evaluate, particularly for those submitted to high-profile journals like Nature, Nature Genetics, etc.?!

    Do you see anything wrong in this system?

  2. Report this comment #21790

    Xavier True said:

    One dirty trick some reviewers play is by demanding unnecessary work and thus delaying/rejecting the publication of the manuscript they review, they could gain more time and better opportunity to have their own similar work published, or even worse take ideas from the manuscript they review and publish them first. This has happened to me a few times for my submissions to Nature and Nature Genetics. This is the problem of anonymous peer review process and the lack of a mechanism preventing such things to occur on the publishers' side.

    On the other hand, the lax peer review process and the "nonexpert" judgement and decision by editors have produced a lot of trash publications in Nature, Nature Genetics, Science, etc., with big claims but no strong evidences or sound logic to support. Sensation is always what Nature and Science want for commercial purpose, but that does not produce good science. I and many of my colleagues would not go to these journals to find the best or even just descent publications in my field.

  3. Report this comment #21817

    Claudiu Bandea said:

    To paraphrase one of the highlights of this Nature Editorial response to the paper by ?Hidde Ploegh?: http://www.nature.com/news/2011/110427/full/472391a.html and the associated avalanche of comments, the current peer review system demands a solution.

    Before adding to the growing list of problems with the current peer review, I want to advance a potential solution inspired by the publication process of this very Editorial. I presume that: (1) one or more members of Nature?s editorial team wrote a draft of the article; (2) the draft was reviewed by some of their colleagues (definitely, not by their distant peers, such as those from the editorial staff of competing journals); and, (3) the manuscript was published in a format accessible to full, open (peer) review and evaluation. What a great model for the scientific publication process!

    One of the major weaknesses of the current peer review system is the limited number of peers reviewing a manuscript, usually 2 or 3, often chosen from a list provided by authors; this not only questions the relevance of the process, but it devaluates the highly honored paper-stamp: ?published in a peer-reviewed journal.? Another major weakness is the absence of public transparency of the reviewing process, which lowers the responsibility and the performance of both, the authors and the reviewers (as extreme examples, some authors submit poorly conceived manuscripts with the hope of eventually incorporating enough suggestions and ideas from reviewers to get the manuscripts published, and, for various reasons, the reviewers can intentionally slowdown the publication process). Moreover, the reviewers get no credit for their contributions, which sometimes can surpass those of the authors; needless to say, in time, this dampens their enthusiasm for making such contributions.

    By increasing the responsibility of the authors, whose reputation/record would depend on the submitted work, by having an open peer review that gets full recognition and become peer-reviewed itself, and by having editorial boards that guide this process, this model might be a viable solution for scientific publishing in the future. Come to think of it, this model might already be in place at Nature, which has opened its articles to a genuine peer review process, although not formalized as such. The question is will progressive scientists take advantage of it?

  4. Report this comment #21877

    R J said:

    The review system for high-impact Journals with professional editors is indeed flawed, and this editorial only highlights what is wrong. The response of the editor to valid criticism is to have a chat in the editorial office and come up with "several cases" (really that many) where "we overruled a referee's request for additional work". Oh so that's fine then, a bit of anecdotal evidence and we, as authors, are meant to be satisfied that all is fine.

    The main problem with professional editors is that in general they are completely out of their depth. A professor in my Department often uses the term ?failed academics? when feeling particularly hard-done by. As a reviewer, one often gets to see the comments of other reviewers. Frequently these contain inaccuracies, errors and in some cases show a startling ignorance of the subject. If professional scientists can get it so wrong, how can an editor, with only a few years research experience, hope to make a meaningful contribution to the review process? At least when the editor is a scientist, one can have scientific discourse about the pros and cons of a particular review. Unfortunately, with a professional editor this is often not the case, because they simply lack sufficient knowledge or understanding to get the point.

  5. Report this comment #21878

    R J said:

    The review system for high-impact Journals with professional editors is indeed flawed, and this editorial only highlights what is wrong. The response of the editor to valid criticism is to have a chat in the editorial office and come up with "several cases" (really that many) where "we overruled a referee's request for additional work". Oh so that's fine then, a bit of anecdotal evidence and we, as authors, are meant to be satisfied that all is fine.
    The main problem with professional editors is that in general they are completely out of their depth. As a reviewer one often gets to see the comments of other reviewers. Frequently these contain inaccuracies, errors and in some cases show a startling ignorance of the subject. If professional scientists can get it so wrong, how can an editor, with only a few years research experience, hope to make a meaningful contribution to the review process? At least when the editor is a scientist, one can have a scientific discourse about the pros and cons of a particular review. Unfortunately, with a professional editor this is often not the case, because they simply lack sufficient knowledge or understanding to get the point.

Subscribe to comments

Additional data