An accusation that referees are too demanding and editors too supine demands a response. Authors, editors and referees all have lessons to learn.
Last week one of our editors received the following from a referee of a paper currently under assessment:
“I guess the issue with this kind of paper is that there are an almost limitless number of changes/additions that could be made, especially considering the complexity of the data presented here. I suspect that this paper might run into a few reviewer 'issues' as it covers so much ground. In my review I have tried to be cognisant of your 27 April Nature article ('End the wasteful tyranny of reviewer experiments') and as such give this a 'yes' vote pending revisions.”
In the same week, we received a note from another reviewer to the effect that the “tyranny of reviewer experiments” had significantly increased the impact of the claims made in a manuscript he assessed, and he hoped that the authors would agree that the further work was worth the effort.
Clearly, some targets of the Nature article have taken note of it. In brief, that column, by Hidde Ploegh at the Massachusetts Institute of Technology in Cambridge, argued that referees too often ask for more experiments, and that editors too passively tend to pursue such requests (see Nature 472, 391; 2011).
But for the paper mentioned above, the question of whether further work is required is still open until the editor decides otherwise. Our editors must ask themselves: would further work lift the paper over a threshold of robustness or significance that justifies publication in Nature, or is it already sufficient? And have other referees differing views about this?
In resolving these questions, the editor will discuss the paper with colleagues and also with the referees.
Referees generally put in very substantial amounts of labour on behalf of their fellow scientists.
The accusation that editors are too passive was not specifically directed at Nature, but we take it seriously. We could too easily discount it on several grounds. Surveys of our published authors, as well as general surveys of scientists conducted independently, overwhelmingly support the view that papers have gained in their passage through peer review. Critics do not realize how much discussion and critical assessment underpins our editorial decisions. And without question, the ever-increasing pressure to publish is far too often leading authors to submit papers that would gain substantially in scientific significance with some further work.
It is important also to acknowledge that our referees generally put in very substantial amounts of labour on behalf of their fellow scientists, and make constructive suggestions that ensure that some of the extraordinary claims that Nature publishes are backed by the necessary evidence.
Nevertheless, a more reflective response is also required.
At Nature and at the Nature research journals, our teams of staff editors are expected to make their own conclusive judgements about a paper's position below or above their journal's threshold, and will often overrule referees' expectations in this respect in either direction. For example, we may decide that even if a paper lacks a new insight into mechanism, it represents a sufficient resource in the novelty of its data or technique to make a significant impact on the discipline. Conversely, we may decide that an additional piece of work would greatly increase a paper's range or depth of impact, and make that a condition of publication — we hope to the ultimate benefit of the community and the authors themselves (see Nature 463, 850; 2010).
But our editors do not necessarily have the expertise to judge whether, for example, an application of a novel technique or reagent has been adequately validated. Authors are free to challenge a request for more work in these circumstances, and an editor may seek technical advice from another expert to resolve the matter.
Spurred by this discussion, we looked back at recent decisions. We soon found several cases in which, with technical guidance where necessary, we overruled a referee's request for additional work — for example, when the editor felt that, contrary to a referee's assertion, the gain in robustness would not be sufficient to justify the effort and delay.
What lessons can be learnt, therefore? By authors: in the interests of robustness and genuine impact, resist the pressure to publish prematurely. By editors everywhere: don't be supine in the face of referees' requests.
And above all, by referees: please don't ignore any impulse to demand more, but be self-critical too.