Two assessments of the refereeing process highlight challenges for journals.
It would be misguided indeed for Nature to have any competitor's sense of Schadenfreude over Science's experiences with two papers on human embryonic stem cells by Woo Suk Hwang and his colleagues. It is possible that we at Nature would have published the papers had they been submitted to us instead.
The external review committee, which included Nature's US executive editor Linda Miller, earlier this month endorsed the general level of care exercised by Science. It also stated that, cumulatively, concerns raised during the refereeing processes should have been pursued more sceptically. Hindsight is always 20/20, however, and it is not at all clear that the questions were sufficiently pressing that any journal would have pursued them, given the degree of trust that is required between authors and editors.
It is precisely this degree of trust that the review panel goes on to challenge. Given the fact that cases of fraud demonstrably make it through refereeing processes, and given the importance of public trust in science, it proposes that journals apply additional scrutiny and risk assessment to papers that are likely to have a significant public impact, such as those with direct implications for policy, public health or climate change. The additional scrutiny recommended by the panel includes greater attention to raw data and a clarification of the contributions of each co-author.
We at Nature, like the editors at Science, accept that this challenge has to be addressed, and we have accepted their invitation to deal with such matters collaboratively. The key is to raise editors' and referees' practical awareness of the risk of deception. A conscious risk assessment, in which the likelihood of deception is explicitly analysed, is much easier said than done. An elementary checklist of risk factors can readily be introduced. But it would, for example, be quite inappropriate to single out papers from Seoul National University, or on human embryonic stem cells, and apply higher thresholds of proof to them.
On the other hand, we do already seek to ensure that major claims are backed by rigorous data and argument. Nevertheless, the sad fact is that high-profile and problematic papers have occasionally slipped through the net, and we accept that this underlines the need for enhanced attention by editors.
Nature and the Nature research journals already encourage the specification of authors' contributions to papers, and the uptake of this by authors has increased greatly in the past year — a fact that is welcomed by some funding agencies. We now intend to conduct a survey to help us decide whether to make this practice compulsory, and we would welcome readers' feedback.
It is an interesting question whether a more open peer-review process might have led to the detection of Hwang's fraud. At present, however, the level of interest in open peer review is too small to hope for such an outcome. That, at least, is the implication of Nature's trial of open peer review, the results of which can be found in an online debate on the subject (http://www.nature.com/nature/peerreview/debate/nature05535.html).
In the trial, the papers selected for traditional peer review were, in a parallel option offered to authors, hosted for public comment. In the event, 5% of authors took up this option. Although most authors found at least some value in the comments they received, they were small in number, and editors did not think they contributed significantly to their decisions.
This was not a controlled experiment, so in no sense does it disprove the hypothesis that open peer review could one day become accepted practice. But this experience, along with informal discussions with researchers, suggests that most of them are too busy, and lack sufficient career incentive, to venture onto a venue such as Nature's website and post public, critical assessments of their peers' work.
Another form of peer review emerges after publication, when work is replicated — or not. If this kind of discussion is to make it into the open, rather than be confined to gossip at conferences, it requires a forum where peers are able to comment on individual papers, with minimal editorial intervention. Would commenting on Nature papers be more widely adopted by researchers after they have been formally published than before? We intend to introduce this function next year, and find out.
About this article