Progress in the transparency of both research and editorial processes is gathering pace. This was demonstrated at the International Congress on Peer Review and Scientific Publication in Chicago, Illinois, earlier this month, and in various discussions that are under way among publishers, researchers and others.
The examples given here relate to initiatives by the Nature Research journals, some of which follow pioneering work by other publishers.
Take the improvements in researchers’ descriptions of what they did and did not do in their experiments. One such initiative is the checklist introduced by Nature and the Nature journals in 2013 for life-sciences submissions.
At the congress, Malcolm Macleod of the University of Edinburgh, UK, and his colleagues discussed the results of an independent study of the impacts that this checklist has had on Nature journals’ content. They looked at the completeness of reporting in journals following the initiatives. They analysed papers published in Nature journals — 223 submitted before May 2013 and 225 after. They looked for whether and how authors had identified and addressed sources of bias. They found that the proportion of papers reporting on all four measures — randomization, blinding, exclusions and sample-size calculations (their selected ways of mitigating bias) — increased from zero to 16%.
There was no such growth in a set of equivalent papers from outside the Nature group. Meanwhile, reporting on individual criteria and statistics increased markedly in the Nature journal papers.
We have highlighted elsewhere some of the further steps we have taken. And we have heard anecdotally from some researchers how this has begun to influence the design of their experiments.
Five steps to transparency
Credit to Macleod and his colleagues: there were no fewer than five welcome types of transparency in this project itself. These embody a gradual trend in which the public release of research results is moving farther away from the traditional form of a single, wrap-up publication.
First, the authors published a formal research protocol in a peer-reviewed journal (F. Cramond et al. Scientometrics 108, 315–328; 2016). Such publications are a mechanism, already established in clinical and other interventions research, by which authors ensure that their research is well designed. Editors report that the peer-review process of these papers is much more collaborative in spirit than it is for papers making claims about results. (Even better if the journal commits to publishing the outcome regardless of its conclusions, which avoids pressures to cherry-pick data or model results. Nature Human Behaviour is so far the only Nature journal publishing such ‘pre-registration protocols’.)
Second, the authors posted the final draft paper describing their conclusions on a preprint server before submission (M. R. Macleod and the NPQIP Collaborative Group. Preprint at bioRxiv http://dx.doi.org/10.1101/187245; 2017). Third and fourth, the group released the data-analysis plan and the analysis code before data collection was completed. These were registered on the Open Science Framework (https://osf.io/mqet6/#). Fifth, the complete data set was publicly deposited on Figshare (http://dx.doi.org/10.6084/m9.figshare.5375275.v1; 2017).
This is an example of the research process being disaggregated, publicly, into its components: peer-reviewed research design, a preprint of outcomes that invites community responses, the release of code and data, and final publication. Such a practice allows greater access to the thinking behind a project. It also provides an opportunity to directly distribute credit to the authors for their efforts on the various components.
“Such a practice allows greater access to the thinking behind a project.”
In peer review, examples of experiments and innovations abound. The Nature Research group has recently run four separate initiatives. One is double-blind peer review, in which authors’ identities are hidden from referees. Since this was introduced for all the Nature journals in 2015 as a standard option, author take-up has been between 9% and 14% across the journals.
Other initiatives pursue greater transparency. On Nature Communications, following the example of other publishers, such as EMBO Press, the default since January 2016 is for authors to have the anonymous referees’ reports and their responses published with their paper. Authors can opt out, and about 60% of papers have their referees’ reports published. Authors in ecology and evolution are the most positive, and those in some areas of physics significantly less so (see Nature Commun. http://dx.doi.org/10.1038/ncomms13626; 2016).
Other exploratory initiatives reflect a community desire for greater transparency. For example, in a trial on Nature, we have since March 2016 allowed referees to be accredited on the published paper if they wish. So far, the proportion of referees across the disciplines who have selected this option has been about 50%. If this is extended to Nature Communications, it will be interesting to see whether referees will want to include their names on the reports that are already displayed with the paper. Other publishers’ experiences suggest that many will not.
In a separate trial that started this month, Nature Communications is being open about its submitted papers. The journal is pointing readers to the authors’ submitted version if it is posted on a preprint server, once the paper has been selected for peer review (see go.nature.com/2fmvtrj).
Are there risks to all this transparency? It may give rise to different sorts of bias. For example, we hear from some authors that they don’t want to know who authored a positive review, so that they can avoid future positive peer-review bias themselves. Meanwhile, some researchers and editors fear that referee identification encourages positively biased or softened peer review.
As Nature and the Nature Research journals explore ever-greater transparency in editorial processes and support it in research processes, we welcome readers’ thoughts and suggestions: firstname.lastname@example.org.
- Journal name:
- Date published: