Four years ago, the world of psychological research was rocked when an attempt to replicate 100 studies resulted in successful replications for just over one-third of them.
Such a discovery is not unique to psychology. Failure to replicate influential results has also been documented in preclinical cancer research, behavioural social science and experimental economics, among other fields.
One reason for this is publication bias: journal editors, reviewers and authors can favour positive, sometimes eye-catching results over negative findings. But negative results waste research funds and researchers’ time if they do not reach the research community through publication, and, in the case of clinical studies, can lead to fruitless patient involvement. According to one estimate, as much as 85% of research investment in the biomedical sciences is wasted because some studies are never published, are unnecessary or are poorly designed.
Researchers and funders in affected fields have been taking steps to address the problem, and one solution to emerge is a publication format called Registered Reports that some journals have eagerly adopted. Among them is Nature Human Behaviour, part of the Nature Research family of journals, which this week has published its first two articles in this format.
What is innovative about Registered Reports is that peer review and the decision to publish precede data collection and analysis. A researcher will submit what is essentially a detailed plan — a protocol — for a research project for publication in a journal. This will include the research question to be asked, along with a description of the study design and methodology, and a detailed analysis plan.
The journal’s editors will then decide whether the proposal merits peer review, based on the importance of the research question — not the potential novelty of the results. If reviewers are satisfied that the study is methodologically robust, the journal commits to publishing the work when it is complete, regardless of the final outcome.
Such a commitment to publish before results are known reduces the potential for publication bias. And the quality of a study is improved when researchers receive constructive peer-review comments before data-gathering begins. Crucially, potential problems in study design can also be identified and addressed before it’s too late to make changes.
One of the Registered Reports in Nature Human Behaviour addresses the long-standing question of whether people who have an accurate judgement of their own abilities have greater life, career and relationship satisfaction (J. C. He and S. Côté Nature Hum. Behav. https://doi.org/c8pr; 2019). The authors now conclude with some confidence that none of the five hypotheses previously floated in the literature is supported. Although this is a null result, it is a positive move forward for the field.
Around 200 journals across different fields now accept Registered Reports. The format, however, is less suited to research in which there is no clearly defined hypothesis or analysis plan, and it is rarely a good fit for exploratory science. That said, support from more journals in the life and social sciences — fields that need emphasis on study design — could accelerate take-up.
Registered Reports can also benefit funders, who want the research they fund to be well designed and robust, and to tackle important, unsolved problems. The Children’s Tumor Foundation, Cancer Research UK and Pfizer have, on occasion, partnered with journals on Registered Reports. The Templeton World Charity Foundation also requires the submission of outputs as Registered Reports for at least one of its funding initiatives. But, at the same time, it is imperative that, in such arrangements, journal editors and publishers retain the ability to make decisions independently of funders.
In research, process matters as much as the final result. That is why more journals should consider Registered Reports, and more funders should alert their grantees to a format that is helping to uphold the integrity of scholarship across disciplines.
Nature 571, 447 (2019)