First analysis of ‘pre-registered’ studies shows sharp rise in null findings

Logging hypotheses and protocols before performing research seems to work as intended: to reduce publication bias for positive results.

30 October 2018

Ariana Lindquist/Bloomberg/Getty

Registering research protocols in advance of data collection could change the findings.

Studies that fail to find a positive result are often filed away, never to see the light of day, which leads to a publication bias that compromises the credibility of scientific literature.

An analysis now suggests that registering and peer-reviewing study protocols before research is conducted could improve this ‘file-drawer problem’, and help to correct the existing publication bias towards positive findings.

Researchers from Cardiff University, UK, report what they say is the first analysis of whether the practice is effective. They find that studies for which protocols were pre-registered are much more likely than the general scientific literature to report null findings. The analysis was posted on 17 October to the PsyArXiv repository.

Better findings

In a registered report, researchers submit a study protocol to a journal before a study begins. If this passes peer review, the journal provisionally commits to publishing a paper when the study is completed, regardless of the results.

The practice has emerged in the science community in the past few years, building on initiatives such as pre-registration in clinical trials, which is required by law in the United States. There are currently around 140 journals using the format, and 130 registered reports that have published final results.

Proponents of the format hope that it will combat dubious research practices, such as formulating hypotheses only after looking at the findings or not reporting negative results.

To see whether registered reports increase the frequency at which null results are reported, psychologists Chris Allen and David Mehler analysed the outcomes of 113 registered reports in the biomedical and psychological sciences.

The pair identified 296 discrete hypotheses across those studies, and found that, overall, 61% of these were not supported by the results that those studies later published. For studies that sought to replicate previous findings, the percentage of null results was slightly higher, at 66%, whereas this figure stood at 55% for original research (see ‘Registered reports cut publication bias’).

These figures are much higher than the proportion of null results presented in the general scientific literature, which the authors estimate to be between 5% and 20%, on the basis of previous research.

Still not representative

The study might still underestimate the true proportion of null findings, says Anne Scheel, a psychology researcher at Eindhoven University of Technology in the Netherlands, who is also investigating registered reports.

Other research has estimated that the proportion of hypotheses tested in psychology which are in fact false is greater than 90%, she says, which would suggest that the rate of null findings in the current scientific literature is “drastically at odds with what we would expect without any publication bias”.

Allen adds that their analysis is exploratory, and that there could be other explanations for the findings.

For instance, says Scheel, people might use this format strategically for hypotheses that they anticipate will not work out, given that registered reports more or less guarantee publication.

Registered reports are a young format, she adds, and it may be that the studies published so far are not representative of the field of psychology more generally. “There are so many unknowns at the moment,” Scheel says. “But that also makes it a very exciting time for meta-scientists.”

The number of registered reports is growing exponentially, and Allen now hopes to conduct another study, with a larger sample size, to answer some of the questions the research has raised. And he plans to register it first.

This article was first published on nature.com