Hannah Hobson, a Lecturer at the University of York, published a Registered Report as part of her PhD and explains how this decision took the stress out of publication and brought the joy back into data collection.
In the windowless suite in which I collected electroencephalogram (EEG) data for my PhD, it could have been any time of day or night. It looked the same whether I was setting up for my first participant in the morning or rinsing the last of the disinfectant from the EEG caps in the evening. As PhD students the world over will attest, collecting data is hard work. Imagine then my heartache when I ran my analyses, having collected over 100 EEG recordings, and found so few significant differences between my experimental conditions.
In fact, there was no heartache at all. I shrugged my shoulders, wrote up the rest of the paper as planned, and it was published within 6 months in a respected journal.
This is, of course, not usually what happens to such results. The jaded midcareer researcher may not bother to even attempt to get them published; instead, the nonsignificant findings rot in the infamous file drawer. However, for the PhD student who still needs those first publications and who does not have as many projects in process as a more senior researcher, it may be critical that there is some publication of these data. Their (statistically) nonsignificant findings thus begin a long and potentially fruitless journey, getting serially submitted, rejected, reformatted, re-submitted, rejected again, reformatted again, and so on. At some stage, the researcher may decide to perform more exhaustive analyses of their data, beyond the scope of their original hypotheses, in an attempt to find a significant result somewhere (p-hacking). Of course, for these new analyses to make narrative sense, the researcher has to change the story of the paper: the original hypotheses fall by the wayside, and new hypotheses are constructed (a practice known as ‘HARKing’: hypothesising after results are known). The paper they are now submitting is quite far from the paper they planned and is now less reproducible than their original nonsignificant results—but at least it has a better chance of getting published.
My findings, however, made it out without me having to resort to such measures. The reason for this is that my project was conducted as a registered report. In this process, authors first submit their introduction, methods and planned analyses, accompanied (usually) by power analyses and possibly some pilot data, before any of the data for the main analyses have been collected. This Stage 1 submission is reviewed and may be accepted in principle, if the reviewers agree that the study has an interesting hypothesis and sound methodology and analyses to test it. Researchers then collect their data and analyse them, as they said they would in their Stage 1 submission.
The overarching goals of registered reports can be argued to be two-fold, acting respectively on the reviewers and the authors. Firstly, reviewers cannot (consciously or unconsciously) recommend rejection of a paper on the basis of nonsignificant findings, as they are reviewing the paper prior to data collection. Secondly, registered reports prevent and dis-incentivise problematic researcher practices (such as p-hacking and HARKing), behaviours largely aimed at finding a significant result somewhere in the data to help get the paper published. Registered reports literally prevent researchers from p-hacking and HARKing, as they cannot adjust their Stage 1 submission, but also: why would researchers need to? If the publication is not dependent on the results being significant and telling a neat story, what would be the point?
Registered reports thus help to prevent both reviewers and authors from engaging in practices that threaten the validity of science. But there are also selfish reasons for doing a registered report, especially as an early-career researcher. As a graduate student or postdoc, it is you who will be collecting the data. Individual projects thus represent huge time investments. Under the traditional publishing process, there is often no guarantee that this investment will pay off. Yet I enjoyed the freedom to just get on and conduct my research study the way I had planned it, without fear that non-significance would throttle my chances of publication. In an academic environment where junior scientists can feel judged on things that are outside the realm of their control (I hope we can agree that p values should be beyond the realm of anyone’s control), registered reports offer an alternative way to do science.
My registered report—my first publication—was published in 2016, and at the time many were still sceptical of registered reports. Since then, over 200 journals have offered authors the opportunity to conduct their work as a registered report, and the format has been adopted in fields beyond psychology and neuroscience, such as ecology and economics.
Registered reports are not appropriate for all research questions (exploratory research, for example) and by themselves are not enough to resolve the issues of reproducibility faced by the biomedical and social sciences. An individual researcher will also need to consider carefully the timing of conducting a registered report, as they must factor in time before data collection to allow the paper to undergo Stage 1 review, though in my experience, this is more than made up for at the end of the process when you do not have to approach multiple journals to try and publish nonsignificant or unexpected findings. Registered reports represent a powerful tool for conducting confirmatory research, and they have unintended but nonetheless great advantages for the early career researcher.
The author declares no competing interests.
About this article
Cite this article
Hobson, H. Registered reports are an ally to early career researchers. Nat Hum Behav 3, 1010 (2019). https://doi.org/10.1038/s41562-019-0701-8