What next for registered reports

As adoption of registered reports is growing, two pieces in this issue take stock, providing recommendations and outlining next steps. We complement these pieces with practical advice on how to prepare a successful stage 1 submission.

When we launched Nature Human Behaviour in January 2017, we adopted the registered report format and have been championing it for confirmatory research ever since. In this issue, Hardwicke and Ioannidis present an audit of the format across journals, mapping its growth as well as current challenges. In an associated Correspondence, Chambers and Mellor provide an update of the efforts of the Registered Reports Committee and the Center for Open Science to address these challenges.

Registered reports grew out of an increasing concern that standard publishing practices unwittingly encourage questionable research practices — an emphasis on the results of research has been argued to promote selective reporting of only those results that fit with a ‘clean’ story, p-hacking (running multiple alternative analyses until a significant result is obtained) and HARK-ing (hypothesizing after the results are known). Registered reports offer a radical alternative: we peer review the research protocol (essentially the first half of a research paper, including an analysis plan) before data collection and commit to publishing the full paper regardless of what the results are, if the research question is important and the methods robust. After data collection, the full paper is peer reviewed again, but can only be rejected if the authors did not adhere to their protocol.

The registered report format allows us to focus on the importance of a research question and the quality of the methods, without any danger of being distracted by the shape of the results. Editorially, evaluation of registered report protocols (or stage 1 submissions) means answering some of the questions that we consider for every submission: Is the research question of broad interest or specialist? Is it central to the field or peripheral? Of particular relevance for registered reports is a third crucial question: Is the research confirmatory or exploratory?

Confirmatory research is anchored in testable theories that allow clear predictions — this is the type of research for which registered reports were created. The data pattern that would be expected if a specific hypothesis was true must be clearly distinct from the data pattern that would confirm the null hypothesis. Whether authors clearly come out in favour of one hypothesis over another does not affect the evaluation. In other words, confirmatory research doesn’t have to confirm hypotheses, or directly replicate past research, but it needs to be able to either confirm or disconfirm specific theories.

Registered reports address publication bias by ensuring that all results, including null, are published. To ensure that the results are robust and interpretable, regardless of their direction, we consider, and ask the reviewers to consider, whether the proposal contains outcome positive tests and if the study or studies are adequately powered to detect the effects of interest. We ask that studies using frequentist statistics have a priori power equal to or greater than 0.95 to capture the smallest theoretically meaningful effect, while studies that implement inference by Bayes factors must collect data until the Bayes factor in favour of either the experimental or the null hypothesis exceeds the value 10.

Registered reports typically undergo two rounds of review before they receive stage 1 in-principle acceptance. They are typically reviewed by two to three experts whose expertise covers the research question and methodology, as are all other research manuscripts. The reviewers are invited to comment on a specific set of questions (see Box 1). In addition, registered reports typically have one additional reviewer who is highly familiar with the format and vets how closely the protocol fulfils the format-specific requirements. In rare cases, we invite an additional expert on aspects of statistics, for example, power analysis. Tips on how to prepare a strong registered report are listed in Box 1.

Box 1 The editors’ tips and requested referee comments contain questions authors should ask of their proposal to prepare a successful stage 1 submission to Nature Human Behaviour

Editors’ tips for authors. Considering the following questions will help authors to prepare a strong stage 1 protocol. These points should be addressed, regardless of whether the protocol describes a direct replication or original confirmatory research.

Merits of the research question:

• Does the proposal represent confirmatory research, with a clear theoretical and/or evidence-based foundation or a direct replication?

• Does the report contain sufficient evidence for the significance of the research question for a broad audience and the plausibility of the research question?

• For direct replications, has the to-be-replicated study been influential and continues to be so, has it inspired significant research and/or theorizing?

Rigorous methodology:

• Are the experiments sufficiently powered? Tip 1: for frequentist power analysis, review the literature and — if available — combine the information with any pilot data that shows feasibility of the approach; tip 2: consider whether Bayesian sampling is preferable and more efficient; tip 3: define hypotheses in repeated-measures designs as simple contrasts, rather than interactions for design efficiency

• Are outcome-neutral conditions or experiments included?

• Is the analysis described to the level of detail that the work could be fully replicable from the details in the report? Tip: include a pre-coded, documented analysis pipeline

• If the registered report contains a model, is the model pre-coded and the search parameter space defined? Tip: include model code and simulated data

Editors’ requested referee comments. The following points are the aspects that reviewers are asked to consider in their reports. Anticipating these questions will help to prepare a strong proposal.

• The significance of the research question(s) and whether it is suitable for a broad, multidisciplinary audience

• The extent to which the proposed study can satisfactorily answer the research question(s)

• The logic, rationale and plausibility of the proposed hypotheses

• The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis)

• Whether the clarity and degree of methodological detail would be sufficient to replicate exactly the proposed experimental procedures and analysis pipeline

• Whether the authors provide a sufficiently clear and detailed description of the methods to prevent undisclosed flexibility in the experimental procedures or analysis pipeline

• Whether the authors have considered sufficient outcome-neutral conditions (for example, absence of floor or ceiling effects; positive controls) for ensuring that the results obtained are able to test the stated hypotheses

A stage 1 registered report will typically be rejected, either before or after review, because the research question is of interest to a specialist audience, the proposed study/studies cannot address the research question, or the project is exploratory and not firmly based in existing theory and evidence. Common issues that can often be addressed in revisions include a lack of detail in the data analysis pipeline, power analysis/sampling issues, and a failure to commit to outcome measures and to spell out in advance what data patterns will be taken as evidence for which hypothesis. Since the research has not been performed yet, reviewers dedicate their efforts towards making a proposal as close to perfect as research can be, contributing their ideas on how authors can cross the ‘t’s and dot the ‘i’s to produce an outstanding piece of work. This process can have a truly collaborative feel that is rare in other formats, where reviewers can easily feel that the purpose of their labour is to list shortcomings after the fact.

As Hardwicke and Ioannidis describe in their Comment, this comparatively young publication format is still developing and journal editors and reviewers are called on to learn from experience and improve the universe of registered reports. A crucial point the authors make, strongly supported by Chambers and Mellor, is that the success of registered reports vitally depends on whether they will be recognized as publications that transparently document the confirmatory, pre-planned nature of the research. Only the ability to compare the stage 1 protocol with the eventual stage 2 publication assures the level of transparency and control that will allow the format to fulfil its promise. Hardwicke and Ioannidis document that this transparency, enabled by committing authors to deposit their stage 1 protocol in an official repository, is not achieved across the board. At Nature Human Behaviour, a condition for stage 1 in-principle acceptance is that the authors deposit their protocol in a public repository to be made publicly available either immediately or after acceptance of their stage 2 submission. As a service to our authors, we offer to deposit their protocol on their behalf on a dedicated space in figshare, either publicly or under embargo. Interested readers can find there the first few examples of stage 1 accepted protocols in the journal that the authors selected to set public immediately (https://springernature.figshare.com/registered-reports_NHB).

We are committed to offering publication of registered reports and to working with the community towards future improvements. We encourage authors, reviewers and readers to contact us with their questions, and are looking forward to publishing our first stage 2 complete submissions.