A new method should be thoroughly tested, applied, described — and peer-reviewed — before biological discoveries generated using the method are published.
Which comes first, the method or the result? We think that most of our readers would agree that this is definitely not a ‘chicken-or-the-egg’ conundrum. It stands to reason that a new method should be carefully and thoroughly characterized and benchmarked — and its full description and these results peer-reviewed — before biological findings generated using this new method can be fully trusted.
As editors of a methods journal, we have observed many instances where this ideal chain of events has not been followed. Certainly it is not surprising that researchers who have discovered something novel and exciting using their new method would prioritize publishing these findings, especially if there is competition from other groups. Further, two groups may collaborate, one developing a method and the other applying the method to a biological question; these groups will have different priorities and may have papers ready for journal submission at different times.
Though we are aware of and sympathetic to these types of situations, we argue that publishing new biological findings generated using a novel method before the methods paper is accepted for publication in a peer-reviewed journal is detrimental to research.
In the most egregious examples, authors of a findings paper that uses an unpublished method or software tool will provide no details about the method and simply cite “manuscript in preparation.” When reading a paper that has been peer-reviewed and undergone various editorial checks at a journal, a reader should be reasonably able to trust the results. But when the results hinge on a method that has not yet been vetted through peer review and communicated via publication, how can such findings be trusted? Even more worrying, how can the biological findings be reproduced by others? We urge peer reviewers to be on the lookout for this poor practice and flag it to the journal editor handling the paper.
Preprint servers allow authors to rapidly share unpublished work to the scientific community, something that we both support and encourage here at Nature Research. However, we argue that it is insufficient to cite a preprint reporting a method as evidence that the method has been properly validated. Our colleagues at Nature Biotechnology, for example, require that methods central to new results in a submitted manuscript be accepted for publication in a peer-reviewed journal before they will publish the manuscript, a stance we applaud. As they wrote in a 2017 Editorial, “peer-reviewed journals must ensure that the integration of minimally reviewed preprints into their papers does not compromise the reproducibility of the science they publish.”
We strongly encourage researchers who want to publish two papers, one reporting a new method and the other a new finding, to prioritize writing up both. If it is not practical to publish the methods paper in a journal before submitting the findings paper, submission should at least be done concurrently. If both papers are submitted to the same journal, or to the same publisher, peer review and publication can often be coordinated. If the papers are submitted to different journals, the other paper should be provided to the editors (note that this is a requirement at Nature Methods). This allows the editors and the reviewers to understand how the method works and also to judge whether there is substantial overlap between the papers.
Even in cases where a methods and a findings paper have been simultaneously submitted to journals, peer review outcomes can be unpredictable. We advise authors to keep their editor informed about the status of the other paper and try to ensure that the methods paper is at least provisionally accepted (if not published) before the findings paper is published.
Authors should also be aware that if they describe a method in some detail in a paper where they report new biological findings, this may prevent them from later being able to publish a dedicated methods paper in a journal (such as Nature Methods) where methodological novelty is an important editorial criterion. If we think a method is sufficiently exciting and important for us to potentially publish a paper focused on the method itself, we will occasionally consider it. But in such cases the methods paper must stand on its own: it must describe a new tool or an optimized workflow, or provide substantial additional characterization or validation data, or describe a novel application. In other words, there must be a good reason to justify publishing a dedicated methods paper following the initial report.
There are many examples of methods, tools and resources that have remained unpublished even for years. You might ask: why bother publishing a dedicated method paper at all? Methods are key to advancing scientific progress, and it’s just as important for the method as for a novel finding, if not even more important, that the work go through a careful vetting process. At Nature Methods, we also uphold strict editorial standards regarding a method or tool’s description (including making software code and unique materials available), its characterization and benchmarking in comparison to existing approaches (including making these data available), and a demonstration of general applicability. We think that these standards help improve the reliability and reproducibility of methods we publish, allowing readers to better trust new biological findings generated by such methods, as well as making the methods themselves more useful and practical for a broader audience.