Weighing and communicating evidence

    As preliminary research findings are shared more widely and at an increasing pace, action is needed to counter the spread of misinformation.

    The COVID-19 pandemic has wrought remarkable changes in the way research is disclosed and the speed with which it is disseminated. Preliminary findings are being rapidly shared through unvetted preprints — which themselves are openly promoted by researchers, institutions and companies. In turn, peer-reviewed journals are publishing COVID-19 papers at an unprecedented pace — and some have relaxed editorial standards — to meet the public’s demand for answers in the face of the worldwide health emergency. As journalists race to report information about SARS-CoV-2, news stories touting unsubstantiated research have multiplied, sometimes leading to misinformation, runs on company stock and even medical scares. Addressing this problem will require efforts on several fronts — from journalists, from publishers, from press officers and from researchers themselves.

    As of 1 July, the preprint servers bioRxiv and medRxiv had hosted 26,584 posts, nearly as many as for the entirety of 2019 (30,446); 19% of these 26,584 preprints relate to COVID-19 research. Over the same period, a search of PubMed reveals that ~30,000 peer-reviewed papers were published on the disease. According to a recent analysis, an average of 367 COVID-19 papers are being published every week, with a median time from submission to acceptance of just 6 days (compared with 84 days for non-COVID-19 content). In contrast, in 2003, only 7% of 311 papers were published before the SARS epidemic ended, and the first preprint only appeared 6 months after the disease was described.

    Certainly, there is a moral imperative for researchers to share papers rapidly with their peers to galvanize progress in tackling the outbreak. And the timeliness of preprints can help decision makers form public health policy in the absence of peer-reviewed studies (see Comment in Lancet Global Health and post on medRxiv). But the willingness of many scientists to tout preliminary findings in press releases has increased the risk that inaccurate work ends up in the mainstream media and in front of readers ill-equipped to evaluate it.

    In February, a preprint claiming to have identified very short “insertions” in the SARS-CoV-2 protein sequence that had an “uncanny similarity” to HIV spurred conspiracy theories that the virus was man-made. Although the authors quickly withdrew the post following an outcry, by then the damage had been done. The following month, a preprint claiming that people with blood type A have significantly higher risk of acquiring COVID-19 was touted in numerous news outlets, only for the study’s flaws to be revealed in later reporting. April saw controversy swarming around a preprint from Stanford University researchers claiming a much lower fatality rate for SARS-CoV-2 on the basis of positive serology in 1.5% of 3,300 people screened in Santa Clara County, California, only for the study’s design and statistical analysis to be widely criticized.

    The accelerated speed with which traditional peer-reviewed COVID-19 papers are published has also created problems. Studies at the New England Journal of Medicine and the Lancet reporting safety issues with the generic drug hydroxychloroquine were retracted after the company (Surgisphere) supplying the health records for these studies declined to share data supporting publication. Elsewhere, a bamboozling drip, drip of papers and preprints describing often contradictory and underpowered clinical trial results for many drugs continues to challenge interpretation. The predilection of some companies to press release incomplete top-line human data lacking necessary detail on study methodology and cohort composition, rather than publish peer-reviewed papers, also muddies the waters.

    To complicate matters, the pandemic has prompted an influx of non-specialist journalists who often do not appreciate the complexity of science reporting. Such reporters may not understand the importance of the venue where scientific findings were reported, let alone inform their readers. Where and how science is disclosed matters and needs to be communicated. Research disclosed on social media or in a press release is not the same as a talk or abstract presented at a meeting, which in turn is not the same as a post on a preprint server or a traditional peer-reviewed paper. Peer-reviewed papers still represent the most reliable mechanism for communicating science in terms of editorial checks and balances and scrutiny from independent scientists, even if their veracity is not guaranteed.

    There is also a lack of appreciation for the varying levels of evidence associated with different types of clinical trial: an isolated case report is inferior to an observational trial; an open-label trial is inferior to a blinded one; and a single randomized, controlled trial is inferior to a systematic review covering many human trials. All of this underlines the importance of educational initiatives underway at the Science Media Centre and HealthNewsReview.org. These organizations are also providing journalists with networks of experts for contextualizing research findings (similar to ProfNet, COVIDscholar and more recent scientist-organized services). For those papers or preprints of particular public interest, it may also be advisable to provide an easy-to-understand ‘media brief’ — perhaps akin to briefs for policymakers pioneered by Nature Energy.

    With the onslaught of COVID-19, many journals made papers available for immediate release without embargo. This is a retrograde step as press embargoes give reporters more time to produce balanced, in-depth reporting. In May, PLOS announced that it was changing its embargo policy so that any peer-reviewed paper associated with a preprint will now be under embargo. Other journals should follow suit.

    Most important, scientists and their institutions or employers have a responsibility to be more selective about the studies they choose to promote and publicize.

    Preprints are an important way to share early versions of scientific papers with other scientists. But credulous and uncritical reporting of preprints is creating misinformation. Institutions should not press release preprints unless there is a compelling public health or policy rationale to do so. Spreading flawed or incomplete information is problematic in the best of times. In the midst of a pandemic, when lives may depend on it, it is inexcusable.

    Rights and permissions

    Reprints and Permissions

    About this article

    Verify currency and authenticity via CrossMark

    Cite this article

    Weighing and communicating evidence. Nat Biotechnol 38, 903 (2020). https://doi.org/10.1038/s41587-020-0650-9

    Download citation