NEWS

How swamped preprint servers are blocking bad coronavirus research

Repositories are rapidly disseminating crucial pandemic science — and they’re screening more closely to guard against poor-quality work.

Search for this author in:

Tumblr illustration.

When Albert-László Barabási, a computational scientist at Northeastern University in Boston, Massachusetts, submitted a paper to the preprint server bioRxiv last month, he received an unexpected response. The biomedical repository would no longer accept manuscripts making predictions about treatments for COVID-19 solely on the basis of computational work. The bioRxiv team suggested that Barabási submit the study to a journal for rapid peer review, instead of posting it as a preprint.

Publication norms are changing rapidly for science related to the coronavirus pandemic, as scientists worldwide conduct research at breakneck speeds to tackle the crisis. Preprint servers — where scientists post manuscripts before peer review — have been flooded with studies. The two most popular for coronavirus research, bioRxiv and medRxiv, have posted nearly 3,000 studies on the topic (see ‘Preprint surge’). The servers’ merits are clear: results can be disseminated quickly, potentially informing policy and speeding up research that could lead to the development of vaccines and treatments. But their popularity is spotlighting the scrutiny that these studies receive. Without peer review, it’s hard to check the quality of the work, and sharing poor science could be harmful, especially when research can have immediate effects on medical practice. That has led platforms including bioRxiv and medRxiv, to enhance their usual screening procedures.

Graphic showing the major preprint servers have posted thousands of studies related to the coronavirus since the outbreak began.

“We’ve seen some crazy claims and predictions about things that might treat COVID-19,” says Richard Sever, a co-founder of both servers.

Much of that speculative work has been based on computational models, says Sever — so, after consulting with several experts in outbreak science, the team decided to bar those papers from bioRxiv. “We can’t check the side effects of all the drugs and we’re not going to peer review to work out whether the modelling they’re using has any basis,” Sever says. “There are some things that should go through peer review, rather than being immediately disseminated as preprints.”

Barabási understands the need to ensure patient safety but disagrees with the decision. “It’s precisely the coronavirus that creates an environment where you need to share,” he says. The purpose of a preprint server, he says, “is that we decide what is interesting, not the referees”. He ended up posting the study on the physical-sciences preprint server arXiv.

Quality control

ArXiv, launched almost 30 years ago, was the first major preprint repository — but in recent years, discipline- and region-specific servers have mushroomed. Screening procedures vary, but an analysis of 44 servers posted last week on bioRxiv1 found that most have quality-control systems. Seventy-five per cent publicly provided information about their screening procedures, and 32% involved researchers in vetting articles for criteria such as relevance of content.

“I think there was perhaps a misconception that there are no screening checks that go on with preprint servers,” says Jamie Kirkham, a biostatistician at the University of Manchester, UK, and a co-author of the study. “We have actually found that most of them do.”

BioRxiv and medRxiv have a two-tiered vetting process. In the first stage, papers are examined by in-house staff who check for issues such as plagiarism and incompleteness. Then manuscripts are examined by volunteer academics or subject specialists who scan for non-scientific content and health or biosecurity risks. BioRxiv mainly uses principal investigators; medRxiv uses health professionals. Occasionally, screeners flag papers for further examination by Sever and other members of the leadership team. On bioRxiv, this is usually completed within 48 hours. On medRxiv, papers are scrutinized more closely because they may be more directly relevant to human health, so the turnaround time is typically four to five days.

Sever emphasizes that the vetting process is mainly used to identify articles that might cause harm — for example, those claiming that vaccines cause autism or that smoking does not cause cancer — rather than to evaluate quality. For medical research, this also includes flagging papers that might contradict widely accepted public-health advice or inappropriately use causal language in reporting on a medical treatment.

But during the pandemic, screeners are watching for other types of content that need extra scrutiny — including papers that might fuel conspiracy theories. This additional screening was put in place at bioRxiv and medRxiv after a backlash against a now-withdrawn bioRxiv preprint that reported similarities between HIV and the new coronavirus, which scientists immediately criticized as poorly conducted science that would prop up a false narrative about the origin of SARS-CoV-2. “Normally, you don’t think of conspiracy theories as something that you should worry about,” Sever says.

These heightened checks and the sheer volume of submissions has meant that the servers have had to draft in more people. But even with the extra help, most bioRxiv and medRxiv staff have been working seven-day weeks, according to Sever. “The reality is that everybody’s working all the time.”

ArXiv and ChemRxiv, a preprint server for chemistry, have also seen their share of COVID-19 papers. ArXiv has posted more than 800 and ChemRxiv has around 200. Both platforms have enhanced their screening procedures for COVID-19-related papers, although neither has stopped posting all studies with treatment-related computational predictions. “If all the [preprint platforms] had the same standards, then we’d be systematically shutting out the same voices,” says Steinn Sigurdsson, arXiv’s scientific director. “We want to have somewhat overlapping domains.”

Marshall Brennan, ChemRxiv’s publishing manager, says that when it comes to papers about treatments, they are “taking much more liberty than we normally would to send those back to the authors to say, ‘Look, this science here is suitable for a preprint server, but you can't make these claims in the context of a public-health crisis.’” He notes that in one such paper, the authors had recommended a home remedy for COVID-19 entirely on the basis of a computational analysis. That paper was swiftly rejected.

Graphic showing how Peer-reviewed journals have accelerated publication of studies on the coronavirus

Expedited publication

The abundance of coronavirus research is also reshaping peer review at journals. Several titles, including Science, journals published by Cell Press, The BMJ and Nature report a surge in coronavirus-related submissions, and many have accelerated the peer-review process to ensure rapid dissemination.

A preprint posted in April on bioRxiv2 found that many medical-research journals had drastically speeded up publication pipelines for COVID-19 papers. The analysis, which included 14 journals, found that average turnaround times had fallen from 117 to 60 days (see ‘Rapid review’). (The study omitted several influential journals, such as JAMA, The Lancet and The New England Journal of Medicine because of a lack of appropriate data.) Some journals went from submission to publication in two weeks or less.

“That really makes one wonder how thorough this process really is,” says the study’s author, Serge Horbach, a doctoral student at Radboud University in Nijmegen, the Netherlands.

Howard Bauchner, the editor-in-chief of JAMA, notes that low-quality submissions are rising. Journals in the JAMA Network have received 53% more submissions in the first quarter of this year than in the same period in 2019. “Many of these are related to COVID-19, but most are of low quality,” Bauchner says.

To address the need for rapid review, a group of publishers and scholarly-communication organizations announced an initiative last month to accelerate the publication of COVID-19 papers using measures such as asking people with relevant expertise to join a list of rapid reviewers. The initiative’s members include Outbreak Science Rapid PREreview, a platform where researchers can request or provide swift reviews of outbreak-related preprints.

Even in light of expedited publication, it is important to remember that “the role of the journal is to say: ‘This has been fairly peer-reviewed, statistically reviewed, and can be relied on,’ rather than, ‘This is coming out at you as fast as it possibly can,’” says Theodora Bloom, executive editor of The BMJ and a co-founder of medRxiv. Still, Bloom notes that the COVID-19 papers submitted to her journal “are being handled at the fastest rate possible”.

Unlike preprint servers, being published in a journal gives papers the appearance of being reliable and valid knowledge, Horbach adds. “Nonsense or incorrect science in one of these papers is potentially much more harmful.”

Nature 581, 130-131 (2020)

doi: 10.1038/d41586-020-01394-6

References

  1. 1.

    Kirkham, J. J. et al. Preprint at bioRxiv https://doi.org/10.1101/2020.04.27.063578 (2020).

  2. 2.

    Horbach, S. P. J. M. Preprint at bioRxiv https://doi.org/10.1101/2020.04.18.045963 (2020).

Download references

Nature Briefing

An essential round-up of science news, opinion and analysis, delivered to your inbox every weekday.