Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

The COVID-19 XPRIZE and the need for scalable, fast, and widespread testing

To the Editor — The US Food and Drug Administration (FDA) Emergency Use Authorization (EUA) and Instructions for Use (IFU) documents outlining the current approved virology tests for SARS-CoV-2 are largely unstandardized. As such, there remains an urgent need for a searchable interface allowing exploration of standardized information reported in these EUA and IFU documents. To gain an improved understanding of the current testing landscape and to galvanize future test development, we present here an online tool (http://www.resiliencehealth.com/tests) that profiles current and emerging virology tests for detecting SARS-CoV-2 (Figs. 1 and 2). We also call on the research community to respond to an open COVID-19 XPRIZE competition, OpenCovidScreen, seeking to identify cheap, high-quality, scalable testing solutions.

Fig. 1: Cumulative number of EUA virology tests approved from February to July 2020.
figure1

NGS, next-generation sequencing; NAAT, nucleic acid amplification test; LFA, lateral flow assay; RT, reverse transcription; ddPCR, Droplet Digital PCR; dPCR, digital PCR; qPCR, quantitative PCR.

Fig. 2: Performance and targets of different EUA virology tests.
figure2

a, Limit of detection (LoD) for tests that reported copies/μL or copies/mL (presented in copies/μL throughout). b, EUA virology test targets. SARS-CoV-2 5′–3′ genome4 on horizontal axis, with tests on the vertical axis and colors indicating whether the test has one target (red) or multiple targets (brown) in the specified region. Each line indicates a test source (company or institution). If an EUA reported different LoDs for different targets, sample types or methods, each is displayed as a separate row. NGS, next-generation sequencing; NAAT, nucleic acid amplification test; LFA, lateral flow assay; RT, reverse transcription; ddPCR, Droplet Digital PCR; dPCR, digital PCR; qPCR, quantitative PCR; RDRP, RNA-dependent RNA polymerase; DSP, diagnostic sample preparation; NP, nasopharyngeal; OP, oropharyngeal; rRT, real-time reverse transcriptase.

As of 27 July 2020, an analysis of the FDA data on EUA SARS-CoV-2 virology tests reveal a wide range of limit of detection (LoD), spanning >5 orders of log10 differences. These metrics are of critical importance because each 10-fold increase in the LoD of a COVID-19 viral diagnostic test is expected to increase the false negative rate by 13%1.

Beyond this variable performance reported in IFUs for EUA tests, key attributes of many tests, such as primer sequences, protocol steps or viral gene targets, are either unclear or missing. Also, most approved EUAs use large multipliers (2- to 200-fold) on their own LoD for contrived or clinical samples to pass the minimum threshold for approval, based on a 95% positive/negative percentage agreement across at least 30 positive and 30 negative samples. As submissions stand, it is difficult to directly compare results and even understand how a test will actually translate into real-world or clinical settings.

Moreover, there has not been an independent assessment of these tests’ abilities or a comprehensive benchmarking of their strengths and weaknesses in different clinical settings, nor a consistent sample type (for example, nasopharyngeal, nasal, saliva) used across the EUAs. Thus, a comprehensive benchmarking effort on all methods on the market would be helpful, similar to ones conducting head-to-head studies of serological tests2 and other sites that annotate and analyze some of these tests, such as FindDx (https://www.finddx.org/), the US National Institute for Standards and Technology (NIST)’s Rapid Microbial Testing Methods Consortium (https://www.nist.gov/programs-projects/nist-rapid-microbial-testing-methods-consortium) and the COVID-19 Testing Project (https://covidtestingproject.org/index.html).

However, even if all current EUA tests for the SARS-CoV-2 virus performed with >95% sensitivity and >95% specificity, their combined capacities would still fall short of enabling large-scale, ubiquitous temporal monitoring (tens of millions per day), involving samples with varying viral load and substrates3. Also, tests with a higher LoD would not be readily applicable to pooling strategies, in which samples are by definition diluted before testing. Even the lower LoD tests, while promising for pooling, have not had independent LoD assessments. Moreover, truly city-scale or even national-level testing to decrease and control infection rates would require fast turnaround times (test result in less than one day), easy processing, and a low cost (per test and capital expense), such that a consumer, employer or government body could easily pay for multiple tests a week for each person.

To address the above issues, we have designed the COVID-19 XPRIZE competition, OpenCovidScreen (https://opencovidscreen.org/), to identify economically viable, high-quality, scalable testing options (Fig. 3). After competitors are selected on the basis of their results, methods, cost, scalability and speed, they will then be sent blinded samples to analyze. Results will be uploaded to the XPRIZE site and analyzed for overall performance, LoD and false positives. The finalists, based on their overall methods, results and innovation, will go into the clinical validation round, in which OpenCovidScreen will follow the methods laid out by the competitors to assess the reproducibility of their results. Top teams from this validation round will be awarded a prize and selected to set up and deploy testing sites, wherein OpenCovidScreen will help scale their tests and expand them into more locations, as well as to coordinate with government, industry and non-profit efforts (for example, the NIH RADx Program and Testing for America; https://www.nih.gov/research-training/medical-research-initiatives/radx).

Fig. 3: The XPRIZE/OpenCovidScreen competition.
figure3

The OpenCovidScreen competition is intended to identify fast, scalable, and cheap SAR-CoV-2 virology tests through a series of phases, starting with submitting test and method information (top), a proficiency test including blinded samples, and a clinical validation phase (middle). The top competitors identified on the basis of their methods, results and scalability will then be chosen for deployment to enable widespread usage of their tests (bottom).

We invite readers to submit solutions for this XPRIZE, which is open to participants from all around the world (https://xprize.org/testing). This prize can serve as a springboard for both new and established technologies that can enable truly global viral testing and surveillance. Successful methods will depend on the availability of reagents, resources and automation, and as such, these metrics will also be used to identify the finalists. Winning methods will help regions increase testing capabilities by orders of magnitude and thus empower schools, businesses and cities to rapidly reopen safely, as well as pioneer technologies and platforms that can be used for future outbreaks. It is crucial to deploy rapid, scalable methods capable of tracking viruses to mitigate their detrimental impacts on society. All are encouraged to help in this fight.

References

  1. 1.

    Arnaout R. et al. SARS-CoV2 testing: the limit of detection matters. Preprint at bioRxiv https://doi.org/10.1101/2020.06.02.131144 (2020).

  2. 2.

    Whitman J. D. et al. Test performance evaluation of SARS-CoV-2 serological assays. Preprint at medRxiv https://doi.org/10.1101/2020.04.25.20074856 (2020).

  3. 3.

    Anonymous. The COVID-19 testing debacle. Nat. Biotechnol. 38, 653 (2020).

    Article  Google Scholar 

  4. 4.

    Khailany, R. A., Safdar, M. & Ozaslan, M. Gene Rep. 19, 100682 (2020).

    Article  Google Scholar 

Download references

Acknowledgements

We thank Scott A. Jackson, Peter M. Vallone, and Kenneth D. Cole from the National Institute of Standards and Technologies (NIST). We thank Testing for America (501c3) for their support, the OpenCovidScreen Foundation, Mayor Cory Mason, the Bert L and N Kuggie Vallee Foundation, Igor Tulchinsky and the WorldQuant Foundation, Bill Ackman and Olivia Flatto and the Pershing Square Sohn Cancer Research Alliance, Ken Griffin and Citadel, the US National Institutes of Health (R01AI125416, R21AI129851, R01AI151059), and the Alfred P. Sloan Foundation (G-2015-13964). Research was supported by the DOE Office of Science through the National Virtual Biotechnology Laboratory, a consortium of DOE national laboratories focused on response to COVID-19, with funding provided by the Coronavirus CARES Act.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Christopher E. Mason.

Ethics declarations

Competing interests

J.K. is cofounder and board member for Gingko Bioworks. C.E.M. is a cofounder and board member for Aevum, Biotia and Onegevity, as well as an advisor or compensated speaker for Abbvie, Acuamark Diagnostics, ArcBio, Bio-Rad, DNA Genotek, Genialis, Genpro, HomoDeus, Karius, Illumina, MilliporeSigma, Nanostring, New England Biolabs, Qiagen, Promega, Resilience Health, Tempus Labs, Whole Biome, and Zymo Research. A.C.H., H.P.J., C.E.M. and M.J.M. advise Resilience Health. J.H. is a board member or shareholder for Grail and Illumina. G.C. has conflicts: http://arep.med.harvard.edu/gmc/tech.html.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

MacKay, M.J., Hooker, A.C., Afshinnekoo, E. et al. The COVID-19 XPRIZE and the need for scalable, fast, and widespread testing. Nat Biotechnol 38, 1021–1024 (2020). https://doi.org/10.1038/s41587-020-0655-4

Download citation

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing