Introduction

Coronavirus disease 2019 (COVID-19), due to the novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), continues to be a global health challenge. As of November 22, 2020, the COVID-19 burden included 57.6 million confirmed cases and 1.3 million deaths worldwide1. Meanwhile, the true extent of exposure to the SARS-CoV-2 infection and how far different national populations are from herd immunity remain poorly understood. Commercial serological assays are increasingly being used to address this gap in evidence. The extent to which such assays can capture ever infection (defined as past or prior infection) in a population remains to be elucidated. Understanding who has been exposed and potentially acquired immunity against this virus may help healthcare providers and public health stakeholders in establishing and implementing more efficient and effective strategies and policies for managing the disease and economic burden associated with the COVID-19 pandemic.

Qatar experienced a large SARS-CoV-2 epidemic with a high rate of laboratory-confirmed infections at > 60,000 infections per million population2,3,4. As part of the national response, the public health authorities expanded serological testing for SARS-CoV-2 antibodies for both healthcare and research purposes. Three automated main serological testing platforms are being used. The first is the Roche Elecsys® Anti SARS CoV 2 (Roche, Switzerland)5 platform at Hamad Medical Corporation (HMC), the main public healthcare provider and the nationally-designated provider for all COVID-19 healthcare needs. The second is the Mindray CL-900i anti-SARS-CoV-2 IgG (Shenzhen Mindray Bio-Medical Electronics Co., China)6 platform at Qatar University (QU), which is used for research purposes. The third is the BioMérieux VidasIII (BioMérieux, Marcy-l’Etoile, France)7 platform at QU, which is also being used for research purposes.

To interpret the emerging results of serological testing and to inform the national response, this study was conducted to compare the performance of these three assays and to assess the implications for measuring SARS-CoV-2 ever infection. The novelty and strength of this study is that it is conducted based on a population-based sample8 in a setting at a high exposure to this infection2,3,9,10,11.

Methods

Blood specimens were collected from 394 volunteering individuals between July 26 and September 9, 2020, as a sub-study of a nationwide survey8 assessing SARS-CoV-2 seroprevalence (IgG antibodies) and current-infection prevalence (using polymerase chain reaction [PCR] testing) in the wider population of craft and manual workers who constitute 60% of the population of Qatar12. Informed by prior work13,14, a sample size of 400 was estimated to be sufficient to ensure narrow confidence intervals for the Cohen’s kappa statistic, but we were able to include and test only 394 specimens. The research work was approved by the ethics review boards at HMC, QU, and Weill Cornell Medicine-Qatar. The study was conducted following the ethics review boards guidelines and regulations. Informed consent was obtained from all study participants.

The automated serological testing was performed using the above indicated three commercial assays. The Roche Elecsys® Anti SARS-CoV-2 (“Elecsys” in short form) assay, our reference assasy, uses a recombinant protein representing the nucleocapsid (N) antigen for the determination of IgG antibodies against SARS-CoV-25. Anti-SARS-CoV-2 results were generated following the manufacturer’s instructions (reactive: optical density cutoff index ≥ 1.0 vs. non-reactive: cutoff index < 1.0)5,15.

The Mindray CL-900i® anti-SARS-CoV-2 IgG (“CL-900i” in short form) assay uses paramagnetic microplates coated with recombinant nucleocapsid (N) and spike (S) antigens for the determination of anti-SARS-CoV-2 IgG antibodies6. The analyzer automatically calculates the analyte concentration of each serum specimen according to a master calibration curve, and the results are shown in the units of U/mL. Anti-SARS-CoV-2 results were generated following the manufacturer’s instructions (reactive: optical density cutoff index ≥ 10.0 vs. non-reactive: cutoff index < 10.0)6,16.

The BioMérieux VidasIII assay (“VidasIII” in short form) uses a VIDASIII® analyzer for anti-SARS-CoV-2 IgG detection through a two-step sandwich ELFA assay7. The IgG in the serum specimen binds to a recombinant spike S1 sub-domain (containing the receptor-binding domain [S1-RBD]) of the SARS-CoV-2 virus coated on a solid phase. Alkaline phosphatase-conjugated anti-human IgG are then added. The fluorescence intensity generated by the substrate is then measured at a wavelength of 450 nm. The intensity of the signal is proportional to the level of IgG. The optical-density cutoff index was calculated according to the manufacturer’s instructions7,15. The ratio between the relative fluorescence value (RFV) measured in the specimen and the RFV from the calibrator was interpreted as positive if the index value was ≥ 1.07,15.

All PCR testing was conducted at HMC Central Laboratory or at Sidra Medicine Laboratory, following standardized protocols. Nasopharyngeal and oropharyngeal swabs (Huachenyang Technology, China) were collected and placed in Universal Transport Medium (UTM). Aliquots of UTM were: extracted on the QIAsymphony platform (QIAGEN, USA) and tested with real-time reverse-transcription PCR (RT-qPCR) using the TaqPath™ COVID-19 Combo Kit (Thermo Fisher Scientific, USA) on an ABI 7500 FAST (ThermoFisher, USA); extracted using a custom protocol17 on a Hamilton Microlab STAR (Hamilton, USA) and tested using the AccuPower SARS-CoV-2 Real-Time RT-PCR Kit (Bioneer, Korea) on an ABI 7500 FAST; or loaded directly to a Roche cobas® 6800 system and assayed with the cobas® SARS-CoV-2 Test (Roche, Switzerland). The first assay targets the S, N, and ORF1ab regions of the virus; the second targets the virus’ RdRp and E-gene regions; and the third targets the ORF1ab and E-gene regions.

Results of the serological and PCR testing were subsequently linked to the HMC centralized and standardized database comprising all SARS-CoV-2 PCR testing conducted in Qatar since the start of the epidemic2,18. The database also includes data on hospitalization and on the World Health Organization (WHO) severity classification19 for the hospitalized PCR-confirmed infections.

Results from the three types of serological testing were cross-tabulated. Four concordance metrics were estimated: overall, positive, and negative percent agreement, as well as Cohen’s kappa statistic. The latter is a robust metric that measures the level of agreement, beyond chance, between two diagnostic testing methods20. The kappa statistic ranges between 0 and 1; a value ≤ 0.40 indicates poor agreement, a value between 0.40 and 0.75 indicates fair/good agreement, and a value ≥ 0.75 indicates excellent agreement20. Level of significance was established at 5%, and a 95% confidence interval (CI) was reported for each metric. A nonparametric statistical method, Spearman correlation, was used to assess the correlation between the optical densities of each pair of antibody assays. Calculations were conducted using Microsoft Excel.

Results

SARS-CoV-2 seroprevalence was estimated at 42.9% (169/394; 95% CI 38.1–47.8%) using the CL-900i assay, 40.6% (160/394; 95% CI 35.9–45.5%) using the VidasIII assay, and 42.4% (167/394; 95% CI 37.6–47.3%) using the Elecsys assay. A total of 183 specimens were seropositive in at least one of the assays for a total sample seroprevalence of 46.5% (183/394; 95% CI 41.6–51.4%).

Supplementary Table S1 shows the results of the serological and PCR testing for each of the 394 participants. A total of 35 specimens showed discordant results between the three antibody assays (Table 1). Of the 35 individuals with discordant antibody results, 9 were PCR-positive at the time of specimen collection. Eleven specimens were seropositive using the CL-900i assay but seronegative using the VidasIII and the Elecsys assays; among these, two were PCR-positive with cycle threshold (Ct) values of 23.9 and 27.0. Five specimens were seropositive using the Elecsys assay but seronegative using the VidasIII and the CL-900i assays; among these, one person was PCR-positive with a Ct value of 21.6. Two specimens were seropositive using the VidasIII assay but seronegative using the CL-900i and the Elecsys assays; among these, one was PCR-positive with a Ct value of 29.2.

Table 1 Characteristics of the specimens that were discordant among the Mindray CL-900i anti-SARS-CoV-2 IgG, BioMérieux VidasIII anti-SARS-CoV-2 IgG, and Roche Elecsys Anti SARS-CoV-2 antibody testing.
Full size table

The overall, positive, and negative percent agreements between the CL-900i and the Elecsys assays were estimated at 93.4% (95% CI 90.5–95.5%), 92.8% (95% CI 87.9–95.8%), and 93.8% (95% CI 89.9–96.3%), respectively (Table 2A). Cohen’s kappa statistic was estimated at 0.87 (95% CI 0.83–0.90), indicating excellent agreement between the two assays (Table 2A). The Spearman correlation between the optical densities was at 0.751 (p-value < 0.001) indicating strong correlation.

Table 2 Concordance metrics between (A) the Mindray CL-900i anti-SARS-CoV-2 IgG testing and the Roche Elecsys Anti SARS-CoV-2 testing, (B) the VidasIII anti-SARS-CoV-2 IgG testing and the Roche Elecsys Anti SARS-CoV-2 testing, and (C) the BioMérieux VidasIII anti-SARS-CoV-2 IgG testing and the Mindray CL-900i anti-SARS-CoV-2 IgG testing.
Full size table

The overall, positive, and negative percent agreements between the VidasIII and the Elecsys assays were estimated at 95.7% (95% CI 93.2–97.3%), 92.8% (95% CI 87.9–95.8%), and 97.8% (95% CI 95.0–99.1%), respectively (Table 2B). Cohen’s kappa statistic was estimated at 0.91 (95% CI 0.88–0.94), indicating excellent agreement between the two assays (Table 2B). The Spearman correlation between the optical densities was at 0.824 (p-value < 0.001) indicating strong correlation.

The overall, positive, and negative percent agreements between the VidasIII and the CL-900i assays were estimated at 93.2% (95% CI 90.2–95.3%), 89.3% (95% CI 83.8–93.2%), and 96.0% (95% CI 92.6–97.9%), respectively (Table 2C). Cohen’s kappa statistic was estimated at 0.86 (95% CI 0.82–0.90), indicating excellent agreement between the two assays (Table 2C). The Spearman correlation between the optical densities was at 0.804 (p-value < 0.001) indicating strong correlation.

A total of 49 swabs were PCR-positive at the time of specimen collection during this study for a current-infection prevalence of 12.5% (49/392; 95% CI 9.6–16.1%)—two individuals declined PCR testing (but not antibody testing) during this study. Figure 1A shows the distribution of the PCR Ct values among those PCR-positive, indicating broad distribution suggestive of these persons being diagnosed at the various stages of infection. The median PCR Ct value was 24.1 (interquartile range [IQR] 20.6–31.9).

Figure 1
figure 1

Distribution of polymerase chain reaction (PCR) cycle threshold (Ct) values of (A) the 49 persons identified as SARS-CoV-2 PCR-positive at the time of specimen collection during the study, (B) the 17 persons identified as SARS-CoV-2 PCR-positive, but were antibody-positive in at least one of the assays, and (C) the 32 persons identified as SARS-CoV-2 PCR-positive, but were antibody-negative in all three assays.

Full size image

Among all those PCR-positive at the time of specimen collection, 30.6% (15/49; 95% CI 19.5–44.5%) were seropositive in the CL-900i assay, 22.5% (11/49; 95% CI 13.0–35.9%) were seropositive in the VidasIII assay, 24.5% (12/49; 95% CI 14.6–38.1%) were seropositive in the Elecsys assay, and 34.7% (17/49; 95% CI 22.9–48.7%) were seropositive in at least one of the assays. Therefore, 32 individuals had a PCR-positive diagnosis at the time of specimen collection, but were antibody-negative in all three assays.

Among all those seropositive in at least one of the assays, 9.3% were PCR-positive (17/183; 95% CI 5.9–14.4%) at the time of specimen collection. The median PCR Ct value was 32.3 (IQR 25.6–35.1), indicative of mostly non-recent infections21,22,23 (Fig. 1B). Among all those seronegative in all three assays, 15.2% were PCR-positive (32/211; 95% CI 11.0–20.6%) at the time of specimen collection. The median PCR Ct value was 21.7 (IQR 19.1–27.4), indicative of mostly recent infections (Fig. 1C).

Through linking with the national SARS-CoV-2 PCR testing database18, and of the 394 participants, 4.3% (17/394; 95% CI 2.7–6.8%) had a record of SARS-CoV-2 PCR-confirmed diagnosis prior to this study (that is any time before the PCR and antibody tests during the study period). All but one of these were antibody-positive by at least one of the assays. The individual testing antibody-negative but had a prior PCR-confirmed diagnosis was diagnosed on July 23, 2020, that is four days prior to the antibody serological test date. This individual declined PCR testing during this study and at the time of the serological test. Only 15 specimens were linked to a PCR-confirmed SARS-CoV-2 diagnosis prior to this study by at least 7 days. All 15 specimens were positive in all three antibody assays, resulting in a sensitivity of 100% (95% CI 78.2–100%) for all three assays 7 days after PCR diagnosis.

Of the 183 persons with an antibody-positive status in at least one assay, 16 persons had a SARS-CoV-2 PCR-confirmed diagnosis prior to this study. Accordingly, the detection rate (the percentage of those antibody-positive who had a prior PCR-confirmed diagnosis) was 8.7% (16/183; 95% CI 5.5–13.7%).

Based on the above, a total of 216 persons had a laboratory-confirmed infection at or prior to this study; that is an antibody-positive result in at least one assay (183 cases), a PCR-positive diagnosis prior to this study but with an antibody-negative status in all three assays (1 case), or a PCR-positive diagnosis at the time of specimen collection during this study but with an antibody-negative status in all three assays (32 cases). Accordingly, the percentage of persons with evidence of ever infection through either PCR or antibody testing was 54.8% (216/394; 95% CI 49.9–59.7%). Moreover, only 78.2% of ever infections were antibody-positive by the CL-900i assay (169/216; 95% CI 72.3–83.2%), 74.1% by the VidasIII assay (160/216; 95% CI 67.8–79.5%), and 77.3% by the Elecsys assay (167/216; 95% CI 71.3–82.4%).

Linking with the national COVID-19 hospitalization database18 identified only one laboratory-confirmed infection through this study to have progressed to severe disease per WHO severity classification19. The person had also a diagnosis of diabetes, hypertension, and coronary artery disease. This person was diagnosed PCR-positive at time of specimen collection, was seronegative in the CL-900i and the VidasIII assays, but was seropositive in the Elecsys assay. No infection was critical per WHO severity classification19 and no COVID-19 death was reported for any of the study participants.

Discussion

A primary finding of this study is that all three antibody assays had comparable performance and excellent agreement. This positive finding, however, conceals important shortcomings about the use and performance of commercial antibody assays in assessing ever infection with SARS-CoV-2 in population-based surveys, especially at times of high SARS-CoV-2 incidence, as is the case at present globally.

The first shortcoming is that each of these three assays missed ≥ 69% of those who were PCR-positive at the time of specimen collection. This finding is explained in large part by the 1–4 weeks delay in development of detectable antibodies after acquiring the infection21,24. This explanation is supported by the low PCR Ct value among those PCR-positive but antibody-negative (Fig. 1C), which indicates recency of infection21,25. At the time of the study and in the population being studied, the outbreak was advancing, so there was a significant proportion of new infections, making the serology assay less useful for estimating population prevalence of ever infection. It is unknown whether the lower sensitivity could have been due in part also to commercial assay development preferentially opting to maximize the specificity of the assay, to avoid a false positive diagnosis with its clinical implications, but at the expense of the sensitivity of the assay.

To explore this conjecture, we investigated the distribution of the optical density values for the three assays, for both the seronegative and seropositive persons (Fig. 2), and derived alternative empirical optical density cutoffs by multiplying the standard deviation of the values among those seronegative by a factor of three26,27 to set the new cutoff. All three new cutoffs for all three assays were lower than those defined by the assays’ manufacturers, supporting the conjecture that the manufacturers may have chosen them to be high to maximize specificity. Having said so, the estimated seroprevalence using the new cutoffs increased only minimally in this sample. The new proportions of antibody-positivity detected by the CL-900i, VidasIII, and Elecsys assays increased to 44.8% (versus 42.9% based on manufacturer’s defined optical density cutoff), 42.3% (versus 40.6%), and 43.8% (versus 42.4%).

Figure 2
figure 2

Distribution of the optical density values in those seronegative versus those seropositive for each assay. Red line indicates the manufacturer’s cut-off values. Blue line indicates the empirically determined optical density cut-off values as informed by the literature26,27.

Full size image

The second shortcoming is that each of these three assays also missed other individuals with evidence of ever infection. Despite excellent agreement overall, nearly 10% of the total sample still showed discordant results between the three antibody assays. Differences in the sensitivity of the assays to diagnose recent infection explains only partially these discordant results. Indeed, most (74.3%) of these persons with discordant results were PCR-negative at the time of specimen collection (Table 1), and thus less likely to have had a recent infection. The extent to which false positivity may explain some of these discordant results is unknown, but the three manufacturers reported essentially perfect specificity for each of these assays5,7,28,29.

As a consequence of these findings, the use of any one of these antibody assays to assess ever infection in a population-based sample, especially at the times of high SARS-CoV-2 incidence, will underestimate ever infection in the sample. In the sample in the present study, at least 20% of the actual infections that occurred were missed. A solution to this challenge is to combine PCR data and serology data together, or that the serology data cannot be adequately interpreted without knowledge of the PCR positivity data, or that serology is less useful when the epidemiology is rapidly changing. With the global pandemic continuing at high SARS-CoV-2 incidence, this finding suggests that ever infection in populations is possibly substantially higher than is currently believed.

The discordant results as well as the differences in the patterns of optical density values observed across the three assays may be attributed to differences in the target antigen used in each assay, as antibody response can vary by target antigen. The CL-900i assay targets both the full S (S1 and S2 subunits) and N proteins6. The VidasIII assay targets the RBD of the S1 subunit7 and the Elecsys assay targets the N protein5. Therefore, the CL-900i assay could be more sensitive in detecting SARS-CoV-2 ever infection than assays that target only the S or N proteins. This was highlighted in a recent study where the CL-900i assay demonstrated the highest sensitivity compared to the other assays. This may also explain the 11 discordant specimens in this study that were seropositive using the CL-900i assay but seronegative using the VidasIII and Elecsys assays (Table 1).

The antibody testing outcome obtained using the Elecsys assay was more similar to that of the CL-900i assay, possibly because the Elecsys assay detects the total antibodies (IgG, IgA, and IgM) against the SARS-CoV-2N protein30,31. An earlier study suggested that the use of S1 or the RBD alone is associated with lower sensitivity than the full S protein32. The greater sensitivity of antibody response found against the trimeric full S protein is likely to result from antibodies binding to the S2 subunit and the conservation of conformational epitopes within the higher-order structure32. Therefore, assays that specifically target the S1 or the RBD may underestimate the seroprevalence of SARS-CoV-2, a finding that may explain the lower seroprevalence obtained through the VidasIII assay.

Earlier studies demonstrated that antibody response against different SARS-CoV-2 antigens may also develop with different kinetics and thus could be used as indicators for the stage of infection. In the acute phase of infection, antibody response against the N and S proteins develops simultaneously, whereas the response against the N protein appears to wane faster compared to the S protein, which tends to persist over time32,33,34. This may also contribute to explaining some of the discordant outcomes found in this study.

This study has some limitations. Two out of the 394 participants included in the study declined PCR testing (but not serological testing) at time of specimen collection. The performance of these antibody assays was compared to each other (and to PCR testing), but not to a gold standard test of seropositivity, as such a test was not available to study investigators. Therefore, we were unable to measure ever infection prevalence to a gold standard, and use this to compare the performance of each assay to the gold standard, nor to assess the sensitivity and specificity of each assay in the study sample. The specificity of the Elecsys assay has previously been reported to be 99.98% and the sensitivity to be 98.80% on day 14 after PCR diagnosis5. A validation study by Public Health England reported a specificity of 100% and a sensitivity of 83.9% for the same assay28. As for the remaining assays, specificity and sensitivity were reported at 94.9% and 82.2%, respectively, for the CL-900i assay29, and at 99.9% and 88.6%, respectively, for the VidasIII assay7.

In conclusion, all three assays had comparable performance and excellent agreement when used in a high SARS-CoV-2 exposure setting, but still missed at least 20% of cases with laboratory-confirmed evidence of ever infection. This suggests that current growing use of commercial antibody assays to assess ever infection in population-based surveys, especially at times of high SARS-CoV-2 incidence when many infections are recent, is likely to substantially underestimate actual infection exposure. The findings demonstrate further the need to interpret the serology testing together with PCR testing.