Analytic comparison between three high-throughput commercial SARS-CoV-2 antibody assays reveals minor discrepancies in a high-incidence population

Performance of three automated commercial serological IgG-based assays was investigated for assessing SARS-CoV-2 “ever” (past or current) infection in a population-based sample in a high exposure setting. PCR and serological testing was performed on 394 individuals. SARS-CoV-2-IgG seroprevalence was 42.9% (95% CI 38.1–47.8%), 40.6% (95% CI 35.9–45.5%), and 42.4% (95% CI 37.6–47.3%) using the CL-900i, VidasIII, and Elecsys assays, respectively. Between the three assays, overall, positive, and negative percent agreements ranged between 93.2–95.7%, 89.3–92.8%, and 93.8–97.8%, respectively; Cohen’s kappa statistic ranged from 0.86 to 0.91; and 35 specimens (8.9%) showed discordant results. Among all individuals, 12.5% (95% CI 9.6–16.1%) had current infection, as assessed by PCR. Of these, only 34.7% (95% CI 22.9–48.7%) were seropositive by at least one assay. A total of 216 individuals (54.8%; 95% CI 49.9–59.7%) had evidence of ever infection using antibody testing and/or PCR during or prior to this study. Of these, only 78.2%, 74.1%, and 77.3% were seropositive in the CL-900i, VidasIII, and Elecsys assays, respectively. All three assays had comparable performance and excellent agreement, but missed at least 20% of individuals with past or current infection. Commercial antibody assays can substantially underestimate ever infection, more so when infection rates are high.


Scientific Reports
| (2021) 11:11837 | https://doi.org/10.1038/s41598-021-91235-x www.nature.com/scientificreports/ healthcare providers and public health stakeholders in establishing and implementing more efficient and effective strategies and policies for managing the disease and economic burden associated with the COVID-19 pandemic. Qatar experienced a large SARS-CoV-2 epidemic with a high rate of laboratory-confirmed infections at > 60,000 infections per million population [2][3][4] . As part of the national response, the public health authorities expanded serological testing for SARS-CoV-2 antibodies for both healthcare and research purposes. Three automated main serological testing platforms are being used. The first is the Roche Elecsys ® Anti SARS CoV 2 (Roche, Switzerland) 5 platform at Hamad Medical Corporation (HMC), the main public healthcare provider and the nationally-designated provider for all COVID-19 healthcare needs. The second is the Mindray CL-900i anti-SARS-CoV-2 IgG (Shenzhen Mindray Bio-Medical Electronics Co., China) 6 platform at Qatar University (QU), which is used for research purposes. The third is the BioMérieux VidasIII (BioMérieux, Marcy-l'Etoile, France) 7 platform at QU, which is also being used for research purposes.
To interpret the emerging results of serological testing and to inform the national response, this study was conducted to compare the performance of these three assays and to assess the implications for measuring SARS-CoV-2 ever infection. The novelty and strength of this study is that it is conducted based on a population-based sample 8 in a setting at a high exposure to this infection 2,3,[9][10][11] .

Methods
Blood specimens were collected from 394 volunteering individuals between July 26 and September 9, 2020, as a sub-study of a nationwide survey 8 assessing SARS-CoV-2 seroprevalence (IgG antibodies) and current-infection prevalence (using polymerase chain reaction [PCR] testing) in the wider population of craft and manual workers who constitute 60% of the population of Qatar 12 . Informed by prior work 13,14 , a sample size of 400 was estimated to be sufficient to ensure narrow confidence intervals for the Cohen's kappa statistic, but we were able to include and test only 394 specimens. The research work was approved by the ethics review boards at HMC, QU, and Weill Cornell Medicine-Qatar. The study was conducted following the ethics review boards guidelines and regulations. Informed consent was obtained from all study participants.
The automated serological testing was performed using the above indicated three commercial assays. The Roche Elecsys ® Anti SARS-CoV-2 ("Elecsys" in short form) assay, our reference assasy, uses a recombinant protein representing the nucleocapsid (N) antigen for the determination of IgG antibodies against SARS-CoV-2 5 . Anti-SARS-CoV-2 results were generated following the manufacturer's instructions (reactive: optical density cutoff index ≥ 1.0 vs. non-reactive: cutoff index < 1.0) 5,15 .
The Mindray CL-900i ® anti-SARS-CoV-2 IgG ("CL-900i" in short form) assay uses paramagnetic microplates coated with recombinant nucleocapsid (N) and spike (S) antigens for the determination of anti-SARS-CoV-2 IgG antibodies 6 . The analyzer automatically calculates the analyte concentration of each serum specimen according to a master calibration curve, and the results are shown in the units of U/mL. Anti-SARS-CoV-2 results were generated following the manufacturer's instructions (reactive: optical density cutoff index ≥ 10.0 vs. non-reactive: cutoff index < 10.0) 6, 16 .
The BioMérieux VidasIII assay ("VidasIII" in short form) uses a VIDASIII ® analyzer for anti-SARS-CoV-2 IgG detection through a two-step sandwich ELFA assay 7 . The IgG in the serum specimen binds to a recombinant spike S1 sub-domain (containing the receptor-binding domain [S1-RBD]) of the SARS-CoV-2 virus coated on a solid phase. Alkaline phosphatase-conjugated anti-human IgG are then added. The fluorescence intensity generated by the substrate is then measured at a wavelength of 450 nm. The intensity of the signal is proportional to the level of IgG. The optical-density cutoff index was calculated according to the manufacturer's instructions 7,15 . The ratio between the relative fluorescence value (RFV) measured in the specimen and the RFV from the calibrator was interpreted as positive if the index value was ≥ 1.0 7,15 .
All PCR testing was conducted at HMC Central Laboratory or at Sidra Medicine Laboratory, following standardized protocols. Nasopharyngeal and oropharyngeal swabs (Huachenyang Technology, China) were collected and placed in Universal Transport Medium (UTM). Aliquots of UTM were: extracted on the QIAsymphony platform (QIAGEN, USA) and tested with real-time reverse-transcription PCR (RT-qPCR) using the TaqPath™ COVID-19 Combo Kit (Thermo Fisher Scientific, USA) on an ABI 7500 FAST (ThermoFisher, USA); extracted using a custom protocol 17 on a Hamilton Microlab STAR (Hamilton, USA) and tested using the AccuPower SARS-CoV-2 Real-Time RT-PCR Kit (Bioneer, Korea) on an ABI 7500 FAST; or loaded directly to a Roche cobas ® 6800 system and assayed with the cobas ® SARS-CoV-2 Test (Roche, Switzerland). The first assay targets the S, N, and ORF1ab regions of the virus; the second targets the virus' RdRp and E-gene regions; and the third targets the ORF1ab and E-gene regions.
Results of the serological and PCR testing were subsequently linked to the HMC centralized and standardized database comprising all SARS-CoV-2 PCR testing conducted in Qatar since the start of the epidemic 2,18 . The database also includes data on hospitalization and on the World Health Organization (WHO) severity classification 19 for the hospitalized PCR-confirmed infections.
Results from the three types of serological testing were cross-tabulated. Four concordance metrics were estimated: overall, positive, and negative percent agreement, as well as Cohen's kappa statistic. The latter is a robust metric that measures the level of agreement, beyond chance, between two diagnostic testing methods 20 . The kappa statistic ranges between 0 and 1; a value ≤ 0.40 indicates poor agreement, a value between 0.40 and 0.75 indicates fair/good agreement, and a value ≥ 0.75 indicates excellent agreement 20 . Level of significance was established at 5%, and a 95% confidence interval (CI) was reported for each metric. A nonparametric statistical method, Spearman correlation, was used to assess the correlation between the optical densities of each pair of antibody assays. Calculations were conducted using Microsoft Excel.  Table S1 shows the results of the serological and PCR testing for each of the 394 participants. A total of 35 specimens showed discordant results between the three antibody assays (Table 1). Of the 35 individuals with discordant antibody results, 9 were PCR-positive at the time of specimen collection. Eleven specimens were seropositive using the CL-900i assay but seronegative using the VidasIII and the Elecsys assays; among these, two were PCR-positive with cycle threshold (Ct) values of 23.9 and 27.0. Five specimens were seropositive using the Elecsys assay but seronegative using the VidasIII and the CL-900i assays; among these, one person was PCR-positive with a Ct value of 21.6. Two specimens were seropositive using the VidasIII assay but seronegative using the CL-900i and the Elecsys assays; among these, one was PCR-positive with a Ct value of 29.2.
A total of 49 swabs were PCR-positive at the time of specimen collection during this study for a current-infection prevalence of 12.5% (49/392; 95% CI 9.6-16.1%)-two individuals declined PCR testing (but not antibody testing) during this study. Figure 1A shows the distribution of the PCR Ct values among those PCR-positive, indicating broad distribution suggestive of these persons being diagnosed at the various stages of infection. The median PCR Ct value was 24.1 (interquartile range [IQR] 20.6-31.9).
Through linking with the national SARS-CoV-2 PCR testing database 18 , and of the 394 participants, 4.3% (17/394; 95% CI 2.7-6.8%) had a record of SARS-CoV-2 PCR-confirmed diagnosis prior to this study (that is any time before the PCR and antibody tests during the study period). All but one of these were antibody-positive by at least one of the assays. The individual testing antibody-negative but had a prior PCR-confirmed diagnosis was diagnosed on July 23, 2020, that is four days prior to the antibody serological test date. This individual declined PCR testing during this study and at the time of the serological test. Only 15 specimens were linked to a PCR-confirmed SARS-CoV-2 diagnosis prior to this study by at least 7 days. All 15 specimens were positive in all three antibody assays, resulting in a sensitivity of 100% (95% CI 78.2-100%) for all three assays 7 days after PCR diagnosis.
Of the 183 persons with an antibody-positive status in at least one assay, 16 persons had a SARS-CoV-2 PCRconfirmed diagnosis prior to this study. Accordingly, the detection rate (the percentage of those antibody-positive who had a prior PCR-confirmed diagnosis) was 8.7% (16/183; 95% CI 5.5-13.7%).
Based on the above, a total of 216 persons had a laboratory-confirmed infection at or prior to this study; that is an antibody-positive result in at least one assay (183 cases), a PCR-positive diagnosis prior to this study but with an antibody-negative status in all three assays (1 case), or a PCR-positive diagnosis at the time of specimen collection during this study but with an antibody-negative status in all three assays (32 cases Linking with the national COVID-19 hospitalization database 18 identified only one laboratory-confirmed infection through this study to have progressed to severe disease per WHO severity classification 19 . The person had also a diagnosis of diabetes, hypertension, and coronary artery disease. This person was diagnosed PCR-positive at time of specimen collection, was seronegative in the CL-900i and the VidasIII assays, but was  19 and no COVID-19 death was reported for any of the study participants.

Discussion
A primary finding of this study is that all three antibody assays had comparable performance and excellent agreement. This positive finding, however, conceals important shortcomings about the use and performance of commercial antibody assays in assessing ever infection with SARS-CoV-2 in population-based surveys, especially at times of high SARS-CoV-2 incidence, as is the case at present globally. The first shortcoming is that each of these three assays missed ≥ 69% of those who were PCR-positive at the time of specimen collection. This finding is explained in large part by the 1-4 weeks delay in development of detectable antibodies after acquiring the infection 21,24 . This explanation is supported by the low PCR Ct value among those PCR-positive but antibody-negative (Fig. 1C), which indicates recency of infection 21,25 . At the time of the study and in the population being studied, the outbreak was advancing, so there was a significant proportion of new infections, making the serology assay less useful for estimating population prevalence of ever infection. It is unknown whether the lower sensitivity could have been due in part also to commercial assay development preferentially opting to maximize the specificity of the assay, to avoid a false positive diagnosis with its clinical implications, but at the expense of the sensitivity of the assay.
To explore this conjecture, we investigated the distribution of the optical density values for the three assays, for both the seronegative and seropositive persons (Fig. 2), and derived alternative empirical optical density cutoffs by multiplying the standard deviation of the values among those seronegative by a factor of three 26,27 to set the new cutoff. All three new cutoffs for all three assays were lower than those defined by the assays' manufacturers, supporting the conjecture that the manufacturers may have chosen them to be high to maximize specificity. Having said so, the estimated seroprevalence using the new cutoffs increased only minimally in this sample. The new proportions of antibody-positivity detected by the CL-900i, VidasIII, and Elecsys assays increased to 44.8% (versus 42.9% based on manufacturer's defined optical density cutoff), 42.3% (versus 40.6%), and 43.8% (versus 42.4%).
The second shortcoming is that each of these three assays also missed other individuals with evidence of ever infection. Despite excellent agreement overall, nearly 10% of the total sample still showed discordant results between the three antibody assays. Differences in the sensitivity of the assays to diagnose recent infection explains only partially these discordant results. Indeed, most (74.3%) of these persons with discordant results were PCRnegative at the time of specimen collection (Table 1), and thus less likely to have had a recent infection. The extent to which false positivity may explain some of these discordant results is unknown, but the three manufacturers reported essentially perfect specificity for each of these assays 5,7,28,29 .
As a consequence of these findings, the use of any one of these antibody assays to assess ever infection in a population-based sample, especially at the times of high SARS-CoV-2 incidence, will underestimate ever infection in the sample. In the sample in the present study, at least 20% of the actual infections that occurred were missed. A solution to this challenge is to combine PCR data and serology data together, or that the serology data cannot be adequately interpreted without knowledge of the PCR positivity data, or that serology is less useful when the epidemiology is rapidly changing. With the global pandemic continuing at high SARS-CoV-2 incidence, this finding suggests that ever infection in populations is possibly substantially higher than is currently believed.
The discordant results as well as the differences in the patterns of optical density values observed across the three assays may be attributed to differences in the target antigen used in each assay, as antibody response can vary by target antigen. The CL-900i assay targets both the full S (S1 and S2 subunits) and N proteins 6 . The VidasIII   www.nature.com/scientificreports/ assay targets the RBD of the S1 subunit 7 and the Elecsys assay targets the N protein 5 . Therefore, the CL-900i assay could be more sensitive in detecting SARS-CoV-2 ever infection than assays that target only the S or N proteins. This was highlighted in a recent study where the CL-900i assay demonstrated the highest sensitivity compared to the other assays. This may also explain the 11 discordant specimens in this study that were seropositive using the CL-900i assay but seronegative using the VidasIII and Elecsys assays ( Table 1). The antibody testing outcome obtained using the Elecsys assay was more similar to that of the CL-900i assay, possibly because the Elecsys assay detects the total antibodies (IgG, IgA, and IgM) against the SARS-CoV-2N protein 30,31 . An earlier study suggested that the use of S1 or the RBD alone is associated with lower sensitivity than the full S protein 32 . The greater sensitivity of antibody response found against the trimeric full S protein is likely to result from antibodies binding to the S2 subunit and the conservation of conformational epitopes within the higher-order structure 32 . Therefore, assays that specifically target the S1 or the RBD may underestimate the seroprevalence of SARS-CoV-2, a finding that may explain the lower seroprevalence obtained through the VidasIII assay.  Earlier studies demonstrated that antibody response against different SARS-CoV-2 antigens may also develop with different kinetics and thus could be used as indicators for the stage of infection. In the acute phase of infection, antibody response against the N and S proteins develops simultaneously, whereas the response against the N protein appears to wane faster compared to the S protein, which tends to persist over time [32][33][34] . This may also contribute to explaining some of the discordant outcomes found in this study.
This study has some limitations. Two out of the 394 participants included in the study declined PCR testing (but not serological testing) at time of specimen collection. The performance of these antibody assays was compared to each other (and to PCR testing), but not to a gold standard test of seropositivity, as such a test was not available to study investigators. Therefore, we were unable to measure ever infection prevalence to a gold standard, and use this to compare the performance of each assay to the gold standard, nor to assess the sensitivity and specificity of each assay in the study sample. The specificity of the Elecsys assay has previously been reported to be 99.98% and the sensitivity to be 98.80% on day 14 after PCR diagnosis 5 . A validation study by Public Health England reported a specificity of 100% and a sensitivity of 83.9% for the same assay 28 . As for the remaining assays, specificity and sensitivity were reported at 94.9% and 82.2%, respectively, for the CL-900i assay 29 , and at 99.9% and 88.6%, respectively, for the VidasIII assay 7 .
In conclusion, all three assays had comparable performance and excellent agreement when used in a high SARS-CoV-2 exposure setting, but still missed at least 20% of cases with laboratory-confirmed evidence of ever infection. This suggests that current growing use of commercial antibody assays to assess ever infection in population-based surveys, especially at times of high SARS-CoV-2 incidence when many infections are recent, is likely to substantially underestimate actual infection exposure. The findings demonstrate further the need to interpret the serology testing together with PCR testing.

Data availability
All relevant data are available within the manuscript and its supplementary materials. License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.