Introduction

HIV incidence determination is a vital component of monitoring and understanding HIV transmission dynamics in high-risk populations and measuring prevention effectiveness1. Traditionally, HIV incidence estimation has been carried out using prospective cohort studies that pose many logistical and cost challenges2,3. With the recent development of laboratory techniques able to distinguish between recent and long-term HIV-1 infections, the use of cross-sectional surveys for HIV-1 incidence estimation has become possible4. Among the many laboratory-based incidence assays developed, the most evaluated assay is the immunoglobulin G (IgG)-capture BED- enzyme immunoassay (BED-CEIA) based on measurement of the proportion of HIV-1-specific IgG to total IgG after seroconversion5,6,7. Licensing and manufacturing of commercial kits with consistent quality began in 2005 and this assay is currently used globally for population surveillance1,8,9,10.

Beginning in 2005, the Chinese National HIV/HCV Reference Laboratory and several other provincial laboratories adopted the BED incidence assay for HIV-1 incidence surveillance. Collectively, they demonstrated high inter-laboratory reproducibility and yielded HIV-1 incidence estimates similar to those obtained by prospective cohort studies in China1,11,12,13,14. As the number of laboratories performing incidence assays rapidly increased, the need to monitor overall and individual performance increased. The National HIV/HCV Reference Laboratory has an extensive quality assurance (QA) and quality control (QC) system for HIV/AIDS testing, including laboratory staff training, supervision, management, accreditation, proficiency testing (PT) and diagnostic kit evaluation for serologic, immunologic and molecular assays15,16.

PT is a key component of external quality assurance, which utilizes inter-laboratory comparison to determine the ability of laboratories to obtain accurate results and has been shown to be effective in monitoring and improving HIV-related laboratory testing quality17,18,19,20,21,22,23. The goals of PT for HIV incidence testing are to assess the performance of participating laboratories and identify issues for corrective action and technical assistance. The National HIV/HCV Reference Laboratory organized provincial BED laboratories to voluntarily participate in the semiannual international BED PT program run by the U.S. Centers for Disease Control and Prevention (CDC) in 2006 and subsequently developed a domestic BED PT program in 2011. In this report, we summarize performance characteristics of the international and domestic BED PT programs.

Results

Participation and overall rating

In addition to the National HIV/HCV Reference Laboratory, 7 other Chinese laboratories participated in the international BED PT program provided by the U.S. CDC in 2006; this number increased to 19 laboratories in 2012 (Fig. 1a). In round 10, one laboratory participated in the PT program at a late stage and shared one panel with another laboratory which resulted in more reports sent than the number of panels received. Overall, there were 182 results, all with valid QC and correct recency classifications and thus all Chinese laboratories received a “pass” rating in the international program over 13 rounds.

Figure 1
figure 1

Participation and performance of laboratories of China in the international BED PT program over 13 rounds (a) and in the domestic BED PT program over 3 rounds (b).

Full size image

For the domestic PT program, there were 31 and 32 participating laboratories in 2011 and 2012, respectively (Fig. 1b). Ninety (95.7%) of the 94 returned results achieved a perfect score over 3 domestic PT rounds. Three (9.7%) laboratories in the initial round and one (3.1%) in the third round failed to attain a pass rating. Three of these four laboratories with failed rating were new PT participants.

PT data analysis

A total of 104 specimens were distributed over 13 international PT rounds and 24 specimens were distributed over 3 domestic PT rounds. Overall, the concordance plots between participant normalized optical density (ODn) and expected ODn results are shown separately for international and domestic PT (Fig. 2a) and by kit manufacturers (Fig. 2b and 2c). The agreements between obtained and expected results were high with correlation coefficients (R2) ranging from 0.942 to 0.998. Bland-Altman plots of the mean differences were all close to zero (−0.029 and 0.199 for international and domestic PT, respectively) and the 95% limits of agreement include 92.3% (96/104) and 91.7% (22/24) of all difference data from international (Fig. 2d) and domestic (Fig. 2e) PT, respectively. Similar results were also obtained between Calypte and Sedia kits (result not shown). Results in the lower left quadrant (Fig. 2a, 2b and 2c) represented incident specimens and those in the upper right quadrant represented long-term infections. There was one specimen in round 12 of the international program and another in round 1 of the domestic program that yielded discordant classification (lower right and upper left). Because ODn values of these specimens were close to the 0.8 cutoff for recency determination, the specimens are expected to fluctuate around the cutoff and laboratory performance was considered satisfactory.

Figure 2
figure 2

Concordance of mean obtained results with expected results for 13 international BED PT rounds (n = 104) and 3 domestic BED PT rounds (n = 24).

Regression analysis: correlation of expected ODn results and Mean ODn results. The best-fit regression line is shown for each PT program. Vertical and horizontal arrows indicate 0.8 cutoffs. Mean ODn results vs. expected ODn results for all participants (a); participants using kits from different manufacturers during international PT (88 Calypte kits, 40 Sedia kits) (b); participants using kits from different manufacturers during domestic PT (Calypte kits, n = 8; Sedia kits, n = 24) (c). Bland-Altman analysis: correlation of expected ODn results and Mean ODn results for international PT program (d) and domestic PT program (e).

Full size image

QC analysis

The BED assay kit contains 4 controls including a calibrator (CAL) to validate each plate run and normalize the optimal density of specimens in relation to CAL optical density (OD) which reduces inter-run and inter-laboratory variability. There were 364 plate assays (210 and 154 by Calypte and Sedia kits, respectively) conducted in 13 rounds of the international PT program. There were 183 plate assays (26 and 157 by Calypte and Sedia kits, respectively) in the 3 domestic PT rounds. The performance characteristics of these runs are shown in Table 1. In the international PT, coefficients of variation (CVs) of the mean OD values of low positive control (LPC) and high positive control (HPC) were between 20.2% and 23.6% and were reduced to 6.3 to 7.1% after normalization, which illustrates a reduction in variability. The ODn readings obtained from Calypte and Sedia kits were very close. For instance, the mean ODn of LPC was 0.611 (CV 7.1%) and 0.594 (CV 6.6%) for Calypte and Sedia, respectively. Similar narrow ODn ranges and CVs were observed in LPC and HPC in the domestic program. Negative control (NC) CV was high, as anticipated, due to their low OD readings.

Table 1 Mean OD and ODn values for CAL and controls, including standard deviation (SD) and coefficient of variation (CV%) from international and domestic BED PT rounds
Full size table

Laboratory reproducibility

In order to assess intra-laboratory reproducibility, overall concordance of initial ODn results with confirmatory ODn results were compared by regression analysis in Fig. 3a and 3b and by Bland-Altman analysis in Fig. 3c and 3d for international and domestic PT, respectively. The R2 of the linear regression lines for international and domestic BED PT were 0.942 and 0.905, respectively; The mean differences were all close to zero (0.015 for international PT and −0.003 for domestic PT) and the 95% limits of agreement include 94.0% (739/786) and 94.2% (425/451) of all difference data reported by international and domestic PT, respectively. Four of 451 results, as depicted in open circles, showed considerable deviation (Fig. 3b and 3d).

Figure 3
figure 3

Overall concordance from all participating laboratories between the initial and confirmatory runs for 786 results during international PT rounds and 451 results during domestic PT rounds.

Regression analysis (a and b) and Bland-Altman analysis (c and d) for international and domestic PT program, respectively. For each PT program, the best-fit regression line is shown with statistics in the upper left (a and b). Four results with large different initial and confirmatory ODn are indicated by open circles (b and d).

Full size image

The presence of 2 identical specimens in each international PT round was used to provide another insight into assay reproducibility within and between laboratories. The specimen used for the reproducibility check in each round was different and thus their ODn value changed from round to round. Fig. 4 shows data on duplicated specimens in Calypte and Sedia kits (Fig. 4a), individual Calypte kit (Fig. 4b) and Sedia kit (Fig. 4c). The mean ODn values of the duplicate specimens were very close to each other in each round, indicating high reproducibility. The average CV of duplicates was 11.4% and mean CV of Calypte and Sedia kits was 12.3% and 10.0%, respectively.

Figure 4
figure 4

Performance characteristics of duplicated specimens in the international PT program rounds, with black and white bars representing the specimen replicates.

Mean ODn with standard deviation (SD) of all participating laboratories (a), laboratories using Calypte kits (b) and laboratories using Sedia kits (c). Sedia kits became available in 2010. In round 13 of the international BED PT program and round 3 of the domestic BED PT program, all participating laboratories used Sedia kits except one. The results from that laboratory in these two rounds were not included.

Full size image

Cause of laboratory error and corrective actions

All Chinese laboratories that participated in 13 rounds of the international PT program from 2006 to 2012 received “pass” ratings. In the domestic program, 4 reports of the 94 returned reports failed over 3 rounds (Table 2). There was one misclassification report in round 1, but because a corrective action report was not returned, the cause is unknown. A kit reagent problem in round 1 was not resolved because the laboratory did not have an additional kit to repeat the testing. An invalid QC result in round 3 was due to a transcriptional error when a technician incorrectly transferred data from the laboratory form to the PT report form. The remaining 2 results with invalid QC were produced by 2 laboratories newly participating in PT.

Table 2 Laboratory failures and associated reasons during the domestic PT programd
Full size table

Discussion

To understand the expanding HIV epidemic, the Chinese government took steps to scale up HIV screening to identify infected persons for early treatment in order to reduce the mortality and transmission24,25,26. To better understand the trajectory of the HIV-1 epidemic in most-at-risk populations, China adopted the laboratory-based BED incidence assay initially developed at the U.S. CDC in 20025. Several incidence measurements were carried out by both BED assay and prospective cohort studies in China and yielded comparable results1,12,13,14. With vigorous laboratory training and quality monitoring, the BED assay platform was quickly expanded to include 32 provincial laboratories16. In 2011, the China CDC formulated a national incidence program to use provincial serologic confirmatory laboratories to conduct its annual national HIV incidence determination27 and designated the National HIV/HCV Reference Laboratory as the principle laboratory to coordinate and monitor overall performance and quality assurance. The use of multiple provincial serologic confirmatory laboratories in China is in sharp contrast to the use of a centralized BED-based incidence laboratory in the U.S. This de-centralized operation mode necessitates the needs of timely coordination, high quality reagent kits, uniform performance across various provincial laboratories and stringent quality monitoring.

In 2006, the National HIV/HCV Reference Laboratory organized 7 provincial laboratories to participate in the U.S. CDC's international BED PT program. To reduce the international transportation costs and make distribution more uniform, the PT panels were collectively shipped from the U.S. CDC to the National HIV/HCV Reference Laboratory. By 2012, the number of participating laboratories of this international program gradually increased to 19. China was then a major participant country in the U.S. CDC's BED PT program, constituting 45% of total program laboratories. Anticipating the need of quality monitoring of the annual national HIV incidence survey, the National HIV/HCV Reference Laboratory took another major step in 2011 and established a domestic PT program. This domestic program was well accepted. There were 32 participating laboratories in 2012 representing a 68% increase over the number of Chinese laboratories in the international program.

As reported by Dobbs et al28, it was critical to demonstrate the quality of commercial BED assay kits and the consistent quality performance across laboratories and operators. Our data from both international and domestic programs indeed demonstrated high correlations between participant and expected results (Fig. 2), outstanding agreement between initial and confirmatory tests (Fig. 3) and very low CV of duplicate specimen results (Fig. 4). In 3 rounds of the domestic program, 95.7% of reports correctly identified the recency status of all PT specimens. The R2, slopes and intercepts of the best-fit regression lines shown in Figs. 2b and 2c and the controls summarized in Table 1 illustrate minimal difference in performance between the two kit manufacturers. As expected, laboratory staff who participated in both the international and domestic PT performed better than those who participated only in the domestic program. Laboratories with invalid results were targeted for more extensive training and corrective remedies were distributed through a nationwide laboratory newsletter.

However, we also found the shortcoming of the domestic BED PT program from the summarized results. All the results of regression analysis over the domestic PT program were slightly inferior to the international PT. The probable reasons include inexperience of staff of some new participating laboratories, the challenges faced in storage and administration of specimens in some resource-limited settings. Additionally, the lack of duplicate specimens in each domestic PT round resulted in simplification of the approach for evaluating the intra-laboratory reproducibility, only through assessing the concordance between initial and confirmatory results. As the domestic BED PT program is in the initial stage, more improvements will be achieved with its development.

Our successful coordination and monitoring of the quality of laboratory-based HIV-1 incidence testing in a large national HIV testing network in China is highly encouraging. Our findings illustrate the high quality of reagents and reproducibility of operators. Our success could be replicated by other countries or regional programs. A new limiting-antigen avidity assay was recently reported to provide more accurate HIV-1 incidence estimation than the BED assay29. We are currently evaluating this new assay and will launch a domestic program based on the lessons learned from our current BED PT experience.

International and domestic BED PT programs were successfully used to monitor the quality of laboratory-based incidence estimation and better understand the causes of testing errors. Recently, several laboratory-based methods have been devised to identify HIV-1 incident infections29,30,31,32,33,34. As the usefulness of these assays is demonstrated and they are adopted, quality monitoring programs such as PT will need to be established. Our experience with the HIV-1 BED incidence assay, first using an international program and subsequently developing a domestic program, will guide the quick establishment of PT programs for other assay platforms to strengthen the Chinese HIV laboratory network. Other countries or regions in the world can also learn from this experience to establish appropriate local PT programs for assay-based HIV incidence estimation.

Methods

PT panel specimens and performance reporting

The international PT program was provided by the U.S. CDC. The heat inactivated (56°C, 30 minutes) serum specimens (n = 8) chosen for each PT round were derived from a repository of 25 undiluted HIV-1 positive bulk volume donor plasma samples28. Their OD values and recency status were determined by HIV-1 BED incidence assay in the U.S. CDC laboratory in Atlanta, Georgia. Specimens were aliquoted 100 μl per tube and stored at −70°C prior to distribution to international participating laboratories. In addition, two of the eight specimens in each panel were duplicates to allow for intra-laboratory reproducibility analysis. PT panels for all Chinese participating laboratories were sent in one shipment in each PT round with dry ice from the U.S. CDC to the National HIV/HCV Reference Laboratory. International transportation and customs clearance typically took 3–5 weeks. Upon receipt at the National HIV/HCV Reference Laboratory, PT panels were repackaged with ice packs and distributed to provincial laboratories. Domestic express mail usually took 2–3 days. Each participating laboratory was requested to report test results to the National HIV/HCV Reference Laboratory, who then transferred results to the U.S. CDC. In reverse manner, the U.S. CDC returned PT results to the National HIV/HCV Reference Laboratory who then distributed results to individual provinces.

For the domestic program, PT specimens were derived from a repository of 40 HIV-1 positive specimens purchased from Kinghawk Pharmaceutical Co., Ltd (Beijing, China). Eight specimens with varying OD values representing recent or long-term infection were randomly chosen from this repository for each domestic PT round and their OD values were verified prior to distribution. Aliquoted specimens (100 μl) were stored at −70°C at the National HIV/HCV Reference Laboratory. The number of specimens with recent or long-term classification varied from round to round and there were no duplicate specimens as in the international PT program. PT specimens were packaged and shipped by express mail service with ice packs. Participants were instructed to store the specimens at −20°C or below prior to testing and not to freeze-thaw them repeatedly. They were also instructed to return results within 10 working days after receipt of specimens.

For both the domestic and international programs, a cover letter and Microsoft Excel data report form were e-mailed to each participant. The report form contained the assay kit name, lot number, expiration date, technician name, assay date, QC and specimen OD and ODn values and result interpretation for recent or long-term status. Laboratory directors were asked to verify data and sign off on the report. The National HIV/HCV Reference Laboratory collected and analyzed results and then summarized overall findings with the names of the laboratories staying anonymous. Each laboratory director was informed of his/her laboratory performance. The summary report was published in the Laboratory Network Newsletter, a China CDC periodical.

BED CEIA assay and incidence classification

All specimens in the PT panel were tested by participating laboratories using BED HIV-1 Incidence EIA kits purchased from either Calypte Biomedical Corporation (Portland, OR) or Sedia Biosciences Corporation (Portland, OR). Kit performance comparisons were conducted for each round except for round 13 of the international program and round 3 of the domestic program in which only one laboratory each used Calypte kit and thus their results were not included in the analysis of Sedia performance. The assay procedure and testing rationale had been previously described5,6,35. Briefly, all specimens were first screened in singlet. A kit CAL, NC, LPC and HPC were run in triplicate on each EIA plate and median values were obtained. Specimen OD values were divided or normalized by median CAL OD to yield ODn. As required by both kits, specimens with initial screening results of ODn 1.2 or less were tested again in triplicate in the confirmatory run. Specimens in the confirmatory run with ODn ≤ 0.8 were classified as recent infection and those with ODn > 0.8 as long-term.

Data analysis and corrective actions

For each round, the National HIV/HCV Reference Laboratory collected information and reports from participating laboratories. The data from all 13 international and 3 domestic rounds was analyzed in aggregate. Overall reported results were compared to the panel's expected ODn values and recent/long-term classification by regression analysis and Bland-Altman analysis. Additional parameters measured include inter-laboratory comparison, inter-assay reproducibility of specimens retested by confirmatory testing and intra-assay reproducibility of duplicate specimens.

To pass a PT round, the OD and ODn values of all QC materials must fall within the established ranges shown in the kit inserts and the final classification of each of the panel specimen should be in complete agreement with expected final result interpretation established by the U.S. CDC in the international PT or by the National HIV/HCV Reference Laboratory in the domestic PT program. A single invalid result in either QC materials or recency classification resulted in laboratory failure. If an error was found, the participating laboratory was notified and required to submit a corrective action form within two weeks to identify the source of the error. If requested, the National HIV/HCV Reference Laboratory assisted participating laboratories in identification of errors.

The summary report for each BED PT round was published in the Laboratory Network Newsletter as mentioned, allowing each participating laboratory to compare their results to the expected results.

Approval of the study

This study was approved by The Academic Committee of The National Center for AIDS/STD Control and Prevention of China CDC and all experiments were carried out in accordance with the approved guidelines. All the PT specimens were from commercial panels (SeraCare Life Sciences, Inc. and Kinghawk Pharmaceutical Co., Ltd for international and domestic PT program, respectively), so there existed no issues about ethics and informed consent.