Introduction

Genetic testing is routinely used for diverse applications, including risk assessment, prognosis, diagnosis, and disease management. The accurate interpretation of DNA variants identified through genetic testing is essential for patients and clinicians to make informed medical decisions. Until recently, variant interpretations generated by experienced diagnostic laboratories have not been made readily available to the broader community, leading to a situation in which classifications for the same variants differ between laboratories.1, 2, 3 Making variant-level data accessible enables knowledge sharing and leads to more consistent variant classifications, to the ultimate benefit of patients, clinicians, and the scientific community.

Professional societies such as the American College of Medical Genetics and Genomics (ACMG) and the Association for Molecular Pathology (AMP) provide variant-interpretation guidelines,4 but these guidelines are qualitative and inexact. Differences in the clinical interpretation of DNA variants across laboratories and databases continue to be a major challenge.1, 2, 3, 5 Global initiatives have been created in recent years to disseminate variant interpretations found in both clinical and research laboratories, including ClinVar,6 the Human Variome Project,7 LOVD,8 and the BRCA Exchange.9

We created the Canadian Open Genetics Repository (COGR; http://www.opengenetics.ca), a collaborative effort for the collection, storage, sharing, and analysis of variants reported by medical diagnostics laboratories across Canada, using a clinical-grade genetic database and a commonly shared platform designed to hold multiple forms of information related to human gene DNA variants and their relationship to disease.10 This database draws from the genetic holdings at clinical laboratories and hospitals across Canada.

We present a COGR initiative to build consensus interpretations on variants identified in Canadian diagnostic laboratories and academic medical centers. The collaborative structure of the COGR promotes real-time sharing between geographically distant laboratories and enhances the exchange of information about DNA variants within the expert community. To date, the COGR has 22 participating institutions across eight provinces: Alberta, British Columbia, Manitoba, Ontario, Quebec, New Brunswick, Nova Scotia, and Newfoundland and Labrador. As of December 2016, COGR has over 18,000 variants (approximately 13,500 single-nucleotide and small-insertion, deletion, and duplication variants and 4,500 copy-number variants), encompassing 1,298 genes and 79 diseases. By creating and disseminating laboratory-specific comparison reports along with standardized assessment procedures, we aimed to provide a mechanism for resolving discordant classifications.

This study focused on a large, cross-laboratory case study in data sharing and resolution through the COGR for variants within the BRCA1 and BRCA2 genes. This type of interinstitutional quality improvement program is increasingly critical as new technologies make the accumulation of genetic knowledge cheaper and more accessible, and will have the added benefits of advancing knowledge through data sharing and providing a rich store of clinical variant data for more effective use in clinical applications.

Materials and methods

The COGR, a clinical genomic database and community resource for standardizing and sharing variant-level interpretations across Canada, has been described previously.10 The COGR platform relies on the GeneInsight software,11 an information technology platform designed for the management of genetic data. To date, there are 22 participating sites across Canada, with 11 sites currently sharing variant data. This case study focused on the BRCA1 and BRCA2 genes. When this initiative began, 12 of the 22 sites were performing BRCA1 and BRCA2 genetic testing. Of the 12 eligible laboratories, 11 took part in this initiative (Figure 1).

Figure 1
figure 1

Geographical location of COGR participants. Map of 22 sites participating in the Canadian Open Genetics Repository (COGR). The yellow markers indicate the sites that participated in this initiative (A, C, E, F, G, H, K, N, O, Q, and U); the red markers indicate the remaining COGR sites. *Accredited laboratories.

Full size image

Data were obtained primarily from accredited (certified) clinical laboratories that perform routine molecular genetic diagnostic testing using validated methodologies following strict provincial and international quality standards (e.g., those of the Institute of Quality Management in Healthcare, the Clinical & Laboratory Standards Institute, the College of American Pathologists, the Clinical Laboratory Improvement Amendments, and ACMG). Scientific evidence for classification of a genetic variant was collected by qualified experts in the clinical genetics community across Canada, including board-certified geneticists at the Canadian College of Medical Genetics and the American Board of Medical Genetics and Genomics. These laboratories are compliant with federal and provincial laws, such as Ontario’s Personal Health Information Protection Act (SO 2004, c 3, Sch A). One academic medical center laboratory with a focus on research testing of BRCA1 and BRCA2 also participated in this study.

Aggregation and interpretation of variants

Between October and November of 2015, each participating site submitted retrospective BRCA1 and BRCA2 variant data and classifications according to the workflow shown in Figure 2a. To facilitate comparison, variant data were downloaded from each GeneInsight instance and chromosomal positions were verified and converted to the GRCh37 (hg19) build using an independent software package (Mutalyzer, release 2.0.14).12 BRCA1 variants were mapped to transcript NM_007294.3 and BRCA2 variants to transcript NM_000059.3.

Figure 2
figure 2

(a) Workflow used to build a consensus on variants identified in Canadian clinical laboratories. (b) The three independent tier models used to determine variant concordance across laboratories based on what was originally used (five-tier), what was suggested by participants (three-tier), and what reflects clinical management (two-tier).

Full size image

An updated version of a variant assessment tool (VAT) was provided to all participants, together with standard operating procedures (SOPs; available via download at http://www.opengenetics.ca/vat2/).13 However, individual laboratories were responsible for maintaining their own interpretation processes (e.g., versioned documentation of variant SOPs) and for keeping pace with new knowledge, such as the availability of new locus-specific databases. Variant-interpretation processes, while different in each laboratory, were based on published guidelines and included validated decision matrices.4, 10, 13, 14 In-depth analyses of genetic variants were performed by qualified experts, including certified genetic counselors and laboratory directors, informed by the available data and the clinical context in which the patients presented. Multiple lines of evidence were integrated to assign variant classifications, including scientific literature, established gene–disease relationships, type and location of DNA alteration, population frequencies, segregation in families, amino acid conservation, functional studies, biochemical properties, and computational predictions. To establish uniformity in interpretation terminology schemes, all laboratory variant-level interpretations were mapped to five classification categories: benign (B), likely benign (LB), variant of uncertain significance (VUS), likely pathogenic (LP), and pathogenic (P).

Variant interpretations were compared and analyzed for discordances across the 11 participating sites. Variant interpretations were considered “concordant” if interpretations of the variant across multiple sites were all in agreement, and were otherwise deemed “discordant.” Variants without interpretations were considered “unclassified” and were omitted from analysis if there were no new interpretations following review of comparison reports. As an example, if three sites (A, B, and C) observed the same variant, with sites A and B having interpretations in agreement and site C having no classification, the variant was considered concordant across two sites. Unclassified variants were included in the comparison reports to encourage reassessment.

Comparison reports

Laboratory-specific comparison reports were assembled into three main sections: concordant variants, discordant variants, and variants unique to their laboratory (available via download at http://www.opengenetics.ca/comparison-report-template/). For each variant, laboratories were provided with details, including the complementary DNA change (in Human Genome Variation Society nomenclature), reference sequence, amino acid change, and variant type (e.g., frameshift, missense, nonsense, silent, insertion, deletion, insertion/deletion, intronic, untranslated region, or splice-site variant), the laboratory’s interpretation, as well as the number of laboratories in agreement. For each discordant variant, laboratories were also provided with data displaying how their variant classification compared with other anonymized laboratories’ classifications and, if available, summary evidence compiled using the VAT without any leading or conclusive statements about the data.

Laboratories were asked to reinterpret each discordant variant and, where relevant, to provide the rationale for the change in interpretation, whether they reviewed other laboratories’ detailed interpretations through the sharing mechanism in GeneInsight or contacted other laboratories to discuss the variant, and which methods were used for the assessment. Laboratories were also given an opportunity to select variants that they wanted to discuss with the COGR group members through an online meeting prior to assigning a final interpretation. Comparison reports were released to each site independently, along with a supplementary interpretation tool created to map evidence according to the ACMG 2015 variant-interpretation guidelines.4

Analysis and consensus discussion

The 11 participating sites completed comparison reports, which were aggregated and analyzed for changes in variant interpretations compared with the initial submission. Three different methods of analysis were applied based on number of tiers (Figure 2b). The first was a five-tier comparison model in which B, LB, VUS, LP, and P were considered independent tiers. The three-tier combines classifications with a similar severity but differing levels of certainty by considering B/LB as one tier, VUS as the second tier, and LP/P as the third. The two-tier comparison model is based on the assumption that LP/P classifications are considered clinically actionable whereas B/LB/VUS are not. Thus, in the two-tier model, B/LB/VUS made up one tier and LP/P another. Directionalities of the classification changes were analyzed to assess the impact of differing comparison models. Preliminary findings were presented and discussed in an online meeting. Several discordant variants were discussed with COGR members at the request of laboratories and were reviewed with participating laboratories during this meeting.

The number of discordant variants compared with the total number of variants submitted by each laboratory was used to determine the discordance rate. The total number of discordant variants was determined by measuring the difference in the number of discordant variants before and after analysis of the comparison report for each laboratory.

Results

Variants submitted

A total of 5,554 BRCA1 and BRCA2 variant classifications were submitted between October and November of 2015 by the 11 participating laboratories from Ontario, British Columbia, Alberta, and Manitoba. The numbers of variants submitted by participating laboratories ranged from 110 to 1,072. The following variant classifications were initially submitted: B (895, 16.1%), LB (487, 8.8%), VUS (1,279, 23.0%), LP (146, 2.6%), P (2,167, 39.0%), and unclassified (580, 10.4%). There were 3,014 independent variants among the 5,554 submissions: 1,866 were classified by only one laboratory, 900 were classified by two or more laboratories, and 248 had no classifications (see Supplementary Figure S1 online). Thus, a total of 2,766 unique classified variants were observed across the participating laboratories (Supplementary Figure S2).

The 900 variants classified by two or more laboratories had, on average, 3.5 observations. Among these variants, 550 (61.1%) had concordant interpretations and 350 (38.9%) were discordant based on the five-tier comparison model (Figure 3). Using the three-tier comparison model resulted in 660 (73.3%) of variants with concordant interpretations and 240 (26.7%) with discordant interpretations, whereas implementing the two-tier comparison model resulted in 855 (95.0%) concordant and 45 (5.0%) discordant interpretations (Figure 3). There was a total of 1,335 observed interpretations among the 350 discordant variants. These interpretations were classified as follows (numbers in parentheses represent the number and percentage of discordant-variant observations, respectively): B (526, 39.4%), LB (271, 20.3%), VUS (342, 25.6%), LP (60, 4.5%), and P (136, 10.2%) (Supplementary Figure S3A). A total of 202 of the 350 five-tier discordant variants had summary evidence descriptions available for the comparison reports.

Figure 3
figure 3

Concordant and discordant variant interpretation results before and after review. Comparison of total concordant and discordant variants submitted versus totals after review of comparison reports, based on the three comparison models: five-tier (blue), three-tier (orange), and two-tier (purple).

Full size image

Analysis of comparison reports

After analysis of the comparison reports, 595 of 1,335 (44.6%) discordant-variant interpretations did not change classification, 317 (23.7%) changed classification owing to reassessment, and 423 (31.7%) were not reassessed. After reclassification, both the B and LB categories increased in number of variants, by 17.3% and 15.5%, respectively, while the number of VUSs decreased by 38.9%. Of variants initially classified as VUS, 50 changed to B, 72 to LB, 14 to LP, and 9 to P (Supplementary Figure S3B). Additionally, there were 75 instances in which a variant that was discordant between at least two laboratories was uploaded but left unclassified by another laboratory. Of these previously unclassified variants, 74 (98.7%) were newly interpreted, with 34 classified as B, 9 as LB, 17 as VUS, 2 as LP, and 12 as P.

Based on the five-tier comparison analysis model, 75 of the 350 (21.4%) discordant variants reached consensus, increasing the total number of concordant variants to 625 of 900 (69.4%) (Figure 3 and Supplementary Figure S1). Of the 75 variants reaching consensus, 26 were seen by two laboratories and 49 by three or more laboratories. On average, there were 1.2 reclassifications per variant among the 49 variants (60 variant classifications were changed on reassessment). These 60 reclassifications were distributed across the participating laboratories.

When the three-tier comparison model was utilized, the number of variants that were concordant increased to 773 (85.9%), a gain of 113 variants (Figure 3). For the two-tier model, the number of concordant variants increased to 892 (99.1%), with 37 additional variants achieving consensus and 8 (0.9%) variants remaining discordant (Figure 3 and Table 1). In general, discordance rates after reclassification were similar in the various laboratories, especially under the three-tier and two-tier schemes (Table 2).

Table 1 Eight variants remaining discordant in the two-tier model after review of the comparison reports
Full size table
Table 2 Lab discordance rates before and after review
Full size table

The method of reassessment was provided for 476 of the 907 reassessed variants; most reassessments were carried out using the laboratory’s own variant-interpretation tools (210 observations or 44.1%) or the variant assessment tool provided by the COGR (239 variant observations or 50.2%), while a smaller number used the 2015 ACMG tool provided in this study (27 variant observations or 5.7%) (Supplementary Figure S4A). While no laboratory indicated direct contact with another laboratory during the assessment process, for six variants a laboratory made use of the variant-sharing infrastructure in GeneInsight to view other laboratories’ specific evidence and classifications.

Of the variant interpretations that changed classification, 51 of 317 (16.1%) variant interpretations were determined to be errors that occurred during the project owing to translation between different classification schemes (Supplementary Figure S4B). Most of the remaining changes were “due to revised interpretation criteria” (112, 35.3%) or based on “new evidence provided or evidence described by other laboratories” (142, 44.8%); the remaining 12 (3.8%) were changed for other reasons (Supplementary Figure S4B).

Seven discordant variants were selected from participants to be discussed during the online meeting. Before this discussion one of the seven variants reached concordance, based on the five-tier comparison model after review of the comparison reports. After discussion, consensus across the five-tier system was reached for one additional variant. Of the remaining five variants, consensus was reached on three variants, based on the three-tiered model, and on two based on the two-tier model. The classification for the 900 variants seen by two or more labs can be found at http://www.opengenetics.ca (Supplementary Table S5).

Discussion

We describe a process for resolving differences in variant-interpretation rates across Canadian laboratories, using the BRCA1 and BRCA2 genes as a case study. We aimed to arrive at a consensus for variant interpretations and reduce variant-classification differences between professionals reporting on BRCA1 and BRCA2 variant data. Initially, based on a five-tier classification model, striking rates of discordance were observed, with 38.9% of variants discordant between at least two laboratories. In the ACMG/AMP interpretation guidelines, the five-tier model is recommended for laboratory use to reflect the variable degree of certainty inherent in variant interpretation. However, in practice, the three-tier and two-tier models may more closely reflect how individual patients are managed in the clinic. For instance, patients with an uncertain significance, likely benign, or benign variant are often managed based on whether they have a family history of breast or ovarian cancer rather than on their genetic test result.15, 16 The rates of interpretation differences were considerably lower when these less granular models were applied, suggesting that the majority of discordant calls did not span boundaries typically used for clinical management. Notably, before reassessing variants, each laboratory had, on average, a 43.6% discordant rate using the five-tier model, with most groups being similarly discordant (Table 2). After the laboratories reassessed variants, the number of discordant classifications across laboratories dropped from 350 to 275 —a reduction of 8.3% using the five-tier discordance model.

To identify the primary reason consensus was reached, we examined the 75 variants for which consensus was achieved after review. Among the variants with three or more observations (49 of 75), concordance was reached after only 1.2 labs, on average, changed classification. These individual laboratory reclassifications were spread across the different laboratories, highlighting how important it was for all laboratories to take part in the reassessment process. The main reason that variant interpretations changed among this set was the availability of new evidence (52.7%), followed by the use of revised classification criteria (28.4%). This suggests that many of the discordant-variant interpretations were made several years ago, emphasizing the need for periodic review and for tracking and maintaining versioned variant information, including a date stamp associated with each variant classification.

Eight variants remained discordant after the comparison report and reinterpretation process in the two-tier comparison model, including variants with VUS and LP and P classifications (Table 1). Interestingly, as of October 2016, five of the variants were discordant in ClinVar (four with conflicting interpretations spanning the two-tier model and one spanning the three-tier model), two had only a single interpretation in ClinVar, and one was absent from ClinVar.6 These variants were representative of some of the more challenging aspects of clinical interpretation, such as lack of segregation data, few functional studies, poorly performing predictive software algorithms, and differences in evaluating risk and/or penetrance. These variant discordances were spread fairly evenly across the laboratories, with no clear outlier in terms of percent discordant variants across the two-tier model (Table 2).

Consensus discussion and tiered analysis

We identified 428 of 1,335 (32.1%) discordant-variant observations that did not undergo reassessment, which accounted for >75% of the discordances after review: 226 observations remained discordant based on the five-tier model, 97 based on the three-tier model, and six based on the two-tier model. Although many discordant interpretations were resolved, additional review of discordant variants may have resolved more conflicts. Unfortunately, clinical laboratories often have limited resources to assess and reassess variants, are not reimbursed for reassessing variant classifications, and do not share common policies for doing so. The reassessment of the variant data herein occurred because of this study, which provided funding and infrastructure to share and compare data. Thus, reassessment of variants may be desirable but not necessarily feasible. Even after reassessment, discordance remained between expert assessors, which points to a lack of available information or to limitations of the ACMG guidelines, which may be misunderstood or improperly applied by assessors. Additionally, some laboratories chose not to reassess variants because they believed discordance would not make a clinically significant difference. For instance, many of the participating laboratories did not review LB versus B or LP versus P discordances, because they were considered “minor discrepancies,” demonstrated by a larger reduction in discordant variants across the three-tier model as compared with the five-tier model. While 317 variants were reclassified, only for 65 (20.5%) variants did the classification change across more than one tier in the five-tier system. Furthermore, apart from typographical errors, none jumped between the extremes of the three-tier system (LP/P to LB/B or vice versa; Supplementary Figure S3B). This highlights the thorough nature of the initial classifications.

Some of the initial discrepancies turned out to be due to translation errors between different classification systems. These errors were not present on the initial patient reports, but rather occurred when data were manually extracted and mapped to a common classification scheme. This issue is solved by laboratories coalescing around a common classification scheme, and, with release of the ACMG/AMP classification guidelines, this is starting to occur. The need for manual extractions and translations highlights the challenge for small laboratories with limited funding to upgrade their informatics solutions along with expanding test content.

In addition to providing tools for the interpretation of variants to the laboratory participants, we also created an online meeting for expert review of variant interpretations. After seven variants were discussed during the meeting, all seven were deemed to be concordant across the two-tier system, five across the three-tier system, and two across the five-tier system. The c.5116G>A (p.Gly1706Arg) in BRCA1 reached a consensus of LP, and highlighted the important nature of data sharing and discussion. This variant had been reported as deleterious in a functional study,17 but the variant nomenclature used in that report made it unclear whether the variant studied was the same as that seen in this data set. Group discussion clarified the finding, and demonstrated the need for journals to enforce the use of Human Genome Variation Society nomenclature for variant descriptions in published reports. The overall feedback from participating laboratories about the online discussion was positive; the forum was viewed as allowing laboratories to share evidence more directly and work through specific concerns. This type of discussion can serve as a useful method for resolving variant interpretations when laboratories are struggling to understand why a particular variant had different interpretations.

The ability to interpret and classify variants is based partly on the availability of tools to facilitate the process as well as clinical judgment. The COGR provided a variant assessment tool to individual laboratories; however, most laboratories continued to use their own proprietary methods. While standardization in variant-classification protocols is the goal, the results of this initiative indicate that for most variants there are no major differences in interpretation between laboratories, and in many cases, significant differences can be resolved through reassessment and data-sharing approaches. Additional refinement of the classification rules, including gene- and disease-specific modifications being developed by the Clinical Genome Resource,18 will also foster more uniform interpretations across laboratories. Further investigation is needed to understand what actions are taken and what decisions are made by clinicians and patients based on genetic results. While the COGR has gone a long way toward standardizing variant interpretations and nomenclature, individual laboratories may still differ in their reporting practices. Additional work is needed to standardize how information is reported back to clinicians and patients.

Submission of genomic variant data may become mandatory as part of proficiency testing and laboratory accreditation, and disclosure of variant data may soon become a requirement for all testing laboratories. By further comparing differences in laboratory interpretation processes, taking account of staffing, software, standard procedures, and other resources, we might be able to better understand the reasons underlying discordant-variant classifications. Thus, we encourage individual institutions to share their data holdings through collaborative networks such as COGR and open databases such as ClinVar. Efforts such as the COGR provide secure and valuable systems for comparison of variant data and an interinstitutional quality assessment program.

This initiative was limited in scope to only two genes across eleven laboratories. However, it was set up as a pilot project to establish the procedures for collecting, comparing, and reassessing variants interpreted by laboratories across Canada. The program is currently being rolled out to cover all genes and diseases across the Canadian clinical laboratory community, with an automated process for developing the discrepancy reports. The data holdings among the participating laboratories may be limited, demonstrating the need to connect this effort with other international variant sharing networks. The COGR is implementing data exchanges with other similar initiatives, including VariantWire, the Beacon Network, and ClinVar, enabling this program to have an even wider impact.

Conclusion

Here, we present an interinstitutional quality improvement program for DNA-variant interpretation at the national level. Sharing of variant knowledge provides additional evidence for clinical diagnostic laboratories to more accurately classify variants, thereby allowing clinicians and patients to make more informed decisions, and may lead to better patient outcomes, such as more consistently correct diagnoses and appropriate patient management. In the case of genetic testing for BRCA1 and BRCA2, surgical, chemotherapeutic, or other management decisions might be based on a test result.19 None of the participating sites were previously aware of their variant-interpretation discordance rates, nor was there a mechanism in place to enable analysis of variant data across Canadian institutions prior to the creation of the COGR.

Sharing data through the COGR has enhanced the ability of Canadian public hospital laboratories to identify differences in interpretations that may have led to preventable errors in the delivery and communication of genetic testing results. As genetic variation plays an ever more important role, in both patient management and treatment, accurate and consistent variant interpretation is crucial. Supporting this, the type of quality assurance program presented here will become increasingly important as the scale of genetic testing grows, and will ultimately improve patient outcomes. Because this program was carried out by professionals who work within the context of routine diagnostic testing, it will also help hospitals across the country to gauge the quality of their genetic testing programs. In addition, this work raises questions around whether laboratories have the duty to update, reclassify, and recontact clinicians when variant classifications change over time.20

In summary, the COGR serves as a focal point for the collaboration of Canadian diagnostic laboratories with one another and with laboratories in other countries in the development of tools and methods that take full advantage of laboratory data in diagnosing and managing genetic diseases. As a continuing effort, the COGR endeavors to increase genetic knowledge and standardization through data sharing and consensus building, ultimately improving our ability to diagnose and treat genetic diseases.