Just five years ago, the world of variant interpretation was very different. There was no standard terminology for variant classification, and laboratories were using many different terms as ambiguous as “mutation” and “polymorphism” to convey pathogenicity, or lack thereof, and a plethora of qualifiers such as “possibly,” “probably,” and “likely” to convey degrees of uncertainty. The only terminology broadly agreed on was “variant of uncertain significance” (VUS). Furthermore, there was substantial variability in how laboratories evaluated evidence. A single publication in a peer-reviewed journal could be cited as justification for declaring pathogenicity, even though on many occasions little evidence to support that assertion was present in the publication beyond, for example, a non-statistically-significant observation in an affected individual. This was perpetuated by the routine deposition of “deleterious mutations” in the Human Gene Mutation Database,1 often without strong evidence supporting these claims. Similarly, exemplar variants included in the Online Mendelian Inheritance in Man database2 to support gene–disease relationships were assumed to be pathogenic.
EVOLVING STANDARDS FOR VARIANT INTERPRETATION
To address these issues, guidance was published in 2015 by the American College of Medical Genetics and Genomics (ACMG) and the Association of Molecular Pathology (AMP) to provide more detailed standards for how to weigh specific types of evidence and combine them to reach a classification for variants in Mendelian disease genes.3 Standardized terms were also published: pathogenic, likely pathogenic, uncertain significance, likely benign, and benign. These guidelines have now been widely adopted by laboratories both nationally and internationally.
While the development of detailed standards has brought greater consistency to variant classification, there remains both a degree of expert judgment and a learning curve in the accurate application of these guidelines.4 The authors of the ACMG–AMP guidelines recognized this and acknowledged that the 2015 guidelines were a first step towards the goal of more objectively defining pathogenicity. An assumption was that these rules would see both general improvements as well as specific details added for individual genes and diseases over time.3 In this month’s issue of Genetics in Medicine, Nykamp et al.5 report on a detailed evolution of the ACMG–AMP guidelines, transitioning to a semiquantitative framework with the use of a point-based scoring system and decision tree matrices. The ACMG–AMP committee had not decided to pursue a point-based system at the time over concerns that assigning points might lead to a false assumption of the quantitative accuracy of the approach, given substantial reliance on professional opinion to assign and weight certain types of evidence. However, it was recognized that a semiquantitative approach would improve the ability to combine both pathogenic and benign evidence, as well as to allow more gradation in the relative strength of each type of evidence. The careful approach taken by Nykamp et al.5 to systematically dissect the evidence types presented in the guideline, as well as account for many additional subtleties in the types of evidence encountered in variant classification, should be commended. I anticipate that ClinGen’s Sequence Variant Interpretation working group will carefully consider this approach as it works to provide additional guidance to the application of the ACMG–AMP guidelines and supports the ongoing evolution of this field.
As a testament to the utility of a standardized approach to variant classification, reported in this same issue of GIM, Harrison et al.6 were able to show the successful resolution of 87% of interpretation differences among four major clinical laboratory submitters to ClinVar, with the majority of variants resolved through application of the ACMG–AMP guideline for variant interpretation. A similar rate of success was previously published with an analysis between two laboratories.7 In an additional paper in this issue of GIM, Lebo et al.8 examine variant interpretation among laboratories in the Canadian Open Genetics Repository and report on a consensus-driven process to variant curation through data sharing and application of the ACMG–AMP guidelines. In this study, the utility of data sharing through the Canadian Open Genetics Repository was demonstrated using BRCA1 and BRCA2 variant classifications from 11 diagnostic laboratories across Canada as a proof of principle. Using a three-tier model, discordant rates were reduced from 27% to 14%, with only 0.9% medically significant discordance after consensus efforts.
Developing a community-centric approach to variant interpretation
In addition to the challenges in standardizing variant interpretation, another barrier arose over the past ten years. Laboratories were having difficulty keeping pace with the resources and expertise required to interpret the millions of variants being identified through high-throughput sequencing. It became clear that the most efficient mechanism for solving the problem was crowd sourcing.9 If each laboratory shared its work in relation to gathering evidence and classifying variants, every laboratory would benefit. Most importantly, very few data from clinical laboratory testing were routinely published, preventing access to a large and important body of data. After a few attempts to solve this problem, a group of laboratories turned to the National Center for Biotechnology Information (NCBI) to consider launching a database of shared variant interpretations, which is now called ClinVar.10 In parallel, these laboratories banded together through a National Institutes of Health U41 Genomic Resources grant (now part of ClinGen) to support data collection, standardization, and community sharing through a partnership with ClinVar. This effort was in part modeled on the successful International Standards for Cytogenomic Arrays Consortium, which brought cytogenetic laboratories together after the disruptive technology launch of cytogenomic microarrays into mainstream clinical genetics.11
Not surprisingly, when laboratories began sharing data in ClinVar, the differences in variant interpretation methods that had been used over the past ten years became clear, as evidenced by the discordant variant interpretation records in ClinVar.12 However, highlighting these differences through a “share and compare” approach allowed laboratories to resolve differences. In the study by Harrison et al.,6 the sharing of unpublished evidence helped resolve differences for 30% of variants. For some variants, the aggregate evidence from multiple laboratories moved variants from a classification of uncertain to either pathogenic or benign.
Is the glass half full or half empty?
In assessments of ClinVar, two conflicting arguments have been made. One argument discredits ClinVar, arguing that the discordance observed makes the database untrustworthy and a risk to the community.13, 14 The other view highlights ClinVar as a tremendous asset, allowing laboratories to share curated evidence used in interpretation, to identify discordant interpretations so they can be resolved, and to stimulate research on variants classified as being of uncertain significance.12, 15, 16, 17 While I am inherently biased in my views as a funded ClinGen investigator who has also worked closely with the NCBI to guide the development of ClinVar, I think the rapid growth and usage statistics for ClinVar, and the absence of any documented harm whatsoever, bring an objective view that the database has become an invaluable positive asset.
That said, we have more work to do. In this issue of GIM, Yang et al.17 report a detailed analysis of the data in ClinVar and note that ClinVar submissions from non-clinical laboratory sources (of which there are many) are far more likely to cause interpretation discrepancies than submissions from clinical laboratories, which are generally concordant. In addition, older submissions tend to disagree with newer ones, and low-penetrance variants (which usually have limited medical management implications) have more disagreement than high-penetrance ones. These factors appear to play a substantial role in the published comparisons that show high discordance among ClinVar entries.13, 14, 18 Comparisons that carefully take such obvious factors into account show substantially higher concordance.16, 17, 19, 20
The same factors could affect any ClinVar user. When ClinVar was launched, a star system was developed to distinguish the level of review that each variant had undergone, with the goal of aiding users in determining the trustworthiness of the variant interpretations in ClinVar.12 Unfortunately, while the distinction between 0/1/2 stars (single laboratory submitters with (1 star) or without (0 star) interpretation criteria provided and all interpretations concordant (2 star)) versus 3/4 stars (ClinGen-approved expert panels (3 star) and professional guidelines (4 star)) is apparent, the distinction between 0- and 1-star submitters has been less clear, with a wider range of quality present, as demonstrated by Yang et al.17 To ameliorate this problem, both Yang et al.17 and ClinGen investigators have proposed alternatives to the representation of data in ClinVar. However, while the staff at the NCBI conduct an enormous effort to ensure the accurate and standardized representation of each variant and the submitted interpretations, ClinVar is an archive and does not itself pass judgment on the accuracy of the interpretations. As such, ClinGen is now pursuing alternative representation of ClinVar data that will support a more nuanced assignment of single-star submitter status, as well as a more facile ability to flag and remove old, outlier, and incorrect interpretations.
A Call to action
As a clinical laboratory director of 15 years, I have seen a broad array of quality in molecular diagnostic laboratories. In my own anecdotal experience, I am now seeing a strong correlation between the quality of interpretations and whether the laboratory is a ClinVar submitter. I have also personally watched laboratories dramatically improve their interpretation methods after engaging in active data sharing in ClinVar. I therefore put out a plea to the community to step up and take action to ensure transparency in variant interpretation, to allow laboratories to subject their interpretations to peer review by the community, and to enable aggregation of evidence to improve knowledge over time. Various stakeholders can help as follows.
Regulatory agencies. Clinical laboratories are required to perform many tasks to ensure the quality of the tests they offer, although most focus on analytical validity, not clinical validity. While quality assurance activities require dedicated resources, there is little doubt in my mind that submission to ClinVar has a more positive impact on the accuracy of tests than checking the temperature on PCR machines and the countless other tasks laboratories perform daily as required by the accreditation bodies. It is time for the College of American Pathologists, which accredits most genetic testing laboratories, to make submission to ClinVar a requirement of quality assurance.
Payers. Payers of health-care services only reimburse tests that have demonstrated clinical utility. If there is insufficient quality assurance around the accurate interpretation of a test, it will not have utility. If you are a payer, it is time to make reimbursement contingent on laboratory data sharing and to strictly enforce the policy. This has already been initiated by at least one payer (https://www.genomeweb.com/informatics/genomic-variant-data-sharing-gains-support-collaboration-seen-key-interpretation).
Health-care providers. It is the duty of providers to act in the patient’s best interest, and this means ordering a test from a laboratory that shares data and helps our community improve knowledge for the benefit of patient care. If you are a hospital or clinician, order your tests from laboratories who share data. Currently, providers can check for a list of submitters on the ClinVar website: https://www.ncbi.nlm.nih.gov/clinvar/docs/submitter_list/. In addition, ClinGen maintains a webpage (www.clinicalgenome.org/lablist) of clinical laboratories meeting minimum requirements of data sharing, which takes into account the percentage of clinically reported variants being deposited and other factors.
Journals. Curating variant evidence from the literature is a time-consuming task and is highly error prone when researchers publish data that are not subject to the data quality checks performed by database curators, such as that done for ClinVar. If you are an editor of a journal or you review manuscripts for journals, please make sure submission of interpreted variants to ClinVar is a requirement of publication. Supplemental inclusion of metadata using standardized formats (genomic coordinates and Human Genome Variation Society nomenclature) should also be required.
Clinical laboratories and researchers. Enough said; it’s time to submit your data!
The end goal
As we continue to explore the best approaches to data sharing and representation of variant evidence, it is important to keep the end goal of variant interpretation in mind, which is to advance medicine and improve patient care. We must recognize that, similar to the rest of the practice of medicine, we will never be completely concordant in our interpretations of variants, which will inevitably continue to rely in part on professional opinion and evolving evidence. Nevertheless, we must ensure that the professional opinion of geneticists is based on a fully shared body of evidence with open and ongoing community review, and that we support this endeavor in a cost-effective manner. A colleague once told me that the airline industry does not compete on safety because safety in flight is a fundamental right, highly regulated by federal law. Instead, the airlines focus on the added services they can provide to differentiate their businesses. It is time we considered the sharing of variant interpretations a fundamental right of patients to ensure that they receive accurate clinical care. Laboratories have plenty of other means to differentiate their services.
Stenson PD, Mort M, Ball EV et al. The Human Gene Mutation Database: towards a comprehensive repository of inherited mutation data for medical research, genetic diagnosis and next-generation sequencing studies. Hum Genet 2017;136:665–677.
Amberger JS, Bocchini CA, Schiettecatte F, Scott AF, Hamosh A . OMIM.org: Online Mendelian Inheritance in Man (OMIM(R)), an online catalog of human genes and genetic disorders. Nucleic Acids Res 2015;43:D789–D798.
Richards S, Aziz N, Bale S et al. Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology. Genet Med 2015;17:405–424.
Amendola LM, Jarvik GP, Leo MC et al. Performance of ACMG–AMP variant-interpretation guidelines among nine laboratories in the Clinical Sequencing Exploratory Research consortium. Am J Hum Genet 2016;98:1067–1076.
Nykamp K, Anderson M, Powers M et al, Sherloc: a comprehensive refinement of the ACMG–AMP variant classification criteria. Genet Med 2017;19 (e-pub ahead of print 11 May 2017).
Harrison SM, Dolinsky JS, Knight Johnson AE et al, Clinical laboratories collaborate to resolve differences in variant interpretations submitted to ClinVar. Genet Med 2017;19 (e-pub ahead of print 16 March 2017).
Garber KB, Vincent LM, Alexander JJ, Bean LJ, Bale S, Hegde M . Reassessment of genomic sequence variation to harmonize interpretation for personalized medicine. Am J Hum Genet 2016;99:1140–1149.
Lebo MS, Zakoor K-R, Chun K et al, Data sharing as a national quality improvement program: reporting on BRCA1 and BRCA2 variant-interpretation comparisons through the Canadian Open Genetics Repository (COGR). Genet Med 2017;19 (in press).
Bale S, Devisscher M, Van Criekinge W et al. MutaDATABASE: a centralized and standardized DNA variation database. Nat Biotechnol 2011;29:117–118.
Landrum MJ, Lee JM, Benson M et al. ClinVar: public archive of interpretations of clinically relevant variants. Nucleic Acids Res 2016;44:D862–D868.
Riggs ER, Wain KE, Riethmaier D et al. Towards a universal clinical genomics database: the 2012 International Standards for Cytogenomic Arrays Consortium meeting. Hum Mutat 2013;34:915–919.
Rehm HL, Berg JS, Brooks LD et al. ClinGen—the clinical genome resource. N Engl J Med 2015;372:2235–2242.
Gradishar W, Johnson K, Brown K, Mundt E, Manley S . Clinical variant classification: a comparison of public databases and a commercial testing laboratory. Oncologist 2017; e-pub ahead of print 13 April 2017.
Vail PJ, Morris B, van Kan A et al. Comparison of locus-specific databases for BRCA1 and BRCA2 variants reveals disparity in variant classification within and among databases. J Community Genet 2015;6:351–359.
Harrison SM, Riggs ER, Maglott DR et al. Using ClinVar as a resource to support variant interpretation. Curr Protoc Hum Genet 2016;89:8.16.1–8.16.23.
Lincoln SE, Yang S, Cline MS et al, Consistency of BRCA1 and BRCA2 variant classifications among clinical diagnostic laboratories. JCO Precis Oncol 2017; e-pub ahead of print 11 April 2017.
Yang S, Lincoln SE, Kobayashi Y et al, Sources of discordance among germ-line variant classifications in ClinVar. Genet Med 2017;19 (e-pub ahead of print 1 June 20).
Balmana J, Digiovanni L, Gaddam P et al. Conflicting interpretation of genetic variants and cancer risk by commercial laboratories as assessed by the Prospective Registry of Multiplex Testing. J Clin Oncol 2016;34:4071–4078.
Nussbaum RL, Yang S, Lincoln SE . Clinical genetics testing laboratories have a remarkably low rate of clinically significant discordance when interpreting variants in hereditary cancer syndrome genes. J Clin Oncol 2017;35:1259–1261.
Yang S, Cline M, Zhang C, Paten B, Lincoln SE . Data sharing and reproducible clinical genetic testing: successes and challenges. Pac Symp Biocomput 2016;22:166–176.
This publication was supported by the National Human Genome Research Institute in conjunction with funding from the Eunice Kennedy Shriver National Institute of Child Health and Human Development under award U41HG006834. The content is solely the responsibility of the author and does not necessarily represent the official views of the National Institutes of Health. The author thanks all ClinVar submitters for sharing their data and the staff at the NCBI for tirelessly processing submissions to ClinVar.
The author is employed by institutions that offer fee-based clinical sequencing, and receives National Institutes of Health grant support for projects discussed in this article.
About this article
Current Genomics (2019)
A survey of undetected, clinically relevant chromosome abnormalities when replacing postnatal karyotyping by Whole Genome Sequencing
European Journal of Medical Genetics (2019)
Systematic evaluation of gene variants linked to hearing loss based on allele frequency threshold and filtering allele frequency
Scientific Reports (2019)
Quantitative approaches to variant classification increase the yield and precision of genetic testing in Mendelian diseases: the case of hypertrophic cardiomyopathy
Genome Medicine (2019)
Genetics in Medicine (2019)