Introduction
To the Editor:
We retrospectively reviewed published variants in the PKD1 genes and detected errors in 39 of 771 variants (5.06% (95% c.i., 3.62–6.85)). All arose from human processing mistakes. As peer-reviewed publication is no safeguard for those considering the clinical significance of an unknown variant, we suggest that reporting of new variants for the proposed Human Variome Project should employ both automated reporting and expert scrutiny.
Measurement of the biological activity of a mutant gene provides the best indication of the functional effect of a gene variant1. As this is seldom possible, less direct measures are used to assess the likely clinical significance of a variant. These include observing the degree of nucleotide sequence conservation in orthologous genes, the nature and position of resultant amino acid changes, its frequency within cases and control populations and whether it exhibits familial segregation with the disease. With the exception of the latter, this information is often present within published mutation reports, providing grounds for the clinical interpretation of a gene variant.
Not surprisingly, guidance from professional organizations representing medical geneticists emphasizes the important role of locus-specific mutation databases (LSDBs)2 (http://www.hgvs.org/dblist/dblist.html) and peer-reviewed publications in the evaluation of unknown variants3 (see http://www.acmg.net/Pages/ACMG_Activities/stds-2002/stdsmenu-n.htm and http://www.cmgs.org/BPGs/Sequencing_new.htm). At this early stage in the development of LSDBs, professional bodies also caution against overreliance on databases to interpret the meaning of an observed variant3. Similar cautionary warnings are not issued about published mutation reports, presumably because the rigor of the peer review process instills a stronger sense of confidence in published clinical mutation reports. Studies of quality control in genetic diagnostic laboratories have so far focused on the quality of DNA sequencing4 and genotypic allele calling5. To our knowledge, there has been no systematic study of the accuracy of variants in peer-reviewed publications.
A recent upgrading of the autosomal dominant polycystic kidney disease (ADPKD) mutation database (PKDB; see http://pkdb.mayo.edu)6 presented an opportunity to evaluate the accuracy of published reports of variants in the ADPKD-associated gene PKD1 (16p13.3). A large number of disease-causing mutations, a large number of polymorphisms not associated with disease and a considerable number of gene variants with an unknown effect are now reported for this gene. This allowed us to assess the accuracy with which 771 variants were reported in 55 peer-reviewed publications (Supplementary Table 1 online). In order to evaluate the accuracy of these variant reports, the numbering of each was first standardized to the ATG start codon of the PKD1 NCBI RefSeq mRNA sequence (NM_000296.2). The nomenclature was also altered where necessary, to comply with current nomenclature standards7 (http://www.hgvs.org/mutnomen/). The accuracy of each reported gene variant and its associated amino acid effect were assessed via an in-house mutation checker tool6 in conjunction with the Artemis DNA sequence visualization software8. Inconsistencies between the reported nucleotide changes and associated reference sequence or reported amino acid change were subjected to closer investigation until resolved. Corresponding authors were invited to recheck their own publication as well as our suggested amendments. Twenty-two of thirty-four corresponding authors (65%) responded. Excluding the time contributed by corresponding authors, approximately 170 h of curator time was required to standardize and check all the gene variant reports.
As a number of reference sequences and numbering conventions had been used to describe the gene variant reports, the nucleotide numbering of 542 (70.3%) variants required updating. Twenty publications (36%) did not provide details of the reference sequence used or indicate the numbering convention employed. This required the empiric determination of these details. A total of 39 errors (Supplementary Table 2 online) were identified (5.06% (95% c.i. 3.62–6.85)), 36 of which were detected using the procedure describe above. The remaining three errors were reported by a corresponding author after re-examination of the primary data. The errors, which have been categorized as either miscounting, misassignment or typographical (Fig. 1), were identified in both the nucleotide and amino acid descriptions of variants. While it was evident the majority of errors identified in this study arose from human copying, the discovery of errors in reporting primary sequence data indicates that this vital first step is also prone to error. As many primary data were not reviewed in this study, it is highly likely the error rate reported here is an underestimate. The 10% administrative error rate detected among diagnostic laboratories reported in ref. 5 and the 13% rate of genotyping and nomenclature errors among 64 diagnostic laboratories participating in an international external quality assessment of sequence-based genetic testing4 could more closely approximate the error rate in peer-reviewed publications.
Figure 1: Errors were grouped into three categories: misassignment, miscounting and typographical.
The number of instances of each category at the nucleotide (light gray) and amino acid (dark gray) levels is shown.
Full size image (24 KB)Our experience of checking published variants, correcting their nomenclature and adjusting their correspondence to a common reference sequence provides some insight into the likely cost of completing the task for other LSDBs. At present, there is little scope for automating the initial steps required to standardize numbering and nomenclature. However, the later steps of checking for inconsistencies in the description of variants are being aided increasingly by mutation-checking and visualization software6, 8 (http://www.ebi.ac.uk/cgi-bin/mutations/check.cgi).
Much is known about the processes that contribute to human errors, as well as the proofreading and feedback processes required to identify mistakes9. Although the most effective way of reducing human error is to introduce automated processes that reduce or eliminate the need for human characterization and transcription of gene variants, software introduces new sources of error: for example, gene name errors introduced through the use of Microsoft Excel10.
In the interim, journal editors and LSDB curators should consider a range of strategies, including encouraging authors of genomic variant reports to recheck gene variant data before releasing it into the public domain. Sequence data may now be checked prospectively with mutation checking software.
We must point out that the scope of our analysis did not extend to scrutinizing the accuracy of clinical interpretation of the reported variants, which adds another dimension of risk to the clinical use of published gene variants. The consequences of being unable to access a report of a rare disease-causing variant may not, in practice, be a major issue because its rarity alone should prompt suspicions about its potential pathogenicity. However, failure to access a report of a rare polymorphic variant, perhaps found only in a specific population, would seem likely to increase the risk of wrongly concluding that the variant is associated with disease. This, in turn, could prompt inappropriate clinical decisions. By contrast, incorrect assignment of a novel pathogenic variant as non–disease associated may prompt unnecessary expenditure on additional molecular screening when the primary disease-causing lesion has already been characterized. Finally, erroneous reporting of a rare non–disease associated polymorphism as disease-causing could result in inappropriate clinical decisions because of the mistake. Potential ramifications of these scenarios are poignantly illustrated in a hypothetical scenario presented in ref. 11.
The error rate detected among these gene variants published in peer-reviewed journals shows that caution must be exercised—particularly by curators of LSDBs, genetic diagnostic laboratories, genetic counselors and other health care practitioners—when relying on published reports to evaluate the likely clinical significance of an unknown variant. The nature of the mistakes demonstrates that steps toward reducing human single-entry recording of sequence variants into peer-reviewed publications or LSDBs will enhance the accuracy and clinical utility of the Human Variome Project.
Note: Supplementary information is available on the Nature Genetics website.
Members of the ADPKD Gene Variant Consortium include the following: Peter C Harris6, Sandro Rossetti6, Dorien Peters7, Martijn Breuning7, Elizabeth Petri Henske8, Akio Koizumi9, Sumiko Inoue9, Yoshiko Shimizu10, Wanna Thongnoppakhun11, Pa-thai Yenchitsomanus11, Constantinos Deltas12, Richard Sandford13, Roser Torra14, Alberto E Turco15, Steve Jeffery16, Michel Fontes17, Stefan Somlo18, Laszlo M Furu18, Yvo M Smulders19, Bernard Mercier20, Claude Ferec20, Stéphane Burtey17, York Pei21, Luba Kalaydjieva22, Nadja Bogdanova23, Marie McCluskey24, Lee Jung Geon25, C H Wouters26, Jana Reiterova27, Jitka Stekrová27, Jose L San Millan28, Gianluca Aguiari29 & Laura Del Senno29
