Nature Biotechnology
22, 1230 - 1235 (2004)
doi:10.1038/nbt1004-1230
Towards quality assurance and harmonization of genetic testing services in the European UnionDolores Ibarreta1, Robert Elles2, Jean-Jacques Cassiman3, Emilio Rodriguez-Cerezo1
& Elisabeth Dequeker31 European Commission, Joint Research Center (JRC), Institute for Prospective Technological Studies (IPTS), Edificio Expo, E-41092 Seville, Spain. 2 National Genetics Reference Laboratory (Manchester), Department of Medical Genetics, St. Mary's Hospital, Hathersage Road, Manchester UK, M13 0JH. 3 Center for Human Genetics, Campus Gasthuisberg, B-3000 Leuven, Belgium.
Correspondence should be addressed to Elisabeth Dequeker Els.Dequeker@med.kuleuven.ac.beAlthough quality assurance schemes have been proven effective in preventing genetic testing errors, a recent survey of services in the EU reveals that laboratory participation in these schemes is fragmented and incomplete.Testing for hereditary diseases has developed over the years into an important aspect of genetic service provision. In a rush to bring these powerful new technologies into medical use, issues of quality have not always been given sufficient attention. There are documented indications that genetic tests as they are performed now do not ensure the quality desired. In Europe, an external quality assessment (EQA) scheme organized by the European Molecular Genetics Quality Network (EMQN) for the diagnosis of Huntington disease showed a misdiagnosis rate of 1.3%1 and in the case of cystic fibrosis testing, the survey found that more than half of the participating labs made mistakes2. According to surveys carried out in 1997 and 1999, the practice standards of a number of laboratories in the United States are in need of improvement as well3,
4.
In response to these concerns, we recently conducted a study, coordinated by the European Commission's (EC, Brussels) Joint Research Center, on the current scope and quality of genetic testing services in Europe5. In general, our research showed high scientific achievement and examples of good practice in Europe in genetic testing. However, the variability of practices both within and between European countries raises concerns. Although genetic specialists and professional organizations have initiated moves to promote quality assessment, genetic testing services are provided under widely varying conditions and regulatory frameworks in different EU countries6,
7,
8. We propose several options to ensure the highest quality of such services, including harmonized quality control, better cross-border cooperation and the establishment of a quality controlled European database of genetic testing services.
Genetic testing services in Europe
As a first step, we assessed through a survey the current dimension—the number of laboratories, number of tests, among other topics—of genetic testing services in Europe. Twenty-one national partners of the EMQN were asked to complete a questionnaire. The survey was not intended to be a comprehensive collection of data but rather a snapshot of the current state of genetic testing in Europe.
Number of active laboratories. The survey results estimate the numbers of laboratories performing genetic tests in Europe to be 751 (Table 1). A further 936 clinical chemistry/ hematology centers were estimated to offer genetic tests, although this data was incomplete; the activity from this sector was more difficult to document and hence, these centers were not considered further.
We compared the results generated in this study to other sources. The European Directory of DNA Diagnostic Laboratories (EDDNAL, http://www.eddnal.com/), an online database covering 17 European countries, which is partially funded by the EC, lists, in 2003, 313 laboratories and 580 genetic conditions for which tests are available. EDDNAL was conceived as an online database of European laboratories, but it has proven difficult to keep it updated. Inclusion in the database is voluntary and there is no review of the information provided.
A comprehensive survey carried out recently in Spain identified 53 centers performing genetic testing in this country9, a figure close to the 47 estimated in our survey.
"Orphanet," a database of services dedicated to rare diseases, including all genetic disorders, with funding from the EC, lists DNA diagnostic laboratories in seven European countries (http://orphanet.infobiogen.fr/). The data are considered comprehensive for two of them: France and Italy, where they reported 195 and 119 labs, respectively, figures close to our findings. Discrepancies in the actual numbers, as depicted in Table 1, reflect the difficulty of identifying laboratories offering genetic tests in Europe. Currently no official registry or reliable and up-to-date database exists for Europe.
An American initiative 'Genetests' (http://www.genetests.org), funded by the National Institutes of Health (Bethesda, MD, USA), listed, in August 2003, 578 laboratories and 1,091 clinics (US and international). Listing in the database is voluntary, so these figures might not represent a complete picture.
Profile of laboratories. Currently, a minority of labs exist outside the public sector (Fig. 1). The majority of genetic tests are carried out routinely in laboratories often closely associated with clinical genetic services and a university or other research facility. This reflects the common transition from research to service.
 | | |
In general, activity in genetic testing in the private sector is likely to be small scale and many small commercial laboratories are not documented or are associated with a larger enterprise. Although there were an estimated 40 private testing laboratories in Germany, a limited number of larger commercial organizations are active in the EU. For example, CERBA, a French laboratory (Cergy Pontoise Cedex), offers an extensive menu of genetic tests (http://www.pasteur-cerba.com/), and Orchid Biosciences (Abingdon, Oxfordshire, UK) completes about 6,000 genetic test reports per year for cystic fibrosis and other indications on a contract basis for customers outside Europe (http://www.orchid.co.uk/). Data on the activity of small private laboratories is difficult to obtain. However in Spain, the private laboratory sector demonstrates a modest level of activity (below 20%) compared to the public sector9.
A recognized but less well ascertained group of genetic testing service providers are the clinical chemistry, immunology, hematology, histopathology, oncology and microbiology laboratories that do not specialize in human genetics. Our estimate from the survey is that there are at least as many laboratories of this type active across Europe as there are specialist clinical molecular genetic laboratories. However, most of their activity is likely concentrated on diagnostic testing for a limited number of diseases, serving local populations and using mainly simple tests such as factor V Leiden, hereditary hemochromatosis and prothrombin variants.
Number of genetic tests carried out in the EU. Data are scarce on the number of tests carried out in Europe. The survey conducted in Spain represents one of the very few attempts to determine this number. That study revealed that over 45,000 tests were carried out in Spain in 2001 (ref. 9). For the United Kingdom, Patton et al.10, quoting the audit of the UK Clinical Molecular Genetics Society, estimated that over 50,000 tests are carried out per year. In Finland, in 1997, about 7,000 tests were carried out in university hospitals and approximately 2,500 by private laboratories11. In the Netherlands, in 2000, over 20,000 genetic tests were carried out for roughly 250 genetic disorders.
In the European public sector, our survey of the EMQN national contact partners estimated the activity in 2002 at 381,000 genetic test reports from 352 specialist clinical molecular genetic laboratories, averaging 1,100 reports per center. A further 316 laboratories in clinical genetics or research centers outside the countries surveyed were reported to be active. Using an average figure of 1,100 reports/laboratory, we estimate the total EU activity in 2002 is 735,000 reports.
In the United States, in 1996, more than 175,000 tests were performed, the number corresponding to an increase of 30% compared to 1995 (ref. 3), which is in line with the increasing trend found in Europe (Fig. 2).
The most frequently performed tests correlate with those diseases or traits with a high incidence in the population (hemochromatosis, coagulation disorders, fragile X, cystic fibrosis). Not surprisingly, testing for these common conditions is done in numerous centers and regions, because the mutations responsible for the diseases are well known and most of the techniques are reliable and relatively easy to learn. However, there seems to be a significant unmet need for genetic testing in single gene disorders of lower incidence.
Progress in molecular genetics has been tremendous, especially in the last few years, its results finding its way into medical applications as diagnostics and therapeutics. This can be shown, for example, by the data bank of Online Mendelian Inheritance in Man (OMIM) (http://www3.ncbi.nlm.nih.gov/Omim/), a catalog of human genes and genetic disorders that lists a constantly increasing number of clinical disorders for which mutations are known (currently over 1,600). However, in practice a smaller range of tests are available and a limited number form the most popular tests among referrers and providers. For example, as shown by the survey conducted in Spain as part of this study, after identifying all centers for genetic testing in the country, it was found that only 214 different tests were available in 2001 (ref. 9). In the United Kingdom, as another example, only 273 diagnostic tests are listed by the UK Genetic Testing Network (http://www.doh.gov.uk/genetics/UKGTN). This lack of supply affects primarily rare diseases; as the incidence goes down, so does the percentage of diagnostic services available. Given that research into genetic mutation is still at a stage where it is relatively laborious and requires expert interpretation, only a few laboratories are in a position to develop and supply an appropriate test for less common diseases.
Assessing quality
Several surveys have shown that the quality of genetic test performance may be insufficient in some centers. An EQA scheme for cystic fibrosis organized by the European Concerted Action on Cystic Fibrosis (Leuven, Belgium) showed that 20% or more of the participating laboratories made at least one or more technical or administrative errors on a total of six different DNA tests for common cystic fibrosis mutations2.
In the United States, a survey carried out in 1997 found that 15% of laboratories offering genetic tests might have suboptimal laboratory practices, especially regarding personnel qualifications and laboratory practice standards, scoring below 70% in overall quality as defined by the American College of Medical Genetics. Ten percent of the labs taking part in the survey and offering genetic tests had not been certified according to the Clinical Laboratory Improvement Amendments of 1988 (CLIA), even though such certification is mandatory3.
There is also evidence that erroneous interpretation of results occurs12. As an example of the possible consequences of such errors, unnecessary amniocenteses with their consequent risks have been done because of confusion over the interpretation of test results of a cystic fibrosis screening program in the United States13.
Existing EQA schemes. During recent years, there has been a growing interest in quality assurance of genetic testing. As a consequence, several organizations offer EQA schemes, of different scales and for different genetic disorders. Schemes are funded either by international groups (the European Commission, European Molecular Genetics Quality Network or the Cystic Fibrosis Thematic Network) or by national governments (Italian scheme) or by private subscription (UK National External Quality Assessment Schemes, German scheme)). In the United States, the College of American Pathologists (Northfield, IL) and Centers for Disease Control and Prevention (CDC; Atlanta, Georgia) are the most recognized. Table 2 summarizes for some genetic disorders the number of organized EQA schemes, stratified by organization level. The range of the number of participants is also indicated. In Europe, participation in EQA is mostly voluntary.
| | |
In Europe, the Clinical Molecular Genetics Society of the UK (http://www.cmgs.org/) was the first group to organize national quality assessment pilot schemes for cystic fibrosis (CF) and Duchenne muscular dystrophy, in 1991 and 1993, respectively12. The first international EQA scheme for genetic diseases in Europe was run in 1994 for CF14. Since 1996, the scheme, with the support of EC funding, has been made available to all laboratories that perform genetic diagnostic testing for CF2,
15. In 1998, an EC-funded project entitled the "European Molecular Genetics Quality Network" (EMQN)16 initiated the organization of pilot schemes for a further group of genetic conditions including familial breast/ovarian cancer17, Huntington disease1,
18, Duchenne and Becker Muscular Dystrophy19, Friedreich Ataxia20 and Charcot-Marie-Tooth disease21.
Clearly, multiple initiatives conducted at different levels are being undertaken for each disorder. However, all schemes are effective. A comparison of the different EQA schemes for CF (UK, European and US) has shown that although the design, analysis and data reporting of each scheme differ, in each case, the genotyping error rate declined with consistent participation. For the EU schemes between 1996 and 2000, errors have steadily decreased from 63 (3.8%) to 29 (1.3%). The proportion of labs making genotype errors decreased from 35% to 10% over the same period12.
Quality assurance of genetic testing services in EU. To evaluate the status of quality assurance (QA) systems in genetic diagnostic laboratories, a survey was carried out within the European EQA scheme for cystic fibrosis in 2001. The survey was sent to 206 CF EQA scheme participants with a 73% response rate. The group surveyed comprised laboratories that are actively involved in diagnostic testing for CF but the majority of these laboratories also do tests for other genetic diseases. As this survey was taken among the laboratories regularly participating in the CF EQA, and thus on a selected population, the results may overestimate the level of quality assurance.
The survey showed that most genetic laboratories participated in EQA schemes for only a few diseases (Fig. 3). Ninety-one labs (60%) participated in EQA schemes for two or more diseases. Thirty-seven laboratories (24%) participated in five or more EQA schemes in the last five years. An inverse correlation was observed between the number of diseases being diagnosed in a given laboratory and its participation rate in EQA schemes of different diseases. For example, 30.3% of the labs that carry out testing for ten or more diseases (Fig. 3, violet bars) participate in EQA for only one disease. Clearly, many laboratories do not participate in all of the available EQA schemes.
| | |
About 73% of the surveyed laboratories (110/151) claimed to have a QA system already in place, or to be in progress of implementing one. Among these centers, 45 followed an international standard, 52 followed a local or national standard and 13 laboratories did not specify the guidelines/standard followed (Fig. 4a). Only 54% of the laboratories were inspected by official bodies: 34% were inspected by a national body and 20% were certified or accredited by an internationally recognized certification or accreditation body (Fig. 4b).
| | |
This survey illustrates that participation in EQA schemes and in certification or accreditation by molecular genetics laboratories in parts of Europe is fragmented and insufficient. The percentage of labs that never undergo an official inspection is high (46%) as is the number of labs offering tests for diseases for which they do not participate in EQA schemes (Fig. 4). Although not all diseases have an available EQA scheme in place, they do exist for some of the most common tests. As technology progresses and the complexity and number of tests increase, this weakness will become more significant.
Uncertain genetic counseling. The quality of genetic testing cannot be considered isolated from either the type and quality of laboratory reporting, or the quality of the counseling that patients receive before and after testing. Two-thirds of the genetic testing centers surveyed for our study reported direct links to clinical genetics services that include counseling.
Our survey among EMQN labs indicated that a request for a genetic test from a family doctor is still relatively rare in most countries. In some countries (for instance the Netherlands) only genetic specialists can request a genetic test8. However the doctor requesting the test is most often not a genetic specialist. In some countries, a test result that confirms the genetic condition is not always followed by the offer of genetic counseling to the patient and relatives9. This might also be a reflection of the fact that the number of clinical geneticists is not adequate in the EU22.
A global solution
The EQA approach has already proven successful in Europe, as EQA schemes already in place have resulted in a gradual reduction of the percentage of laboratories making errors2. Nevertheless, the error rate for some disease services still remains unacceptably high and insufficient availability of EQA schemes and low levels of participation suggest that continued efforts are needed to further improve the quality of genetic services provided to the community. Existing initiatives are fragmented and their continuity is not assured. As a possible solution, we propose that an effort should be made to bring together all these initiatives under a European 'umbrella' to harmonize EQA standards and practices. This could also ensure that the EQA schemes support changing technical needs and new applications of genetic testing.
EQA schemes can evaluate technical accuracy, and the ability of the laboratory to correctly interpret the generated data and produce clear and informative reports. Reporting in native languages is important, a fact that argues strongly for the continuation and further development of existing national EQA schemes. As an additional advantage, the EQA schemes organized at a national or regional level allow analytical assessment using mutations of particular relevance for that region. Harmonization of the national or regional schemes could help ensure that the national/regional EQA schemes can be compared with international schemes.
On the other hand, the low cost-efficiency of small EQA schemes argues against the establishment of new national schemes in each EU country. To date most EQA schemes have been supported by EC research grants or governmental initiatives, complemented with the unpaid time of the scheme organizers and quality assessor volunteers. This has allowed EQA schemes to be free of charge to participants or available for a modest fee not reflecting the actual costs. The real costs of organizing EQA schemes have been calculated in a study of the European CF network and the European Concerted Action for CF23. According to this study, fixed personnel costs required for the EQA organization are high compared to the additional personnel and the variable material cost generated per additional laboratory (less than 1% and 0.5% of the fixed cost, respectively)23. Given the relatively high fixed costs for the organization of an EQA, a large number of participants (100−200) is necessary for a cost-efficient organization of EQAs.
A cooperation between the European/ international EQA schemes and the national organizers could combine the advantages of both systems. A similar model has been recommended by the US CDC as a result of a panel on quality assurance in human molecular genetic testing24.
Conclusions
Genetic testing services in the EU have substantially increased their activity in the past few years. Our study has estimated that more than 700,000 tests were done in 2002, with a greater than 100% increase per year in some member states (Fig. 2). Testing for genetic diseases has rapidly moved from the laboratory into medical practice and, in this process, issues of quality require adequate attention. There are documented indications that errors and deficiencies in the overall process occur in the EU and the United States. There is also evidence that interpretation of results is not always adequate. The EU population has a very positive view of the potential of medical genetics25. To help maintain this, action is required to ensure the highest quality of tests and testing centers, and to make the best use of existing EU expertise to widen coverage at the lowest cost possible.
There is ample evidence in our study for the potential of EU-wide coordination of efforts to ensure that genetic tests and the counseling that accompanies them are accurate, of high quality and standardized across the EU. Although in almost every EU country there are examples of good practices, these objectives are not being consistently achieved. The efforts initiated so far have shown excellent results. However the challenge is to extend these efforts and ensure their long-term continuity and to reach out beyond the 'core' genetics community to related disciplines and laboratories, which are not involved in the existing networks. The organizations providing EQA schemes for genetic tests are fragmented both in terms of geographical area and their time scope (mostly depending on external funding for a limited time).
An internal quality management system would improve the quality of testing. Implementing quality management systems is a multiphase process and requires a lot of energy and time from laboratory personnel. A detailed study estimating costs for implementing and maintaining a quality management system (including participating in an EQA scheme) has not been completed but experience indicates that a medium-size laboratory requires the salary of a quality manager and support personnel and a support budget of approximately  25,000 ($31,000)/year. Additionally, our study made evident the need for a reliable and updated database of laboratories performing genetic testing in the EU, listing the conditions for which tests are offered in each laboratory. This database should also provide information about the participation of the listed labs in EQA and other quality assurance items like accreditation or certification of the lab.
In summary, given the variability and shortcomings in quality assurance systems in genetic testing identified and the difficulties added by fragmentation, a harmonized approach at European, or higher international level is indicated. Networking and integration at different levels in the process of genetic testing is required to fill the gaps identified. This idea of a European 'network of networks' could be developed by one of the tools offered by the European Commission's 6th Framework Research Program: the Network of Excellence (NoE). This specific instrument, the NoE, is designed to tackle the fragmentation of European research, helping to construct a durable structure for the way that research is carried out and it could be used on the quality of genetic tests in this case. This would be an important first attempt towards sharing information on quality assurance in Europe and would result in creating a 'virtual' body or infrastructure for harmonization of practices. This approach will require a significant coordination effort and long-term funding. It is likely that in the future EU member states would consider discussing a common regulatory framework for the provision of quality genetic testing services. On the other hand, the low cost-efficiency of small EQA schemes argues against the establishment of new national schemes in each EU country. If so, the initiative to network research on the quality of genetic tests would be of even more importance.
Disclaimer: The views expressed in this study do not necessarily reflect those of the European Commission (EC).
REFERENCES
- Losekoot, M., Bakker, B., Laccone, F., Stenhouse, S. & Elles, R. A European pilot quality assessment scheme for molecular diagnosis of Huntington's disease. Eur. J. Hum. Genet. 7, 217−222 (1999). | PubMed | ISI | ChemPort |
- Dequeker, E. & Cassiman, J.J. Genetic testing and quality control in diagnostic laboratories. Nat. Genet. 25, 259−260 (2000). | Article | PubMed | ISI | ChemPort |
- McGovern, M.M., Benach, M.O., Wallenstein, S., Desnick, R.J. & Keenlyside, R. Quality assurance in molecular genetic testing laboratories. J. Am. Med. Assoc. 281, 835−840 (1999). | Article | ISI | ChemPort |
- McGovern, M.M., Benach, M., Wallenstein, S., Boone, J. & Lubin, I.M. Personnel standards and quality assurance practices of biochemical genetic testing laboratories in the United States. Arch. Pathol. Lab. Med. 127, 71−76 (2003). | PubMed | ISI |
- Ibarreta, D., Bock, A.K., Klein, C. & Rodriguez-Cerezo, E. Towards quality assurance and harmonization of genetic testing services in the EU. ESTO Report EUR 20977, Joint Research Centre, European Commission (2003). http://www.jrc.es/home/publications/publication.cfm?pub=1124/
- Harris, R. & Reid, M. Medical Genetic Services in 31 countries: an overview. Eur. J. Hum. Genet. 5, 3−21 (1997). | PubMed | ISI |
- Donnai, D. & Elles, R. Integrated regional genetic services:current and future provision. Br. Med. J. 322, 1048−1051 (2001). | Article | ISI | ChemPort |
- Health Council of the Netherlands. DNA Diagnostics publication num. 1998/11E Bezoekadres Parnassusplein 5, 2511 VX. Den Haag, Netherlands. (1998).
- Rueda, J. & Briones, E. Genetic Testing Services for Hereditary Diseases in Spain: Results from a Survey (eds. Ibarreta, D., Bock, A.K. and Rodriguez-Cerezo, E.) (Joint Research Center, European Commission, Seville, Spain, 2002).
- Patton, S., Barton, D. & Elles, R. Retaining the confidence of the public in molecular genetic testing—quality assurance. Community Genet. 3, 164−169 (2000). | Article |
- Genetic Testing in Europe. Harmonisation of Standards and Regulations. (Forschungsberichte 01/99, Vienna, 1999).
- Dequeker, E., Ramsden, S., Grody, W.W., Stenzel, T.T. & Barton, D.E. Quality control in molecular genetic testing. Nat. Rev. Genet. 2, 717−723 (2001). | Article | PubMed | ISI | ChemPort |
- Vastag, B. Cystic fibrosis gene testing a challenge: experts say widespread use is creating unnecessary risks. J. Am. Med. Assoc. 289, 2923−2924 (2003). | Article | ISI |
- Cuppens, H. & Cassiman, J.J. A quality control study of CFTR mutation screening in 40 different European laboratories. The European Concerted Action on Cystic Fibrosis. Eur. J. Hum. Genet. 3, 235−245 (1995). | PubMed | ISI | ChemPort |
- Dequeker, E. & Cassiman, J.J. Evaluation of CFTR gene mutation testing methods in 136 diagnostic laboratories: report of a large European external quality assessment. Eur. J. Hum. Genet. 6, 165−175 (1998). | Article | PubMed | ISI | ChemPort |
- Patton, S. et al. The European Molecular Genetics Quality Network (EMQN) in 10th International Congress on Human Genetics, Vienna, Austria. Eur. J. Hum. Genet. 9 (S1), 311 (2001). | Article | PubMed | ChemPort |
- Mueller, C., Stoppa-Lyonnet, D. & Kristoffersson, U. An equality assessment scheme for genetic testing of familial breast/ovarian cancer in 10th International Congress on Human Genetics, Vienna, Austria. Eur. J. Hum. Genet. 9 (S1), 311 (2001). | Article | PubMed | ChemPort |
- Losekoot, M. et al. An External Quality Assessment scheme for genetic testing of Huntington Disease in 10th International Congress on Human Genetics, Vienna, Austria. Eur. J. Hum. Genet. 9 (S1), 311−312 (2001). | Article | PubMed | ChemPort |
- Stenhouse, S. & Ramsden, S.G.M. & Jean-Pierre, M. An External Quality Assessment scheme for genetic testing of Duchenne and Becker Muscular Dystrophy in 10th International Congress on Human Genetics, Vienna, Austria. Eur. J. Hum. Genet. 9 (S1), 312 (2001).
- Barton, D., Sistermans, E. & Claustres, M. An External Quality Assessment Scheme for genetic testing of Friedreich Ataxia in 10th International Congress on Human Genetics, Vienna, Austria. Eur. J. Hum. Genet. 9 (S1), 312 (2001).
- Rautenstrauss, B., Lupski, J. & Timmerman, V. An External Quality Assessment scheme for genetic testing of Charcot-Marie-Tooth disease in 10th International Congress on Human Genetics, Vienna, Austria. Eur. J. Hum. Genet. 9 (S1), 312 (2001).
- European Society of Human Genetics. Provision of genetic practices in Europe: current practices and issues. Eur. J. Hum. Genet. 11 (S2), S2−S4 (2001).
- Dequeker, E. & Cassiman, J. Cost-efficient organisation of external quality assessments in molecular diagnostic laboratories in 10th International Congress on Human Genetics, Vienna, Austria. Eur. J. Hum. Genet. 9 (S1), 302 (2001).
- Williams, L.O. et al. Quality assurance in human molecular genetics testing: status and recommendations. Arch. Pathol. Lab. Med. 127, 1353−1358 (2003). | PubMed | ISI |
- Gaskell, G., Allum, N. & Stares, S. Europeans and Biotechnology in 2002. Eurobarometer 58.0 (European Commission, Brussels, 2003). http://europa.eu.int/comm/public opinion/archives/eb/ebs 177 en.pdf
Acknowledgments
The study was done within the framework of the European Science and Technology Observatory Network. The European Molecular Genetics Quality Network and the Cystic Fibrosis Thematic Network were main cornerstones in producing the surveys.
|
