Introduction

Internal and external quality control are essential for evaluating the reliability and accuracy of a diagnostic cytogenetics laboratory. In addition, a satisfactory performance in EQA gives assurance both to patients and referring clinicians that the diagnostic laboratory results are reliable and accurate. EQA is recognized by international standards and accreditation bodies as a tangible measure of the quality of a laboratory's performance.1 Accredited laboratories are required to participate in a recognized EQA scheme for all aspects of the diagnostic service, if available.1, 2

EQA in cytogenetics is generally undertaken in one of two ways, retrospective or prospective. With retrospective assessment the EQA scheme requests material and documentation from specific types of diagnostic case analysed by the laboratory. This form of assessment is valuable insofar as it examines the real practices of the laboratory, but has the disadvantage that inter-laboratory comparisons may not be possible because of differences in the complexity of cases submitted. Prospective assessment provides distribution of a consistent batch of material so that inter-laboratory comparison can be made dependably; the disadvantage of this approach is that EQA cases may be given special treatment or priority and not reflect the real practices of the laboratory. As it is difficult to source sufficient appropriate tissue, the material distributed more often comprises slide preparations, images, or case scenarios.

The first cytogenetics EQA scheme in Europe was the UK pilot in 1981, which has developed into the present comprehensive scheme. Although it is a national EQA scheme, it is open to other countries, having participants from 21 nations. The pilot initially involved a retrospective audit of report times and success rates, before expanding to include a retrospective assessment of cytogenetic cases. More recently the UK scheme has introduced prospective assessment with the distribution of case scenarios and diagnostic slides for analysis and interpretation, and currently includes both retrospective and prospective components. Assessment includes scrutiny of technical preparation quality (retrospective only) and appropriateness of tests undertaken, interpretation of abnormalities, significance of the abnormality to the patient and family as revealed by the composition of the report, and use of standard nomenclature.

Europe now has seven nationally based EQA schemes and there is also one multinational European EQA scheme (Cytogenetics European Quality Assessment scheme, CEQA). Some national schemes have a separate scheme organizer and management structure for constitutional and haematological (or oncology) cytogenetics EQA schemes. The German constitutional scheme started in 1989, basing its structure and EQA format on the UK scheme. This constitutional scheme later expanded to include fluorescence in situ hybridization (FISH) EQA. Finland and Spain started EQA schemes for constitutional cytogenetics in 1994 and 1998. More recent schemes include the Italian scheme, the haematological Spanish scheme and the French and German haematological schemes.3 Czech Republic and France started a haematological and constitutional EQA scheme, respectively in 2006. Finally a European EQA, CEQA was also piloted in 2006 based on the UK NEQAS web-based scheme, which has set out to provide EQA, through multiple language submissions and recruitment of assessors from many European countries.

Some national schemes use prospective EQA, either by sending images, or slides. Only the French and German haematological schemes continue to distribute cell suspensions. For larger schemes, access to the volumes of blood or cell suspensions necessary is not practical due to the limited source of appropriate material. Thus prospective EQA is difficult to organize and consequently, most European EQA schemes undertake retrospective assessment of images or slides from cases reported by the participating laboratories.

It is usual for an EQA scheme to have an organizer and a steering committee consisting of senior members of the profession, and to make use of a panel of expert assessors chosen from the profession by the steering committee for their seniority, experience, and skills. The steering committee decides the scope and operation of the EQA scheme.

This novel web-based EQA was designed to mimic the stepwise process undertaken by a laboratory analysing a real diagnostic case. This document describes the Internet-based EQA system, the different approaches to the online EQA taken by laboratories and a summary of the recent EQA results. While this approach to EQA is feasible, the initial costs to develop and implement this form of EQA are likely to be too prohibitive for small voluntary contribution EQA schemes without external funding.

Methods

Software

The data from the web pages are collected in a PostgreSQL database. The system is written using open source software like Apache and Tomcat/cocoon (XML/XSLT) and the website is run on a dedicated Pentium 4 2.8 GHz server. JPEG images are provided for prospective analysis. These images can be evaluated by the participant online or downloaded to an image analysis system, as the majority of commercially marketed image analysis systems have a facility for downloading and manipulating JPEG images.

Case selection

The EQA involves a selection of normal and abnormal cases. Cases can be tailored to a specific clinical scenario with the option of multiple additional investigations, for example metaphase images, parental bloods, FISH investigations, additional banding techniques, and previous family or cytogenetic history. An example of the multiple options is shown in Figure 1.

Figure 1
figure 1

Diagrammatic representation of the multiple options.

Full size image

Each EQA case is made available for analysis in accordance with European cytogenetic report times for diagnostic results.4

Participant analysis procedure

The website, designed for easy participant navigation, is protected with a unique entry password allocated to each laboratory. After entry, the first screen seen by the participant shows the referral card with the patient clinical details. The participant then selects the appropriate tests from the menu bar to analyse images, selects further tests as necessary to supplement the analysis, and ascertains the result based on the clinical information given. Once a specific supplementary test has been selected, the participant will view either a series of images as magnifiable thumbnails, or a table or diagram, or some text depending on the type of test requested.

The participant is able to make comments as each test or image is reviewed, as well as being able to track, by clicking the appropriate button on the screen, those images already viewed or analysed. All comments are visible to the participant and can be amended at any time until completion of the case. The software gives each image a unique number and enables the participant to colour code the periphery of the image so that when logging on at a later stage it is immediately clear which images have been previously analysed or just viewed.

Reporting

Participants are required to submit a report online including the correct nomenclature to describe the chromosome rearrangement,5 a written description and clinical interpretation of the abnormality. The participant can write a draft report at any stage of the analysis which, provided it is saved, will be available the next time they enter the website. Once the case is analysed and the checking procedure completed, the final report is submitted by entering a second password (an administrator-defined exit code). The exit code ensures against any accidental submission before the case is complete, but once submitted, the case cannot be revisited for revision; however, participants who have completed their submission can view the case for educational purposes after the final submission date has passed.

Case administration

The administrator creates the EQA case with the relevant images and assigns a cost to each diagnostic test (see ‘Cost allocation system’ below). The administrator also assigns participants to specific cases, so only registered participants have access to the case when it is available. Once the submission date has passed, expert assessors can be assigned cases submitted by all laboratories, or to specific cases, to avoid assessing their own laboratory, and to exploit their particular areas of expertise.

Assessors’ actions

Submissions are examined by the assessors’ panel and marked according to set criteria of marking and cost allocation (see below). Any poor performance is identified at this meeting and ratified by a steering committee.

Cost allocation system

As the software system audits website usage, the administrator and assessors can visualize diagrammatically the web page usage for each case made by each participant. Web page usage by the participant is identified both by a ‘cost’ and an extensive web trail making it possible for the assessors to comment when an inappropriate or excessive choice of tests has been made, indicative of inefficient laboratory practice. This audit trail is of particular use when the incorrect diagnosis is obtained as it may be possible to ascertain where in the decision-making process the analysis or interpretation of the abnormality was incorrect.

Units of cost take into account working practice and ease of collecting data within a laboratory, and are based on the draft UK currencies for cytogenetic testing (Tony Parkin: personal communication). A different cost is given for the analysis of a blood (10 units) as opposed to an oncology sample (13 units) or amniotic fluid sample (11 units). Different FISH tests are weighted on their complexity (M-FISH=50 units vs a microdeletion probe=7 units).

A participant may select the same test several times to visualize the results but only incurs a cost on the initial visit to the web page. Participants attempting to select every test to obtain the result are thereby distinguishable from those participants that obtain the answer using the appropriate tests to ascertain the chromosome abnormality.

Marking system

Laboratories were assessed for their analytical and interpretative performance according to performance criteria previously set by UK NEQAS for Clinical Cytogenetics and CEQA EQA schemes. The performance criteria are based on International System for Human Cytogenetics Nomenclature (ISCN),5 national or European Professional Guidelines.4, 6, 7 Marks based on accuracy of analysis, appropriate choice of tests, interpretation of the significance of the result, content, and clarity of the report. Minor omissions receive 0.2–1.0 penalty points. However, an incorrect analysis such as missing a visible chromosome abnormality or a serious misinterpretation would be given 0 marks and result to a poor performance.

Results

EQA results

The online EQA was initially piloted with constitutional cases before expanding to include oncology (www.ccneqas.org.uk).

Five pilot EQAs involving eight EQA cases were trialled on the Internet system for UK NEQAS (www.ccneqas.org.uk) and CEQA schemes (www.ceqa-cyto.eu) (Table 1). A total of 80 laboratories from 24 countries participated in different EQAs from across the world. Pilot EQAs are voluntary and 14% of laboratories failed to submit their report before the deadline due to lack of time or unavailability of staff. CEQA was successfully piloted using the Internet-based EQA in 2006. Twenty-five participants enrolled and 19 laboratories from 15 countries submitted a report. Seven laboratories had never participated in EQA before and successfully navigated the web pages to attain the correct answer. The CEQA scheme will continue to expand over the coming years to enable more laboratories to participate in EQA.

Table 1 Summary of pilot EQAs undertaken using the internet system
Full size table

Since these pilots, UK NEQAS has completed a further 7 EQA involving 14 EQA cases for constitutional and oncology cytogenetics (Table 2). A total of 82 laboratories from 20 countries participated from across the five continents (Americas, Asia, Australasia, Africa, and Europe) and of these Europe was the largest continent with 12 different countries participating.

Table 2 Summary of results and performance for the constitutional and Oncology EQAs
Full size table

Significant differences in reporting of analysis and interpretation of the normal and abnormal results were evident. All laboratories correctly analysed the normal results and mosaics but inappropriate advice was given in a small proportion of cases. One unexpected finding was the reporting of a 46,XX,del(2)(q37.1) blood case as a normal female (46,XX) by three laboratories (Figure 2; Table 2). This may indicate a poor level of analysis or unfamiliarity of synchronized blood cultures that produce longer metaphases and enable the analyst to visualize the chromosomes at a higher resolution. Slides from three cases were sent to these laboratories to follow up the poor performance and one laboratory still failed to detect a terminal deletion in the long arm of chromosome 2, del(2)(q37.1). A total of 6 analytical errors resulting in a misdiagnosis occurred with 668 assessments across the 14 EQA cases involving various tissues/diseases. This gives an error rate of 6/688=0.8%. If interpretative errors are included this makes an error rate of 14/688=2%. This 2% error rate compares favourably with the UK NEQAS analytical and interpretative error rate of 1.7% for the retrospective slide distribution EQA for 766 assessments over 12 EQAs in the past 3 years.

Figure 2
figure 2

A representative metaphase from the website. The normal chromosome 2 and the del(2)(q37.1) are arrowed.

Full size image

Participant usage of the website

In 2005, the peak usage of the website occurred when the EQA cases went live. Further breakdown showed that peak usage was in the last few days before the closing date when all three cases were online. As the main load on the server was when an EQA case was about to close, start and close dates of subsequent UK NEQAS cases were staggered to spread the load. Most laboratories viewed a case within the first week of it opening but the final reporting of a case was left by the majority to the last 5 days (Figure 3). The regular periods of low activity correspond to the weekends. The majority of laboratories took 2 weeks to report the EQA case online, the main exception to this being the eight laboratories that did not start the case until the last week (Figure 3).

Figure 3
figure 3

Postnatal blood external quality control (EQA) case—website hit distribution showing when a laboratory started and finished the EQA case.

Full size image

Participants also fell into two basic categories, those that analysed every test available on the website and those that were more discriminating in the tests they selected. While the majority of participants ascertained the correct answer, not all made efficient use of their time and resources assuming that such extensive web page usage reflects their diagnostic practice. The online EQA differs from the real diagnostic situation in that all possible tests are available at once (for example FISH tests or parental blood metaphases). Consequently, laboratories may have approached the analysis of the case differently from their routine practice to ensure they got the correct answer. After a cost allocation for each investigation was introduced in 2006, a wide variation in costs was found. For one case involving the diagnosis of a balanced interchromosomal insertion, with material from the long arm of chromosome 8 being inserted into the long arm of chromosome 18, 46,XY,ins(18;8)(q11.2;q13q24.1), the costs ranged from 10 to 764 units. The expected answer could have been obtained within a maximum cost of 44 units assuming initial G-banded analysis (5–10 units), parental bloods (20 units) and confirmatory FISH studies (14 units) were undertaken. The modal cost for this particular blood EQA was between 40 and 60 units with a mean cost of 153.8. While 27 of 42 (64%) laboratories incurred a cost of <100 units, 13 of 42 (25%) of laboratories incurred costs >200 units with two of these laboratories incurring unit costs>730. These additional costs were incurred because laboratories used M-FISH (50 units) or selected virtually all the FISH probes that were available (microdeletion probes, individual telomere probes, or individual centromere probes).

There was a very wide variation in the content of submitted reports, with some participants not explaining or interpreting their findings. For instance, few laboratories related the chromosomal findings to the presenting clinical features at presentation, or discussed the significance of the result. Some participants had difficulty reporting a normal result succinctly and suggested inappropriate and costly investigations to ensure nothing had been missed.

Discussion

Internet-based EQA system

The way in which this online EQA has been developed follows closely the stepwise processes involved in the diagnostic situation and thus overcomes some of the limitations of other forms of prospective EQA. It allows the participant to choose appropriate follow-up tests (for example, FISH) to elucidate the chromosome abnormality. This is a major improvement on conventional distribution of slides or images where the inclusion of additional test material inevitably reveals the answer.

Scheme administration

The Internet EQA is easy for the administrator to manage as each case can be individually created and the IT tools allow total flexibility to add or delete investigations, images, or graphics. In addition it is possible to duplicate a previous EQA case or create a template case, which can then be modified to create a new EQA case scenario, reducing the administration for the scheme organizer.

Cost allocation

The ability to allocate a cost to each investigation or test and the total cost analysis undertaken by the software has proved to be a valuable tool in assessing the efficiency of the procedures followed by the participant when selecting the different tests. The introduction of this cost allocation will hopefully educate laboratories in future EQA rounds not to approach an EQA case differently from a diagnostic case.

Reporting

The wide variation that was found in reporting style and content was unexpected, both within and between countries. The variation between different countries may be explained by different national requirements/standards, or indeed by absence of defined standards in some countries. However, for countries such as the United Kingdom and the Netherlands that do have national standards variation in reporting styles was still evident. The introduction of the European cytogenetic guidelines4, 6 may result in further harmonization in reporting over time where national laws allow. In some European countries (eg Greece), the cytogenetic laboratories are only able to report the analysis: by law the interpretation is undertaken by the referring clinician. Elsewhere the interpretation is made by the laboratory or is discussed with a clinical geneticist. In some European countries the head of the laboratory also has to be clinically qualified. Where the interpretation is made by a clinician outside the laboratory, these practitioners are now being encouraged to be involved in the EQA process, so the complete cytogenetic result is assessed for EQA.

Poor performance

Identification of poor performance is an essential part of EQA and enables a laboratory to address the internal quality issues and improve the diagnostic service offered. Poor performance can only be addressed by an EQA scheme with a reliable and discriminatory marking system. It is unfortunate that UK NEQAS has observed over a number of years that a small percentage (2 of 82 laboratories to date) of laboratories fail to register for EQA the following year. On each occasion the laboratory has had a poor performance within the first year of participating, suggesting they prefer to ignore the problem rather than address it.

Further applications

While this software programme has been used for Internet-based EQA, its flexibility allows for other applications such as continuing professional development, competency testing, or independent learning programmes. The website can also be adapted for educational purposes to give rare case scenarios.

The EuroGentest network has identified more than 700 cytogenetic laboratories in Europe of which fewer than half participate in existing EQA schemes. The potential for laboratories to have access to EQA is overcome with this Internet-based EQA as submissions can be in several languages. In the longer term, EQA participation will be mandatory as accreditation of cytogenetic laboratories becomes a requirement1, 2 and availability of an accessible EQA will be beneficial. The recent OECD guidelines provide a framework for quality assurance for genetic laboratories and are equally relevant and applicable to cytogenetics and biochemical genetics.2 These guidelines also lay out principles and best practices for proficiency testing (i.e. EQA) and accreditation, and adoption of these recommendations will improve the quality of the genetic laboratory and EQA services.2

Most national EQA schemes are currently free or charge a nominal fee, but only with the introduction of registration fees to fund administrative staff and cover running costs will EQA schemes be sustainable in the long term. All but two national schemes are run on a voluntary basis, jeopardizing their sustainability, as schemes are susceptible to closure or interruption when the scheme organizer retires or moves from their diagnostic cytogenetic post. Our experience indicates that the recurrent costs for a national or European-based cytogenetics EQA scheme are similar regardless of whether the EQA involves a paper exercise, CD-ROM, or an Internet-based system. However, there is an additional cost to initially set up a website.

Conclusions

The online Internet-based EQA has provided a varied EQA programme for a national and European EQA scheme, covering oncology and/or constitutional cytogenetics. The EQAs from the UK NEQAS and CEQA schemes have demonstrated that this form of EQA is able to detect poor performance. The vast majority of laboratories obtain a satisfactory performance in all their EQA rounds giving assurance to patients and clinicians that the laboratory is providing a quality service with reliable and accurate results.