Introduction

About 3–5% of breast cancers are caused by germline mutations in the breast cancer genes BRCA1 (OMIM 113705) and BRCA2 (OMIM 600185).1 After the BRCA1 and BRCA2 genes were identified in 1994 and 1995,2, 3 genetic testing became available and is now routinely offered to women from high-risk families. Estimates on the frequency of BRCA1/2 mutations in the unselected population range between 1:2501 and 1:840.4 In certain populations, Ashkenazim and Icelandic as most poignant examples, clear founder effects have been observed that lead to mutation frequencies as high as 1:40.5 In these populations, the strategy for mutation detection differs in that frequent founder mutations have to be ruled out before analyzing the whole BRCA1 and BRCA2 genes.

Screening for BRCA1 and BRCA2 mutations is both time- and cost-intensive. The methods used should be able to detect disease-related mutations, that are distributed throughout the whole BRCA1 and BRCA2 genes, and also be able to identify missense mutations: although most BRCA1 and BRCA2 mutations lead to a truncated protein, there is also a significant proportion of pathogenic missense mutations6 (BIC Database). The rather frequent mutation C61G in exon 5 of the BRCA1 gene is such an example. Direct sequencing allows specification of the sequence alteration and is considered the gold standard. Because direct sequencing is time-consuming and costly,7 there is a need to establish alternative faster and less expensive methods for BRCA1 and BRCA2 routine diagnostics with comparable accuracy. Several scanning methods are available for the detection of sequence alterations, without giving information about the specific alteration (for review8, 9, 10). The most commonly used techniques are the single-strand conformation polymorphism (SSCP), restriction endonuclease fingerprinting (REF)-SSCP, conformation-sensitive gel electrophoresis (CSGE), fluorescence-based conformation-sensitive gel electrophoresis (F-CSGE), two-dimensional gene scanning (TDGS), protein truncation test (PTT), and denaturing high-performance liquid chromatography (DHPLC). The first five methods are based on the different motility of the mutant DNA fragment. PTT is an in vitro transcription–translation assay for the detection of truncating mutations. DHPLC uses chromatography to separate hetero- from homoduplices under high pressure and partially denaturing temperature conditions. Numerous other methods have been developed for the detection of sequence alterations. These include the restriction enzyme-based techniques enzymatic mutation detection (EMD) and multiple-dye cleavase fragment length polymorphism (MD-CFLP), the yeast-based stop codon (SC) assay, which detects truncating mutations and requires RNA, RNA-based sequencing, the micronucleus test (MNT), originally established to determine DNA damage. Recently, there have been reports on further techniques for the detection of BRCA1 and BRCA2 mutations, namely a real-time PCR-based approach using sequence-specific probes, the array-based chip technology using SNP-specific oligonucleotides, and MALDI-TOF mass spectrometry.

This systematic review was performed as part of a Health Technology Assessment (HTA)11 to evaluate the diagnostic accuracy and the potential for implementation in a routine setting of different scanning methods for BRCA1 and BRCA2 mutations in comparison to direct sequencing. A systematic review – as opposed to traditional reviews of the literature – starts with a clearly formulated research question and uses explicit and transparent methods to identify, select, and appraise studies. It is the essential tool of evidence-based medicine. Its purpose is to provide clinicians and health managers with a more objective basis for decision-making and to identify the lack of evidence.12 Therefore, many national health systems and insurers are now basing their decisions on the reimbursement of new technologies on the results of systematic reviews.

Materials and methods

Literature search

A literature search was carried out between March and April 2004 in the following databases: Medline, Embase, Biosis, Pascal, Cancerlit, Cochrane Library and the databases of the member organizations of the International Network of Agencies for Health Technology Assessment using the key words ‘BRCA’, ‘mutation’, ‘tests’ and their synonyms, adapting them to each database. There was no limit on the date of publication. In addition, the journals Human Mutation, Cancer Research, Human Molecular Genetics, Clinical Chemistry, Genetic Testing, Clinical Genetics, Nucleic Acid Research, Journal of Medical Genetics, and Human Genetics were screened by hand. Furthermore, the reference lists of the included studies were screened. In October 2006, an update of the literature search was performed. As all included articles of the original search were identified through Medline, the update focussed on this database. The literature search was performed by HS, employing rather broad inclusion criteria. Borderline publications were discussed by all authors until a consensus for inclusion or exclusion was reached.

Inclusion and exclusion criteria

Primary studies, systematic reviews, and health technology assessments were considered. The a priori criteria for the inclusion of primary studies were: (1) the study population reflects a population at risk in a routine setting, (2) direct sequencing was used as a reference standard for assessing sensitivity and specificity, (3) independent, blinded investigators, (4) prospective trials, (5) sensitivity and specificity are reported or can be calculated from the results reported, (6) language is English, German, French, Spanish, Italian or Dutch. Techniques for scanning were included if they allow the complete coding region of BRCA1 and BRCA2 to be screened and they detect the same types of mutations that can be identified by direct sequencing (DS). A priori exclusion criteria were (1) studies performed using only direct sequencing or comparing DS with techniques that are only able to identify known mutations (ie allele-specific-oligonucleotide analysis), (2) studies that did not include negative controls, (3) studies using samples with artificially created sequences, (4) studies investigating only parts of the BRCA1 or BRCA2 gene, (5) studies that only aimed at detecting polymorphisms, and (6) studies that only aimed at detecting large deletions within the BRCA1 or BRCA2 genes, because this type of mutation cannot be detected by the reference standard direct sequencing. Nonsystematic reviews (‘narrative reviews’), letters, comments, and meeting abstracts were also excluded.

Strictly applying these criteria, only two studies, evaluating four methods, would have remained.13, 14 The inclusion criteria were therefore extended and the consequences for the internal and external validity are considered in the discussion section. Additional studies were included (1) with a retrospective design, (2) if only parts of the BRCA1 or BRCA2 gene were investigated, (3) if only parts of the study were compared to the reference standard direct sequencing – in this case only these parts were included, (4) if the authors only reported that there were no false positive or false negative results, so that conclusions on sensitivity and specificity could be drawn, even if it was not possible to calculate these parameters, (5) if techniques were used that are suitable to identify only specific types of mutations, that is, truncating mutations. If it was not clearly described whether the study design was retrospective or prospective, or whether the investigators were blinded, the authors of the publications were contacted. In the case of failed or inconclusive response, the publication was given the ‘benefit of doubt’ and this indefinite status was reported.

Assessment of the internal and external validity

For the assessment of the internal and external validity of diagnostic studies, a scheme suggested by Flynn and Adams15 was adapted. For the assessment of internal validity, it was recorded if the evaluation of the reference diagnosis (reference sequencing) was carried out independently, and note was made of the perspective (prospective or retrospective) of the study and sample size (>35 patients with mutation and >35 patients without mutation are needed to yield 95% confidence intervals whose lower bound is >90% sensitivity if the sensitivity is 100%). For the assessment of the external validity, it was recorded if the composition of the sample can be reproduced, if the proportion of individuals with mutations lies between 20 and 40% (as this number reflects the population of current testing programs that apply a minimum of 10–20% BRCA mutation probability as inclusion criteria),16, 17 if the study aimed at all mutations detectable by direct sequencing, and if the whole gene BRCA1 or BRCA2 was considered.

Outcome parameters

Outcome parameters were sensitivity, specificity, positive predictive value and negative predictive value. Different definitions of ‘a positive result’ were accepted (ie ‘positive’ can mean any alteration like mutations, truncating mutations, unclassified variants (UV), and polymorphisms) as a basis for the calculation of these parameters.

Data extraction and data synthesis

Data were extracted and synthesized qualitatively first by HS and then counterchecked by the other authors. Because of the heterogeneity of methods, study designs, selected gene fragments, and outcomes, a meta-analysis was not performed.

Results

Results of the literature search

The results of the literature search in 2004 and the stepwise exclusion process are illustrated in Figure 1. Out of 3016 references, only 10 articles met our inclusion criteria. The update in October 2006 yielded another 800 citations, but none of them fulfilled the inclusion criteria. Of the 122 publications excluded, 84 were evaluations of scanning methods. Among these publications, excluded from this review for methodological reasons, the distribution of scanning methods was as follows: 12 SSCP, 12 PTT, 10 microarrays, 10 heteroduplex analysis, eight CSGE, six enzymatic assays, five DHPLC, four yeast-based assays, three DGGE and three TDGS. Another 18 methods were evaluated only once. Four HTA reports and one guideline were identified.17, 18, 19, 20, 21

Figure 1
figure 1

Flow chart: selection of the literature.

Study description

The ten studies included report on 12 different scanning methods for the detection of BRCA1 and BRCA2 mutations (see Table 1): Only for three techniques (DHPLC, SSCP, TDGS) was more than one evaluation study identified. The study populations were affected and non-affected individuals from families with an increased risk for BRCA1 and BRCA2 mutations. Only in four studies14, 22, 23, 24 could the prevalence of individuals with mutations be deduced from the data. The numbers between 48 and 80% indicated that study populations had a higher prevalence than what is to be expected in current testing programs. In four studies, the whole BRCA1 gene was scanned for all types of sequence alterations that can be detected by DS. The only study evaluating the scanning of all types of sequence alterations within the BRCA2 gene was limited to exon 11.25 One study was performed in seven laboratories, one study in four laboratories, whereas the remaining eight studies were carried out in single laboratories. Definitions of positive results were heterogeneous: all mutations, truncating mutations only, mutations, and UVs, or any alteration from the wild type. One study24 used truncating mutations as denominator. However, detailed information in the paper allowed for a recalculation based on cell lines as denominator (Table 1).

Table 1 Study description

Assessment of the internal and external validity

The results of the assessment of the internal and external validity are presented in Table 2. An independent, blinded evaluation was reported in the publications or communicated by the authors in nine studies; in one study26 it could not be confirmed. In nine studies, the sample size was too small to allow results that exclude a sensitivity of <90% with a P<0.05 (provided that measured sensitivity is 100%), and for one publication27 the group-specific sample size was not clear. It was possible to reproduce the study cohort from the description of inclusion and exclusion criteria in only three publications.23, 27, 28

Table 2 Internal and external validity of the studies

Results of the evaluations

For seven methods, sensitivities of 100% were reported (DHPLC, EMD, MD-CFLP, F-CSGE, RNA-based sequencing, REF-SSCP, SC assay). Among them, DHPLC was the only method that was evaluated in more than one single study.13, 14, 24, 28

For five methods, sensitivities of less than 100% were found (SSCP, TDGS, MNT, PTT, and CSGE). Detailed information on these outcomes is presented in Table 3.

Table 3 Results

Discussion

Molecular diagnostics for families with hereditary breast and ovarian cancer is a rapidly evolving field. After the breast cancer genes BRCA1 and BRCA2 were identified and genetic testing became possible, there was a call for reliable and easy-to-perform diagnostic tests.17 Because of the importance of a test result, great demands are made on the diagnostic accuracy. As direct sequencing is costly, an appropriate diagnostic approach should include a scanning method that can be carried out quickly, automatically and ideally allows the detection of all different types of mutations with a sensitivity and specificity of 100%. To evaluate the currently available scanning techniques, this systematic review was carried out.

Methods of this review

The very sensitive search finally led to 10 articles that met the inclusion criteria. Results were compared with the systematic searches of Noorani and McGahan,18 McGahan et al21 and the NICE guideline.17 No additional article was found that would have met the inclusion criteria. The risk of bias in the search was therefore assumed to be negligible.

The small number of pertinent studies in itself highlights an important problem in this area: The experience with a method necessarily remains anecdotal when only a single valid evaluation study can be identified. However, although carefully designed evaluation studies are needed, for the last 3 years no such published study could be identified.

Quality of the included studies

The reporting quality of the results of the studies was quite poor. No publication presented the statistical significance or confidence intervals of the results.

Only four studies13, 14, 22, 23 evaluated the scanning of all mutations that can be detected by direct sequencing for a complete gene, the other studies either evaluated only the scanning for selected exons25, 26, 27, 28 or were limited to the detection of truncating mutations.24, 29 Thus, it is not possible to extrapolate findings from one gene or parts of one gene to other genes or even to other parts of the same gene.30, 31 This has to be taken into account because in none of the included studies was the whole BRCA2 gene scanned.

All included studies had small sample sizes. For DHPLC, four studies had a total sample size of 159 individuals but they were too heterogeneous regarding the sequences analyzed and their outcome parameters to allow for quantitative pooling.

The inclusion of studies, which were either retrospective22, 23 or whose perspective was not clear,26 should not have a major impact on internal validity according to a study by Lijmer et al.32 In contrast, doing the reference test while knowing the results of the candidate test, leads to a bias towards better results of diagnostic accuracy.32 The publication of Casadei et al26 does not mention blinding explicitly and we were not able to obtain this information from the authors. Although the description of this study implied an independent evaluation, the unconfirmed status has to be taken into account when interpreting the results.

For comparison of the results, the heterogeneous election of the definition of a ‘positive result’ of the studies is also problematic. Four studies22, 23, 24, 29 only aimed at the detection of disease-related mutations, whereas the other studies reported on the detection of all sequence alterations, therefore also on UV, that is, sequence variants with unknown pathogenetic function with a frequency of less than 1% in normal controls, and polymorphisms, that is, sequence variants with a frequency of more than 1% in normal controls. The disease-related mutations are certainly the most important sequence variants, but screening for all sequence alterations is the more suitable approach for comparison of different screening methods, because there is no certainty that all possible disease-related mutations are as yet identified.

Current status of the scanning methods evaluated

In the included studies, the sensitivities of the methods SSCP, MNT, PTT, CSGE, and TDGS were too low to recommend them for current routine use: the gel-based techniques SSCP, CSGE, and TDGS separate mutated from wild-type DNA fragments during an electrophoretic run owing to their different conformations. Performance of these techniques is time-consuming, interpretation of the results sometimes difficult and dependent on the investigator. PTT is a widely used in vitro transcription–translation technique for the detection of truncated proteins resulting from a premature stop codon. However, most, but not all disease-causing BRCA1 and BRCA2 mutations are nonsense mutations.6 Andrulis et al,24 referring to earlier experiences, did not expect PTT to detect mutations in the 5′ region and therefore had one laboratory in the study use complementary direct sequencing for this region. Therefore, PTT is not suitable for use as the only method in a routine diagnostic setting.

MNT detects micronuclei that occur in cells after DNA damage. As BRCA1 and BRCA2 are involved in DNA repair via homologous recombination, defects in their function result in the occurrence of micronuclei. The main application field for MNT is the scanning for unspecific alterations in DNA repair but, because of its low sensitivity and specificity, this method cannot be recommended as a diagnostic tool. Regarding this question, the sensitivity, and specificity are very low, being 58 and 80% respectively.

F-CSGE25 and REF-SSCP27 are advanced modifications of the gel-based techniques, developed to accelerate and to automate the established techniques. Because the critical points and therefore the limits will still remain the same, irrespective of whether the method is used in the classical or in the advanced manner, sensitivities should not vary significantly.

EMD24 and MD-CFLP26 are enzyme-based procedures using the properties of restriction enzymes. Because these enzymes are highly temperature-sensitive and unstable, a high throughput application in a routine setting seems critical.

Jakubowska et al23 report on RNA-based sequencing of BRCA1. The study had a good overall reporting quality. However, only 21 patients were included, 10 of them with and 11 without a mutation so that more extensive investigations will be necessary. For diagnostic procedures in a routine setting, RNA-based sequencing has the disadvantage that RNA rapidly undergoes degradation, unless it is immediately transcribed into the stable cDNA.

The only study evaluating the yeast-based stop codon assay29 was on a sample cohort with merely three mutations. The SC assay requires mainly RNA as origin material and is limited to the detection of truncating mutations. It can therefore not be employed as the only scanning test.

At present, only DHPLC can be considered to be used as a scanning test in a routine diagnostic setting. However, none of the four studies evaluating DHPLC was performed on BRCA2 and some authors reported problems in specific fragments. The DHPLC is a chromatography-based method and is able to distinguish between homoduplices – in the case of two wild-type alleles – and heteroduplices in the case of a heterozygous sequence alteration. A DHPLC analysis can be performed easily and quickly. This technique is automated and enables high-throughput analyses. The DHPLC is able to detect all types of mutations and the interpretation of results does not depend on the investigator. Thus, it is not surprising that DHPLC is increasingly used in mutation analyses in familial breast cancer.33, 34, 35

None of the techniques included in this systematic review allows the detection of large rearrangements, like deletions of whole exons. Recently, the MLPA (Multiplex Ligation-dependent Probe Amplification) assay was introduced as an additive diagnostic tool36 to detect this type of mutation, which is estimated to occur in up to 10% in BRCA1 and less in BRCA2.37, 38 Therefore, MLPA is currently used in addition to either scanning, for example by DHPLC, and subsequent direct sequencing or direct sequencing only.

Costs related to direct sequencing and DHPLC

Few studies have evaluated the costs for personnel and material related to direct sequencing and DHPLC. One study in France, limited to BRCA1, calculated 991 euro for direct sequencing, and 174 euro for DHPLC.7 Costs for overheads were not considered in the French study. A German study, covering both genes, calculated 2070 euro for direct sequencing and 1848 euro for DHPLC in 2004.39In the same year the price of Myriad laboratories for direct sequencing was at 2975 US dollar (2403 euro at the time), whereas Bioscientia, a German-based company operating with a license from Myriad, charged 3980 euro.39 In October 2006 the prices of Myriad have increased to 3120 US dollars, corresponding to 2456 euro, whereas the prices of Bioscentia decreased to 1890 euro (personal communications) during the same interval (the offer of Bioscientia includes a search for large rearrangements and deletions in case of a negative result with direct sequencing in high-risk patients). The large variations in costs and prices between countries as well as different points in time precludes definite conclusions regarding the most cost-effective strategy at present. Purchasers will need to compare prices in their local context.

In summary, this systematic review compares the diagnostic accuracy of 12 different scanning methods for the detection of mutations in BRCA1 and BRCA2. On the basis of the currently available controlled studies that matched our inclusion criteria, DHPLC outperforms all other studied methods. However, its assumed ability to detect mutations with a high sensitivity in BRCA2 as it does in BRCA1, needs to be confirmed through adequate studies.