Main

Genetic testing has the potential to benefit the population in many ways, including enabling early detection of patients with hereditary predispositions to specific diseases, individualization of drug therapy, and development of gene-based drug therapies.1 However, this is not without controversy. For example, some claim genetic testing has had little impact on current clinical practice, a situation reflecting ethical, financial, medical, scientific, and/or sociologic impediments to genetic screening and service programs.2 One such scientific impediment includes the lack of validation studies on the associations between genetic testing and outcomes. Clarifying the economic implications associated with genetic testing provides additional information for decision makers with scarce resources.

In an environment of limited health care resources, costs and the clinical effectiveness of medical interventions are factors considered in funding decisions. Economic evaluation provides a potentially valuable tool to prioritize health interventions, allowing identification of interventions that maximize health gains per dollar. Economic evaluation is defined as “the comparative analysis of alternative courses of action in terms of both their costs and consequences.”3 Cost-effectiveness analysis (CEA), a type of economic evaluation, provides a quantitative framework for evaluating interventions or treatments and is used by decision makers to evaluate new health technologies. As a result, there is a corresponding increase in the number of economic evaluations published and an increased demand for systematic reviews of these studies. This is particularly true of genetic testing and services, an area now undergoing an unprecedented pace of new discovery with potential clinical application.4

By reviewing economic evaluations of genetic testing services published between January 1990 and August 2004, Carlson et al.5 provided meaningful insight into the quality and interpretation of such studies. Since 2004, there have been a considerable number of economic evaluations published on genetic testing and interventions; however, there have been few published reviews on the health economic evidence pertaining to these types of investments. Moreover, these reviews focused primarily on pharmacogenetic strategies in cancer,6,7 cystic fibrosis,8 and screening,2,9 and none assessed the quality of the economic evaluations.

Quality assessment of economic evaluations of new medical interventions is important to the health care community and in particular to decision makers because it may indicate which studies should be included when making evidence-informed policy or decisions pertaining to funding. For these reasons, we assessed more recent contributions in this area and accordingly undertook a review and quality assessment of economic evaluations of genetic testing and interventions published from 2004 to 2009.

METHODS

In July 2009, we performed a literature search for economic evaluations on genetic testing and interventions published between September 2004 and June 2009. We used the same search terms and databases as Carlson et al.,5 which included terms as follows: “cost-effectiveness,” “cost-benefit,” “cost-minimization,” “cost-utility,” “economics,” “gene therapy,” “pharmacogenetics,” “genetic screening,” and “genotype” and used databases such as PubMed, Medline, Proquest, LexisNexis, Expanded Academic Index, The Harvard Review of Economic Analyses (http://www.hsph.harvard.edu/cearegistry/), PsycINFO, National Institute for Clinical Excellence (http://www.nice.org.uk), and the Canadian Agency for Drugs and Technologies in Health (http://www.cadth.ca/index.php/en/home).

For inclusion in our review, studies had to meet the following criteria: (1) a focus on detecting heritable disease, (2) reporting on an economic evaluation of a genetic test or service, and (3) publication in a peer-reviewed journal. Studies that were not published in English, full-sized articles, or related to interventions intended for humans were excluded. Studies on the genotyping of viral genomes such as hepatitis or human immunodeficiency virus were also excluded.

All studies meeting inclusion criteria were reviewed and information on design and model characteristics, incremental cost-effectiveness ratios (ICERs), and conclusions then extracted. Summary statistics were calculated for the type of intervention, economic analysis, model, perspective, and sensitivity analysis. Conclusions for studies in similar clinical areas were summarized. ICERs extracted in this review and those cited by Carlson et al.5 were graphed to observe trends through time. Each article identified through the literature review was also assessed for quality by two reviewers (S.D. and Z.M.) using the Quality of Health Economic Studies (QHES) instrument developed by Chiou et al.10 This grading system has 16 criteria, each of which is associated with a specific value, and the total quality scores can range from 0 to 100. Before grading, both reviewers reached consensus on the interpretation of the criteria. Articles were then graded individually by each reviewer, with points deducted from the maximum score (100) for each criterion not addressed. After the independent reviews, the results were discussed, and either a consensus was reached for each score or a third reviewer (J.H.) was consulted to achieve a consensus score.

Once scores were obtained using the QHES grading system, the overall mean and subgroup means were calculated and compared with those reported by Carlson et al.5 The difference between mean scores was assessed for statistical significance using a one sample t test. Because the magnitude of the quality score is based on the deduction of points for missed criteria, the number of studies that failed to address each quality criterion was tracked to identify major contributors to poor quality scores in the reviewed studies.

RESULTS

Descriptive statistics of included studies

The search strategy identified 631 publications. Among these, 54 satisfied the inclusion criteria based on their titles and abstracts. However, more complete review of the 54 full-text articles revealed only 26 to be economic evaluations, and, accordingly, all subsequent analyses focused on these 26 studies.11–36

The majority of these 26 studies were focused on genetic interventions for preventive screening (46%) or increasing treatment efficacy (46%). Most of the preventive screening interventions were for detecting specific genetic mutations in carriers with high risk of developing the associated disease. Genetic interventions aimed at increasing treatment efficacy mostly incorporated a pharmacogenomics approach. Of the only two studies (8%) addressing other types of genetic interventions, one focused on prenatal screening to detect fragile X syndrome and the other on a diagnostic test for children with learning and developmental disabilities.

Table 1 presents descriptive statistics by type of economic analysis, model, perspective, and sensitivity analysis. All the studies reviewed used one of four types of economic analysis: CEA, cost-utility analysis (CUA), cost-benefit analysis (CBA), and cost-consequence analysis (CCA). CEA, which calculates cost per single unit of effect or outcome, was the most common method (65%), and cost-utility, which uses quality-adjusted life year, was the least common (12%) method used across these studies.

Table 1 Descriptive statistics and quality scores
Full size table

Economic evaluations often use a model, such as a decision tree—which calculates “payoffs” of various possible choices represented as branches. Also Markov models are frequently used, which use a series of “states” that represent the disease process and are associated with a benefit and cost.3 Decision tree analysis was the most commonly used in the 26 reviewed studies (46%). These models may also incorporate different study perspectives depending on the objectives of the study. Although the societal perspective, which includes all costs borne by the health system, patient, and society, is highly recommended by many guidelines,37–39 the studies reviewed here most frequently used the health care system perspective (50%) with relatively few studies using a societal perspective (27%).

Approaches to sensitivity analysis were also explored. Such analyses provide a means to characterize uncertainty in the model. The most simple sensitivity analysis is a one-way sensitivity analysis, where each model input is varied individually to investigate its impact. Two-way or multiway sensitivity analyses are extensions of a one-way sensitivity analysis.3 More than half of the studies included in this review used a one-way or two-way sensitivity analysis. Economic evaluation guidelines recommend the use of probabilistic sensitivity analysis (PSA),37 wherein a probability distribution for each model input is defined and simulations are run with the value of the model input being randomly drawn from the defined distributions.37 PSA was performed in 15% of the economic evaluations that were reviewed. Of the studies that did not conduct any sensitivity analysis, three were CBAs or CCAs and one used a modular simulation program to simulate all possible screening scenarios.

Sensitivity analysis also highlights which parameters have the largest impact on study results. The parameters that were chosen to be varied in the sensitivity analyses were different in each of the studies reviewed. Approximately 60% of studies varied the cost of genetic testing, but only a few included the specificity and sensitivity of the genetic test in their sensitivity analysis. Although prevalence of genotype distribution among a population can have a significant impact on CEA results, this factor was considered in only 25% of the studies.

Overview of included studies

The studies included in this review span a number of clinical indications, genetic tests and interventions that are summarized in Table 2. Conclusions pertaining to cost-effectiveness of the genetic testing and interventions are listed according to clinical area. As summarized in the table, four studies evaluated the cost-effectiveness of colorectal cancer screening or surveillance of high-risk individuals,26,28,30,31 and, within the specified scenarios, genetic testing and screening of high-risk individuals were considered to be cost-effective.

Table 2 Overview of economic evaluations for genetic testing and interventions (2004–2009)
Full size table

The cost-effectiveness of prenatal screening for cystic fibrosis was also examined in two studies.15,16 Of these, one study found a school-based screening program for genetic carriers of cystic fibrosis to be of comparable cost-effectiveness with other prenatal cystic fibrosis screening programs.15 The other study concluded that screening for cystic fibrosis resulted in cost-savings compared with nonscreening; in other words, the cost of screening was less than the lifetime costs incurred by patients with cystic fibrosis.16

Two other studies performed economic evaluations of different population screening scenarios for hereditary hemochromatosis.17,36 Both studies not only concluded that certain scenarios of population-based screening can be cost-effective but also noted considerable uncertainty in interpreting the data. One study, for example, cited discounting of outcomes and adherence of patients to preventative phlebotomy as influential parameters.36 The other study concluded that cost-effectiveness was dependent on the mutation penetrance and prevalence in a population.17

Three more studies evaluated the cost-effectiveness of genetic testing for thiopurine methyltransferase variants in patients with inflammatory bowel disease.19–21 Among these studies, one concluded that pretreatment genetic screening in inflammatory bowel disease patients represents good value for money with cost-effectiveness estimates that compare favorably to other health interventions.19 In a second study, it was suggested that phenotype testing alone results in cost-savings compared with genotype testing.21 The third study compared genetic testing, metabolite monitoring, and usual care and found that genetic testing and metabolite monitoring dominate usual care.20

The cost-effectiveness of pharmacogenetic testing for smoking cessation treatment was evaluated by two other of the studies reviewed.22,23 Both studies included nicotine replacement and bupropion pharmacotherapy among the strategies evaluated. Results of one study indicated the use of genetic testing in tailoring smoking cessation treatment to be cost-effective under certain assumptions.23 By contrast, the other study cautioned against assuming that genetic tailoring of treatment would be more cost-effective than current models for pharmacotherapy.22

Six studies included in this review were CUAs, but only five of these stated ICERs. The ICERs identified in this review (2004–2009) and those from the review by Carlson et al.5 (1997–2004) were plotted to identify any trends over time (Fig. 1). The trend appears to be in an upward direction.

Fig. 1
figure 1

Trends in ICERs ($/QALY) for genetic testing interventions (1998–2009). QALY, quality-adjusted life year.

Full size image

Quality of included studies

A quality score was assigned for each article by each reviewer using the QHES method.10 The correlation between the two reviewers' scores is 0.77. The mean quality score for the reviewed studies was 89.8 with a range of 41–100. The mean quality score and range were also calculated by subgroups: type of economic analysis, model, sensitivity analysis, and perspective (Table 1). Cost-utility studies had the highest mean quality score of 96.8, and, when compared with studies that used other types of economic analysis, cost analyses received the lowest mean quality score within this subgroup. Studies using a combined decision tree and Markov model received a higher mean quality score of 94.5 relative to studies that used only a decision tree or a simulation model. Studies that did not include a sensitivity analysis received a lower mean quality score than those that did.

Although the mean quality score from this review (89.77) was 2.72 points higher than the mean quality score from the review by Carlson et al.5 (87.05), this difference is not statistically significant (P value <0.35).

The number of studies in this review that did not address specific quality criteria were also calculated (Table 3). This analysis revealed that the most frequently missed criteria was a discussion of bias (53.9%) disclosure of funding source (30.8%), and statement of study perspective (19.2%).

Table 3 Missing quality criteria in reviewed studies
Full size table

DISCUSSION

The studies included in this review of economic evaluations for genetic testing and interventions were all published between 2004 and 2009, spanned a number of clinical indications, and used a variety of modeling techniques and assumptions. A few reviews of economic evaluation of genetic tests have been published since 2004, but the focus in these articles has been on pharmacogenetic strategies.2,4–9 One review by Vegter et al.,9 for example, examined the adherence of pharmacogenetic screening economic analyses to current guidelines. This study considered four aspects of pharmacoeconomic analysis when assessing the adherence to guidelines: (1) sensitivity analysis, (2) time horizon and discounting, (3) study perspective, and (4) review of evidence. The authors found that most economic evaluations of genetic interventions lacked standardization with respect to analysis perspective, sensitivity analysis specifics, and discount rates.9 In contrast, our review on the quality of economic evaluations of genetic services incorporated a more comprehensive quantitative methodology that includes not only the four aspects considered in the study by Vegter et al.9 but also 12 other quality criteria such as bias, assumptions, limitations, etc.

The analysis of subgroup quality scores revealed a number of interesting trends. Cost-benefit and cost-consequence studies, for example, had the lowest mean quality score, when compared with CEA and CUA. This finding may reflect the fact that many QHES criteria pertain to outcomes, and CBA and CCA consider outcomes differently. As is consistent with current recommendations by economics evaluation guidelines, the studies that performed PSA had higher quality scores, when compared with studies that used other types of sensitivity analysis. It is likely that authors who heeded guideline recommendations for sensitivity analysis also followed guidelines when constructing other parts of their evaluation, which would explain the observed trend between the use of PSA and higher quality scores.

Although there was a small difference between the overall mean quality score from the period 1990–2004 to the period 2004–2009, the difference was found to be statistically insignificant, implying little change in the quality of genetic intervention-focused economic evaluations between 1990–2004 and 2004–2009. One would have predicted an improvement in quality over this period in view of the increase in published guidelines for economic evaluation. The basis for the lack of such improvement is not clear but may reflect our study's higher proportion of CBA and CCA studies (which tend to have lower quality scores) or a ceiling effect.

The most commonly missed quality criterion in our review— discussion of bias—differs from that most commonly missed in the review by Carlson et al.5—statement of perspective. This discrepancy may be due to changes in journal policy, with a statement of perspective now being mandated by many journals.

Importantly, our study has a few limitations that need to be considered in terms of interpreting our findings. One such limitation relates to the terms genetics and genomics that are very broad and include a wide field of biomedical research that is not necessarily fully captured by our review. In addition, economic evaluations of genetic interventions tend to be improperly indexed and may be hidden in databases under disease-specific terms. Circumventing this problem to ensure all studies pertaining to genetic interventions were captured would require a manual review of all economic evaluations in a plethora of disease fields. Another study limitation derives from the application of QHES grading methodology. We found some QHES criteria to be vague and difficult to apply. Some criteria are open to multiple interpretations, and many criteria include several considerations, which a single score may fail to capture. An example of such a criterion is “Did the analytic horizon allow time for all relevant and important outcomes? Were benefits and costs that went beyond 1 year discounted (3–5%) and justification given for the discount rate?”10

CONCLUSIONS

Our review of economic evaluations on genetic testing and interventions found that the studies published between 2004 and 2009 were of fairly high quality with a mean quality score of 89.8 of 100. Our findings suggest that no statistically significant improvement in quality of such evaluations occurred from the period 1995–2004 to 2004–2009. Although the mean quality score was fairly high, assessment of missing quality criteria identified a number of areas for improvement, including discussion of the direction and magnitude of potential biases. In addition, we found aspects of QHES methodology for grading the quality of economic evaluations to be ambiguous, such as the inclusion of multiple distinct issues in one criterion or the use of poorly identified terminology. Additional research is needed to modify, clarify, and optimize the QHES quality grading system.