Introduction

Exome sequencing (ES) is highly effective at identifying the specific etiology of disease in heterogeneous pediatric patient populations with suspected genetic disorders.1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18 While it is usually performed as a last-resort diagnostic test due to its comparatively high cost, several recent studies suggest that when ES is performed as a first-line test, the additional investigations avoided for diagnosed patients may fully offset the cost of the test.13,19,20,21 However, as we discuss below, these cost-effectiveness estimates may reflect atypical diagnostic yields and are highly sensitive to the cost of delivering ES. For example, one study found that offering first-line ES would either save US$10.1 million for an insurance plan with 1 million members or cost an additional US$7.5 million, depending on which clinical laboratory’s estimate for the cost of delivering ES is used.19 For last-resort ES, averted testing is arguably a less relevant consideration because most alternative investigations will already have been completed, so the cost-effectiveness of a last-resort ES service is likely to be even more sensitive to the diagnostic yield and the cost of delivering the service.

One way of simultaneously comparing the way in which the cost and yield of different ES services vary is by using the cost per positive diagnosis (CPPD), which is calculated by dividing the per-patient cost of delivering an ES service by its diagnostic yield. A number of recently published ES studies have reported both cost and yield, with the associated CPPDs varying widely from C$5,696 (ref.22) to C$37,469 (ref.23) in 2016 Canadian dollars. Assuming these figures approximate the incremental cost-effectiveness of last-resort ES, the former service would have a much higher probability of being deemed cost-effective than the latter, given societal willingness to pay for an additional genetic diagnosis in Canada (one study estimated a willingness to pay of C$14,665 in 2016 dollars).24 A key question, then, when assessing these types of reports, as well as formal cost-effectiveness analyses for diagnostic ES services, is whether the cost and diagnostic yield of ES included in the analysis are representative of the typical experience for a diagnostic ES service.

To facilitate interpretation of the broader relevance and generalizability of existing and future cost-effectiveness analyses for diagnostic ES services as well as individual reports describing the delivery of diagnostic ES services, we conducted a benchmarking study. We used data extracted from a set of studies reporting on last-resort diagnostic ES performed on heterogeneous pediatric patient cohorts to generate benchmark estimates for the cost (from a health-care provider perspective) of delivering a last-resort ES service, its yield, and its CPPD, as well as to characterize the variability of these performance indicators. In addition, we explored the potential drivers of this variability by reporting on a proposed delivery model for a diagnostic ES service developed in a clinical research setting in British Columbia, Canada, and considering hypothetical scenarios in which we estimate the impact of changes in the design of the service on cost, yield, and CPPD.

Materials and methods

Benchmarking study

To create benchmarks for the cost, yield, and CPPD of diagnostic ES services, we conducted a review of the literature to identify studies published between 2013 and 2017 that reported on (i) the diagnostic yield of ES for majority pediatric patient cohorts with heterogeneous clinical presentations and (ii) the cost of delivering a diagnostic ES service (see the Supplementary Materials and Methods online for a detailed discussion of our search strategy and inclusion criteria). We extracted reported diagnostic yields, ES strategy, and sample characteristics, and calculated the simple average and standard deviation of the reported yields. Reported costs for ES were extracted from all studies in which they were reported, and converted to 2016 Canadian dollars using exchange rates drawn from the Bank of Canada’s Year Average of Exchange Rates for 2016 and consumer price inflation rates for each country. For studies that only reported the per-sample cost of singleton ES, estimated trio ES costs in the same context were calculated by subtracting costs that are only accrued once independent of the number of family members sequenced (e.g., interpretation, counseling, and consultation costs) from the per-sample cost, multiplying the remaining quantity by three, and adding the one-time costs back in.

To generate estimates for the benchmark cost, yield, and CPPD of diagnostic ES, we performed Monte Carlo simulations. Following standard practice in health economics, the diagnostic yield for each type of ES service was assumed to follow a beta distribution and the cost was modeled as a log-normal distribution.25 We fit the reported costs for solo and trio ES to a log-normal distribution using log-linear regression to estimate the parameters for the log-normal cost distributions. We conducted 10,000 runs of the simulation separately for solo and trio exome sequencing, taking independent random draws from both the cost and yield distributions, and dividing the cost by the yield to generate a CPPD estimate. The simulations were performed using R (3.1.1). We used the means of the resulting distributions as the benchmark values, and their 2.5th and 97.5th percentiles as the bounds of the 95% confidence intervals (CI).

CAUSES study cost model

The Clinical Assessment of the Utility of Sequencing and Evaluation as a Service (CAUSES) study is a 3-year translational research initiative at BC Children’s & Women’s Hospitals in Vancouver, British Columbia to perform and assess the clinical value of genome-wide (exome or whole-genome) sequencing on 500 pediatric patients with suspected genetic disorders. Figure 1 illustrates the workflow of the CAUSES study. Notably, a clinical genomic consultation service (GCS; step 1) involving a committee of clinical geneticists, a pediatric subspecialist, genetic counselors, and a molecular geneticist was established to review the appropriateness of ES for patients referred to CAUSES by their managing physicians (who were located mainly at BC Children’s & Women’s Hospitals, but also in the community), with the goal of maximizing the efficient use of resources. For patients for whom ES was deemed to be indicated, an initial visit (step 2) was scheduled with patients and their parents, during which a clinical geneticist and genetic counselor discussed the research study, provided pretest genetic counseling, and obtained informed consent for enrollment.

Figure 1: The eight steps involved in the CAUSES study’s diagnostic ES service workflow, with figures indicating the actual number of families who were processed through each step of the workflow in 2016.
figure 1

For costing purposes, 5a and 5b are aggregated into a single bioinformatics analysis step (step 5). Note that the change from 146 families in step 6 to 115 families in step 8 reflects the substantial time period required to make a final determination and communicate it to families, not attrition in the number of participants. Similarly, the reduction in the number of families between steps 4 and 5a does not represent attrition, but rather indicates that the bioinformatics analyses for these families were not completed in 2016. CAUSES, Clinical Assessment of the Utility of Sequencing and Evaluation as a Service study.

Full size image

Subsequent to consent, patients and their parents provided blood samples (step 3). DNA was extracted by the BC Children’s Hospital clinical laboratory and sent for research-based sequencing by an external vendor (step 4) that provided raw data as FASTQ files to the study’s bioinformatics team. A genomic analyst generated a candidate variant list using a custom bioinformatics pipeline that focused on known disease genes (step 5a),26,27 which was then reviewed by the study’s expert panel composed of clinical geneticists, a pediatric subspecialist, a molecular geneticist, genetic counselors, a genomic analyst, and the referring physician (step 6). Any variants thought to contribute to a definite or probable diagnosis (and occasional highly relevant variants of uncertain causality) were clinically confirmed by Sanger sequencing, interpreted by the clinical molecular diagnostic laboratory at BC Children’s & Women’s Hospitals according to American College of Medical Genetics and Genomics criteria,28 and submitted to the patient’s electronic medical record (step 7) before being discussed with participants. Results were discussed with families either in a face-to-face or telehealth meeting with the treating physician and study genetic counselor (if a definite or probable causal variant was clinically confirmed) or in a telephone conversation with the genetic counselor (for null findings) (step 8). For cases in which no causal variants were identified, a bioinformatics re-analysis of the sequence data was performed every 6 to 12 months until the end of the study (step 5b). Note that for costing purposes, inputs into steps 5a and 5b were merged into a single “Bioinformatics” category.

We generated an estimate of the per-patient cost of delivering the CAUSES diagnostic ES service using data from all patients evaluated in 2016, the study’s first full year of operation. The cost model allocated some resources at the patient level (e.g., time spent reviewing a patient’s chart in the GCS) and others at the aggregate or overhead level (e.g., information technology infrastructure) (Supplementary Tables S1 and S2 online). To calculate a per-patient cost for each step in the CAUSES workflow (Figure 1), we summed the cost of all resources committed to a particular step and added a share of the total overhead costs based on the number of patients who had been processed through that step in 2016. We then divided the total cost for each step by the number of patients who completed that step to obtain an average per-patient cost for each step in the workflow (Supplementary Table S3 online, column 7). Finally, to calculate the cost of processing one patient through the entire CAUSES workflow, we summed the average per-patient cost for each stage, weighting the GCS (step 1) cost by the number of patients who would need to be reviewed in the GCS to result in one initial clinic visit (step 2) and the Sanger clinical confirmation and interpretation (step 7) cost by the proportion of families that had a suspected causal variant sent for validation.

For the purposes of calculating the diagnostic yield of the CAUSES ES service, we counted only one index patient per family (a small number of families had two or more siblings with similar presentations). In the interpretation stage (step 6), the CAUSES expert panel classified all candidate variants by consensus as definitely causal of the index patient’s condition (Definite), probably causal (Probable), having uncertain significance (Uncertain), or not causal (Negative). Patients for whom at least one Definite or Probable variant was identified and subsequently confirmed by Sanger sequencing were classified as having received a molecular diagnosis according to the CAUSES research protocol.

Alternative scenarios

In addition to the base model, which describes the CAUSES study as it currently operates, we estimated two alternative scenarios to illustrate how different designs of a diagnostic ES service in the same clinical environment might influence cost, yield, and CPPD. In scenario 1 (no GCS), the GCS, which is used to screen referrals for appropriateness, was eliminated. In scenario 2, singleton ES was performed instead of trio ES. We estimated the diagnostic yield in these scenarios by estimating a reduced yield for patients deemed inappropriate for ES by the GCS (scenario 1) and by adjusting the actual yield by the average of the singleton/trio yield ratios in two studies that use both approaches9,15 (scenario 2).

For scenario 1, we would expect to have provided the CAUSES ES service to an additional 53 families who were referred to CAUSES but deemed inappropriate for ES. The most frequent reason for a patient being deemed inappropriate for CAUSES by the GCS was if the patient had nonsyndromic autism spectrum disorder without intellectual disability. We assumed that the diagnostic yield for this group of patients would have been 6.45% (6/93), based on Tammimies et al.’s29 reported diagnostic yield for an autism spectrum disorder cohort with average IQ in the normal range and a negative chromosome microarray. The second most frequent reason for a patient being deemed inappropriate for ES by the GCS was that an appropriate clinical gene panel was available at a lower cost. As an alternative estimate for the hypothetical diagnostic yield of CAUSES without the GCS, we assume that half the excluded sample would have had a yield of 6.45% but the other half (those for whom a gene panel was indicated) would have had the same diagnostic yield as for actual CAUSES patients.

For scenario 2, we estimated a yield for singleton ES of 0.66 times the actual CAUSES yield. The 0.66 factor was estimated by averaging the singleton/trio yield ratios reported by Farwell et al.9 (20.6%/37.3% = 0.552) and Retterer et al.15 (23.6%/31.0% = 0.761). For this scenario, we assumed that exome sequencing was performed only on the proband and also assumed that the time spent on bioinformatics analysis per patient would triple and that the average number of single-nucleotide variants submitted for Sanger confirmation for families for whom a possible variant was detected would double, due to the larger number of possible de novo or biallelic variants that would have to be followed up in a singleton analysis. These assumptions were based on estimates provided by our bioinformatics team members. The estimate for the increase in the number of Sanger-sequenced single-nucleotide variants relative to the actual trio ES scenario is meant to be a conservative estimate of the degree to which the use of Sanger sequencing might increase in a singleton ES scenario.

As a sensitivity analysis, we also estimated the cost and CPPD of both the base case and the two hypothetical scenarios when yields were varied by 10% in either direction.

Results

The literature search yielded a total of 18 studies that reported diagnostic yields in heterogeneous, primarily pediatric patient cohorts: 2 studies that included both singleton and trio ES, 5 with only singleton ES, and 11 with only trio ES (Table 1). Most studies (12/18) were based in the United States, but Australian, Canadian, and European patient cohorts were also included. The proportion of patients with developmental delay or intellectual disability varied widely (from 25% to 100%), but 11/13 studies for which data was available reported that more than 50% of patients had developmental delay/intellectual disability. A majority of patients were 18 years old or younger in all 18 studies.

Table 1 Recent studies reporting a diagnostic yield for ES in heterogeneous pediatric patient populations
Full size table

We also identified 10 studies that reported 12 estimates of the cost of delivering a diagnostic ES service (Table 2). Column 1 (per-sample cost) and column 3 (per-trio cost) in Table 2 provide the raw cost estimates reported in the studies, depending on whether the study in question performed singleton or trio ES. For singleton ES studies, column 3 also includes an estimate of what trio ES would cost in the same context, calculated by separating out one-time costs and tripling sequencing costs. Note that a detailed breakdown of costs by one-time versus per-sample costs was not available for all studies, so the cost estimates for trio ES may be somewhat inflated. All costs were converted to 2016 Canadian dollars (see Table 2 footnote).

Table 2 Recent studies reporting the cost of delivering a diagnostic ES service
Full size table

The average of the diagnostic yields reported in the studies identified by our review was 34.4% (SD = 5.9%, n = 13) for trio ES and 26.6% (SD = 8.0%, n = 7) for singleton ES (Table 1).1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18 The mean estimated cost of delivering a diagnostic ES service was $6,720 per patient (SD = $2,080, n = 12) for trio ES and $2,726 (SD = $974, n = 10) for singleton ES (Table 2).11,13,18,19,22,23,30,31,32,33 The simulation-based benchmark estimates for diagnostic trio ES services were $6,437 (95% CI: $5,305–$7,704) for the cost of delivery, 34.3% (95% CI: 23.2–46.5) for yield, and $19,340 for CPPD (95% CI: $13,024–$28,640). For singleton ES, the benchmark estimates were $2,576 (95% CI: $1,993–$3,270) for the cost of delivery, 26.5% (95% CI: 12.9–42.9) for yield, and $10,700 for CPPD (95% CI: $5,628– $20,628). Notably, the CPPDs ($18,203 (ref.11) and $20,043 (ref.18) for trio ES and $7,112 (ref.13) for singleton ES) for the three studies in Table 1 that also reported a cost of service delivery fall within the benchmark confidence intervals generated by the simulations, which lends face validity to our benchmark estimates.

The base case cost model (Table 3) resulted in a total average cost of $6,138 per patient (or equivalently, per-family, since only one index patient is assumed per family) to deliver the CAUSES study’s ES service model (including pretest screening and consultations, laboratory, bioinformatics, and interpretation costs, and posttest counseling). The CAUSES study’s diagnostic yield of 42.6% (49/115) in 2016 resulted in a CPPD of $14,405, which lies at the 8th percentile of the benchmark trio ES CPPD distribution. As expected, the cost of exome sequencing (step 4) was the most expensive component of the CAUSES study, representing 36% of the total cost, but the bioinformatics analysis and the GCS each represented about 15% of the total cost. Interpretation of the annotated variant list by the expert panel accounted for 10% of the total cost.

Table 3 Per-patient cost, diagnostic yield, and CPPD of the CAUSES diagnostic ES service
Full size table

Table 3 also shows the results of the two alternative scenarios. Note that the estimated cost of delivery varies both directly as a result of the design and indirectly as a result of the estimated diagnostic yield (i.e., if more causal variants are identified, more families’ results are subject to Sanger confirmation and clinical interpretation, which increases total costs). Scenario 1 reduced labor inputs by highly qualified personnel, resulting in lower per-patient costs, but this did not make up for the estimated reduction in yield (34.0% in scenario 1 versus 42.6% in the base case), resulting in a somewhat higher CPPD ($15,495 for scenario 1 versus $14,405 for the base case). For scenario 2, the overall cost per patient was also lower, although it is worth noting that the two-thirds drop in sequencing costs was partly offset by the increased time that would be spent on the bioinformatics analysis and the expected increase in the number of Sanger validations required to recognize and rule out possible de novo variants. Again, the reduced estimated yield (28.1% versus 42.6% in the base case) in the singleton ES scenario resulted in a higher CPPD ($18,223) than in the base case.

Notably, the sensitivity analysis for yield suggests that singleton ES (scenario 2) will have a higher CPPD than the base case even when comparing the best-case yield for singleton ES (30.9%) with the worst-case yield for the base case (38.3%). For the no GCS scenario, however, CPPD was approximately equal to the base case CPPD estimate when the yield for scenario 1 was increased by 10%.

Discussion

Our benchmark estimates help to contextualize the results of recently published cost-effectiveness studies of diagnostic ES. For example, Stark et al.13 and Tan et al.31 find that offering ES as a first-line test for children with suspected Mendelian disorders would result in significant cost savings to the health-care system by averting the costs of less effective conventional tests. While these studies have other limitations (e.g., they do not appear to account for the possibility of ongoing investigations following negative first-line ES), the estimated cost savings are partly driven by very high diagnostic yields (44.4%13 and 52%31) that lie beyond the upper bound of the CI surrounding our benchmark yield estimate for last-resort singleton ES (26.5% (12.9 to 42.9)). These high yields may in part reflect the stringent patient selection criteria used in these studies (in fact, Tan et al. excluded patients who had previously had single-gene or gene panel tests, which is why this study was not included in Table 1), but our results nevertheless suggest the need for caution when making general inferences about the cost-effectiveness of ES services based on this type of study.

Our literature review found significant variation in the estimated cost of ES, with trio ES costs ranging from C$3,218 to C$10,866 (Table 2). This likely results in part from the decline in the cost of sequencing over time,34,35 cross-national price differences, and measurement error, but also reflects differences in which inputs were accounted for. For example, only 4 of 10 studies included the cost of consultations with clinicians and genetic counseling in their cost estimates, even though they are fundamental elements of most diagnostic ES service delivery models. Our costing model for the CAUSES study did attempt to capture all relevant costs (including referral screening, genetic counseling, physician consultations, and team-based interpretation costs). However, some unit costs (e.g., the genomic analyst’s salary, the sequencing cost) were provided on a research basis and might be substantially higher in a clinical implementation of the CAUSES service. Other costs would likely be lower in a clinical implementation (e.g., higher volumes could increase efficiency), so it is difficult to assess what the impact on the overall cost of providing this service on a purely clinical basis would be.

In addition, the implementation of each ES service component is likely to drive the observed variability in reported costs. For example, 5/18 studies in Table 1 indicated that each patient’s variant list was reviewed by a multidisciplinary panel of experts. As the CAUSES cost model indicates, this approach to variant review can add substantial labor costs to an ES service—it accounts for about 10% of the total cost for CAUSES, and would account for a greater percentage of a singleton ES service’s total cost. This is reflected by the cost estimates for the three laboratories studied by Sabatini et al.19 in their evaluation of ES for children with neurodevelopmental disorders (Table 2). These ranged from C$2,016 to C$4,556 (for singleton ES), and depending on which cost was used, their analysis indicated that offering first-line ES would either be cost-saving or have an incremental cost. The first estimate is near the lower bound of the confidence interval for our singleton ES cost benchmark (C$1,993 to C$3,270), while the second is far beyond the upper bound, and as Sabatini et al.19 note, “Cost variability was observed for the technical sequencing and variant interpretation even with the same platform and method and appeared to be related to the extent of subsequent analysis (e.g., through group review and interpretation versus individual laboratory director review and sign out)” (p. 324) (though as we explore in our scenario models for CAUSES, group interpretation may result in a greater diagnostic rate and potentially reduce the CPPD).

Taken together, these results illustrate the importance of comprehensively costing an ES service (including labor costs for genetic counseling, physician consultations, and team interpretation) when conducting an evaluation, and suggest the need for the development of standardized criteria to be used in costing the delivery of ES services to supplement ongoing efforts to standardize the reporting of diagnostic yield.36 This would promote the comparability of costs reported by different studies and facilitate future cost-effectiveness analyses of clinical applications of exome and whole-genome sequencing.

That said, the rapid pace of progress in genomics makes the cost performance of ES a moving target, which introduces significant uncertainty into evaluations of diagnostic ES services and may limit the applicability of our benchmark estimates. Notably, the cost of sequencing a human genome dropped from about US$29,000 in 2010 to approximately US$1,200 in 2015,35 a decline that has driven down the overall cost of clinical applications of genome-wide sequencing.34 In fact, the cost of exome sequencing for the CAUSES study declined by about 25% during the first 2 years of the study. If this trend continues over the coming years, the gap in benchmark cost and CPPD between trio and solo ES will likely narrow, since the difference in cost between the two approaches to ES would be reduced. Second, it is important to note that the costs and the downstream effects on health-care utilization of returning incidental or secondary findings (including cascade testing) are not included in our cost estimates (nor in the existing economic evaluations of ES, which do account for the cost of other diagnostic investigations13,18,19,21,22). Only about 4% of families in the CAUSES study have received incidental findings, so including the cost of reporting incidental findings would probably only marginally change the reported cost estimate for delivering the CAUSES study ES service, but if the number of reportable variants increases over time, both the number of incidental findings per patient and the proportion of patients who opt to receive incidental findings may increase, in which case it will be crucial to include these costs in economic evaluations of ES. Finally, the diagnostic yield of ES may improve over time as new disease genes and pathogenic variants are discovered, though the studies in Table 1 are too heterogeneous and not numerous enough to test this hypothesis.

In conclusion, even in situations in which the cost of averted testing is taken into account, the cost-effectiveness of specific diagnostic ES services will likely depend heavily on the cost of service delivery and on the diagnostic yield. As our results show, these two parameters are highly variable. The benchmark estimates presented in this study provide a snapshot of the current range of costs, yield, and CPPD reported by diagnostic ES services in a diverse set of countries and clinical contexts and can be used to evaluate the broader applicability of current and future cost-effectiveness estimates for diagnostic ES.