Introduction

The rapid development of genomic technology in medicine is enhancing understanding of how genomic variation introduces heterogeneity in response to clinical and public health interventions. This knowledge presents the opportunity to incorporate genomic factors into clinical decision making; however, genomic technology translation has been limited by numerous factors, among which include a lack of prospective trial evidence comparing the health outcomes of genomic testing strategies with standard clinical care.1,2,3

Comparative effectiveness research (CER), generally defined as “systematic research comparing different interventions and strategies to…inform patients, providers, and decision makers…about which interventions are most effective for which patients under specific circumstances,” has the potential to address these evidence gaps.3 Prospective CER trials are often conducted to compare the average effects of interventions or strategies in “real-world” settings.4,5 Prospective CER trial designs can be structured to generate comparative evidence about the outcomes of genomic versus standard clinical care strategies.3 Few CER studies of this type have been conducted to date. However, given stakeholder requirements for more rigorous evidence of the relative benefits and harms of genomic technologies, prospective trials will likely play an important role in future evaluation of clinical utility (defined as “improvement in life expectancy or quality of life for patients and their families”6) and decisions about appropriate translation of candidate technologies.3

Ideally, researchers would conduct prospective CER trials to evaluate all promising genomic technologies, and those with demonstrated clinical utility could move forward toward clinical application. However, this scenario is not realistic because of the vast number of existing and developing technologies, the prohibitive costs of such studies, and the limited number of patients with targeted diseases who are willing and able to participate in trials.7,8,9 For example, a recent horizon scan identified 250 new genomic tests during the period from May 2009 to May 2010.8 An example of this reality is the SWOG RxPONDER (Rx for Positive Node Endocrine Responsive Breast Cancer) trial to investigate chemotherapy benefit in patients with node-positive breast cancer and low to intermediate Oncotype Dx recurrence scores.10 This prospective CER trial plans to screen 9,000 patients with breast cancer to recruit 4,000 participants.10 The trial is estimated to cost ~$27 million due to the substantial screening effort required to reach target accrual, high cost of the Oncotype Dx test, and considerable infrastructure requirements. Clearly, trials of this size and cost cannot be conducted to evaluate all new technologies. Consequently, stakeholders must make decisions about how to best invest genomic research resources to improve public health. This necessitates robust priority-setting frameworks that engage the producers and end users of genomic technologies and facilitate decisions about which technologies should move forward on the translational pathway.11,12

Clinical trial prioritization and decision making has traditionally been carried out by a limited subset of stakeholders, namely researchers and physicians. However, the recent momentum of CER has increased efforts to involve a more comprehensive array of stakeholder constituencies in research prioritization and decision making.13,14,15 These constituencies include clinicians, researchers, patient/consumer advocates, policy makers, funders, payers, and industry.16 An important challenge to stakeholder engagement is the involvement of individuals with diverse expertise, including those with potentially little research experience.14,15 Successful engagement requires thoughtful selection of stakeholders in relation to the project’s needs, education, and expert facilitation. This type of approach has the potential to yield more robust research evidence, but little formal understanding exists regarding what factors these diverse stakeholder groups should consider, how their distinct perspectives meld into shared understanding and/or consensus, and how these factors relate to final priority setting.9

In this study, we performed a qualitative analysis of discussions from a stakeholder workshop in which six candidate genomic technologies were priority ranked for evaluation in a prospective CER trial. The study objective was to identify criteria that a diverse group of stakeholders identified as important for the prioritization of potential cancer genomics projects and to report major themes of the stakeholder discussion. Although an important end point of a prioritization process is choosing which studies actually receive funding, these criteria need to be discussed, evaluated, and used before funding decisions can ultimately be made. The findings provide a basic framework for systematically prioritizing genomic technologies for additional evaluation and illustrate the potential impact of engaging a diverse group of stakeholders in research prioritization, which is increasingly important given the stakeholders’ central role in CER.

Materials and Methods

Setting

This study was conducted in the setting of the Center for Comparative Effectiveness Research in Cancer Genomics (CANCERGEN), a multidisciplinary consortium including the Fred Hutchinson Cancer Research Center, University of Washington, the Center for Medical Technology Policy, and Southwest Oncology Group, one of the largest of the National Cancer Institute–supported cancer clinical trial cooperative groups in the United States. Under the leadership of the Fred Hutchinson Cancer Research Center, CANCERGEN brings expertise in CER, clinical trial design, and cancer genomics to facilitate the rapid design and implementation of prospective comparative effectiveness studies of genomic technologies. The overall mission of CANCERGEN is to generate high-quality evidence regarding the clinical utility and economic value of genomic technology applications as compared with standard care. As a part of this project, we held a 1-day meeting for an external stakeholder advisory group (ESAG) to produce a priority ranking of six candidate genomic technologies in cancer for evaluation in a prospective CER trial to be carried out by SWOG.

Population

This study involved CANCERGEN’s ESAG, which consisted of 13 individuals selected to represent the perspectives of a diverse range of constituencies related to cancer genomic technologies: professional patient/consumer advocates (two; one of whom is a cancer survivor), payers (three), clinicians (three), policymakers/regulators (three), and the life sciences and diagnostic industry (two).17 One of the primary aims of CANCERGEN was to implement a stakeholder engagement process to inform the prioritization and design of CER studies in the area of cancer genomics for the evaluation in prospective CER trials to be carried out by SWOG. The project was approved by the institutional review board of the Fred Hutchinson Cancer Research Center.

Prioritization process

The prioritization activities involved ranking six candidate genomic technologies that were identified through a landscape analysis during an earlier phase of the CANCERGEN project.17 Landscape analysis is a process that involves evaluation of the literature coupled with domain-specific expert consultation.17 The study investigators selected the six candidate technologies because they believed them to have the greatest potential for clinical impact if prospective CER evidence was generated. These technologies are shown in Table 1 .

Table 1 Overview of candidate genomic technologies under consideration
Full size table

To facilitate the ranking of the genomic tests by the ESAG, we developed an initial list of priority-setting criteria on the basis of previous CER prioritization efforts by the Agency for Healthcare Research and Quality.18 These criteria were also informed by feedback from the ESAG. The final priority-setting criteria are shown in Table 2 .17 The rationale and process for selecting these criteria have been described previously.17,18

Table 2 Description of prespecified priority-setting criteria
Full size table

A modified Delphi process was used to assist the ESAG in creating a priority ranking of the genomic tests, which involved a mix of online voting and a full-day in-person meeting.19

The method enables group problem solving through an iterative process of discussion, feedback, and revisions. Four weeks before the in-person stakeholder meeting, the ESAG was provided information on each of the six candidate technologies in the form of background briefs and test target profiles ( Supplementary Appendix S1 online). These materials were intended to provide stakeholders with a basic understanding of the candidate technologies and the key factors associated with each of the nine priority-setting criteria. Study investigators ensured that materials and engagement activities were accessible to all participants. ESAG members were asked to review the materials and initially rate each genomic test according to the nine priority-setting criteria using a Web-based survey tool.

The in-person, full-day stakeholder meeting was held in Seattle, WA, on 14 June 2010. The agenda included an explanation of meeting objectives from study investigators, overview of the priority-setting process, presentation by an ethics expert of the benefits and drawbacks of using deliberative methods to prioritize research, patient perspectives presentations by patient and consumer advocate ESAG representatives, and detailed presentation and discussion of the six candidate genomic technologies facilitated by study investigators. The test target profiles were used during the meeting to inform the discussion. Workshop discussions were recorded and transcribed.

At the conclusion of the in-person meeting, ESAG members were asked to reprioritize the genomic tests and to list the priority-setting criteria that carried the most weight in their decisions. After 2 weeks, the ESAG was requested to reprioritize the tests after reading the meeting summary and using an online survey to allow the opportunity to reflect on the meeting and their rankings.

Qualitative analysis

Using the transcripts from the stakeholder meeting, a directed, qualitative content-analysis approach was used to analyze the discussion.20 Coding was initially informed by the original nine priority-setting criteria and subsequently evolved as new themes emerged from the data. To increase the validity and accuracy of the results, a qualitative research technique called analytical triangulation was used. This technique is an iterative process in which two researchers independently code the text, then compare and adjust their findings to converge to the same analysis.21,22 NVivo (QSR International, Cambridge, MA), a qualitative software program, was used to conduct the analysis.23

Results

All the 13 ESAG members completed the premeeting priority ranking of the candidate technologies based on the background briefs and test target profiles. Table 3 shows the priority ranking. These results were presented to the ESAG at the in-person meeting.

Table 3 Premeeting and postmeeting priority-ranking results
Full size table

Eight stakeholders representing four different stakeholder groups attended the 1-day meeting: patient/consumer advocates (two), clinician/researchers (three), payers (one private, one government health technology assessment), and industry (one). Five ESAG members were unable to attend because of scheduling conflicts. All the 13 ESAG members completed the postmeeting online survey.

Discussion themes by prioritization criteria

Population impact. The population impact of the candidate technologies was frequently discussed ( Table 4 ). Because the incidence of the various cancers and prevalence of the genetic variants differed greatly across candidate technologies, much of the discussion focused on how these criteria should be defined. A physician stakeholder noted the underlying issue in many of the candidate technology disease areas:

Table 4 Major discussion themes from the workshop session by priority-setting criteria
Full size table

“It’s tough to treat patients knowing you’re probably not going to benefit the majority of them…so finding markers like these would be extremely helpful.”

There was a large degree of consensus that disease incidence, variant prevalence, and potential for prolonging survival are important in prioritizing candidate technologies. Population impact was often discussed in terms of number needed to treat.

Similarly, a payer stakeholder argued that that genomic testing was more appropriate to inform use of targeted therapies like erlotinib, rather than cytotoxic agents like cisplatin in the ERCC1 testing example because “cytotoxic agents …are more of a shotgun, rather than a laser beam.” Several stakeholders noted that rare diseases should not be ruled out as potential candidates for CER studies.

Analytical/clinical validity. Stakeholders acknowledged many problems with the analytic and clinical validity evidence for the candidate technologies, which impeded translational efforts in this area. A patient representative noted the lack of robust genomic technology validation, suggesting it must be addressed before these technologies are implemented in clinical practice. Much discussion focused on laboratory-developed tests that are performed at a single institution and are not subject to review by the US Food and Drug Administration. A physician noted:

“I do have concerns…(about) a local, homegrown test and quality control, it is always very difficult, and exploiting a single institution’s technique is difficult.”

Specific concerns about analytic validity were discussed in the case of immunohistochemistry testing for ERCC1 expression in non-small-cell lung cancer (NSCLC) and fluorescence in situ hybridization testing for epidermal growth factor receptor in NSCLC. Stakeholders suggested that additional assay/test validation studies are needed before these technologies can move forward in a CER trial. In terms of clinical validity, several stakeholders considered EGFR mutation testing in NSCLC as a good candidate for a CER trial due to its strong association with survival in patients treated with erlotinib.

Potential clinical benefits. From a variety of stakeholder perspectives, the overall benefits of genomic tests were twofold: (i) to facilitate and select the most effective therapies for given patients and (ii) to reduce the time, cost, and quality of life lost to pursuing ineffective treatments. One patient stakeholder stated:

“…As the field of testing moves forward, it is becoming possible to do a better job of matching treatment to the profile of a particular patient’s disease…It increases the chance of a strong positive response to treatment but…it will also reduce the opportunity cost of time lost with an ineffective treatment.”

Other discussions focused on reducing harm, particularly in relation to EGFR mutation testing. Clinicians and payers thought this test would enrich a population of responders to a targeted therapy, making it an attractive test case for CER. A physician representative stated:

“You’ve got a situation here where even though the test is expensive, you can show that it spares 90% of the patients from the expense and the side effects of the treatment.”

Economic impacts. Economic impacts were also mentioned frequently. Stakeholders focused on the aggregate costs faced by payers as opposed to the out-of-pocket expenses of patients. They noted the importance of economic considerations given the escalating costs of new diagnostic tests and cancer therapies and subsequent impacts on insurance premiums and ability to pay for alternative treatments. A payer representative noted:

“If I could find the people [who benefit most]…there would be a lot of savings, which could then be recycled to find the right drugs for other people.”

On the basis of this rationale, stakeholders showed considerable interest in EGFR mutation testing in NSCLC due to the potential for cost savings associated with erlotinib. In addition, when discussing the economic impact of the breast cancer tumor markers, stakeholders were concerned with the possibility of prescribing of tests that might not provide any clinical benefit. In this context, many stakeholders saw a CER trial as an opportunity to potentially highlight an ineffective technology. In addition, discussions about gene expression profiling in multiple myeloma highlighted the specific challenge in rare disease that therapy is often very expensive, underscoring the need for predictive markers to target therapy.

Clinical trial implementation and feasibility. A major discussion theme under this criterion was whether genomic test results could be withheld from trial participants as part of the randomized study design. Stakeholders felt this could complicate the process of receiving institutional review board approval, lead to suboptimal treatment based on current standards of care, and limit patient willingness to participate in a trial.

Considerable discussion took place regarding the feasibility of various trials based on the design, required number of patient screenings, and required duration of follow-up. For example, many stakeholders thought a trial on ERCC1 expression testing was desirable due to the high prevalence of ERCC1-positive patients and rapid progression of NSCLC, which would facilitate achievement of trial end points in a relatively short period of time. Alternatively, several stakeholders noted that a breast cancer tumor marker study would need to be very large, long, and resource intensive given the slower progression of early-stage breast cancer. With respect to EGFR mutation testing, stakeholders viewed the targeted nature of erlotinib presenting additional complexities to study design. The low prevalence of the EGFR mutation (~10% of the NSCLC population) could present significant enrollment issues.

“I don’t like the design of including all patients because fortunately or unfortunately, we have a population that we know tends to be selected for the mutation.”

One clinician stated that physicians would not be willing to enroll patients into a study in which they might not be randomized into an erlotinib treatment group even though they might be part of the 10% that would most likely benefit from the therapy.

Market factors. Stakeholders discussed the high level of evidence that would be required to delist the breast cancer tumor markers. The political challenges of delisting these markers were emphasized as well, citing the political backlash with the US Preventive Services Task Force mammogram recommendations.

“…Realistically, payers can’t make change until they have some legitimate study backing, especially when you are talking about things that have already disseminated and have a lot of emotional influence such as this.”

From one payer’s perspective, the low clinical validity of the test paired with high dissemination was one impetus for prioritizing the test for a CER study. On the basis of the stakeholder discussions, we subsequently revised the definition of “market factors” to “there is a high degree of pressure on payers to cover this technology (or continue to cover this technology), even with limited evidence.”

Potential clinical harms. Potential clinical harms were infrequently mentioned. Several brief discussions occurred regarding the potential for negative psychological impacts related to anxiety about test results and the harms that could result from treatments given on the basis of false-positive or false-negative genomic test results. Much of this discussion was focused on breast cancer tumor markers. Given the low levels of evidence, poor performance of some markers (e.g., CEA 15-3 marker sensitivity, 36%), and the potential for widespread use, the ESAG suggested that there was significant potential for harm.

Evidence of need for a randomized controlled trial. Since the initial list of candidate tests was limited to tests that appeared to have significant evidence gaps, little explicit discussion took place regarding the evidence of need for a randomized controlled trial (RCT). Stakeholders generally acknowledged that every study under consideration had an insufficient evidence base to support clinical implementation. Much of this discussion was interrelated with considerations of clinical and analytic validity, which was reported above.

New priority-setting criteria identified

Several new criteria emerged from the qualitative analysis and did not directly fall under any of the prespecified priority-setting criteria ( Table 5 ).

Table 5 New criteria identified from the workshop session
Full size table

Patient-reported outcomes. First, there was considerable discussion about patient quality of life, and the potential role of patient-reported outcomes, particularly in relation to breast cancer tumor markers, for which several stakeholders suggested that patient concerns about recurrence and their desire for information drove the demand for testing. In fact, many stakeholders thought that the psychological impact of tumor marker testing and results was a compelling research area in itself. As a patient representative stated:

“When we’re looking at the benefits of these tests…how the tests impact choices and decision making in care for both the patient and the provider…you really need to look at the decision-making process for the individual…and weigh that…in the incorporation of these tests.”

Recruitment. The topic of trial recruitment was often discussed as well. Stakeholders noted the potential difficulty of conducting CER trials in NSCLC because many of the potential participants are recruited to competing industry-sponsored trials. In addition, stakeholders thought recruitment was a key issue in fluorescence in situ hybridization testing for epidermal growth factor receptor in NSCLC, arguing that the small target population would make enrollment quite slow.

Clinical trial ethics. Clinical trial ethics also emerged as a key theme. Stakeholders challenged the ethics of designing an ERCC1 expression testing study with a no treatment arm, particularly if enrolling stage II NSCLC patients who would receive treatment according to current standard of care.

“…Nobody will put a stage II patient on study that’s not getting chemotherapy.”

Clinician representatives agreed that it would be difficult to obtain the institutional review board approval and to find stage II patients willing to be randomized in a study in which they might not receive chemotherapy (the current standard of care).

Final priority-ranking of candidate genomic technologies

On completion of the workshop, stakeholders identified and ranked the three highest priority candidate technologies ( Table 3 ), on the basis of the rationale below:

  1. 1

    ERCC1 testing: high population impact, high clinical validity, ability to quickly obtain end points in a trial due to rapid disease progression.

  2. 2

    EGFR mutation testing: relatively high population impact, high clinical validity, opportunity costs are large as erlotinib is expensive and has significant harms.

  3. 3

    Breast cancer tumor markers: high population impact, high degree of dissemination into current clinical practice, low clinical validity, opportunity to curb inappropriate use of markers and realize savings.

Discussion

Overview

We evaluated a stakeholder discussion to prioritize six cancer genomic technologies for prospective CER trials. The in-person deliberations led to a revision of the priority ranking, suggesting that in-depth face-to-face deliberation can impact stakeholder priority rankings.

Of the nine original priority-setting criteria, six were the focus: clinical benefits, population health impacts, economic impacts, analytical and clinical validity, clinical trial implementation and feasibility, and market factors. Several criteria were rarely mentioned, with the potential harms of tests most notably absent. Failing to discuss the potential harms could lead to the underestimation of the impact of false positives and negatives of a genomic test on clinical practice and patient outcomes. We also identified three new themes from the workshop discussion: patient-reported outcomes, clinical trial ethics, and trial recruitment. The discussion about patient-reported outcomes highlighted the importance of the impact of genomic testing on quality of life and the selection of trial end points. By considering patient-reported outcomes, potential study designs could be broadened beyond RCTs to address, for example, behavior-related research questions. The theme of clinical trial ethics focused on the ethics of withholding the results of relatively established genomic tests in CER trials and how this might have an impact on patient and physician participation. Finally, the theme of trial recruitment emerged from concerns about adequate accrual given competition from industry-sponsored trials and finding patients with rare genomic variants.

Implications

On the basis of this study, future priority-setting processes may consider these nine prespecified and three new criteria into an explicit framework. Although the inclusion of our new priority-setting criteria can enhance the ability of our prespecified framework to facilitate informed decisions about CER trial priorities, future efforts should be made to identify additional priority-setting criteria that enhance stakeholders’ ability to prioritize research investments.

Comparison with previous findings

Few accounts of cancer genomics research prioritization are in the literature to date. The Evaluation of Genomic Applications in Practice and Prevention (EGAPP) program of the Centers for Disease Control and Prevention and Food and Drug Administration is the most notable example but differs from this study in its objective in key ways.9,24 Primarily, this study sought to prioritize candidate genomic technologies for evaluation in a CER trial, whereas the EGAPP effort sought to prioritize genomic technologies for evidence reviews.24 Although overlap exists in the priority-setting criteria involved in each, the difference in objective renders different criteria more relevant in each context. In addition, the EGAPP prioritization primarily involves academically oriented stakeholders, whereas this study involved a diverse array of genomic technology end users, including patients, payers, and technology developers.24

Limitations

This study had limitations that are worth noting. First, the perspectives reported herein are not representative of all potential stakeholders in CER of cancer genomic technologies; however, stakeholders were purposely selected to best help the investigators obtain in-depth understanding of the issues that must be considered when prioritizing and designing CER studies in this area.17 Obtaining in-depth feedback and a high level of stakeholder engagement, which requires significant interaction among participants, would have been difficult to achieve with a larger group. Second, despite their participation through online voting, the 5 of the 13 ESAG members who could not attend in-person meeting did not have a chance to influence the deliberations, which may have had an impact on the final ranking. Still, given the multiple stakeholder groups already represented during the meeting, we believe that the goal of stakeholder engagement—to seek diverse perspectives to help make research more relevant for its end users—was achieved. To see if the results are similar or different, this process can be replicated with other stakeholders. Furthermore, to generalize these findings more broadly, the framework could be tested on a larger sample through alternative methods such as large-scale surveys. Finally, because we started with a preexisting list of criteria, and presented information about those criteria, we may have influenced and/or restricted the range of factors that stakeholders considered and discussed.

The process of prioritizing genomic technologies for future CER can be an extremely complex task to perform. Our findings and revised priority-setting framework can serve as a guide to researchers and stakeholders considering the use of priority-setting frameworks to guide research investments. Although this study specifically examined cancer genomic technologies, this type of framework can also be adapted to other disease areas. Ultimately, this type of approach can potentially assist decision makers in systematically allocating limited resources to CER studies with the intent to maximize public health benefit.

Disclosure

The authors declare no conflict of interest.