Introduction

The National Academies of Sciences, Engineering, and Medicine’s (the Academies) Roundtable on Genomics and Precision Health formed the Genomics and Population Health Action Collaborative (GPHAC) in 2015, as an ad hoc activity.1 Key stakeholders at the state and federal levels, researchers, patient advocates, and clinical practitioners with expertise in public/population health, health disparities, health literacy, implementation science, medical genetics, and patient advocacy came together to work toward the effective and timely integration of genomics into existing public health programs.

This publication describes one activity involving a subset of the Action Collaborative members to develop potential performance objectives and outcome measurements for successful implementation of genomic activities within public health practice. These will become part of a genomics public health toolkit available online, and are intended to stimulate further dialogue among public health, population health, and academic researchers to determine feasibility and refine with baseline and target measures.

Organizing framework for genomic implementation

The implementation of evidence-based genomic services requires identification of outcome measurements at the level of implementation, in addition to those at the system and client levels. In 2009, Enola Proctor and colleagues,2 primarily working in mental health services research, published a paper laying out a framework for the emerging field of implementation research. The anchoring framework, reproduced in Figure 1, explained the distinction between a focus on interventions at the individual level and implementation strategies employed to get those interventions incorporated into service systems.

Figure 1
figure 1

The implementation framework developed by Proctor et al.2

Full size image

The framework recognizes that in order to ultimately achieve successful client outcomes (e.g., improved health status, functioning, and satisfaction with care) at a population level, one must first ensure that evidence-based health interventions are successfully implemented. In turn, these actions will improve the performance of the health system. Health system improvement, here summarized by pursuit of the Institute of Medicine’s standards of care,3 cascades from the achievement of key implementation outcomes, such as feasibility, fidelity, penetration, sustainability, uptake, and costs.

If we apply this framework to genomic services, we define the “interventions” as evidence-based approaches to the delivery of genetic/genomic screening at a population level. Examples of these include (i) genetic testing for Lynch syndrome (LS), a hereditary colon cancer syndrome that is also associated with other early-onset malignancies, for all individuals with newly diagnosed colorectal cancer (CRC) recommended by the Evaluation of Genomic Applications in Practice and Prevention (EGAPP) Working Group,4 and (ii) screening to identify family history associated with hereditary breast and ovarian cancer (HBOC) syndrome, with the majority of cases caused by mutations in BRCA1 or BRCA2, as recommended by the US Preventive Services Task Force.5 Furthermore, we define the implementation strategies as the efforts to embed screening within existing health-care services (e.g., provider training, referral to certified genetic counseling, ordering of tests), and public health efforts to monitor and evaluate those services (e.g., monitoring the frequency of tests, provider awareness, consumer awareness). The set of outcomes in the framework could then be used to establish baseline, realistic targets and ultimately to record progress and population health impact of genomics.

Materials and methods

The eight GPHAC members who were actively involved in working on outcome metrics represented varied perspectives including patient advocacy, professional organizations, state health departments, and federal agencies. The members focused on genomic applications specific to HBOC and LS, recognizing that these examples may be adapted to other genetic conditions. The initial proposal was to research and summarize existing metrics (process, performance, and outcomes measures), identify broad domains for outcome metrics, and then document the findings. Searching was conducted by members individually assigned to review (i) Healthy People 2020 objectives,6 (ii) National Cancer Institute–funded grants, (iii) Centers for Disease Control and Prevention (CDC)–funded grants, (iv) peer-reviewed literature, and (v) Web-based searches using key words (i.e., outcome measures, HBOC, LS, implementation, genetic testing).

During the literature-review phase, articles presenting potential frameworks for guidance emerged, including the paper by Proctor et al.2 described above and one specific to genetic services outcomes, by Silvey et al.7 We developed a modified framework for state public health genomic program performance objectives, and defined outcome measures specific to public health genomics activities, with a major emphasis on HBOC and LS. In our modified framework, outcomes were organized per the three broad outcome categories in the framework proposed by Proctor et al. (i.e., implementation outcomes, system level outcomes, client outcomes), three additional outcome categories—“access to services,” “health-care performance,” and “public health infrastructure,” based on the article by Silvey et al.7—and the 10 Essential Public Health Services.8 “Access to services” identifies public health–specific measures related to efforts to embed and monitor screening and testing within health-care systems. “Health-care performance” focuses on the uptake of health providers or health systems implementing evidence-based and recommended health services. The third category, “public health infrastructure,” spotlights processes essential to the successes of the implementing public health agency. Because our task was to focus specifically on prevention efforts concerning HBOC and LS, two additional objectives were added to the framework—expanding universal screening for LS for all CRC tumors. One focused on LS tumor screening adoption is considered a measure states can readily perform. The other is focused on LS tumor screening reach and is viewed as aspirational.

Through an iterative process, members defined performance objectives and identified relevant outcome measures and possible data sources. We also discussed the likelihood that states or academic researchers would readily have access to data sources such as the Behavioral Risk Factor Surveillance System (BRFSS),9 state-operated medical claims databases, or other service-utilization data. We then posited a state genomics program’s potential “readiness” or capacity for measuring each specific outcome. Through consensus, we categorized each outcome according to capacity and identified top-priority outcomes, with any disagreements resolved by discussion. The categories are as follows:

  1. 1.

    Top-priority outcomes that all states are encouraged to pursue

  2. 2.

    Outcomes that states should pursue

  3. 3.

    Outcomes that states should pursue if data sources are available

  4. 4.

    Aspirational outcomes

As the outcome measures were formulated, discussion followed regarding health disparities. Although specific measures for health disparities were identified in the “equity” outcome category, it was recognized that for other outcome measures, specific health-disparity measures could be developed.

Results

In the preliminary research to identify existing measures, we found that the majority were process measures rather than outcome measures. Only two measures had existing baseline data and targets, both part of Healthy People 2020, including increases in the proportion of (i) women with a family history of HBOC/LS who receive genetic counseling and (ii) persons with newly diagnosed CRC who receive genetic testing to identify LS (or other familial CRC syndromes). Therefore, using the framework described above, additional outcomes measurement included outlining multiple potential metrics and prioritizing each. These are provided in Table 1.

Table 1 Performance objectives, state outcome measures, and their data sources for genomic services
Full size table

Overall, we identified 38 outcome measures, each with a specified performance objective. Eighteen of these measures could be applicable to more general genomic applications, including multiple cancer genetic tests. Twelve were specific to both HBOC and LS, seven specific only to LS, and one specific only to HBOC.

Three of the performance objectives and associated outcome measures were considered priority items that all states are encouraged to pursue at present. These objectives include increases in (i) the proportion of persons with newly diagnosed CRC who receive genetic testing to identify LS, (ii) the proportion of women with a family history of HBOC/LS who receive genetic counseling, and (iii) the use of cascade screening for HBOC/LS. Sixteen additional outcomes were considered measures that state programs could currently capture but were not identified as a priority. There were 2 service-related outcome measures that states could readily perform if data sources are available, and 17 measures that were considered aspirational because no obvious data source are currently available for them. Data sources for 10 outcome measures included survey data, including surveys of individuals, providers, health systems, hospitals, and at the state level through the BRFSS. Additional data sources identified included service-utilization data such as the American Medical Association’s Current Procedural Terminology (CPT) codes, payer policies, state licensure data, hospital credentialing data, professional society and/or network data (e.g., Lynch Syndrome Screening Network), state cancer registries, or Food and Drug Administration guidelines for testing validity. All survey data sources were considered “potential,” meaning that validated surveys may or may not exist but could be developed and initiated if resources permitted.

Two measures identified as potential sources of health equity were identifying the number of (i) genetic tests performed and (ii) genetic counseling visits for underserved subgroups of the population. These were both categorized as measures that individual states can pursue if data sources are available. Stratifying other outcome measures for underserved individuals/populations was not performed, although it was identified as a future activity.

Costs to implement the outcome measures were also discussed, including individual state costs (e.g., cancer registry maintenance, development and maintenance of state-specific all-payer claims database, survey development and data collection and analysis, and staff capacity); costs for researchers and others who develop, conduct, and analyze survey data; and costs for networks such as the Lynch Syndrome Screening Network (LSSN). Costs associated with meeting performance objectives were often shared costs: payments to individuals, families, and/or payers for services, and to health systems and/or labs for services.

Discussion

One of the major activities of the GPHAC Implementation Working Group was to develop a common understanding of appropriate metrics for health departments to assess ongoing implementation of genetic and genomic services (medical evaluation, genetic counseling and testing) within their state or local systems. The project members identified a dearth of existing metrics and therefore used an overarching framework taken largely from implementation research, crafted and categorized metrics, which fostered discussion both of objectives for genetic and genomic services and of available measures to meet those objectives. The group discussed existing data sources where available, as well as potential data sources that could be mined in the future. This summary lays out the rationale and context for the final set of performance objectives and outcome measures created. It describes the metrics developed for population-based genomics focused on HBOC and LS, which may also be applicable for other genetic conditions. It is expected that similar, yet unique metrics can be applied to future evidence-based public health genomic activities. The group stopped short of actually proposing targets because states vary widely in their current application of public health genomics activities and actual baseline data are not currently available for many of the metrics proposed. Our goal in proposing such metrics is to stimulate further dialogue and accelerate efforts to further refine population health genomics metrics.

The three performance objectives and outcome measures that all states are encouraged to pursue consistent with Healthy People 2020 are increasing (i) the number of women with a family history of HBOC/LS who receive genetic counseling, (ii) the number of persons with newly diagnosed CRC who receive genetic testing to identify LS or familial CRC syndromes, and (iii) the number of family members screened following identification of HBOC/LS mutations (i.e., cascade screening). This limited number of performance objectives suggests that the current capacity for measurement of genomic testing implementation is still at an early phase. However, the identification of these three performance measures is considered feasible for implementation and provides strong targets for assessment of effective public health impact.

Although the full list of performance objectives and outcome measures identified is extensive, it is not complete, nor has any validity or effectiveness testing been performed. Further, most performance objective descriptions will warrant additional detail. For example, under the broad category of implementation, within the “Acceptability” subgroup, a more detailed definition of what “comfortable” entails under the objective “increase in the number of providers who are comfortable providing HBOC/LS screening services” is required. In this same category, “appropriately” needs to be defined under an “increase the number of providers who appropriately refer HBOC/LS at-risk families.” Additional suggestions include the possibility of subdividing a more general outcome into more specific outcome measures; for example, under the broad category of “Client Outcomes,” within the “satisfaction” measures, more specific patient subgroups (e.g., disease groups) could be measured.

Other outcome measures such as costs for direct or indirect effect are also of importance because they may impact feasibility and implementation of public health genomic programs. Frequently, return on investment analysis is performed at the “back end” of program evaluation; however, within economic conditions, the importance of conducting return on investment analysis prior to program design and implementation will become even more critical to support, sustain, and expand a program. In cases where return on investment may be challenging to determine, social return on investment is another essential outcome measure that incorporates benefits that may not have immediate monetary value but have substantial social value.

Several state genomics programs are already addressing some of the performance objectives identified. For example, one of the key objectives of the CDC-funded cancer genomics program at the Michigan Department of Health and Human Services is to reduce the impact of hereditary cancer in the state.10 They have collected data through the 2015 Michigan BRFSS to assess personal and family history of breast and ovarian cancer and utilization of cancer genetic counseling.11 Similarly, the Washington State Department of Health Screening and Genetics Unit, in partnership with the CDC-funded Oregon Health Authority Genomics Program, surveyed gastroenterologists to determine how often clinicians routinely screen the tumors of patients found to have CRC to identify LS, as well as identify any resources this group of practitioners would need to initiate universal screening protocols among newly diagnosed cases of CRC (D.L. Doyle, personal communication, February 2017). The results of the survey were returned to the practitioners along with an educational resource guide that included materials specific for the clinician as well as materials that can be shared with families.

State public health genomic activities are viewable through the CDC’s Public Health Genomics Knowledgebase State Implementation Map,12 a clickable map identifying implemented state-specific genomics applications. These cover applications for HBOC syndrome, LS, familial hypercholesterolemia, newborn screening, and more. Results can be filtered for data sources, programs, education, policy, tools, and general information. In addition, the CDC has a useful Genomics Applications Toolkit for Public Health Departments13 that allows states that are just beginning to implement strategies to benefit from work already conducted by the five CDC-funded states.

Given the current state of genomics measurement within state public health systems, the GPHAC Implementation Working Group members suggest three “next steps” for consideration, with the goal of increasing the number of measures that a plurality of states and territories could integrate within their operations.

  1. 1.

    Refinement and adoption of common measures, particularly for the outcomes that all states are encouraged to pursue as well as those that most programs would be expected to capture. The group felt that establishing common measures across states for currently captured data would improve the ability to track progress in genomic testing and enable states to learn from one another where system improvements are made.

  2. 2.

    Developing new data-collection sources; given the number of measures that had no obvious existing data source, the group recommended the development of new mechanisms to collect data, particularly around key service system outcomes such as safety, efficiency, and patient-centeredness.

  3. 3.

    Pooling data; where common measures are already being collected, the group recommended consideration of a broader effort to pool data. This could enhance existing consortia efforts (e.g., the Lynch Syndrome Screening Network) to include system-level performance outcomes, or it could represent a new effort among states to collaborate.

As we enter an era of precision medicine, organizing and planning to mitigate disparities in access to genetic/genomic services is of utmost importance. The GPHAC efforts included consideration of health disparities at every juncture. One of the Academies’ six domains of health-care quality is equity, defined as “providing care that does not vary in quality because of personal characteristics such as gender, ethnicity, geographic location, and socioeconomic status.”3 We have included equity as an outcome measure; however, it is recognized that additional identification and measurement of predictors that give rise to disparities in access need to be further elucidated. A substantial body of literature has documented health disparities in the utilization of genetic tests for hereditary cancers,14,15,16,17 but most of this work to date has focused on individual-level factors that affect the way patients and providers make decisions about testing (e.g., a patient’s health literacy, risk comprehension, social supports, or a provider’s knowledge of clinical practice guidelines, cultural competence, or implicit bias). Research on these health inequalities, however, has tended to ignore system-level factors that give rise to disparities, even though those are viable targets for public health surveillance and policy-level action. Future work by the GPHAC will identify and incorporate these predictors in the state public health genomics toolkit.

In summary, we are proposing multiple genomics outcome measures specific to HBOC and LS, to help guide the implementation of genomic evidence-based tools, programs, and policies within public health systems. This is clearly a starting point intended to engage more states and academic researchers in refining and adopting population-based measures for use in assessing the implementation of evidence-based genomic medicine and its population-level impact on health outcomes.