Implementation research provides a structure for evaluating the clinical integration of genomic medicine interventions. This paper describes the Implementing Genomics in Practice (IGNITE) Network’s efforts to promote (i) a broader understanding of genomic medicine implementation research and (ii) the sharing of knowledge generated in the network.
To facilitate this goal, the IGNITE Network Common Measures Working Group (CMG) members adopted the Consolidated Framework for Implementation Research (CFIR) to guide its approach to identifying constructs and measures relevant to evaluating genomic medicine as a whole, standardizing data collection across projects, and combining data in a centralized resource for cross-network analyses.
CMG identified 10 high-priority CFIR constructs as important for genomic medicine. Of those, eight did not have standardized measurement instruments. Therefore, we developed four survey tools to address this gap. In addition, we identified seven high-priority constructs related to patients, families, and communities that did not map to CFIR constructs. Both sets of constructs were combined to create a draft genomic medicine implementation model.
We developed processes to identify constructs deemed valuable for genomic medicine implementation and codified them in a model. These resources are freely available to facilitate knowledge generation and sharing across the field.
In 2013, the National Human Genome Research Institute established the IGNITE (Implementing Genomics in Practice) Network to support development, implementation, and dissemination of methods that incorporate genomic medicine information into clinical care.1 The network includes six projects, affiliate members, a coordinating center, and working groups to facilitate cross-network collaboration.1 Through the work of network members, IGNITE is poised to provide a substantial impact on genomic medicine in the real world. Its stated goals are “to expand and link existing genomic medicine implementation efforts; develop new collaborative projects and methods for genomic medicine implementation in diverse settings and populations; contribute to the evidence base of outcomes following the use of genomic information for clinical care; and define, share and disseminate best-practices of genomic medicine implementation, diffusion and sustainability in diverse settings.”
While all six IGNITE projects are aligned through their work in genomic medicine, each differs in their intervention (i.e., pharmacogenomics, disease risk assessment, or disease diagnosis), and patient and provider populations. For specifics on each project’s study, refer to the IGNITE website (http://www.ignite-genomics.org). IGNITE’s diversity is, in large part, a direct reflection of the current state of affairs in genomic medicine: an explosion of research results that have not yet sufficiently infiltrated clinical practice to reach patients or providers, particularly those with the fewest resources and greatest challenges to achieving better health. IGNITE members are committed to understanding and addressing barriers to dissemination of actionable genomics findings, and to understanding population impact, particularly among underrepresented populations. IGNITE is thus comprised of a substantial network of diverse clinicians, practice settings, patients, and investigators in geographically widespread areas with variable levels of exposure to genomics. As such, IGNITE affords a rich opportunity to explore, test, institutionalize, and disseminate programs to translate genomic medicine into routine practice. Together, IGNITE investigators, representing a wide array of genomic, clinical, stakeholder engagement, and technological expertise, are discovering best practices for conducting research, building systems to make genomic information more accessible and actionable, and rigorously measuring their barriers, facilitators, and impact.2
The wealth of evidence being generated by each IGNITE project is beginning to lay the groundwork for understanding relevant issues around barriers to implementation, effectiveness, and stakeholder value. However, given the diversity of projects, common themes could be difficult to discern. To help facilitate identification of foundational evidence that could broadly guide implementation of genomic medicine, the network developed the Common Measures Working Group (CMG) (https://ignite-genomics.org/network/working-groups-interest-groups/) at its inaugural meeting in June 2013. The mission of the working group is “to gather data, evaluate, and disseminate methods of genomic medicine implementation research across diverse projects conducted by IGNITE members.” To achieve this mission, the working group developed a plan to identify constructs and measures relevant to evaluating genomic medicine interventions as a whole, standardize data collection across projects, combine data in a centralized resource for cross-network analyses, and develop a testable genomic medicine implementation research model based on the findings of IGNITE research. The database, containing data derived from the body of implementation research and standardized across contexts, is the type of queriable resource that could guide implementations in a learning health-care system as described by Chambers et al.3 In this way, the group felt it could balance the diversity, enhance external validity, and increase statistical power over what could be generated by each project alone. The objective of this paper is to describe the process the CMG is employing to meet its mission and the products of the work to date.
Methods and results
The CMG, composed of 23 members (representing every project plus interested IGNITE affiliates) met twice monthly via WebEx. To achieve its goals, the CMG established a process for identifying common measures, defined steps to evaluate those measures, and reviewed existing literature for similar projects2 (Figure 1).
After reviewing the implementation research literature, the CMG selected the Consolidated Framework for Implementation Research (CFIR)4 (http://cfirguide.org/) to guide our approach, since the framework allowed us to (i) think broadly about common measures, ensuring that it addressed the full spectrum of implementation characteristics; and (ii) share knowledge generated by IGNITE with other genomic medicine projects and networks. CFIR is a framework developed to clearly describe terminology and definitions for a comprehensive collection of constructs, drawn from published models (http://cfirguide.org/).5 As such, it pulls together all existing implementation models into a comprehensive ontology of overarching themes (domains and constructs) that are postulated to be important in implementation. Unlike implementation models, it does not assume there is any specific relationship between individual constructs or constructs and outcomes. It simply lists them all in an organized (consolidated) format, thus its name “Consolidated Framework.” It comprises 39 constructs organized across five domains (Figure 2): (i) innovation characteristics, (ii) outer setting, (iii) inner setting, (iv) characteristics of individuals, and (v) process, which are hypothesized to interact in dynamic and complex ways. Thus the CFIR is an ideal mechanism for establishing a common structure to facilitate knowledge generation and sharing across the diverse settings, populations, and innovations that make up the IGNITE Network projects.6, 7
The methods and results below are organized according to the CMG’s goals outlined in the introduction: identify constructs and measures relevant to genomic medicine as a whole, standardize data collection across projects, combine data in a centralized resource for cross-network analyses, and develop a testable genomic medicine implementation research model based on the findings of IGNITE research.
Identify constructs and measures relevant to genomic medicine
The CMG followed three steps to identify and develop common measures that could facilitate genomic implementation research: (i) review CFIR constructs and prioritize them in order of perceived value for genomic medicine, (ii) compile a list of all measures collected by each project and map them to CFIR constructs (if able), and (iii) identify constructs relevant to genomic medicine that are not represented in CFIR. In this way, we would be able to assess two different but equally important aspects of the existing projects’ protocols: where projects were assessing similar constructs, and where constructs that were deemed valuable to the network were not being measured.
Step 1 (methods)
Each project team independently rated (in order of perceived value for genomic medicine) the 39 CFIR constructs across all five domains using a 3-point scale: 3 = very important, 2 = somewhat important, 1 = not important. Next, ratings for each of the 39 constructs were summed and ranked in order of value (highest summed score to lowest). The 10 highest ranked constructs, plus any constructs not in the top 10 that had been rated as “very important” by at least one project, were identified as high priority. These constructs were presented to the full working group and discussed to ensure consensus on the list of high-priority CFIR constructs.
Step 1 (results)
Initial rankings identified 16 CFIR constructs or subconstructs supported by all projects: trialability, adaptability, relative advantage, and cost (domain: INTERVENTION CHARACTERISTICS); patient needs and resources (domain: OUTER SETTING); organization’s structural characteristics, networks and communications, culture, implementation climate (relative priority), and readiness for implementation (leadership engagement, available resources, access to knowledge and information) (domain: INNER SETTING); knowledge and beliefs about the intervention, and self-efficacy (domain: INDIVIDUALS’ CHARACTERISTICS); and planning, engaging, executing, and reflecting and evaluating (domain: PROCESS) as potential high-priority constructs for the network. Just as importantly, no projects supported intervention source, design quality and packaging, peer pressure, organizational incentives/rewards, other personal attributes, or external change agents as high- priority constructs. Table 1 shows the final results of the ranking procedure, identifying the 10 highest-ranking constructs, which were then used to frame data collection for the network.
Step 2 (methods)
Each project listed all of the domains (and corresponding measures) that they planned to assess or were already assessing (depending upon how far along they were in initiating their study) within the patient, provider, and health system levels. For example, domains in the patient level included demographics, quality of life, laboratory data, decisional processes, social influences, health literacy, attitudes, beliefs, knowledge, readiness to change, behavior, satisfaction, intention, genetic results, family history, and health outcomes. This classification aided in identifying projects that were measuring the same domains but with different instruments, facilitating discussions among projects as to whether (i) projects not currently measuring a domain where others were, would want to include that domain in their assessments and (ii) projects assessing the same domain with different measures would be willing to use a common instrument. Next, all measures were mapped to their corresponding CFIR construct. Mapping was initiated first by each project independently, and then by one of the CFIR developers (L.J.D.). During this process, if a measure was consistently mapped to the same construct, it was automatically assigned to the construct; if there was disagreement, consensus was reached through discussions facilitated by L.J.D. Since, as is often the case, multiple measures were mapped to the same construct, measures within constructs were prioritized using the same process as that for the ranking constructs (step 1).
Step 2 (results)
Table 1 lists the highest-priority CFIR domains associated with each of the high-priority constructs. Note that most of the measures initially planned for inclusion in study protocols did not assess high-priority CFIR constructs and most of the high-priority constructs did not have published measurement instruments.
Step 3 (methods)
In many cases, measures in step 2 could not be mapped to a CFIR construct. Since existing evidence suggests that identifying and describing key contextual factors, even those that fall outside of established CFIR constructs, is valuable because they can affect patient outcomes through their influence on implementation, we included these domains and associated measures as high-priority constructs when they were considered critical for understanding genomic medicine. This determination was made in the same way that CFIR constructs were determined to be high priority (step 1). To do this, when measures could not be mapped to CFIR constructs, they were “mapped” to their underlying domain and the domain added as a construct to the list of high-priority genomic medicine constructs (from step 1).
Step 3 (results)
After comparing CFIR constructs to the projects’ compiled list of domains and measures, it became clear that CFIR lacks a well-defined representation of patient-related domains. The CFIR domain Individuals’ Characteristics reflects individuals within the organization and while patients are part of the health-care organization and are essential to assessing intervention effectiveness, they are a less influential component of implementation success in health-care settings than administrators and physicians. The working group therefore identified patient domains highly relevant to genomic medicine implementation and included these in Table 1 as non-CFIR high-priority constructs.
The ultimate result was a matrix that consisted of CFIR and non-CFIR constructs prioritized for their value to the network with a list of associated measures (also ranked). The matrix provided a simple method for identifying which high-priority constructs the network was already measuring and which it was not. When measures were listed under a high-priority construct, the highest-ranking measure was adopted as the preferred measure. For high-priority constructs with no corresponding measures, a literature search was undertaken to identify if existing instruments could be adapted to genomic medicine, and, if none was identified, then measures were developed. These processes are described in the next section.
Standardize data collection across projects
To maximize the potential value of cross-network analyses, the working group needed to standardize data collection across projects wherever possible. This process occurred as an extension of step 2 in the “identify constructs and measures relevant to genomic medicine” process (above). Two synergistic approaches were employed to facilitate standardized data collection. The first was to promote adoption of a single measure for each domain being assessed by more than one project, and the second was to develop measures for high-priority CFIR and non-CFIR constructs if no published standardized measure existed.
Promote adoption of a single measure for each domain (methods)
As described earlier, all projects listed the domains they were measuring across the spectrum of stakeholders: patients, providers, and health systems. During this process it became clear that, in several cases, two or more projects were planning on assessing a domain (such as patient quality of life) but were using different measures to do so. To promote adoption of a single measure for as many measures as possible, regardless of whether they fell into a high-priority construct/domain, the working group encouraged discussion among the projects by specifically calling attention to those domains and enabling project leadership to discuss them on a case-by-case basis.
Promote adoption of a single measure for each domain (results)
During this process, common measures for several different domains were adopted by projects measuring them. Specifically, the network adopted the following measures: a single item from the SF-128 for self-related health, Voils’s survey for patient medication adherence,9 a single item for patient health literacy,10 the Patient Activation Measure,11 and the Organizational Readiness for Implementing Change.12 In addition, two projects initially planning on only assessing patient measures through the electronic medical record modified their protocol to incorporate patient/provider surveys using common measures identified here and below to enhance the feasibility of cross-network analyses.
Develop measures for high-priority CFIR and non-CFIR constructs (methods)
As described in the “identify constructs and measures relevant to genomic medicine” process, Table 1 represents high-priority CFIR and non-CFIR constructs important for genomic medicine. Importantly, this process identified constructs as valuable that no project had previously considered measuring. To facilitate measurement, the CMG conducted literature reviews. When measures were identified, they were described and coded for standardized data collection across projects; when measures could not be identified, the working group developed measures with the help of a psychometrician.
Develop measures for high-priority CFIR and non-CFIR constructs (results)
Four surveys were standardized and coded for use across the projects planning to survey patients and/or providers; all are available for download from the IGNITE Supporting Practice through Application, Resources, and Knowledge (SPARK) toolbox (https://ignite-genomics.org/spark-toolbox/). The first survey agreed on and coded patient demographic and education measurement using the National Institutes of Health–preferred measures. The second is a postintervention patient survey that incorporates measures for patient attitudes toward a specific genomic intervention, intentions for sharing intervention results, and preferences for return of results. The third codifies provider demographics and practice/setting characteristics, including age, gender, race, ethnicity, practice setting, profession, specialty, and years in practice. The fourth is a provider survey designed as a preimplementation survey that could also be employed postintervention as well. This survey, initially 14 items that were then refined to 10, was developed by the CMG to address high-priority CFIR constructs (Table 1) including implementation climate, readiness for implementation, knowledge about the intervention, self-efficacy, and relative advantage.
Lastly, to promote the cross-network analysis, we categorized study interventions (i.e., as pharmacogenomic, disease risk assessment, or diagnostic) and identified several cross-study outcomes that could be assessed despite the disparate interventions. These included uptake of the genomic intervention by providers, frequency providers act on intervention recommendations, success of intervention on each study’s primary patient endpoint, pre/post change in patient’s diet and/or exercise habits, frequency that intervention results are actionable, and impact on mortality, disease incidence, hospitalizations, or emergency room visits.
Combine data in a centralized resource for cross-network analyses
Common data were stored in a relational database (details provided in the Supplementary Material online). To demonstrate the potential value of such a database, we performed a preliminary analysis using R software on patient-level data deposited in the resource database prior to 12 August 2016. Patient demographic features and responses to the patient survey were summarized within and across the IGNITE projects using summary statistics. Differences across projects were evaluated using chi-squared tests for project/intervention, sex, race, ethnicity, and education, and analysis of variance F-test for age.
Of the six projects, one was not collecting individual patient data and therefore was not able to contribute to the patient data analysis. Of the remaining five projects, one was collecting data on paper and was not able to provide an electronic version by the time of manuscript completion, and another opted not to implement the demographic or patient survey. Therefore, the patient analysis represents data contributed from three projects.
Across these three projects, there were 2,430 patients (Table 2). All demographic features were reported by two projects, while one project reported all demographic features except for education level. Overall, the demographic features had high response rates: age and sex were reported for nearly all patients (99.5% for both age and sex), while ethnicity, race, and education response rates were slightly lower (77.0%, 90.0%, and 89.9% respectively). We note that nearly all demographic features, except sex, differ by project (age/race/education P values <1.0 × 10−5, ethnicity P value = 0.002). Patient responses to the preimplementation common measures questions were less complete than the demographics responses. Four of the seven common measures questions had responses submitted by at least one project, and no single question had responses from all three projects. Three questions had responses from two different projects, and one question had response from only one project. The variability in number of projects reporting any given question resulted in patient response rates ranging from 9.87% to 73.4% of the 2,340 patients with demographic information.
Develop a testable genomic medicine implementation research model based on the findings of IGNITE research
By following the steps outlined above, the working group gathered an abundance of data that, once organized, were able to inform a structural understanding of genomic medicine research implementation. We therefore leveraged these data to develop an IGNITE genomic medicine implementation research conceptual model derived from published high-level models and informed by findings across the six IGNITE projects. The goal of the model is to serve as a resource for researchers, such that new evidence can inform and refine the model over time. The initial model was developed based upon high-priority constructs, both within and outside of CFIR. Strategies used by the team to implement genomic medicine interventions were conceptualized as mechanisms of action aimed at effective implementation but that would be moderated (positively or negatively) by high-priority constructs at the organization, physician, and patient levels. More effective implementations of genomic medicine interventions were hypothesized to lead to better clinical or process outcomes.
Based on high-priority constructs identified by the project teams, moderators that are hypothesized to influence implementation effectiveness were defined across four domains: (i) intervention characteristics, (ii) characteristics of the organization’s inner setting, (iii) characteristics of clinicians involved with delivering the genomic medicine intervention, and (iv) characteristics of the process of implementation. Therefore, the model flow suggests measuring those domains prior to implementation so they can inform the selection of the optimal implementation strategy. Figure 3 displays how the domains relate to each other and the implementation strategy, but specific constructs are not listed in the figure as they may differ in other implementation projects (e.g., assessing experience with return of results rather than intention to share results). For IGNITE they are reported in Table 1.
The IGNITE Network’s Common Measures Working Group brought together six disparate projects to synthesize a model for understanding the process of genomic medicine implementation into clinical practice and to identify/develop standardized measures that may help to inform a broader understanding of genomic medicine implementation research. The key steps taken by the group to advance understanding beyond each individual project are formalized in this paper (i) to outline a pathway for other networks that are interested in using a similar methodology, and (ii) to provide tools for individual institutions/projects interested in genomic medicine implementation research. The model (Figure 3), developed with input from providers with diverse clinical settings, environments, and populations, is a simplified linear depiction of our complex nonlinear efforts that is “testable” in that it is meant to be refined as results from ongoing work inform our understanding of this fast-moving field. For example, if other implementation projects identify different or additional high-priority constructs, those should be added to the list of options for that domain, or if researchers find that to be successful all genomic implementation processes should include specific characteristics that are unrelated to the constructs, then the model should be updated to represent those findings.
One key finding of our work is that while the CFIR provides an excellent foundation upon which to conceptualize implementation and as a framework provides representation of constructs from across the entire field of implementation science, it does not fully capture all domains pertinent to genomic medicine. To address this, we expanded beyond the CFIR to incorporate domains relevant to patients, families, and local communities, as CFIR (and thus the implementation models upon which it derived its list of constructs) is more specifically focused on factors relevant to the health system rather than local community values. In genomic medicine, the patient and the local culture around the patient are extremely important, with a large body of work devoted to understanding patient perceptions, anxiety, and personal utility. While these are not unique to genomic medicine, they have special emphasis when interventions address topics related to genomics—particularly for patients who are generally healthy and not pursuing a diagnostic odyssey.
Another key finding, that most projects were not initially measuring any of the CFIR constructs they had identified as high priority, is an essential insight into genomic medicine’s current perceptions of implementation and forays into the field of implementation science. This work may help to guide other implementation projects to consider a broader conceptual view of implementation and which constructs could be valuable for them to address when selecting measures and outcomes. It is important to address these issues early, prior to projects developing protocols, to improve the ability to synergize data collection and methods. Five projects changed their protocols to incorporate relevant measures: two to incorporate patient surveys and three to adopt high-priority measures (the “common measure”) for constructs that they were initially planning to measure, but with different measurement instruments. This shows the value of the measures developed; however, we were still limited in the data we were able to capture given that most projects had already begun when the CMG completed its evaluations.
By thoroughly addressing these questions early in the network formation, we were able to enhance the scientific rigor of the network as a whole and facilitate cross-study analyses that can address a wide variety of questions around implementation and effectiveness, such as (i) provider demographics, attributes and attitudes that predict uptake of genomic medicine; (ii) the extent to which patients plan to and actually share their genetic information with relatives, friends, and others; (iii) patient attitudes toward genetic testing and its potential benefits and risks as perceived by those undergoing a variety of interventions; (iv) influence of the genomic intervention and type of genomic intervention on health, well-being, and self-care/preventive behaviors by patients and provider practices. It is important to note that, while the diversity of the projects is desirable and leads to greater generalizability of cross-network findings, it is also a challenge for combining data. Therefore, a number of questions cannot be answered by analyzing data from across the network; many can only be analyzed within each project. Yet, whenever possible, combining data across the network will provide unique insight into the field as a whole. We have defined cross-network measures as those that are being collected from three or more projects. To demonstrate the potential for these analyses, we reported a preliminary analysis of patient demographics and responses to the CMG-developed patient survey (Table 2). These results are not meant to be taken as definitive “findings,” but, rather, as an example of what these data might look like.
The results of the CMG’s efforts, which include the implementation model, surveys, and listing of high-priority constructs, are available in the IGNITE Network’s SPARK toolbox (https://ignite-genomics.org/spark-toolbox/). SPARK provides free access to the products of our work, which may be useful to clinicians, patients, and educators, as well as researchers. It contains resources to educate and inform providers and patients, as well as recommended measures to assess implementation effectiveness. The implementation research model is unique in that it is specifically tailored to genomics, a rapidly growing but still relatively underutilized tool within health care.
Weitzel KW, Alexander M, Bernhardt BA et al. The IGNITE network: a model for genomic medicine implementation and research. BMC Med Genomics 2016;9:1.
Manolio TA, Chisholm RL, Ozenberger B et al. Implementing genomic medicine in the clinic: the future is here. Genet Med 2013;15:258–267.
Chambers DA, Feero W, Khoury MJ . Convergence of implementation science, precision medicine, and the learning health care system: a new model for biomedical research. JAMA. 2016;315:1941–1942.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC . Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4:50.
Kirk MA, Kelley C, Yankey N, Birken SA, Abadie B, Damschroder L . A systematic review of the use of the Consolidated Framework for Implementation Research. Implement Sci 2016;11:72.
Larsen KR, Voronovich ZA, Cook PF, Pedro LW . Addicted to constructs: science in reverse? Addiction. 2013;108:1532–1533.
Wacker JG . A definition of theory: research guidelines for different theory-building research methods in operations management. J Oper Manag 1998;16:361–385.
Busija L, Pausenberger E, Haines TP, Haymes S, Buchbinder R, Osborne RH . Adult measures of general health and health-related quality of life: Medical Outcomes Study Short Form 36-Item (SF-36) and Short Form 12-Item (SF-12) Health Surveys, Nottingham Health Profile (NHP), Sickness Impact Profile (SIP), Medical Outcomes Study Short Form 6D (SF-6D), Health Utilities Index Mark 3 (HUI3), Quality of Well-Being Scale (QWB), and Assessment of Quality of Life (AQoL). Arthritis Care Res (Hoboken) 2011;63(suppl 11):S383–412.
Voils CI, Maciejewski ML, Hoyle RH et al. Initial validation of a self-report measure of the extent of and reasons for medication nonadherence. Med Care. 2012;50:1013–1019.
Wallace LS, Rogers ES, Roskos SE, Holiday DB, Weiss BD . Brief report: screening items to identify patients with limited health literacy skills. J Gen Intern Med 2006;21:874–877.
Hibbard JH, Stockard J, Mahoney ER, Tusler M . Development of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res 2004;39:1005–1025.
Shea CM, Jacobs SR, Esserman DA, Bruce K, Weiner BJ . Organizational readiness for implementing change: a psychometric assessment of a new measure. Implement Sci 2014;9:7.
Funding for the Common Measures Working Group was provided through a supplement (NIH grant 3U01HG007282-01S1) to Duke University’s IGNITE study “Implementation, Adoption, and Utility of Family History in Diverse Care Settings” (NIH grant 1U01HG007282-01).
The authors declare no conflict of interest.
Supplementary material is linked to the online version of the paper at
About this article
Barriers and Facilitators to Genetic Testing for Familial Hypercholesterolemia in the United States: A Review
Journal of Personalized Medicine (2019)
Molecular Genetics & Genomic Medicine (2019)
A logic model for precision medicine implementation informed by stakeholder views and implementation science
Genetics in Medicine (2019)
Opportunities to implement a sustainable genomic medicine program: lessons learned from the IGNITE Network
Genetics in Medicine (2019)
Drug–gene and drug–drug interactions associated with tramadol and codeine therapy in the INGENIOUS trial