Main

The appropriate integration of genomics into routine practice stands to significantly impact medicine.1 However, substantial barriers must be overcome to drive the integration of genomic technologies to clinical care. One is the gap in genetics knowledge and skills among physicians.2,3,4,5 Despite a consistently positive view among physicians about the importance of genomics in medicine,5 many lack contemporary background knowledge in genetics and are not adequately prepared to use genetic information to treat patients.5,6,7 A noted lack of confidence about their own genetics knowledge has prevented physicians from having comprehensive discussions about it with their patients.8 Furthermore, physicians may order genetic testing that is inappropriate for the clinical situation and that they do not correctly interpret, potentially leading to wasted health-care resources and less than optimal patient outcomes.9,10 Physicians also have reported a lack of understanding about when and how to make referrals for genetic counseling.11,12

Physicians have endorsed the need for more training in areas such as genetic risk assessment, test ordering and interpretation, and counseling.5,7 Many groups have developed continuing education programs designed to address these topics.13,14,15,16 Historically, though, much continuing medical education (CME) has been passive and didactic.17 Programs that are grounded in evidence-based principles of adult learning and medical education have a greater impact on physician performance than programs that do not incorporate such principles.18,19,20 The strongest gains are seen in CME activities that are more interactive, use a variety of instructional techniques, involve multiple exposures, and focus on outcomes considered important by physicians.21,22 Not all educational activities are developed with the goal of impacting practice change.1,17 The Accreditation Council for Continuing Medical Education has reported that, of activities offered by its accredited CME providers in 2014, slightly more than half were designed to change performance and only 43% were analyzed to determine their impact on performance.23

Given the need for education designed to support practice change, the National Coalition for Health Professional Education in Genetics, Genetic Alliance, and the American Medical Association partnered with a community hospital that had identified the need for and motivation among its physicians to improve genomics knowledge and skills. Using evidence-based adult-learning principles as a framework, along with data collected from an institutional advisory committee and a pilot program, a genomics educational program for physicians at El Camino Hospital (ECH) in Mountain View, California, was developed, implemented, and evaluated. The goal of the 10-module, in-person program, entitled Medicine’s Future: Genomics for Practicing Doctors (referred to as Medicine’s Future), was to train ECH physicians to be confident incorporating genetic principles into routine care.

Development and Implementation of Medicine’s Future

Inception and content development

The ECH Genomic Medicine Institute (GMI) advisory board identified an institutional need for physicians who could demonstrate competence in basic genetics skills and knowledge. In 2011, it released a request for proposals to fund the development of a program to address this need. The National Coalition for Health Professional Education in Genetics and the Genetic Alliance—groups with expertise in health professional education and patient advocacy, respectively—were selected. Needs assessment and current practice data were gathered through a project-specific GMI advisory committee that included ECH leadership and physicians. The process revealed that the outcomes of interest were improvements in foundational genomics clinical skills and knowledge, including collecting family history and assessing risk, assessing clinical validity and utility of genetic testing for specific clinical situations, ordering and interpreting genetic tests, and integrating genetic information into management decisions. A case-based, patient-centered approach was deemed to be important to providers. Foundational modules were developed to introduce these skills using a variety of conditions and case studies. Subsequent modules were developed to focus on application of the foundational skills to several conditions, including common diseases, pharmacogenomics, cancer, neurology, cardiology, pediatrics, and prenatal genetics.

Instructional design and delivery of the Medicine’s Future curriculum

The design of the workshop curriculum was rooted in Merrill’s First Principles of Instruction, a set of prescriptive problem-based learning principles that are common to instructional design theories.24,25 The principles promote learning when (i) learners are engaged in solving real-world problems, (ii) existing knowledge is activated as a foundation for new knowledge, (iii) knowledge is demonstrated to the learner, (iv) new knowledge is applied by the learner, and (v) new knowledge is integrated into the learner’s world. The application of Merrill’s First Principles to the Medicine’s Future curriculum is summarized in Table 1 . Each 2-hour module included a model case study in which an expert facilitator led large-group discussion through the case, small-group cases to practice application, and a patient video with large-group discussion to incorporate concepts related to patient experiences.

Table 1 Merrill’s First Principles of Instruction as applied to Medicine’s Future instructional strategies
Full size table

To further improve learning outcomes, the training was sequenced over several months to capitalize on learn–work–learn opportunities, which are characterized by learning events followed by application in practice and expert feedback.26 The training incorporated methods including elements of narrative medicine, cognitive apprenticeship, collaborative learning, point-of-care job aids, and ultimately community of practice.27 Each module used evidence-based active-learning strategies including peer discussion, expert feedback, audience response systems, and case-based problem solving, which support retention, transfer, conceptual understanding, and improved attitudes.28,29,30,31 In addition, videos of individuals affected by genetic conditions talking about their experience with the condition and interactions with health-care providers were included in every session. These experiences were used to explore the impact of genetic information on patients and family members.

Once developed, the Medicine’s Future curriculum was piloted at ECH from September 2011 through June 2012, with a 2-hour module presented monthly. All ECH physicians were invited to attend the course. Formative evaluation was conducted after each module, at the midpoint of the curriculum, and at the end of the pilot period. Modifications to the curriculum focused on unifying instructional messages, selecting and training facilitators, simplifying genetic testing methodologies, and increasing opportunities to practice risk stratification. The revised Medicine’s Future curriculum, on which we report in this article, was offered monthly from October 2013 through July 2014. As with the pilot curriculum, all physicians at ECH were eligible to participate in the course. Supplementary Table S1 online lists the title and main focal points of each module. A clinical geneticist facilitated each 2-hour module, leading discussions and interactive case studies. Each module was awarded a maximum of 2.0 AMA Physician Recognition Award Category 1 Credits. Participants attending and completing a postactivity assessment and evaluation were eligible to claim CME credit.

Data collection and evaluation

Enrollees in the revised Medicine’s Future program were asked to complete a preactivity assessment approximately 1 week before the first module was presented. This assessment, administered through the American Medical Association’s Online Learning Center, consisted of 5 demographic questions, 12 Likert-type questions about current genetics-related practice experiences and behaviors, 14 semantic differential-scale questions exploring attitudes about genetics in medicine and confidence in genetics-related practice skills, and 20 multiple-choice questions testing knowledge of genetics concepts. These questions were a combination of items used in a similar audience and items developed specifically to assess concepts related to this curriculum’s desired outcomes. Items were piloted with the 2011–2012 modules. Knowledge items with limited or no spread were dropped or modified. Three new items were created to reflect the revised content’s increased focus on pharmacogenomics. The preactivity assessment questions are in Supplementary Figure S1 online.

Postactivity assessment and evaluation were administered through the American Medical Association’s Online Learning Center following each module. Assessment questions consisted of the same questions exploring attitudes and confidence as those encountered in the preactivity assessment. All postactivity knowledge questions were drawn from the 20 knowledge questions from the preactivity assessment. For modules 1 through 9, 3 or 4 multiple-choice knowledge questions were included in the postactivity assessment; for module 10, 10 multiple-choice knowledge questions were included. Evaluation questions asked about the general effectiveness of the module and whether and how participants intended to change their practice based on it. Approximately 6 months after module 10 was complete, an online survey was sent to participants containing a subset of attitude, confidence, and knowledge questions previously encountered, as well as four new questions about changes in practice behaviors due to Medicine’s Future. Genetic counseling referral data were collected by assessing the genetic counseling utilization and billing records of the ECH GMI, which administered the counseling program.

Data analysis

Participants’ responses on each of the assessments and the 6-month follow-up survey were aggregated and compared to determine changes in attitudes about genetics in medicine and confidence in genetics-related practice skills. For semantic differential-scale questions, categories at the high and low ends of the scale were combined. For example, on a scale of 1 to 5 where 1 = strongly disagree and 5 = strongly agree, responses of 4 and 5 were combined. To determine significance, the Fisher’s exact test was used; differences were considered significant when P < 0.05. To analyze genetics knowledge gains, the number of correct answers on the preactivity assessment was compared with the number of correct answers to the same questions on postactivity assessments following each module and 6 months after module 10. Additionally, to determine knowledge gains in broad genetics-related practice components, questions were categorized into the topic areas of risk assessment, development of testing strategy, test interpretation, and application of results to patient care, and the number of correct answers to questions in each of these categories on the preactivity assessment was compared with the number correct on postactivity assessments. To determine significance, a chi-squared test was used; differences were considered significant when P < 0.05.

Results

Participants

Participant demographic and practice characteristics are summarized in Supplementary Table S2 online. Thirty-four participants enrolled in the revised Medicine’s Future curriculum and completed the preactivity assessment; 33 were physicians and 1 was a physician assistant. Participants represented a mix of ages, with those aged 41–50 and 61–70 years making up approximately 70% of the group. Of note, 17 specialties were represented, with more than half of the total enrollees representing primary-care specialties: family practice (5), internal medicine (8), obstetrics and gynecology (4), and pediatrics (2). Participant attendance ranged from 20 to 30 for each module. On average, 75–90% of participants attending each module completed the postactivity assessments and evaluations. Response rate for the 6-month follow-up survey was 38.2%.

Participants’ prior experience with genetics is summarized in Table 2 . Less than 20% of participants reported receiving formal training in genetics, and approximately 30% reported having attended a continuing education course or event in genetics. The majority of participants reported a genetic concern arising among their patients several times per year; however, more than one-quarter reported concerns several times each week. Most participants reported that patients ask genetics-related questions at least several times per year, with topics frequently including family history, risk for a disease in themselves, risk for a disease in their family members, genetic etiology of a disease, genetic testing, resources for themselves or family members, and specific conditions. The vast majority of participants (94.1%) reported that they review family history for new patients. Approximately one-quarter (26.5%) order genetic testing more than monthly. Direct consultation with a genetics expert (geneticist or genetic counselor) or genetic testing laboratory was infrequent, with more than half reporting that they never consult. Referral of patients to genetics experts was more frequent but was not common.

Table 2 Participants’ experience with genetics prior to beginning Medicine’s Future
Full size table

Knowledge related to genetics practice skills

Significant gains in knowledge related to genetics practice skills were observed during the course. Prior to module 1, 54.6% of participants’ answers to knowledge questions were correct, whereas 85.2% of answers were correct following the module in which the question’s main topic was covered (P < 0.05) (data not shown). To determine retention of gains, knowledge was assessed directly following module 10 and approximately 6 months after module 10. Knowledge gained was retained, as demonstrated by significant increases in correct answers to knowledge questions directly following module 10 (85.0%) and 6 months after module 10 (75.0%) compared with before module 1 (P < 0.05 for both; data not shown).

Throughout the 10 modules, four broad practice components were emphasized: risk assessment, development of a genetic testing strategy, test interpretation, and application of results to patient care. Significant knowledge gains were observed in all components following the module in which the concept was addressed ( Figure 1 ). Knowledge gains were sustained throughout the course; directly following module 10, the number of correct answers to knowledge questions in the four practice components was significantly higher compared with prior to module 1 ( Figure 1 ). Knowledge retention in the individual practice components could not be assessed 6 months after module 10 owing to limitations in the number of questions administered at that stage.

Figure 1
figure 1

Change in knowledge in selected genetics concepts following the module in which the concept was covered and following module 10. Asterisks indicate statistical significance compared with values observed prior to module 1 (P < 0.05).

Full size image

Confidence in genetics-related practice skills

Confidence in genetics-related practice skills was self-reported by participants prior to module 1, following each module, and 6 months after module 10. Participants were asked to rate their confidence using a scale of 1 to 5, where 1 represented “not at all confident” and 5 represented “very confident.” Self-reported confidence in eight separate practice skills significantly increased from prior to module 1 to directly following module 10 ( Figure 2 ). Confidence in these skills increased sharply over the first four modules (data not shown). Interestingly, confidence levels had reached or were near post–module 10 levels as early as after module 4 ( Figure 2 ). Self-reported confidence in two other practice skills—the ability to take a family history and the ability to identify patients at increased risk for a condition based on family history—showed small but insignificant increases ( Figure 2 ).

Figure 2
figure 2

Confidence in practice skills related to genetics prior to module 1, following module 4, following module 10, and 6 months after module 10. Asterisks indicate statistical significance compared with values observed prior to module 1 (P < 0.05). P values for nonsignificant comparisons for collecting a family history: following module 4, P = 1.0; directly following module 10, P = 1.0; 6 months following module 10, P = 1.0. P values for nonsignificant comparisons for ability to identify patients at increased risk for a condition based on family history: following module 4, P = 0.21; directly following module 10, P = 0.29; 6 months following module 10, P = 0.73. P values for other nonsignificant comparisons 6 months following module 10: ability to order genetic testing, P = 0.43; ability to interpret test results, P = 0.33; ability to use genetic information in management decisions, P = 0.67.

Full size image

Six months after module 10, confidence in five practice skills continued to be significantly increased. Confidence was increased compared with pre–module 1 levels, but not significantly so, for the other practice skills showing significant differences immediately following module 10, such as the ability to order genetic testing, ability to interpret test results, and ability to use genetic information in management decisions ( Figure 2 ).

Change in practice behavior

Following each of the 10 modules, 50.0 to 88.9% of participants reported that they planned to change their practice based on the course ( Table 3 ); 92.3% of respondents to the 6-month follow-up survey reported that they had changed their practice based on the skills acquired during Medicine’s Future ( Table 3 ), and 92.3% of participants reported, in the form of free-text responses, that they engaged in other activities that impact genetics practice skills, including reading more journal articles about genetics, attending genetics conferences and lectures, and talking to colleagues about genetics issues more often ( Table 3 ). Participants were asked after each module whether they believed Medicine’s Future was more effective than other CME courses in which they had engaged. On a scale of 1 to 5, where 1 represented “much less effective” and 5 represented “much more effective,” 86.2% rated effectiveness as a 4 or 5 averaged across all 10 modules (data not shown).

Table 3 Percentage of participants reporting that they planned to change practice as a result of each Medicine’s Future module and percentage of participants reporting that they had changed practice and engaged in other genetics-related activities 6 months after completing module 10
Full size table

In addition to self-reported practice change, observed behaviors among ECH physicians suggest that Medicine’s Future impacted practice change. Independently, participants from the pilot course started a monthly genetics case conference that was also attended by participants from the revised course and other ECH physicians. Also, in the 10 months preceding module 1, a total of 173 genetic counseling referrals were made by ECH physicians; during the 10 months coinciding with the course, 222 referrals were made—an increase of nearly 30% (data not shown). Referral data for the 10-month period following the course were not available.

Discussion

For genetics to impact the health of the population, providers in all settings must have the skills to use genetic information to identify individuals at increased risk for disease and implement appropriate management, whether that is referral for further work-up, ordering testing, or adjusting screening regimens. We developed Medicine’s Future to enhance these skills in community-based providers of many specialties. To address the known challenges of implementing genetics into practice,6 the curriculum maximized engagement through application of instructional design best principles. Our results add to the evidence that this approach was successful.13,14,32 In Medicine’s Future, knowledge and confidence in practice skills related to risk assessment, development of a genetic testing strategy, interpretation of results, and application of results to patient care all improved significantly. These improvements were sustained over the 10-month duration of the course and, for some skills, for 6 months following the course. In addition, participants reported the intent to improve several genetics-related practice behaviors, and after 6 months they reported that they had changed a subset of these. Our findings are consistent with those from other education programs that show sustained improvements when the program includes substantial learn–work–learn time.13,32 For skills in which improvements were not sustained, the low frequency of opportunities to use such skills in daily clinical routines may be an explanation. For example, determining when genetic counseling is indicated is a skill that can be practiced with nearly every patient, but the need to interpret genetic test results arises much less frequently. Other studies have indicated that the ongoing use of tools to assist in triaging risk-assessment decisions at the point of care can result in longer-term practice changes.33,34

Beyond the quantitative measures directly related to the course objectives, we observed changes to genetic counseling referral practice and the establishment of a genetics-focused case conference. Although neither result can be directly or solely attributed to the course itself, they are associated with participants’ improved confidence in identifying patients who could benefit from genetic counseling and intention to increase attendance at genetics conferences and lectures. The change in referral pattern is aligned with the larger goals that informed the development and design of Medicine’s Future, namely, emphasizing risk-assessment skills to identify individuals appropriate for additional work-up and the added value of referring to genetics professionals. The spontaneous initiation of a monthly genetics case conference by attendees suggests shared interests and comfort with group learning. The development of a community of learners is one goal of adult education because it facilitates sustainable peer learning.35 We speculate that the course structure and the skill of the facilitator in encouraging discussion provided a forum for participants interested in genomics to build relationships and identify shared interests. The observed changes are particularly meaningful because they occurred within the context of a mixed population of specialties, which suggests that the foundational nature of genomic information may be one way to provide a bridge between specialists.

The identification of learners’ educational needs has been demonstrated as an important component for successful outcomes, including for genomics educational programs.21,36 Because this course was intended for any physician at ECH, it was necessary to identify common educational gaps and clinical goals across several specialties. Working with the GMI advisory board, we identified outcomes that focused on building knowledge, skills, and confidence to assess the validity and utility of genetic testing across a wide range of conditions. The course’s participant mix, from primary care to nuclear medicine, suggests that these areas are of interest across specialties. This interest corresponds to an educational gap demonstrated by data showing that, prior to the course, participants correctly answered only approximately 43–56% of knowledge measures grouped by concept.

We believe that the 10-month multiple-module model of Medicine’s Future contributed to its positive outcomes because it reinforced skills, provided opportunity for application in different contexts, and facilitated the development of a community of learners. Although it may not be practical in all settings to engage providers over 10 months, instructional design principles can be applied to the development of genomics education of any length or delivery type. At a minimum, the engagement of a subset of the target audience to inform objectives and skills and the use of active learning are critical steps in developing a program that maximizes the possibility of success.

Our results show that after only four modules confidence in genetics skills had reached or were near levels observed at the completion of the curriculum, raising the possibility that a subset of modules, including those that cover foundational genetics issues, might achieve the desired results. However, we cannot know whether sustained improvements would have occurred with fewer modules; reducing the number of modules might result in fewer opportunities for reinforcement and decrease the likelihood of developing an ongoing community of practice. Additionally, the modules that are focused on specific conditions and testing (modules 5–9) may have provided particular benefit to participants who were in need of more specialized skills. The challenges of finding time, money, and experience to develop content in this way can be difficult to overcome. Partnering with education experts, such as those within institutional CME or medical/nursing education departments, is one way to address these challenges.

We designed Medicine’s Future to maximize participant engagement with each other, the expert facilitator, and the content. Improvement of practice is most effectively accomplished through interactivity, which is best facilitated in small-group settings.21,22 The modular structure of the curriculum allows it to be effectively implemented in multiple locations, without incurring additional development costs. In this way, education developed to change practice can reach a large number of providers when widely implemented.

Our intervention succeeded in meeting ECH’s desired outcomes, and we believe it could be successful in other provider populations. In general, Medicine’s Future participants had a moderate amount of exposure to genetics prior to their involvement in the course, likely impacting their motivation to engage in genetics education. Despite this motivation, the precurriculum knowledge measures suggest that this engaged group still had significant educational needs. The structure of the course and its application of adult-learning best practices could be appropriate for a wide range of locations and provider groups. One challenge in other settings may be identifying physicians motivated to participate, especially given competing priorities.

Limitations

The design of this study was a pre- and postactivity comparison without a control group. It is possible that knowledge, skills, and confidence could have improved through increased awareness of genomics over the same time period due to other factors. However, the improvements demonstrated by participants directly following each module suggest that the modules themselves were responsible. Respondents to the 6-month follow-up survey were self-selected, and the 38% who responded were not representative of the larger participant group.

The data presented represent aggregate results from all participants completing postactivity assessments for each module. Changes in individual participant knowledge, confidence, and practice change were not assessed or reported, owing to the small number of participants who could be followed through every session and assurance to participants that only aggregate data would be used for analysis. Additionally, confidence in genetics-related skills, intention to change behavior, and long-term behavior change were measured using participant self-reporting mechanisms. Although some evidence indicates that self-reported confidence tends to be overestimated,37 the consistent confidence increases reported by participants suggest that Medicine’s Future positively impacted their confidence in performing genetics-related skills. We do not know the case circumstances for each genetic counseling referral that occurred during Medicine’s Future, so it is not possible to determine whether the increased number of referrals reflected improved awareness of the availability and utility of genetic counseling, better recognition of cases for which genetic counseling was indicated, or both. Direct measurement of participants’ behavior by analyzing electronic medical record information about family-history collection, genetic test ordering, and follow-up patient management or assessment using standardized patients38 would be ideal to determine whether the course facilitated integration of genomic practice into the clinic and improvement of skills.

Conclusions

As the national discussion about the challenges of integrating genomics into clinical practice continues, critical evaluation of the approaches being taken to move providers from awareness to implementation is needed. We report on the Medicine’s Future experience as an example of applying best practices in adult learning to genomics education. These principles are well known but have not been implemented extensively for genomics education. Partnerships with organizations experienced in delivering effective education to clinicians should be undertaken as a step toward maximizing positive outcomes. Additionally, education that can be modified for providers with differing levels of experience and skill needs will help to prepare the broad health-care workforce for genomic medicine. The development and dissemination of quality, reusable education will further the goal of integrating genomics broadly and appropriately into clinical settings.

Disclosure

Development and delivery of Medicine’s Future were funded by El Camino Hospital under a grant to the National Coalition for Health Professional Education in Genetics (NCHPEG) and Genetic Alliance. For the described project, E.K.R., T.I.N., and J.A.S. were supported by NCHPEG; currently, E.K.R. and T.I.N. are supported by The Jackson Laboratory and J.A.S. is supported by the Health Resources and Services Administration (HRSA). E.K.R. received an honorarium for participation in an HRSA grant review committee. She has received travel reimbursement for speaking engagements in clinical settings but no honoraria. The work of K.J.T. is supported by the American Medical Association. She receives travel reimbursement for participation on the Institute of Medicine’s Roundtable on Translating Genomic-Based Research for Health and the Veterans Administration National Research Advisory Council, as well as for speaking engagements at selected conferences; she does not accept honoraria for these activities. For the described project, L.O.D. was supported by a consulting contract with ECH. She is currently an adviser to Vitagene, KineticDX, and Health Gorilla, for which she receives no compensation but has options to purchase future shares. She also receives travel reimbursement for speaking activities at selected conferences. For the described project, S.S. was supported by a consulting contract with NCHPEG. She is currently supported by a consulting contract with The Jackson Laboratory and by the Fullerton Genetics Center (Mission Health System). The work of J.C.O. is supported by Genetic Alliance and by grants to Genetic Alliance from HRSA and the Centers for Disease Control and Prevention. He receives travel reimbursement for speaking activities at selected conferences.

The authors declare no conflict of interest.