Main

Within the primary care community, there is an enormous unmet educational need for practical, understandable, and accessible learning materials around clinical genetics. At present, more than 1800 conditions are known to have at least some genetic basis,1 with thousands more having both environmental and genetic contributors.2 In the United States, more than a quarter-million practicing primary care physicians (PCPs) provide medical care to 300 million individuals3; however, most physicians have little training in clinical genetics. At the same time, there are only 5000 genetic specialists in the United States,4,5 the majority of whom (66%) work in academic health centers,6 with only 8% working in unaffiliated hospitals. Therefore, few genetic professionals are available for either medical education or direct patient counseling, with smaller hospital systems and residency training programs disproportionately underrepresented. Given this mismatch, PCPs will bear the bulk of initial screening and counseling patients on medical, ethical, legal, and social implications surrounding genetic disorders and testing.7 PCPs report the need for increased skills in these areas8 and report that barriers to providing genetic services to patients by PCPs include lack of knowledge, clinically relevant resources, and confidence in their skills in genetic medicine.9 Meanwhile, these problems are exacerbated because patients are bombarded with direct-to-consumer genetic testing options, without appropriate understanding of implications of such tests on their lives.10

To address these needs, the National Institutes of Health-National Human Genome Research Institute and other organizations11,12 funded resource development in medical genetics (such as GeneClinics.org13), curricula on medical genetics (such as Genetics in Primary Care14,15), and more recently, curricula on ethical, legal, and social implications (ELSI)16,17 of genetics in medicine.

Carefully developed, engaging, interactive, case/problem-focused, web-based curricula may fill in educational gaps for content areas with low educator availability, such as in ELSI genetics. Well-designed web tools use practical, active learning, and engagement,18 especially useful where local expert faculty are unavailable to facilitate discussion. In other settings, interactive computer technologies have successfully enhanced medical decision making,19,20 promoted examination21 or procedural skills,22 and presented virtual patient cases.23 For instance, a commercial web-based tool for medical students on genetic counseling and ELSI genetics improved learner knowledge and self-efficacy (SE).24

We developed and tested an interactive, web-based, case-driven curriculum on ELSI genetics for PCPs. We implemented the ELSI genetics curriculum in nine primary care residencies, in three specialties at three institutions. Our two research goals were to (1) study changes in resident physician SE after ELSI genetics participation and (2) understand participant viewpoints about advantages/disadvantages of this type of web-based curriculum.

MATERIALS AND METHODS

During a 4-year period, we developed and implemented an interactive web-based primary care curriculum on ELSI genetics, which we then evaluated using a pre-post design. Our curriculum featured 10 clinical cases and five content-specific tutorials geared toward residents in pediatrics, internal medicine (IM), and family medicine (FM). Implementation occurred at three university hospitals: University of California Davis (UCD), University of California Los Angeles (UCLA), and University of Washington, Seattle (UW). All three institutional review boards approved the study.

Curriculum development

Using Kern's curricular development model,25 we developed curricular content with the help of 20 experts in medical genetics, medical ethics, law, medical anthropology, clinical medicine, educational assessment, and educational technology, from six institutions. Essential PCP-ELSI genetics competencies were identified from organizational recommendations26 and published reports.7 We created 10 cases to illustrate genetic conditions, linked to competencies/learning objectives. Content needing in-depth explanation was developed into five content tutorials (Table 1). An electronic portfolio housed content and allowed for curriculum management, with separate entry for learners and program directors/administrators. Screen shots of cases, tutorial, portfolio, and other features are found in Appendix, Supplemental Digital Content 1, http://links.lww.com/GIM/A153.

Table 1 Cases and tutorials on ethical, legal, and social implications (ELSI) in medical genetics
Full size table

Cases

The 10 cases covered 10 conditions (Alzheimer disease, androgen insensitivity syndrome, breast cancer, colon cancer, cystic fibrosis, Down syndrome, hemochromatosis, Huntington disease, Klinefelter syndrome, and thrombophilia), and each took 30–60 minutes to complete. Each case used multiple high-quality video vignettes (e.g., interactions between patients and physicians) to highlight ELSI genetics tensions. Each video or interactive exercise was followed by a relevant discussion between animated characters. Within the cases, learners answered multiple-choice and open-ended questions to promote understanding and engagement. Some video clips also contained unscripted interviews with patients or experts.

Tutorials

Each of the five tutorials took 30–120 minutes to complete. Tutorials used shorter written or video cases in a didactic, but still interactive, format focused on ELSI or medical genetics content. Five tutorials were (1) core concepts in genetics, (2) ELSI cultural issues, (3) medical ethics and law, (4) risk metrics/disease screening, and (5) maternal fetal medicine.

ePortfolio

After completing a case, learners accessed an electronic portfolio (eDoctoring), where they could identify their own challenges and strengths relevant to case competencies and address what they hope to do differently in their future interactions with patients based on case content. After completion of all portions of the tool, learners printed a “certificate of completion” detailing their participation, cases, and competencies. Additionally, program directors could send learners reminders, comments about their learning goals, and generate site-specific progress reports.

Programming was done at the University of Newcastle, UK. The eDoctoring software was programmed in Flash and Python, linked to MySQL database using Zope and housed on multiple load-balanced servers in the United Kingdom and the United States allowing hundreds of learners to be online simultaneously. The curricula were alpha tested with six 4th year medical students to improve flow, consistency, and navigation.

Survey instrument development

Before participation, learners participated in a precurricular survey to collect information about gender, prior experience (three aggregate items), SE (23 items) and knowledge (14 items) about ELSI genetics. Prior experience had three components, each an aggregate (Appendix, Supplemental Digital Content 2, http://links.lww.com/GIM/A154). To assess prior personal experience, learners were asked if they had personal experience with family or friends with genetic disorders, and if so, how involved they were in their care no personal experience (0), experience with no involvement (1—not or slightly involved), and involvement (2—moderately or significantly involved). Prior clinical exposure was the sum of experience with the 10 conditions highlighted in our curriculum (range: 0–10). Prior direct clinical experience was the sum of how frequently (from never = 0 to often = 3) learners had done four things: worked with a patient whose main problem was a genetic disorder, ordered a genetic test, referred to a genetic professional, or interacted with a genetic professional. We did not define “genetic disorder” for participants.

SE and knowledge items were drawn from learning objectives. SE items27 asked, “How confident are you about your current skills in the following areas?” Learners rated their confidence from “not at all confident” (1) to “extremely confident” (5). Sixteen items were classified into five domains. Psychometric item analysis and maximum likelihood factor analysis (with varimax rotation) were used to assess scale internal consistency and dimensionality. Scale refinement was conducted by using baseline data to develop initial scales and follow-up data for validation. We partitioned one domain, transferred four items to new domains, and deleted three items. Cronbach α values for each domain ranged from 0.79 to 0.91 in postcurricular dataset. An overall SE scale comprised the sum of item scores from the six domains (range 23–115; α = 0.95; Table, Supplemental Digital Content 3, http://links.lww.com/GIM/A155). The first extracted factor explained over 90% of common variance, supporting underlying scale score unidimensionality. Knowledge questions addressed ELSI issues (not genetic medical information) by presenting a clinical scenario, followed by true/false or single best answer responses (Appendix, Supplemental Digital Content 4, http://links.lww.com/GIM/A156).

After each case, learners described how thoroughly their current residency and the ELSI genetics curricula covered the clinical condition (1 = not at all covered and 5 = extremely well covered), and how they would apply case content in their practice. A postcurricular survey included SE, knowledge items, and participant viewpoints. Participant viewpoints included recommending curricula to other residents (one item), overall satisfaction (one item), preferred learning formats (one item), comparison with traditional curricula (nine items), curricular features (four items), and learning application (six items). Comparison with traditional curricula asked, “In comparison with traditional curricula, how likely is this web-based program …” followed by nine items (5-point scales: 1 = much less likely; 3 = same as; and 5 = much more likely). Six learning application questions were preceded by the stem, “When you participated in this curriculum, how often did you …” (never, occasionally, and somewhat, often, always).

Implementation

We implemented the ELSI genetics curricula at nine residency programs in pediatrics, IM, and FM at three institutions (UCD, UCLA, and UW). Each residency program selected five cases and two tutorials, with common pre- and postcurricular surveys. Implementation varied by site (case conferences to ambulatory or research blocks). At UCD and UCLA, curriculum participation was mandatory and integrated into existing curricula. UW made curricular or research participation voluntary. All participating residents received face-to-face orientation from program staff/faculty. At sites with mandatory participation, noncompleting residents received 1–3 e-mails or pages from program staff or faculty to complete the curriculum. Curricula could be accessed from any web-enabled computer with a high-speed internet connection. Local computers were loaded with the appropriate hardware (soundcards, etc.) and software (Flash, Quicktime™). Hospitals/clinic technical staff worked with our program staff to overcome minor technical issues, such as firewalls issues disallowing internet access or video streaming.

Analysis

Analysis of variance was used to compare baseline levels of prior personal experience levels between specialties. Our primary outcome was changes in SE scores, assessed using paired t tests. Associations between precurriculum SE and change in SE scores with independent variables were assessed using multiple regression. Independent variables included gender (male), institution (precurricular survey: UCD, UW, and UCLA [reference]; postcurricular survey: UCD and UCLA), residency (IM, FM, and pediatrics), prior personal experience, prior condition exposure, and prior clinical experience. Similar analyses were conducted with precurricular knowledge and change in knowledge scores. Postcurricular viewpoints were reported with descriptive statistics. Bivariate comparisons of viewpoints by specialty revealed only minor differences and are not reported. For learning applications, we collapsed responses to “never, occasionally/sometimes, and often/always” based on score distribution. After each case, learners rated extent of medical and ELSI content coverage provided by our web-based curricula and by traditional curricula for each of several areas. We used mixed-effect regression models to analyze ratings for a given area, across all cases/topics. These models allowed us to assess mean within-learner differences (ELSI versus traditional curriculum), while properly accounting for repeated measurements collected from each learner (one set for each case). In these models, learners were specified as random effects, and topic was specified as a fixed effect.

RESULTS

Enrollment and completion

We enrolled 80% of eligible residents at UCD (95/120), 75% at UCLA (154/205), and 38% at UW (30/78). Fifty-eight percent of enrolled residents were women. Of enrolled residents, 77% at UCD (73/95) and 89% at UCLA (137/154) completed the curriculum, in comparison with only 7% at UW (2/30). Curricular participation was voluntary at all UW sites. Because of competing time demands for residents, UW IM (n = 20 interns) stopped ELSI genetics curricular participation after 2 months. UW pediatrics successfully launched but did not complete curricular participation on two rotations (0/42 resident completing), whereas UW FM had only one resident finish the cases/modules and survey. UW sites were dropped from postcurricular analysis due to limited participation.

Prior experiences

Based on precurricular survey, 75% of all enrolled residents (n = 279) had no personal experience with family or friends with genetic disorders. Of those with personal experience, approximately half (34/279) were not involved in the care of that friend/family, whereas half (35/279) were involved. We found approximately 10% variation by specialty in prior clinical exposure to the conditions in our curriculum, with pediatric residents having the most exposure (Table, Supplemental Digital Content 5, http://links.lww.com/GIM/A157).

Overall, 45% of residents had never ordered a genetic test, across specialties and sites (Table, Supplemental Digital Content 6, http://links.lww.com/GIM/A158). Of those who had ordered genetic tests, 21% had not performed any pretest counseling, 43% performed only brief counseling, and only 35% counseled for a few minutes or more. All pediatric residents (99/99 [100%]) but fewer IM (98/113 [87%]) and FM (51/67 [76%]) residents reported having cared for patients whose primary problem was a genetic disorder. Referrals varied by specialty, with 75% of IM residents and 40% of FM residents reporting having never referred a patient to a genetic professional, whereas 92% of pediatric residents had referred. Of those referring to a genetics profession (a few times, sometimes, and often), 97% (152/157) had interacted with a genetics professional by phone or in person. Additionally, many IM (49%) and FM (34%) residents knew nothing about the background, skills, or access to genetic professionals (counselors or specialists in genetic medicine), in contrast to only 6% of pediatric residents who reported no such knowledge. In analysis of variance models, there were no significant differences between residents at UW and the other two sites for prior clinical exposure, personal exposure, or direct clinical experience.

Self-efficacy

Initially, residents were moderately confident about their genetic skills (initial SE scores: all residents = 71; UCD/UCLA residents = 66; scale range 23–115). Using multivariate linear regression, higher baseline SE scores were associated with prior exposure to genetic conditions (parameter estimate = 1.7; P < 0.01) and more personal experience (1.9; P < 0.01) but not with site or prior direct professional experience. One specialty (FM) independently had lower initial SE scores (−6.2; P < 0.01). Prior disease exposure and personal experience were associated with higher SE in all six subscales, whereas direct clinical experience was associated with higher SE in the genetic knowledge subscale alone (1.3; P < 0.05). For those residents completing the curriculum (n = 210), their overall confidence increased by approximately 15% after participation (UCD/UCLA residents, pretest score 71, to posttest 83.4). As listed in Table 2, using paired two-tailed t tests, residents' SE improved significantly (P < 0.01) for all six subscales in all specialties, except one, with greatest improvement in “uncertainty and decision-making” and in “communication and information sharing.” FM residents had a trend toward improved SE in regards to ELSI cultural issues (P = 0.07). Improvements in overall and subscale postcurricular SE were unassociated with gender, site, prior exposure, prior direct clinical experience, and number of cases completed. Pediatric residents had slightly lower SE change scores in uncertainty and decision making.

Table 2 Self-efficacy scores before and after curricular participation, for residents completing the curriculum (n = 210)
Full size table

Knowledge

Initially, residents from all three sites answered 9.6 (of 14) questions correct. Knowledge scores did not change significantly after participation. Completing residents answered 10.0 (of 14) questions correctly afterward. There was no association between gender, specialty, institution, prior exposure, prior personal experience, prior clinical experience, and changes in knowledge scores.

Participant viewpoints

There were minor or no differences between specialties.

Satisfaction and preferences

Residents were likely to recommend the curriculum to other residents (3.8/5, 5 = definitely recommend) and were satisfied overall with the curriculum (3.7/5, 5 = very satisfied). Residents reported that they would prefer to participate in this web-based program as delivered (78%), compared with the options of small group discussions of ELSI material (33%) or having both web based and small group review (8%). Representative positive and critical comments are included in Appendix, Supplemental Digital Content 7, http://links.lww.com/GIM/A159.

Advantages/disadvantages

Based on end-of-case survey responses, residents felt that this web-based curriculum covered all medical management, genetics, ethical/legal, and social/cultural issues significantly better than their usual curricula (Table 3). Overall, in comparison with traditional lecture-based curricula, residents felt that the web-based curriculum was better at providing flexibility, fitting into their schedule, and demonstrating good and bad communication strategies (Table 4). For most other domains (including exploring content and facilitating long-term retention of knowledge), residents felt that the web-based curriculum was about the same or better than the traditional curriculum. Sixty-seven percent, however, felt that this curriculum provided less faculty contact than traditional curricula.

Table 3 Resident perceptions of how thoroughly the ELSI genetics curricula and their usual residency curricula covered the conditions discussed in each case (n = 210)
Full size table
Table 4 Resident's comparison between web-based curriculum and traditional residency curriculum, after ELSI genetics curricular participation (n = 210)
Full size table

Application

Residents reported significant use and application of curricular content in the following areas (Table 5): using curricular content in patient care settings (occasionally/sometimes [occ] 57% and often/always [often]13%), to create personal learning goals (occ 52% and often 13%), and act on their learning goals (occ 50% and often 13%). Ninety percent of residents thought about the content covered in the tool later in their training (occ 72% and often 17%), and more than 60% thought about their learning goals later (occ 50% and often 14%).

Table 5 Resident's report of their utilization of ELSI genetics content, after curricular participation (n = 210)
Full size table

DISCUSSION

Our interactive, web-based, case-driven curricula on ELSI genetics modestly improved the SE of ELSI genetics of participating primary care residents in three specialties. Prior clinical experience with genetic tests, patients with genetic disorders, or genetic professionals was associated with increased baseline SE; increases in SE were unassociated with prior experience and were likely driven by curricular participation. Residents identified a variety of advantages of the ELSI genetics web-based curricula over traditional curricula, especially in promoting self-directed learning and providing flexibility for participation. Before participation, residents across specialties reported little personal experience with family member or friends with genetic disorders. Pediatric residents had more clinical experience with patients with genetic disorders. After participation, residents reported applying the ELSI genetics material in patient care settings, thinking about content after the curricula was over, and creating and acting on their action plans created after module completion.

For those residents who completed the curriculum, we found an improvement in resident SE across all six major factors in all three specialties. SE asks an individual to reflect about their confidence level in a particular skill. Changes in SE have been shown to predict behavioral change. In one study, SE variation explained a large amount of variance in US health educators' adoption of genomic practices into their practices.28 Although not an endpoint, this SE improvement may be an important mediator in appropriate utilization of ELSI genetics in primary care practices. The ELSI focus of our genetics curriculum contrasted strongly with the strictly disease-oriented focus of residents' usual curricula. Despite this shift, participating residents were largely satisfied with the content and quality of the ELSI genetics curricula. As Cook29–31 identifies, web-based technologies have specific advantages but may not be better than other educational techniques to promote learning. Our residents noted this web-based curriculum was as good as, or better than, traditional curricula for many domains, especially in promoting self-directed learning and providing flexibility of implementation and learning in an applied case-based format. Similarly, medical students participating in a commercial web-based tool (cases featuring interviews with patients with genetic disorders, focusing both on medical and ELSI genetics) were well satisfied with their learning experience.24

Our study results should be considered in light of several limitations. Although there were no baseline differences between residents at UW and the other two implementation sites for our dependent variables or precurricular SE, we cannot assess the effect of UW resident drop-off on our results. Although completing residents reported that they would apply what they learned in their practices, we did not assess changes in their clinical behaviors. Our learners had no significant changes in multiple-choice or true/false ELSI knowledge scores after curricular participation. In part, this may be due to difficulty writing test questions for higher order medical ethics decision making in a true-false or multiple-choice format. For instance, Kim et al.32 demonstrated that medical students selected patient-centered approaches to care on multiple-choice questions (socially desirable responses), whereas having more doctor-centered approaches to care on free text responses (less socially desirable). Although scoring burden is increased by using free text responses to ethical dilemmas, appropriate differentiation between groups is more likely with open-ended responses for higher order decision making. Given the heterogeneity of implementation methods by program, we did not stratify our results by learner year. Earlier learners may have been more likely to find our curriculum novel than senior residents who had more clinical exposure. We also relied on local champions to implement the curriculum in a manner most suitable for their learners—single rotation, conference schedule, etc. Thus, although all learners participated in same volume of content, they did so in several ways and over different time intervals (1–6 months). Given our sample size, we did not examine the effect of implementation method on our outcomes.

From a developmental perspective, this curricular creation was unlike usual collaboration around articles or creation of small group teaching seminars. More akin to new product development in industry, we defined the infrastructure, architecture, and teaching methods simultaneously and iteratively. As we did not have an example of the final product as a model for development, investigators often struggled to create appropriate material with a conceptual framework alone or with changing interim mock-ups. Future development will ensure closer collaboration between the content experts, designers, and technology experts to provide more efficient model of multisite web-technology collaboration.

From a curricular perspective, our findings suggest that web-based curricula have advantages (ease of scheduling, effective modeling by video vignettes, etc.) that can supplement but should not replace traditional didactic education (especially small group learning). Our multisite implementation was challenged by technological and implementation issues. Early curricular versions had programming glitches that were debugged before and during implementation. Local firewalls often prohibited internet access or slowed video streaming. Successful implementation was dependent on local faculty champions taking ownership of the curriculum, program leaders who facilitated learner participation, local technical personnel working closely with our centralized technology staff, and local staff presence to troubleshoot. Integration of this curriculum into existing learning experiences improved participation by letting residents learn ELSI genetics during time already set aside for learning. By integrating or substituting our material into existing time slots, we did not increase the educational burden of residents or their program directors. In settings where participation was not expected or integrated, completion rates were low.

From an administrative perspective, participating program directors informally reported that they found the curricular content useful. They were pleased with curricular reporting functions, which could help satisfy Accreditation Council for Graduate Medical Education requirements to demonstrate practice-based learning and improvement. Additionally, they found several features very useful, including the ability to track learner participation with ease and send e-mail reminders from within the curriculum.

In summary, the ELSI genetics curricular development provides a model for multiinstitutional web development, and lessons learned herein can improve future multisite implementation around web-based curricula. Within the next year, this program will be freely available to medical educators. Although technological advances make it possible to reach a larger group of learners on topics such as ELSI genetics, which are locally underresourced, attention to core educational principles enhances participation, completion, and subsequently, learning. Future work will expand on action planning functionality to enhance content implementation in learner's actual practice.