Main

The rapid evolution of genetic and genomic technologies has revolutionized our approach to cancer risk assessment, screening, prevention, and targeted cancer therapies, heralding the era of personalized medicine. Genetic cancer risk assessment (GCRA) is a specialized clinical service to identify and prescribe high-risk screening and prevention care for individuals with increased cancer risk. Initially delivered primarily through academic health centers, commercial availability of genetic testing, direct-to-physician, and direct-to-patient marketing have accelerated the uptake of predictive testing for cancer risk in the community setting, where clinicians are often inadequately prepared to select and apply genetic tests, and interpret their results.1–6

In response to the national need for specialized training in GCRA, the Division of Clinical Cancer Genetics (CCG) at City of Hope Medical Center (COH) developed a National Cancer Institute-funded (R25 CA112486) Intensive Course in Community Cancer Genetics and Research Training for community-based clinicians. The goals of the course are to increase the number of clinicians with practitioner-level competence in GCRA and to promote community-based research participation. These goals are accomplished through a three-phase continuing medical education (CME)-accredited program of distance didactic learning, interdisciplinary face-to-face training, and continuing professional development activities to support the integration of high-quality, evidence-based GCRA services and research into practice. Development of the multimodal course was guided by the tenets of adult learning and medical education theory.7–16 The course design was also informed by feedback from participants in a 2-week traditional face-to-face pilot course who identified interdisciplinary case-based training and practice-centered support as the greatest strengths of the course but found the 2 weeks of face-to-face training challenging in terms of the compressed content and time away from practice and family.17

This article reports the impact of the course on clinician knowledge, skills, professional self-efficacy, and practice change and describes the evolution of a distance-mediated community of practice (CoP) in CCG to support enduring professional development for community-based clinicians. The innovative course design models the goals of lifelong learning and improved patient care envisioned by the Institute of Medicine (IOM) and other leading CME stakeholders.18–22

MATERIALS AND METHODS

Course description

The multimodal course incorporates the curricular and training resources of the City of Hope Cancer Genetics Education Program,23–25 clinical expertise from the Clinical Cancer Genetics Cancer Screening & Prevention Program Network, which has provided cancer risk assessment services to more than 7000 at-risk families,26 and institutional research and informatics resources. The didactic curriculum, previously described,17 incorporates key elements of the American Society of Clinical Oncologists cancer genetics curriculum27,28 and encompasses content covering basic genetics and oncology, documentation and verification of the family cancer history, hereditary syndrome recognition, genetic testing strategy, test results interpretation and communication, development of evidence-based personalized risk management guidelines, ethical, legal, and social implications (ELSI), and applied research methodology. In the initial course session (2006), the curriculum was delivered through traditional face-to-face lectures that were digitally recorded and converted to QuickTime™ (Apple Inc., Cupertino, CA) video. Recorded sessions were distributed by CD-ROMs and web-based download for distance learning in subsequent sessions (2007–2010). The multimodal course curriculum was delivered in three phases during a 14-month period (Fig. 1). Phase 1 includes 27 modules of distance-learning didactics delivered over 2 months, supplemented with discussion board and weekly 2-hour web conference sessions for evidence-based updates on each topic. Phase 2 consists of 5 days of face-to-face interdisciplinary training at the COH campus in Duarte, California. Participants are awarded up to 90.5 CME/CEU credit hours on completion of Phases 1 and 2. Phase 3 comprises 12 months of prescribed professional development activities to facilitate the integration of new learning and skills into practice.

Fig. 1
figure 1

Multimodal delivery and outcomes assessment schema for the City of Hope Intensive Course in Community Cancer Genetics and Research Training. The course promotes interdisciplinary knowledge acquisition, case-based skills development, and integration of new knowledge and skills into practice through a combined program of synchronous (web conferencing) and asynchronous (discussion board) distance learning and traditional face-to-face training workshops. Alumni are invited to participate in CME-accredited professional development activities beyond the year prescribed by the course, as part of a growing distance-mediated clinical cancer genetics community of practice (CoP). Using the framework of improved CME outcomes described by Moore et al.,21 multiple assessments in Phases 1–3 of the course go beyond levels 1–3A assessments of participation, satisfaction, and declarative knowledge (outcomes most frequently assessed in CME learning activities) to include assessments of procedural knowledge (level 3B), competence in the learning environment (level 4), and longitudinal performance in practice outcomes (level 5). Ongoing tracking of alumni practice activities after completion of prescribed Phase 3 professional development provides an objective source of long-term patient health outcomes tracking (Level 6), which is currently in progress.

Full size image

Phase 3 professional development activities are centered on participation in two CME-accredited web conference series hosted by the interdisciplinary course faculty and facilitated by Microsoft Liveâ„¢ (Microsoft Corporation, Redmond, WA) interface: CCG Working Group and Topics in Cancer Genetic Research (TICGR). CCG Working Group is a weekly 2-hour interdisciplinary cancer genetics case conference series in which COH and affiliated clinicians across the United States present cases from their community practices to discuss cancer risk assessment, surveillance, risk management, and research eligibility for cases covering the full spectrum of hereditary cancers. TICGR is a weekly 1-hour web conference seminar series focused on timely issues in cancer genetics research presented by COH faculty, guest lecturers from other academic institutions, and clinical affiliates. Among prescribed Phase 3 activities, course alumni are expected to participate and present cases from their practices in a minimum of two working group sessions, participate in at least two and present at one TICGR session, and disseminate evidence-based GCRA information in their community settings through professional and lay presentations.

Participants and recruitment

The course is purposefully designed to bring clinicians from different disciplines and practice backgrounds together for collaborative interdisciplinary GCRA training. Physicians (MDs), masters-degreed genetic counselors (GCs), advanced-practice nurses (APNs), or other advanced-degreed clinicians were competitively selected by an interdisciplinary committee of course faculty (one onco-geneticist, one PhD nurse, and two board-certified GCs). Selection criteria were academic and clinical background, need for GCRA services and/or an underserved population in their practice area, and documentation of institutional support for participants to initiate or expand their GCRA practices on return to their community practice settings. Announcements for the course were delivered by e-mail, listservs, and at annual meetings for national medical societies and organizations, and posted on the course website.

Instruments

Course assessment and evaluation instruments address five of the seven levels of expanded CME outcomes described by Moore et al.21,29 (listed in Fig. 1) and recently endorsed by leading CME stakeholders as essential to quality assessment of all CME and professional development activities.18 The learning and skills outcomes assessment instruments listed below were administered by SelectSurvey.Net 2006 (ClassApps.com, Overland Park, KS) at the time points indicated in Figure 1.

Cancer genetics knowledge test (assessment Level 3A/B)

A 96-item multiple choice test (coefficient alpha of 0.70 for internal consistency reliability) covering eight content domains: basic genetics, cancer history documentation, cancer genetics, oncology, recognition of hereditary cancer features, differential diagnoses, genetic testing strategy/test interpretation, ELSI and human subjects protections, administered at baseline (precourse) and post-Phase 2 (final day of face-to-face training).

Professional self-efficacy survey (assessment Level 4)

A 34-item Likert scale survey measuring participant perceptions about their knowledge and skills in six core competency domains: genetics/oncology, hereditary cancer risk assessment, genetic testing strategy/test interpretation, risk management, counseling/ELSI, and research collaboration, administered at baseline and post-Phases 2 (immediate postcourse) and 3 (14 months postcourse). Respondents select from five response choices per item, ranging from 0 = no experience/cannot assess, to 5 = very confident. Reliability of the domains ranged from coefficient alpha = 0.77–0.97 and was 0.97 for the overall composite score.

Case-based skills scenarios (assessment Level 4)

Beginning with the 2008 course session, skills related to the process of cancer history documentation and verification, hereditary cancer risk assessment, and ordering and interpreting genetic tests were assessed using two case scenarios of similar complexity and content, followed by open-ended questions. Scenarios were administered at baseline and post-Phase 2 and were scored independently by two CCG clinical faculty members using a point system based on accuracy, breadth, and depth of responses in three knowledge/skills domains: pedigree construction, cancer history documentation/verification, and risk assessment/testing strategy. Subscores were combined for a total score. Twelve randomly selected pre and postscenario sets were scored by both raters to provide an estimate of interrater agreement. Intraclass correlation coefficients (ICCs) for baseline and postscores were high (baseline ICC = 0.98; post-ICC = 0.99) and significantly different than zero (P < 0.001).

Practice and professional development survey (assessment Level 5)

Comparisons of participant-reported baseline and post–Phase 3 practice changes were assessed based on number of patients seen for GCRA services, number of patients given cancer prevention information, provision of personalized risk management recommendations, facilitation of patient participation in research, professional and lay educational outreach, and participation in prescribed Phase 3 activities (CCG Working Group and TICGR web conference participation and presentations).

Practice and professional development tracking log (assessment Level 5)

Participation in Phase 3 practice and professional development activities was also tracked by the host institution using Microsoft Live electronic attendance logs and summarized at the completion of Phase 3. Continued participation in these activities beyond Phase 3 was also tracked to examine longer-term professional development behaviors and practice activities (GCRA case presentations and enrollment of patients in cancer genetics research registries).

Statistical analyses

All data were entered into spreadsheets, audited for accuracy, and imported into SPSS v18.0 (Statistical Package for the Social Sciences, SPSS, Inc., Chicago, IL) for statistical analysis. The pre and postknowledge tests were scored according to the answer keys. Knowledge gains in the first year of the course (delivered entirely face-to-face) and the year of transition to the multimodal design were compared in order to assure equal learning efficacy. The self-efficacy instrument subscales were computed based on content validity: genetics and oncology, hereditary risk, testing strategies, risk management, counseling/ELSI, research collaboration, and total score. Total scores were used for statistical comparisons.

Descriptive statistics were computed for all scores and Practice Survey items. Two-way repeated measures analysis of variance (ANOVA) was used to compare knowledge change by course session and by practice discipline. Multivariate ANOVA was used to compare self-efficacy subscale scores by session and by discipline. Analyses on the three measurement points for self-efficacy incorporated the participants (n = 52) who completed the post–Phase 3 professional practice survey. A Bonferroni correction was used to adjust for experiment-wise inflation error. The McNemar test or paired t tests were used to compare differences in dichotomous or continuous practice survey items, respectively. A two-way repeated measures ANOVA was used to compare baseline versus post–Phase 2 score performance by discipline on case-based skills scenarios.

RESULTS

Descriptive demographics

Course announcements for sessions held between January 2006 and March 2010 yielded 402 formal applicants. Of these, 135 were competitively selected to participate, 131 accepted, and three deferred participation to a later session due to schedule conflicts. Participant demographics are summarized in Table 1. All 131 participants (48 MDs, 41 APNs, and 42 GCs) completed all Phase 1 distance learning and Phase 2 face-to-face training requirements. One-hundred twenty-four participants practiced in the United States (43 states represented), and seven (5.3%) were from countries outside of the United States. Participants were predominantly women (86%), white (82%), had more than 10 years of clinical experience (58%), and were providing GCRA as part of their practice at the time they participated in the course (76%). Nineteen percent reported African American or Hispanic ancestry and 57% practiced in a geographically underserved area and/or provided minority outreach services. MDs, APNs, and GCs were represented in roughly equal proportions for all sessions. Twelve (six pairs) clinicians were trained as teams, wherein a physician and mid-level APN or GC colleague from the same practice setting completed the course together.

Table 1 Participant demographics and baseline clinical characteristics
Full size table

No significant differences were evidenced in knowledge acquisition or professional self-efficacy between the 2006 session (delivered through a traditional lecture-based format) and the 2007 session (conversion to the three-phase multimodal format). As shown in Figure 2A, Level 3A/B declarative/procedural knowledge significantly increased from baseline to post-Phase 2 among participants overall (P < 0.001) and in all sessions (mean increase 28%; range: 12.2–38.5%). By-discipline comparisons of baseline to postcourse knowledge demonstrated statistically significant increases for GCs, MDs, and APNs (P < 0.001 for all disciplines). Differences in between-discipline comparisons of baseline knowledge scores were statistically significant (P < 0.001), with the highest baseline scores among GCs (72%), followed by MDs (67%) and APNs (59%). Postcourse knowledge scores for GCs and MDs (85%) were significantly higher (P < 0.001) than for APNs (75%).

Fig. 2
figure 2

Intensive course knowledge and cased-based skills outcomes overall and by practice discipline. A, Posttest knowledge scores were significantly higher than pretest scores overall (P < 0.001) and by discipline (MDs, APNs, and GCs, P < 0.001). B, Post–case scenario scores were significantly higher than pre–case scenario scores overall (P < 0.001) and by discipline (MDs and APNs, P < 0.001; GCs, P = 0.007). *Case scenarios were administered beginning with the 2008 course session. Only complete datasets were analyzed (n = 83).

Full size image

As illustrated in Figure 2B, Level 4 assessment of competency and skills in an education setting, measured by case-based skills scenarios, increased significantly from baseline to post-Phase 2 for all disciplines (P < 0.001 for MDs and APNs; P = 0.007 for GCs). Between-discipline comparisons of skills scores were significantly different, with the highest baseline scores attained by GCs (59%), followed by APNs (51%) and MDs (44%). GC and APN postcourse skills scores were significantly higher (64% for both) than postscores for MDs (58%).

Table 2 summarizes Level 4 changes in self-efficacy overall and by practice discipline. All 105 participants who reached the post–Phase 3 time point (14 months from baseline) at the time of analysis had completed baseline and post–Phase 2 professional self-efficacy surveys. Of these, 52 (50% response rate) also completed the post–Phase 3 self-efficacy survey. Overall and subscale self-efficacy scores increased significantly from baseline to the post–Phase 2 measurement for all disciplines (P < 0.001). These increases were maintained or further increased at the post–Phase 3 time point. Between-discipline comparisons of self-efficacy scores demonstrated a significant interaction effect in the overall (P = 0.003) and three of the six subscale scores: genetics/oncology (P = 0.002), hereditary cancer risk assessment (P = 0.001), and test strategy/interpretation (P < 0.001).

Table 2 Baseline to 14-mo postcourse professional self-efficacy outcomes (Level 4 self-assessment of competence in practice)
Full size table

More than 85% of alumni fulfilled course-prescribed web-based professional development activities (Working Group and TICGR participation), and all 57 participants (54% response rate) who completed postcourse professional practice surveys completed all prescribed Phase 3 Level professional development activities. As summarized in Table 3, there were significant baseline to post–Phase 3 increases in the number of patients seen for GCRA (P = 0.018), number of patients given information or resources for cancer prevention studies (P = 0.005), and number of patients enrolled in hereditary cancer registries (P = 0.007). There were also significant increases in the number of participants who provided high-risk screening and prevention recommendations to patients (P = 0.004), and who conducted professional education (P = 0.003) and lay educational outreach (P = 0.041). Additionally, ongoing professional development tracking revealed that more than 30% of alumni continue to participate in Phase 3 web-based professional development activities beyond the prescribed 12-month period. Course alumni have presented more than 800 cases from their community-based practices in weekly CCG Working Group web conferences and have enrolled more than 2000 patients in cancer genetics research registries or protocols. Moreover, 14 course alumni (eight MDs, three APNs, and three GCs) established formal interinstitutional collaborations with the COH hereditary cancer registry30 to promote community-based accrual of patients into cancer research protocols. Six additional alumni (four APNs and two GCs) attempted to develop collaborations but were unable to work through administrative and regulatory barriers (such as lack of support by their administration or prohibitive institutional review board fees).

Table 3 Comparisons of baseline to 14-mo postcourse professional practice outcomes (Level 5 performance assessment)
Full size table

DISCUSSION

The forces of genetic discovery and direct-to-consumer and physician marketing are driving increasing demands for genetic tests that predict health risks.1–6 Technology and market-driven pressures notwithstanding, leading oncology and genetics stakeholders strongly recommend that predictive genetic testing be conducted by qualified cancer genetics providers.31–33 Oncology, genetics, nursing and government health organizations, and some academic institutions offer cancer genetics seminars, workshops, and web-based GCRA resources, and the American Society of Clinical Oncologists Cancer Genetics & Cancer Predisposition Testing Curriculum is a self-teaching resource for oncologists and other healthcare providers.27,28 At present, even within the field of medical genetics, there is no practice credential offered for the subspecialty of cancer genetics. Programs such as the Intensive Course in Community Cancer Genetics and Research Training described in this report are, therefore, needed to train clinicians for practitioner-level competence in GCRA.

The course outcomes demonstrate an effective transition from the face-to-face pilot course that served as its foundation17 to a multimodal three-phase course that combines the flexibility and convenience of distance learning with interdisciplinary face-to-face training and continuing practice-centered support. The course design was informed by the tenets of adult learning theory,34,35 evidence demonstrating the ineffectiveness of traditional, didactic-focused approaches to CME,36–39 and calls from leading stakeholders across the CME enterprise for practice-centered models of learning that enhance quality healthcare, support professional activities, and produce measurable outcomes.13,18,40–42

The majority of participants in the six sessions included in this analysis were women, in part reflecting the dominance of women in the genetic counseling and nursing professions. Despite the fact that 29 of 48 (60%) of the physician participants were also women (possibly reflecting a motivation for training among clinicians who practice in women's health programs, where market-driven demands for BRCA testing are concentrated), there has been a steady increase in the number of male physicians applying to and participating in the course over time. For example, of the 21 participants in the January to March 2011 session, 8 of the 14 physicians are men (four are providing GCRA services within women's health programs, and four are training to develop GCRA programs in broader-spectrum practice settings).

Significant pre-to-post gains were demonstrated overall and by practice discipline in procedural knowledge, professional self-efficacy (a recognized surrogate for actual performance),43–45 and case-based skills. Similar to outcomes from the pilot intensive course, GCs had the highest baseline scores on all measures, and postcourse knowledge scores were significantly higher among GCs and MDs compared with APNs.17 However, MDs scored significantly lower than GCs and APNs on postcourse case-based skills. Although generally accurate, open-ended responses by some MDs were cursory and lacked detail, whereas GCs and APNs more often provided detailed pedigrees and nuanced responses. A more in-depth review of the case scenario responses may help elucidate ways to improve methods for assessing GCRA skills outcomes and better identify discipline-specific learning needs.

Beyond knowledge and skills acquisition, 14-month postcourse assessments demonstrated anticipated gains in the delivery of GCRA services, educational activities, and research participation among course alumni. Although only 50% of the 105 post–Phase 3 alumni completed all Phase 3 professional development practice and self efficacy surveys, internal tracking of engagement in CCG Working Group and TICGR conferences revealed that 85% fulfilled or surpassed the minimum prescribed participation in these activities. Tracking records and alumni feedback suggest that clinicians who practice in procedurally dominated specialties (such as surgeons and medical oncologists) are less likely to participate fully in postcourse professional development activities, in large part due to time constraints and schedule conflicts. Among the six pairs of clinicians who trained as teams, the mid-level clinicians more frequently engaged in Phase 3 web-based activities and continue to represent the team in these activities beyond the required Phase 3 timeframe. This finding reflects the value of our multidisciplinary training model and supports the rationale for cross-disciplinary team learning described as a priority by the IOM (2010).

An associated observation is that although many mid-level clinicians expressed interest in establishing formal cancer genetics research collaborations after completing the course, few were able to get through the regulatory process required for participation in the research network. This suggests that a physician champion is more likely to be empowered to commit the practice, time, and resources required and to successfully navigate the administrative and regulatory process involved in establishing these collaborations, and points to the value of the multidiciplinary team in the development of GCRA programs.

The growing roster of course alumni who participate in Phase 3 CCG Working Group and other professional development activities beyond the prescribed 12-month Phase 3 period is an unexpected outgrowth of the course experience. Findings from an action research project conducted with course alumni revealed that many continue to participate in Phase 3 activities to support evidence-based GCRA patient care and research collaboration in their practices.46

Situated learning theory47 describes how learning is transformative when professionals connected by common practice-centered needs and learning goals come together in informal CoP settings. Through regular CoP participation, new learners move from peripheral to deeper engagement in the learning culture, eventually assuming the role of expert.48 Ongoing participation in CCG web-based activities by a motivated subset of course alumni extends the impact of the course experience beyond near-term changes on knowledge and skills and now forms the nexus of a thriving distance-mediated CoP in CCG for clinicians across the United States.

CoPs have been a recognized professional development resource in the business and education sectors for decades, but the study of their efficacy in healthcare settings has only gained momentum in recent years.49,50 Increases in professional self-efficacy scores at 14 months postcourse may be attributed in part to participation in CoP activities. Presentation of more than 800 GCRA cases by course alumni provides an objective measure of CME assessment Level 5 performance in practice, tracked through engagement in enduring CoP activities that directly inform evidence-based patient care. CCG clinical faculties have observed progressive improvements in GCRA-specific skills among alumni who participate regularly in COH-hosted CoP activities. Future efforts will expand examination of the effects of enduring CoP participation on Level 5 performance outcomes, and additional methods are currently being piloted for longitudinal Level 6 patient health outcomes assessments. Additional interfaces, including a web-based and topic-specific discussion group and listserv, are also under development to augment the distance-mediated CoP experience.

Limitations

The majority of course participants had training in oncology or genetics; consequently, generalizability of the findings and adaptability to more primary care audiences is uncertain. Although not analyzed separately, our limited experience with training primary care and obstetrics–gynecology clinicians (n = 13) suggests similar gains in knowledge, professional self-efficacy, and skills for these participants. The breadth of topics covered in the course is comprehensive and might be beyond the scope of need for some clinicians, but the modular nature of the course lends itself to tailoring for focused interest tracks, such as a breast cancer track for clinicians working exclusively in a breast center setting. Although the case scenario instrument has demonstrated content validity and scoring reproducibility, additional validation is warranted with a more diverse set of clinicians.

CONCLUSIONS

The significant impact of the COH Community Cancer Genetics and Research Training initiative on multiple practice-based outcomes supports the efficacy of the multimodal course design for clinical proficiency training in GCRA. The course was originally funded to provide for 15 competitively selected community-based participants yearly. However, CCG Division resources, aided by American Recovery and Reinvestment Act funds (ARRA no. 3R25CA112486), have been extended to help address the steady increase in demand for cancer genetics training. To date, the expanded program has trained 131 clinicians from 43 states across the United States—more than double the initial target accrual goal; however, there is still unmet need for GCRA training both in the United States and abroad. Seven clinicians representing CCG initiatives in Brazil, Canada, Chile, Hong Kong, and Spain completed the course and continue to participate in web-based CoP activities introduced through the course. Access to the genetic and genomic tools of personalized medicine is nascent in many of these regions. Future efforts should include locally relevant cancer genetics training to address disparities in other countries.

As recently outlined by the IOM, leading CME stakeholders across the full spectrum of healthcare have set forth a mandate to promote CME activities that are purposefully designed to improve practice-based competencies and address lifelong professional learning needs.18–22,51 Beyond the near-term learning efficacy of the course, initial assessments of sustained participation in CoP activities demonstrate the potential of web-based CME for continuing professional development.

The multimodal, multimetric design of the course and its distance-mediated CoP successfully leverage the expertise of the academic health center to help address the need for a genetically literate workforce. This dynamic continuum of advanced training and enduring practice-centered support models the vision of lifelong professional learning that promotes quality care and improved patient outcomes. Future efforts will focus on the direct impact of CoP-generated learning with changes in patient outcomes, in alignment with the highest levels of CME accountability.