Main

Approximately 3% to 5% of colorectal cancer (CRC) is caused by the autosomal dominantly inherited cancer syndrome known as hereditary nonpolyposis colon cancer (HNPCC). HNPCC is caused by germline mutations in one of at least five known DNA mismatch repair genes. Individuals with HNPCC have a lifetime risk of 70% to 80% for colon cancer, 30% to 61% for endometrial cancer, and an increased risk for a spectrum of other extracolonic cancers, particularly those of the ovary, stomach, small bowel, ureter, and renal pelvis.1–3 The risk for cancers of the colon and the endometrium before the age of 50 years is greater than 25% and 20%, respectively.1 Frequent colonoscopy (interval <3 years) initiated at an early age is recommended,4–6 and when combined with polypectomies, has been shown to reduce the rate of CRC by 62% in families with HNPCC.7 Unfortunately, HNPCC can be difficult to distinguish from sporadic CRC because of a lack of pathognomonic features.

Diagnosis of HNPCC is based primarily on family history. Perhaps the most widely known criteria for establishing the diagnosis are the Amsterdam I criteria8: (1) three or more relatives with histologically verified CRC, one of whom is a first-degree relative of the other two; (2) familial adenomatous polyposis should be excluded; (3) CRC involving at least two generations; and (4) at least one CRC case diagnosed before the age of 50.

Genetic testing for the genes associated with HNPCC is clinically available and increasingly being used as a basis for diagnosis. Of importance, genetic testing has the potential to improve clinical management of patients putatively affected and their families through targeted surveillance and surgical options. Anecdotal experience suggests that positive genetic test results may even improve patient compliance with the intensive surveillance regimen.7 However, inappropriately applied genetic testing may do harm if, for example, patients with noninformative negative results are falsely reassured. Because frequent surveillance can reduce the incidence of cancer and mortality, some have proposed population screening for the genes that cause HNPCC9; however, this approach is not widely agreed upon. Recently, the American Gastroenterological Association (AGA) developed a set of guidelines called the modified Bethesda Criteria10 (Table 1) to aid physicians in identifying which individuals in their practices may benefit most from genetic testing for HNPCC. Microsatellite instability testing (MSI) may be performed on the colorectal tumors of individuals whose histories satisfy the modified Bethesda Criteria to determine which of them should proceed to germline genetic testing for HNPCC.10 Individuals with tumors that exhibit high MSI, as defined by a consensus document of the National Cancer Institute, would be advised to undergo germline genetic testing for specific mutations in the genes associated with the majority of HNPCC.10–12

Table 1 Modified Bethesda criteria10
Full size table

Unfortunately, the skills necessary to implement such guidelines, such as taking a thorough family history and interpreting pedigrees, have not received enough attention in medical schools. Consequently, deficits in physicians’ knowledge of genetics have been demonstrated in various studies.13–15 Batra et al.14 surveyed gastroenterologists practicing in New York State regarding their knowledge of genetic testing and genetic counseling for CRC. Of the 285 physicians who responded, 66% were unaware of genetic testing for HNPCC, 21% could not identify a pedigree consistent with a diagnosis of HNPCC, 74% were unaware of the limitations of genetic testing, and 84% were unaware of cancer surveillance guidelines for individuals with HNPCC.14 To address this area of need, the American Medical Association (AMA) and the AGA have jointly developed a continuing medical education (CME) monograph on clinical cancer risk management for HNPCC, which includes the AGA guidelines and a strategy for implementing genetic testing for this disease.16

Improving the knowledge base of medical professionals involved in the care of patients with HNPCC is clearly necessary, but it may not be sufficient to change their behavior. Analysis of formal CME has demonstrated that short didactic interventions, such as conferences and workshops, are ineffective in changing physician performance. Greater effectiveness is achieved through formal CME interventions that provide active learning opportunities.17 A review of the effects of printed educational materials alone showed only a small effect in improving the behavior of health care professionals.18

Computer-based CME, including CD-ROM and Web-based approaches, can provide an active learning experience. Although many Web-based CME programs are available,19 few studies have measured their effectiveness. In one study, a Web-based CME program on skin cancer was developed using a problem-based learning approach that asked users to apply their knowledge to analyze clinical scenarios. This CME was found to be effective in improving primary care physicians’ confidence in managing primary skin lesions, increasing skin cancer knowledge, and improving physicians’ decision-making skills.20 However, a study that compared the effectiveness of Web-based and print-based learning to learn and retain guidelines for management of acute myocardial infarction did not show an increase in overall knowledge or knowledge retention.21 Both studies reported a high level of efficiency in and satisfaction with Web-based learning, which could encourage repeated use of the program and therefore greater reinforcement of knowledge over time. Other potential advantages of computer-based CME are convenience, lower costs, and recognized expert faculty.

We developed a Web-based tutorial on HNPCC and genetic testing to assist medical professionals in applying their knowledge to clinical practice. Our design uses a case-based learning approach and is interactive. In this study, we report the results of a pilot of this tutorial. Our main objectives were (1) to determine whether an interactive Web-based learning program can improve gastroenterologists’ and genetic counselors’ knowledge and clinical skill in cancer risk management of HNPCC, and (2) to evaluate the user acceptability of this learning program.

MATERIALS AND METHODS

Study population

Fellows (n = 13) currently training in adult gastroenterology at the Mount Sinai Medical Center and genetic counseling students (n = 12) currently attending the Genetic Counseling Training Program of the Mount Sinai School of Medicine were invited to participate in this study. One gastroenterology fellow and two genetic counseling students were unavailable at the time of the tutorial administration. This study was approved by the Institutional Review Board of the Mount Sinai Medical Center. Participation in the study was strictly voluntary, with complete maintenance of anonymity.

Tutorial design

The learning objectives of the tutorial address several core competencies for clinical cancer risk management of HNPCC, including (1) eliciting a thorough family history, (2) indications for genetic testing as recommended by the AGA, (3) strategy for initiating genetic testing in a family, (4) limitations of genetic testing, (5) cancer surveillance as recommended by the American Cancer Society, and (6) identifying at-risk family members from a pedigree. We implemented a case-based teaching approach to instruct users in applying their knowledge to clinical practice. Didactic information contained in the tutorial is limited and is intended to be an adjunct to written materials. The tutorial begins with a brief introduction and then proceeds to six hypothetical clinical vignettes. Within each vignette, clinical and family history information in the form of both text and a pedigree is presented to the user, who is then asked to make several management decisions as he or she proceeds through the case. The AGA guidelines for genetic testing are also displayed along with each case. A computer version was written in hypertext markup language by the authors using standard Web authoring tools. The tutorial is interactive and provides customized feedback to the user’s responses. For the purpose of this study, the tutorial was run from the standard 1.44-MB floppy disk drive and not published on the Internet.

Study design

A pretest/post-test design was used to evaluate the effectiveness of our tutorial. Ten multiple-choice questions were devised by the authors to assess clinical management skills. The test used the same case-based design as the tutorial, requiring subjects to analyze clinical data and pedigrees to select the most appropriate management option. The identical set of 10 questions was asked on both the pretest and post-test questionnaires. We elected to use an identical pre- and post-test to ensure that the skills being tested remained the same and could therefore be reliably tested. However, subjects were not made aware in advance that the tests were the same nor were they apprised of their test scores. Demographic information, opinions of the quality of the tutorial, and opinions on the importance of the learning objectives to clinical practice were also elicited. For the opinion questions, subjects were asked to rate their agreement with each statement on a 5-point scale: 1 = “strongly agree,” 2 = “agree,” 3 = “neutral,” 4 = “disagree,” 5 = “strongly disagree.” To maintain anonymity the questionnaires did not contain any personal identifiers; however, they were coded with a unique number to allow pairwise analysis of pretest and post-test scores.

Informed consent was obtained from the subjects 1 week before they took the tutorial, at which time they were provided with the AMA/AGA CME monograph. Permission to use this monograph in conjunction with this study was obtained. We chose this monograph because it contains comprehensive information needed for the clinical management of HNPCC and is consistent with information presented in our tutorial. Subjects were asked to read these written materials before taking the tutorial. The tutorial was given during an in-house computer workshop that was conducted by one of the authors (K. Barnes). The pretest was administered to subjects immediately before they took the tutorial. After the pretest, subjects were allowed ample time (up to 3 hours) to complete the tutorial at their own pace. After the tutorial, the subjects were asked to immediately complete the post-test questionnaire. AGA guidelines for genetic testing that were displayed to users during the tutorial were not made available during the pretest and post-test evaluation.

SPSS was used to perform statistical analysis. The differences in pre- and post-test scores were tested for normality with the Kolmogorov-Smirnov test, and the results indicated that the data were consistent with a normal distribution. Paired t test analysis was used to compare pre- and post-test scores for the total study population and by subgroup. All reported P values are two-tailed. The McNemar test (binomial distribution assumed) was used to compare pretest and post-test responses for each test question. Means and standard deviations were calculated for each opinion question.

Validation of the test instrument

Three experts from outside institutions reviewed the clinical management questions on our pretest/post-test. On 9 of 10 questions, all three expert reviewers agreed on the response designed to be the correct answer. The wording of one question was thought to be unclear; therefore, responses to this question were not included in our analysis.

RESULTS

The characteristics of the study population are presented in Table 2. Twenty-two subjects agreed to participate in the study. Almost 60% of the gastroenterology fellows saw one to four patients per month with CRC, and the rest saw five or more patients per month. Of the gastroenterology fellow group, 41.7% did not refer any patients with suspected HNPCC for genetic testing or counseling during the previous year, 50% referred between 1 and 4 patients during the previous year, and 8.3% referred between 5 and 10 patients. The genetic counseling students were not asked questions regarding patients counseled for HNPCC because it was expected that the number of such cases would be very small at this time in their training. The majority of all subjects (63.7%) had previous experience with computer-based medical education, and 100% of those believed it was effective.

Table 2 Demographics
Full size table

Mean scores on the clinical skill questions are summarized in Table 3. Scores for the total study population improved from 63.9% on the pretest to 81.1% on the post-test (P < 0.001). All subgroups also significantly improved their scores after taking the tutorial (Table 3).

Table 3 Mean scores on clinical management skills questionnaire by group
Full size table

Table 4 summarizes the percent correct responses for each clinical skill question before and after taking the tutorial. The questions pertained to understanding guidelines for genetic testing and cancer surveillance in the setting of HNPCC. An analysis of correct responses by question indicates weaknesses in particular applied knowledge domains before taking the tutorial. Overall, these weaknesses improved after the tutorial, although only the improvement in Question 6 reached statistical significance. Question 6 pertains to understanding that germline genetic testing is indicated for individuals who meet the Amsterdam I criteria and that testing should be initiated in an affected proband. Only 54.5% made the correct management decision on the pretest. After the tutorial, the number of correct responses improved to 95.5% (P = 0.004). Question 2 pertains to pedigree analysis to identify individuals at 50% risk for HNPCC. Before the tutorial, 63.6% of subjects responded correctly, whereas after the tutorial, the percent correct responses declined slightly to 59.1%. Of those who answered incorrectly on the pretest, most (88%) failed to identify any or all individuals at 50% risk. However, on the post-test all incorrect responses were attributable to inclusion of one family member at 25% risk. Questions 8 and 9 required subjects to have committed to memory the AGA guidelines for genetic testing. In both instances, post-test scores revealed little or no improvement.

Table 4 Clinical management skills test by question
Full size table

Attitudes regarding the importance of the learning objectives of this tutorial were favorable. After taking the tutorial, subjects were asked to rate their agreement with three questions on a 5-point scale, where 1 = “strongly agree” and 5 = “strongly disagree.” Subjects were likely to agree that it is important to spend enough time to take a thorough family history and to construct a pedigree (mean = 1.41, SD = 1.18). Subjects were also likely to agree that genetic testing for CRC predisposition was useful for some of their patients (mean = 1.45, SD = 1.18). Subjects also believed the learning objectives of this tutorial were important to their clinical practice (mean = 1.45, SD = 1.06). There were no significant differences in responses between the genetic counseling student group and the gastroenterology fellow group.

The user acceptance of the tutorial was quite high. Subjects spent an average of 38.9 minutes to complete the tutorial, with a range from 26 to 61 minutes. On the post-test questionnaire, subjects were asked to write in any comments about the tutorial. The majority of written comments on the post-test questionnaire were positive, i.e., “that was fun,” and “enjoyable.” Subjects were also asked to rate their agreement with four questions regarding the quality of the tutorial on a 5-point scale, where 1 = “strongly agree” and 5 = “strongly disagree.” Subjects agreed that the tutorial was easy to use (mean = 1.41, SD = 0.91), that the cases and explanations were written in a clear and comprehensible manner (mean = 1.55, SD = 0.96), and that by working through the cases and explanations, the learning objectives were met (mean = 1.73, SD = 1.08). Subjects also were likely to agree that taking the tutorial had increased their confidence to recognize HNPCC and to appropriately offer genetic testing (mean = 1.83, SD = 1.01). There were no significant differences in responses between the genetic counseling subgroup and the gastroenterology subgroup. The large standard deviation on responses to the 5-point scale statements may be attributable to two subjects improperly reading the scale. We infer this because the two subjects who responded negatively to the above statements also responded very positively to a single question (not using the 5-point scale) asking for their overall opinion of the tutorial.

DISCUSSION

We have demonstrated that an interactive case-based tutorial can improve the clinical knowledge that is required to recognize and care for patients with, and at-risk for, HNPCC. Our tutorial was easy to use and could be completed in about 40 minutes. Users were highly favorable in their opinion of the tutorial. Even though our questionnaires were anonymous, it is possible that responses to these opinion questions may be somewhat positively biased because of relationships between the subjects and the authors.

In our group of gastroenterology fellows and genetic counselor trainees, scores on our test of clinical management skills improved overall by 17.2% (P < 0.001) after they took the tutorial. All subjects were requested to read published materials on HNPCC prior to taking the tutorial. Among even the small subgroup (n = 13) who reported reading the written materials thoroughly, statistically significant improvement was achieved. While assignment to the subgroup of thorough readers depended on subjects’ self-assessment, since the study was anonymous, we believe that subjects made accurate assessments. Thus our results suggest that improvements in test scores are the result of the interactive learning experience. To clarify the benefits of the tutorial, further testing with a larger population should compare improvement in test scores after reading published materials alone versus published materials plus the tutorial. Furthermore, although our tutorial was designed as a supplement to published materials, it was effective in improving scores for individuals who said that they did not read the monograph thoroughly. Post-test scores for this group still lagged behind those of thorough readers, even though statistical significance could not be evaluated because of the small sample size.

Our study identified weaknesses in areas of clinical management that have been identified in larger surveys of physicians’ knowledge of genetic testing, particularly understanding the limitations of negative genetic test results.13,14 It is important to note that knowledge that an affected family member should be tested first (one of the basic tenets of genetic testing) was lacking in half of respondents on the pretest, but improved dramatically to nearly 100% on the post-test. With few exceptions, after our tutorial the percent correct responses in all skill domains we assessed improved.

Similar to the results of a Web-based tutorial, which sought to teach guidelines for management of acute myocardial infarction,21 we saw little improvement in scores when memorization of criteria was required to answer the question. During the tutorial, participants received instruction on the appropriate clinical application of the AGA guidelines; however, memorization of these guidelines was not required to proceed through the case vignettes. It is possible that memorization of guidelines may require multiple or longer educational exposures than are provided by our 40-minute tutorial. However, an uncomplicated guideline, such as the indication for testing when a single family member is affected with early-onset CRC, was recalled by our subjects after the tutorial. Not only is this guideline simple, but conceptually it is one of the hallmarks of hereditary cancer syndromes which may have been taught during other coursework.

Web-based CME approaches allow widespread distribution and timely updating of information. Despite these potential benefits, a survey of physicians’ CE/CME practices showed that of 1120 respondents, the most frequently cited methods of acquiring CE/CME were conferences (93%) and print-based methods (66%). The least reported mode was computer-based CME/CE through the Internet (13%). The major barrier identified in this survey was the lack of adequate computer skills rather than a lack of preference to the new technology,22 so as physicians’ computer skills improve, use of computer CME will likely grow. This tutorial might be a useful means for updating a larger group of physicians and for further studying the effectiveness of Web-based educational programs.

Although the improvements in clinical skills after the tutorial are encouraging, our sample size is small. Furthermore, our subjects differ from practicing gastroenterologists and genetic counselors in that they are in training programs, are relatively younger, and in general may have more favorable opinions of computer-based education. Our evaluation used the common, yet artificial, device of presenting clinical vignettes on a questionnaire. Whether these gains will translate into the ultimate goal of improvements in clinical practice behaviors and patient care will require further study.