Main

During recent decades, rapid developments have changed the relevance of genetics in medicine. Several studies have shown that many nongenetic health care providers lack the necessary knowledge and skills to react appropriately to the increasing demands of patients.1–7 One possible cause of this reported lack of knowledge and skills is deficiency in the medical curriculum. In the Netherlands, medical undergraduate education takes a minimum of 6 years of theoretical and practical work at one of eight universities. Curricula vary but universities share a general scheme of examinations and a general framework of final goals. Five problems (goals) that deal more or less specifically with genetics are listed.8 A survey in 2001 among all medical schools in the Netherlands showed that only limited attention in the curricula is paid to genetics (Plass AMC et al., unpublished data, 2005). This is not surprising, because genetic issues are scarcely mentioned in the official final goals of basic medical training in the Netherlands. In most universities, the number of hours spent on genetic education is small and genetics is relatively invisible, being integrated within several courses and comprising only a small proportion of these courses (1% to 21%, with a mean of 8%).8 A computerized test was designed to investigate whether there are serious deficiencies in important aspects of genetic education in the Dutch medical curricula.

MATERIALS AND METHODS

Instrument and procedure

To investigate medical students' nearing graduation knowledge of genetics that is relevant for daily practice, a specific examination was designed. Figure 1 shows the development of this instrument. Questions on genetics, used in several levels of examination in the medical curricula in the period 1997 to 1999, were collected from seven out of the eight medical schools in the Netherlands. Questions from questionnaires described in the literature were also selected.1,9 In total, 855 examination questions were available. A group of four experienced clinical geneticists (group I, designers) from three different Dutch Clinical Genetics Centres assessed these questions independently and classified them into four categories: (1) “essential” knowledge, (2) “desirable” knowledge, (3) knowledge “too specialized” for daily practice, and (4) “redundant” knowledge. Redundant knowledge refers to questions which were, e.g., out of date or questions not dealing with genetic problems: this category was ignored. “Essential” was defined as knowledge that every MD should know (> 95% answers correct), “desirable” as questions the majority (> 60%) of which every MD should be able to answer correctly, and “too specialized” as knowledge that was unnecessary in daily practice. Agreement on the classification by at least three out of the four clinical geneticists was reached for 413 questions. The principal investigator (M.B.) then rearranged these according to topic. After rejecting overlapping questions, 266 questions were reviewed by two educational experts and discussed again with the clinical geneticists (group I). The remaining 225 questions were presented to three other clinical geneticists (group II) involved in genetic education programs at the medical schools. These questions were also translated into English and evaluated by five foreign geneticists. After their input, 215 questions (in categories 1, n = 103; 2, n = 81; and 3, n = 31) were used in the final version of the examination. Box 1 shows an example of a question for each category. The 215 questions were asked in the context of 194 short case-histories. Formats were multiple choice (n = 72), true-false (n = 108), and matching (n = 13). Open-ended questions (n = 22) were also included and hand-scored afterward. The examination was administered by computer, and the time-allowance was 2 hours. In some case-histories, additional information was presented after the first question was answered. The computer program did not permit return to previously answered questions. Participants also completed a questionnaire concerning sociodemographic characteristics and variables potentially influencing the examination scores. These variables were as follows: gender, age, course duration (year of participation in the genetic examination to year of starting medical education), whether or not an elective course in genetics had been completed, whether or not an elective clerkship in clinical genetics had been done, medical school, current type of clerkship, and desired specialty training after graduation.

Fig. 1
figure 1

Procedure for the selection of questions used in the genetics examination.

Full size image

Subjects

Seven of the eight medical schools in the Netherlands that were asked to provide the addresses of their students in the final years of clerkships (fifth and sixth years of the medical curriculum) made postal or e-mail addresses (n = 2377) available. The 8th medical school was not willing to participate in the study due to changes in the curriculum at that time. The students were invited to participate in the examination on a preset date. An incentive of 25 Euros was offered to all participants. The first 50 students per medical school who were willing and able to participate were included. The numbers of participants was limited by the availability of computers in the schools. All participants gave their permission for perusal of their regular medical school examination results, which made it possible to compare participants and nonparticipants to check the representativeness of the participants. All tests were taken between the middle of October and the end of November. The time of the test was approximately similar at all medical schools (start early in the evening).

Validation study

To test the construct validity of the genetic examination, the results of a group of medical students just starting their clerkships (n = 12) were compared to the results of nonmedical students. The latter (n = 11) were recruited through advertisements in a university newspaper. The results were also compared to those of a group of clinical geneticists (group III, n = 10) from three different Clinical Genetics Centres. Due to problems with the computer administration, the results of one medical student could not be saved. In addition, 22 items were not included in the analysis because of storage problems during administration. Because the total number of items was high, the results for the remaining questions (n = 193) were still considered adequate for the analysis.

To set a standard for competence in this examination, independent of the opinions of the designers, the Angoff method was applied because the aim was to investigate whether or not students lack essential knowledge of genetics.10,11 In an Angoff procedure, expert judges were asked to judge the difficulty of each individual item, based on the ability of the marginally competent student to answer the item correctly. The individual judgments were then averaged per item and these averages were used to calculate the grand mean, i.e., a cut-off score. The reproducibility of standards over groups and time was considered to be high.12

To judge the items and set a standard with the Angoff procedure, a group of clinical geneticists involved in medical education programs at the medical schools or with particular interest in medical education (group IV, n = 4) and a second group consisting of gynecologists, pediatricians, and general practitioners (n = 6) were invited to participate.

Data analysis

For each case-history, a score of 1 for a correct answer or 0 for an incorrect answer could be obtained. Raw scores were converted into percentage correct scores. In addition, three subscores were calculated for each of the categories of relevance. Reliability of the test was estimated with Cronbach's alpha. Although in some case-histories more than one question was asked, the content of each was so different that they could be treated as locally independent. Therefore, 215 individual items were entered into the reliability analysis.

Differences between the mean knowledge scores of the various groups of students and clinical geneticists, as well as between the students of the different medical school (A through G) were tested, using one-way ANOVA with post hoc Bonferroni correction for multiple comparisons. Simple linear correlations were used to find variables influencing the scores of the participating students. To adjust for covariance, variables with simple linear correlations with the scores and P values of < 0.1 were included in a multivariate linear regression analysis, with the scores as the dependent variable. To check the representativieness of the participating students their regular examination results were compared to those of nonparticipating students. The scores on regular examinations were transformed into Z-scores, and t tests were performed on the mean Z-scores of participating and nonparticipating students. To test the concentration of the medical students nearing graduation, a paired sample t test was used to compare the results of the first half of the examination with those of the second half. A P value < 0.05 was considered to be statistically significant (two-sided). All analyses were performed with SPSS 9.0 for Windows.

The passing scores according to the Angoff groups were established, first, by averaging the judges' estimate per item (4 × 215 and 6 × 215). Second, all individual item scores were averaged to estimate an overall passing score. Third, the same procedure was performed to estimate passing scores per category of relevance. The generalizability theory was applied to estimate the root mean squared error (RMSE),13 which was also estimated with the GENOVA package. The RMSE is an estimate of the standard error of the mean of the Angoff estimates across items and judges and indicates the error involved in the test's passing score.13

RESULTS

Validation, reliability, and Angoff standards

Table 1 shows the mean percentage of correct answers, confidence interval (95% CI), and standard deviation (SD) per category of relevance for nonmedical students (n = 11), medical students just starting their clerkships (n = 11), and clinical geneticists (group III, n = 10). As expected, the scores of the nonmedical students were lower than those of the medical students, which were lower than the scores of the clinical geneticists. In all three groups the scores were highest for “essential” knowledge and lowest for “too specialized” knowledge. There was a statistically significant difference between the scores of the clinical geneticists and every other group (P < 0.001), as well as between the medical and nonmedical students (P < 0.001). Therefore, it can be concluded that the examination can differentiate between groups with different levels of knowledge. Crohnbach's alpha for the 291 medical students nearing graduation was 0.82.

Table 1 Examination scores (% correct answers, 95% CI and SD) of the participants in the validation study for the categories of relevance and the total genetic examination
Full size table

Table 2 shows the passing scores set by the designers and the two Angoff groups for the total examination and the categories of relevance. The RMSE of the panel of clinical geneticists was 0.86; the 95% CI of this passing score is 54.0% to 57.4%. The RMSE of the panel of nongeneticist judges was 0.017; the 95% CI of this passing score is 69.5% to 76.1%.

Table 2 Passing scores set by the designers (clinical geneticists group I), Angoff group of clinical geneticists (group IV) and Angoff group of nongeneticists for categories of relevance and the total genetic examination
Full size table

Response rate and demographic characteristics

The response rates of the medical students in their last year of clerkships in the seven participating medical schools (A through G) were as follows: 28% (104/372), 33% (158/477), 24% (26/117), 50% (186/370), 29% (154/531), 29% (43/147), and 37% (134/363), respectively. Responders are those students who sent back a letter confirming that they were interested in taking part in this study. A total of 291 students participated in the examination. The number of students who completed the examination was 49, 47, 19, 47, 55, 27, and 47 students per medical school A through G, respectively. Of the participants, 35% were male, and their mean age was 25 years. There were no significant differences in gender or age distributions between participants of the various schools. Over half of the participants started to study medicine directly after completing their secondary education. The median year of starting to study medicine was 1996. Only 3% of the participants had followed an elective course in genetics and only 1% had followed an elective clerkship in clinical genetics. Thirty-four percent of the participants had just started their clerkships, 40% were halfway through, and 26% were nearing the end of their clerkships. There was a statistical significant difference between the medical schools. A smaller percentage of participants of medical school A and a higher percentage of participants of medical schools E and F had just started their clerkships compared to the other schools. A smaller percentage of participants of medical schools C and F and a higher percentage of participants of medical schools A and D were near the end of their clerkships compared to the other medical schools.

Representativeness

Comparison of the results for the regular examinations between participants and nonparticipants showed that participating students from medical school D were representative of their classes, whereas the students from schools A, B, C, E, F, and G were among the better students in their classes.

Genetic examination scores

The answers to 13 questions were not included in the analysis because of storage problems during administration. Therefore, the genetics examination score is calculated for 189 case-histories (202 items). The mean percentage of correct answers on the genetics examination score was 62.19% (Table 3). Table 3 also shows the mean percentage of correct answers, 95% CI of the mean and SD for the different categories of relevance. As expected, the mean score for “essential” knowledge was higher than that for “desirable” knowledge, which was higher than that for “too specialized” knowledge. There was a small but significant difference in the mean percentage of correct answers in the first half of the examination compared to the second half (63% vs. 62% correct answers; P = 0.01). There was no evidence of clustering per subject of correct/incorrect answers. None of the students scored over 95% for “essential” knowledge, approximately a quarter of the students scored 60% or more for “desirable” knowledge, and most of the students scored over 40% for “too specialized” knowledge (Fig. 2).

Table 3 Examination scores (% correct answers, 95% CI and SD) for categories of relevance and the total genetic examination
Full size table
Fig. 2
figure 2

Cumulative percentage of correct scores for the categories of relevance: 1, essential knowledge (♦); 2, desirable knowledge (▪); 3, too specialized knowledge (▴).

Full size image

Genetic knowledge scores in relation to the Angoff passing scores

The results of the two Angoff groups differed significantly. Of the participants, 15% failed according to the clinical geneticists and 93% failed according to the nongenetic health care providers. Based only on the two Angoff groups scores for “essential” knowledge, 26% would have passed according to the clinical geneticists and 3% according to the nongenetic health care providers.

Factors influencing the genetic examination score

Multivariate linear regression analysis showed a significant negative association between age and the genetic examination score (β = [minus]0.13, P = 0.02) and significant positive associations between female gender and the genetic examination score (β = 0.15, P = 0.01), and between course duration and the genetic examination score (β = 0.14, P = 0.02). As shown in Figure 3, students from one medical school (A) scored significantly lower than students from two of the other six medical schools (E and G). No other significant differences were found between the scores of participants from the different medical schools.

Fig. 3
figure 3

Genetic examination scores (mean and 95% CI) of the medical students nearing graduation per medical school (A-G).

Full size image

DISCUSSION

The knowledge of genetics that is relevant for daily practice among medical students nearing graduation in the Netherlands was investigated in a specifically designed genetics examination. As expected, in the validation study, genetic knowledge scores of nonmedical students were lower than those of medical students, which were lower than those of clinical geneticists. The trend in the knowledge scores in these groups, and the decrease in the knowledge scores for less relevant categories in each group, confirm the validity of the measurement instrument. There was great similarity between the relative relevancy ratings of the judges in the two Angoff groups and the clinical geneticists who designed the genetic examination. According to the standards set by the Angoff groups and those set by the designers, a considerable percentage (74%–100%) of the medical students nearing graduation lacked “essential” genetic knowledge. The “essential” knowledge questions were designed and considered to be relevant for daily practice outside genetic clinics, i.e., in the primary care setting. Therefore, the knowledge scores of all medical students nearing graduation reflected deficiencies.

The method of standard setting for this examination is debatable. The standards set by the designers (> 95%) and the judges differed (76.20% vs. 85.26). One explanation could be that the Angoff group of clinical geneticists had a better understanding of what, in view of the deficiency in genetic education in the curricula, could be reasonably expected from a medical student, and adapted their judgments accordingly. Another explanation may be that because the other judges were unfamiliar with the field of genetics they may have unrealistic expectations with regard to the students' knowledge. The RMSE of the panel of clinical geneticists was higher (0.86) than that of the panel of nongenetic judges (0.017). This could be related to the higher number of participants in the latter panel. One could speculate whether the group of designers classified the relevance of the questions correctly. However, both groups of judges relatively classified the relevance of the questions much in the same way.

The number of participants was limited by the availability of computers in the medical schools. The date and time of the test also had to be set in advance to avoid problems with regular use of the computers. Therefore, not all students who were willing to participate had the opportunity to do so. On the other hand, the total number of participants was large enough to indicate true deficiencies in genetic knowledge. Participants from one medical school were representative of their classes, and participants from the other six medical schools were among the better achievers, which is in line with findings in another study concerning the anatomy knowledge among medical students in the Netherlands.14 In that study, participants of four medical schools were among the better students of their classes, and participants of two medical schools were representative of their classes. The fact that all participants were among the better performers except the participants of school D suggests that the level of knowledge of genetics among the “average” medical students nearing graduation might even be worse than among the study participants.

Scores for both “essential” knowledge and for “desirable” knowledge reflect deficiencies among all medical students nearing graduation. These deficiencies may be caused by the limited amount of attention (genetics is integrated in on average 8% of a limited amount of courses) paid to genetics in the different curricula and the relative invisibility (Plass AMC et al., unpublished data, 2005).8 On the other hand, scores for “too specialized” knowledge were higher than expected. It could be that in genetic education too much attention is paid to specialized topics.

Younger students tended to obtain higher scores. This could not be explained by the fact that better students tend to complete their study more quickly, but might be related to changes in high school curricula in recent years. Moreover, female students tended to score higher than male students. This association between knowledge scores and gender was not found in other studies on genetics knowledge among physicians in the Netherlands (Baars MJH et al., unpublished data, 2005).4,6

In general, no differences were found between students from different medical schools, except that students from one medical school (A) scored significantly lower, compared to those from two of the other six medical schools (E and G).

The results of this study are in line with reports of shortcomings in genetic knowledge among nongenetic health care providers, such as general practitioners (Baars MJH et al., unpublished data, 2005).1–4,7 However, genetic knowledge and skills are relevant because the recent rapid developments in this field have given rise to increasing patient demands for management of the genetic aspects of their illnesses, requiring doctors to be increasingly more equipped to deal with these demands.15 This situation could be improved by allocating more time to genetics in the curricula. Because time is needed for other subjects as well, the time spent on genetics should be spent more efficiently and should focus on knowledge that is relevant for daily practice. Another solution could be to make the study methods more efficient, i.e., acquire knowledge on a more problem-orientated basis, in which genetics is more integrated with other subjects. It is therefore important that end-objectives for the genetic education in the medical curriculum are defined and agreed upon.

As argued by Korf,16 there should be a clearer connection between the basic genetics and the clinical approaches. In addition, Boreham17 concluded that the medical students' lack of knowledge of pharmacotherapeutics gave rise to the same argument to promote the trend to integrate basic medical science into clinical cases, as in a problem-oriented curriculum. In the present study, only a very small minority chose to take either an elective course (3%) or an elective clerkship in genetics (1%). A possible solution is to introduce genetics not only as a more visible element of one of the compulsory clerkships, such as gynecology, pediatrics, or internal medicine, but to integrate genetics in the basic teaching of these subjects as well. Genetic education should therefore also be offered to residents and fellows in other disciplines. Patient organizations could become stakeholders to promote the integration of genetic knowledge, and general practitioners could intermediate between public and genetic centers.

The results of this study are specific for the current situation in the Netherlands. Genetic knowledge that is essential for daily practice in other countries should also be studied to identify training needs elsewhere and to identify more efficient training methods.

This article reports on the use of an instrument to measure genetic knowledge that is relevant for daily practice in the Netherlands. We believe that it could also be used in other countries to evaluate and compare genetic knowledge among medical students.

Because at least 74% of the medical students (nearing graduation) in the Netherlands lack genetic knowledge judged to be essential for daily practice, medical schools should consider increasing the amount of genetics incorporated in the medical curricula.