Introduction

In-service examinations are a commonly used educational tool in postgraduate medical education. These exams are meant to simulate the respective board examination for the specialty in an effort to gauge a trainees’ knowledge level. Furthermore, they can provide feedback to the residency program director (PD) regarding the effectiveness of the program’s curriculum. The test is typically composed of representative questions from each of the major areas of a specialty’s core fund of knowledge. For example, the in-service exam for internal medicine includes questions from cardiology, endocrinology, and geriatrics, as well as general internal medicine (for a complete list, see ref. 1). The exam is administered at some point during the academic year to all residents, with the exact timing differing for each specialty—pediatrics gives the test in July, radiology in February, and internal medicine in October—and residents from all years are expected to take the test.

An in-service exam has several goals. For the trainee, the test is used to help residents assess and track his/her improvement in medical knowledge.2 This is accomplished by tracking the improvements in trainees’ scores over the course of their residency training but also in comparing their scores to those of their peers both within their program as well as nationally. In this manner, the in-service can be used to identify residents at risk for failing their certification exam. The results of the in-service can also be used by the PD to evaluate the training program: Is their curriculum covering the required material, as determined by the specialty’s governing board and the Accreditation Council for Graduate Medical Education, and is it doing so effectively?

It is apparent that to accomplish each of these goals there must be a single exam utilized by all training programs for a given specialty that is administered in a uniform manner. Only then can each resident accurately gauge his/her progress in medical knowledge, and each program can determine just how effectively it is teaching its trainees. Unfortunately, this has not been the case for medical genetics. There is no American Board of Medical Genetics (ABMG)- or Accreditation Council for Graduate Medical Education–mandated requirement for an in-service exam. Each program has been left to design its own exam, or to have no exam at all. This deficiency was addressed in the spring of 2009 at the first meeting of the Medical Genetics Residency Directors Group, a special-interest group within the Association of Professors of Human and Molecular Genetics. At that time, it was decided that a single in-service exam should be developed and implemented. Here, we describe that effort, the aggregate results of the exam for 2010 and 2011, and what steps must be taken for this to move forward.

Exam Development

The initial step involved reviewing the section on “Genetics Knowledge” of the ABMG Learning Guide for clinical geneticists.3 The items/learning objectives were placed in to five subcategories: basic science/molecular genetics, pediatric genetics (including dysmorphology), biochemical genetics, adult genetics (including cancer genetics), and prenatal genetics ( Table 1 ). This list was then distributed through the medical genetics PDs’ electronic mailing list with a description of the plan to develop a single in-service exam that would be available to all trainees of the medical genetics residency programs free of charge. However, in exchange for the use of this exam each PD had to agree to central administration of the exam, as well as to provide 15 questions. The questions were to be written according to the format of National Board of Medical Examiners’.4 The 15 questions were the fee for participation in the test. Some programs sent in their own in-service examination. In total, over 1,300 questions were submitted to a central location. These questions were organized by subcategory and preliminary edits were made, including eliminating duplicate questions and correcting formatting errors.

Table 1 Category of questions and the specific topics covered
Full size table

During that time, decisions were made on the final composition of the exam. As it is recommended to allot 75 seconds per question, in order to keep the exam to a maximum of 3 h it was decided to limit the test to 125 total questions. To reflect the relative proportions indicated on the ABMG Learning Guide, the composition of the in-service exam was to be basic/molecular genetics: 40 questions; cancer and adult genetics: 20; prenatal genetics: 20; biochemical genetics: 20; and pediatric genetics/dysmorphology: 25. The final composition was to be slightly different, although many questions required knowledge applicable to multiple categories.

Also at this time, subcommittees were established comprising two to three ABMG-certified clinical geneticists (the list of subcommittee members is included at the end of the article). The subcommittee members were sent all of the questions that had been placed in their subsection and asked to choose the best questions to represent their subcategory in the final in-service exam. For example, the members of the biochemistry subcommittee received ~200 questions. They were asked to return 20 that covered as many of the items on the Learning Guide as possible, and that these questions be edited and re-written in the format of the National Board of Examiners. The same process was followed both years, and although the exam items were kept secure (trainees did not see the actual questions after they completed the exam), we wrote entirely new questions for 2011.

The questions from each subcategory were returned in January and reviewed and re-edited and assembled into a single exam, which was placed on the University of Alabama at Birmingham server. The online exam was reviewed by several board-certified clinical geneticists for both content and clarity, and appropriate modifications were made based on their feedback.

Through the Medical Genetics Residency PD’s listserv, each program was notified that it would be required to register each trainee who wished to take the exam. This included medical residents and PhD clinical laboratory fellows. In 2010, demographic information was obtained during the registration process, but this was found to be inadequate as the exam results were evaluated. For that reason, in 2011 basic demographic questions were inserted in the beginning of the exam so that they had to be completed before the rest of the exam could be begun.

In both 2010 and 2011, instructions including log-on information were sent in late January to each PD, and the exam was made available in early February for a 2-week period. Each program was asked to reserve a 4-h block for their trainees to take the exam. The test was taken on site, and it was required that the exam be proctored.

Participation

In 2010, trainees from 35 programs completed the exam. There are 51 Accreditation Council for Graduate Medical Education–accredited medical genetics residency programs, but only 38 had a resident enrolled in 2009–2010. Therefore, the overall program participation rate was 92% (35/38). In total, 174 trainees completed the exam.

Overall participation increased in 2011. This occurred despite the fact that there was a fee charged for the exam—a onetime $25 fee per program and a $100 charge per trainee. This was necessary to offset the technical cost and data analysis associated with the exam. In addition, the Canadian Medical Genetics residency programs were invited to participate in the exam process. For 2011, trainees from 39 of the 40 programs with a resident participated, as well as 4 of the 5 Canadian programs. (The one “missing” US program had its trainees take the exam at a later time for logistical reasons. Those results are not included in this report.) In total, 214 trainees completed the exam, an increase of 23%. This included 127 clinical trainees (+20%) and 85 laboratory trainees (+25%).

Results

In 2010, the mean score of all exam takers was 52%, or 65 questions correct of 125 total, with a standard deviation of 12%. The mean for the clinical group was 54% (s.d. 11), as compared with 49% (s.d. 12) for the lab group. The scores were nearly identical in 2011. The mean score for all groups was 56% (s.d. 12). The lab group mean score was 53% (s.d. 11), and the clinical group’s mean score was 58% (s.d. 11). Similarly, there was little variation between 2010 and 2011 in the scores of the subgroups. A complete report of the scores is provided in Tables 2 and 3 .

Table 2 2011 Scores by training track and year of experience
Full size table
Table 3 Comparison of scores for 2010 and 2011, grouped by training track
Full size table

We could not accurately set a “passing” score. One common method is to set passing at 2 s.d. below the mean. In 2010, only one taker scored <−2 s.d., none in 2011. In an effort to better judge the scores, for the 2011 exam we asked five recent graduates of medical genetics residency programs who took and passed the most recent ABMG general and clinical exams to complete this exam. Their mean score was 74, with a range from 70 to 77% correct ( Table 2 ). In addition, three also reported their scores for the ABMG general and clinical exams, and each had scores that easily surpassed the minimum passing score (only the actual score and the minimum passing score is reported to the trainees). Although far from definitive, this provided a very approximate target score for the medical genetics residency graduate who is competent for independent practice.

We also were able to look at the scores for each category of exam question by medical discipline ( Table 4 ). This led to several interesting observations, some of which must be considered with caution due to the small numbers in the groups. Categorical medical genetics residents had little deviation among their subject scores, with only biochemical genetics being significantly lower at 51% as compared with their overall score of 60%. Similarly, biochemical genetics was the lowest score for both pediatric genetics residents, by 9%, and maternal fetal–medical genetics residents, where it was 20% lower than their mean. Internal medicine genetics residents also scored 7% lower for biochemical genetics, but this was higher than their scores on pediatric/dysmorphology genetics (12% lower) and basic/molecular genetics (10% lower).

Table 4 Mean percentage correct scores by category (2011 exam)
Full size table

Trainee survey

After completing the test, trainees were asked to complete a brief 10-question survey on their experience with the in-service exam. In total, 98 trainees responded in 2010, 100 in 2011. For each year they offered overall very favorable opinions ( Table 5 ).

Table 5 Responses of the trainees to the in-service exam (n = 98 in 2010, 100 in 2011)
Full size table

Trainee feedback

One of the stated goals of the in-service exam was to provide trainees meaningful feedback as to their relative strengths and weaknesses in medical knowledge. We were unable to provide this in 2010, but we sought to provide this in 2011. Each question was “tagged” with a general topic. For example, a question that requires an understanding of the differences between a deformation and a malformation was coded as “Pediatric/Dysmorphology 1b. Clinical presentation–physical exam findings.” Each trainee received a list of questions by their tag and noted by which they answered correctly and incorrectly. This was intended to be shared with the PD and to be used to identify areas of knowledge weakness so as to guide future learning.

Discussion

For each of the past two years we offered a 125-question exam that represented the breadth of clinical and basic medical genetics knowledge. The exam was available online, hosted by the University of Alabama School of Medicine, with few complications. It has shown increasing acceptance, as 174 trainees completed the exam in 2010, representing 36 training programs, of the 39 that had residents enrolled. In 2011, 215 trainees participated, a 24% increase, with only one program not participating. We believe that this process has been a success in that we have established a common exam that has gained widespread acceptance among medical genetics residency programs.

Although the exam was oriented to clinical trainees, the group scores of the laboratory fellows were close to their clinical counterparts in 2010 (49% vs. 54%). Although both groups improved in 2011, the difference stayed the same (53% vs. 58%). This followed a general trend, as most subgroups had an overall improvement in their scores.

For both 2010 and 2011, the overall scores were fairly low (52% in 2010, 56% in 2011), and the range of scores was narrow (s.d. 11 for both years). In 2010, only one score was below 2 s.d., none were in 2011. We do not know how this compares to the actual ABMG exam scores, as the information has not been made available.

It was reassuring that in general the more experienced trainees performed better ( Table 2 ), suggesting that the exam is testing information that is being taught and is relevant to medical genetics residency training. However, there were some differences noted in the scores by training track. Not surprisingly, laboratory fellows were disproportionately represented in the lower scoring groups, whereas categorical medical genetics residents were overrepresented in the top scoring group (see Supplementary Table S1 online). Among the clinical trainees, the categorical residents still scored the best, but the overall margin was small. In 2011, categorical geneticist residents had a mean score of 60%, internal medicine-genetics residents 59%, pediatric genetics residents 56%, and maternal fetal–medical genetics residents 55%.

The breakdown by subcategory scores also provided some interesting results. Some were not too surprising: biochemical genetics scores were the lowest subscores in three of the four groups. For categorical and pediatric genetics residents, it was 9% below the overall mean, for maternal fetal–medical genetics residents, it was 20%. For internal medicine-genetics residents, it was only 7% below the overall mean for pediatric genetics residents, but it was a better score than they had in basic/molecular and pediatric/dysmorphology.

Any conclusions drawn from these data must be made with some caution because there were relatively small numbers in each group. Based on the overall data, it seems that some trends can be observed. In general, apart from the maternal fetal–medical genetics score in biochemical genetics, there was relatively little variation among subscores from the mean. However, biochemical genetics was consistently a relative weakness across all training tracks. Such breakdowns of the subscores will be something that we will place more focus on in future iterations of the exam.

Future directions

It is hoped that the in-service exam will become a standard tool used by all medical genetics residency programs. Although the evidence suggests the exam is testing material that is relevant to genetics residency training, there is no way to correlate whether it indeed is helpful in preparing for the ABMG examination. One way to address that would be to do as many specialties do, which is to use the same question bank for the board certification exam as for the in-service exam. This would have several benefits. It would ensure that the in-service questions more closely reflect those the trainees will see on the board exam. We have tried to do this, but our effort is at best an approximation. It would also lessen the strain of finding people to write test questions. Currently, anyone who writes questions for the ABMG exam is prohibited to do so for the in-service exam as well. Furthermore, this would allow us to utilize questions that have been vetted and scored. Currently, each in-service question is reviewed (and edited as needed) several times before its inclusion in the final exam. After the exam, the results associated with each question are reviewed, and questions that score poorly (e.g., that have poor discriminators) are more carefully reviewed. Some are discarded, whereas others have a second answer accepted as correct. In 2010, there were six such poor questions, and in 2011 there were two. The questions developed through the ABMG have already been tested and scrutinized to a level that is impossible to duplicate in our current process. Finally, it may allow us to create a more realistic spread of scores, which would enhance the use of the test as an evaluation instrument. Although the mean number of correct answers rose between 2010 and 2011, it remained relatively low at ~60%. We would prefer this to rise, and professional item-writing and test-developing expertise would certainly help in that effort. Whether the Association of Professors of Human and Molecular Genetics in-service exam committee is able to join with ABMG, or remain an independent effort, it is vital to continue to improve the in-service exam by improving the quality of the individual questions and increasing the feedback given to both trainees and their PDs. Joining with ABMG is most attractive, not only for the aforementioned reasons, but it would facilitate one important future goal for the exam: tracking in-service exam scores with eventual board scores.

Members of the Subcommittees

Basic/Molecular Genetics Subcommittee

  • Peter Byers

  • Jay Ellison

  • Gerald Feldman

Prenatal Genetics Subcommittee

  • Laurie Demmer

  • Sue Gross

  • Susan Klugman

Pediatric Genetics/Dysmorphology Subcommittee

  • Margo Adam

  • Kim Keppler-Noreuil

  • Rob Hopkin

Biochemical Genetics Subcommittee

  • Shawn McCandless

  • Daniel Sharer

Adult/Cancer Genetics Subcommittee

  • Georgia Wiesner

  • Reed Pyeritz

  • Judith Westerman

Disclosure

The authors declare no conflict of interest.