Key points

  • Analyses data from multiple mini interviews and identifies the factors being captured by this approach.

  • Discusses how the measured traits map on to the competencies required for dentistry.

  • Highlights the implications for the efficiency and efficacy of the interview process.

Introduction

The process of undergraduate dental education in the UK is both lengthy, typically five years, and expensive, with the total cost of training being approximately £200,000. Therefore, the identification of students with the necessary aptitude for the profession is essential for the training institution and the trainee.1 The selection of the best suited students will ultimately ensure that the best educated graduates will be entering the dental profession, and thereby benefit patient care for the public in the future.2,3

Some dental educationalists have developed lists of domains required for prospective students to become competent dental practitioners. The purpose of these lists is to guide processes aimed at identifying those students with the most potential. For example, the American Dental Education Association (ADEA) has identified the following skills as essential for a dental student: critical thinking, professionalism, communication, interpersonal skills, health promotion, practice management, informatics and patient care.4 Similarly, Cowpe et al. identified seven domains in profile and competence for the graduating European dentist comprising: professionalism; interpersonal, communication and social skills; knowledge base, information and information literacy; clinical information gathering; diagnosis and treatment planning; therapy; establishing and maintaining oral health; and prevention and health promotion.5 This list has subsequently been approved by the general assembly of the Association for Dental Education in Europe (ADEE). The General Dental Council (GDC) has also setup learning outcomes for potential registrants which are grouped in four domains: clinical, communication, professionalism, and management and leadership, along with nine key principles (standards for dental team).6 The issue is then how to best evaluate the core traits that will allow a student to take advantage of opportunities to acquire these skills over their educational journey.

The traditional approach to undergraduate selection in UK dental schools has been through unstructured interviews. This method has strong face validity7 but has many failings, including a lack of standardisation, poor predictive value and the potential for interviewer and social bias.8,9 Moreover, unstructured interviews fail to systematically capture the wide-ranging skills required for dentistry. These problems have led many dental schools to switch to standardised selection processes designed to map to the specific set of skills and aptitudes that are believed to be required for dentistry.

Structured interviews have, therefore, been gaining traction in recent years.10,11 Perhaps the most popular form of structured interview is the 'multiple mini interview' (MMI). MMIs involve short independent assessments, typically in timed circuits. These assessments are designed to resemble the objective structured clinical examination (OSCE) and are rated by one or two assessors.12 MMIs have been successfully introduced by several health disciplines across the world as well as within in a number of dental schools.13,14,15

Importantly, MMIs have been found to be fair and acceptable to students, with students reporting they enjoyed this interview format, and stating that the process allowed them to be competitive. Students also reported that MMIs helped them present their strengths free from any social bias.8,13,16,17 The MMIs are also perceived positively by assessors who have reported that MMIs are effective and provide a format that allows them to evaluate soft skills, candidate abilities and thought processes. The assessors suggested that overall MMIs evaluate a better range of competencies when compared to traditional interviews.18,19 In terms of reliability, recent reviews for student selection in health profession training have suggested that MMIs have moderate to high reliability and have the added benefit of allowing additional analyses to be conducted.17,20 The effectiveness of MMIs in predicting future undergraduate and postgraduate performance has also been reported to be good.21,22

In dentistry, a number of studies, focusing on the perception of applicants and interviewers,23,24 have suggested that MMIs are potentially a better predictor of ultimate dental performance than traditional interviews25,26 and indicate that MMIs are particularly useful in testing cognitive reasoning skills.14 The potential advantages of MMIs have meant this selection approach has been adopted by a number of dental schools within the UK. Nevertheless, no studies have been conducted to establish exactly what skills and abilities these stations are assessing. Nor have any studies ventured into the related issue of whether the purported assessment at a given station corresponds to the appropriate underlying construct. Here, we take an important step in promoting an evidence-based approach to prospective student assessment by providing a systematic examination of the underlying factors being assessed in a current MMI.

Materials and methods

Admission process

Applicants were selected for interview based on their UCAS form (Universities and Colleges Admissions Service). The UCAS form assigns numerical scores for each of its components, which include academic performance, medical experience, work experience, activities and reference report; each application was then ranked. The marking was performed by experienced members of the admissions team and marked twice to ensure there were no discrepancies.

Participants

From a total of 1,409 applicants, 245 candidates were invited to compete via MMIs for a place on the five-year master and bachelor of dental surgery (MChD/ BChD), and bachelor of science (BSc) programmes at the University of Leeds, UK for 2013/14 entry. Two hundred and thirty-nine students attended and 87 were successful in their application. We retrospectively retrieved anonymised data on all 239 applicants for the purposes of this study (approved by the School of Dentistry research ethics committee at the University of Leeds DREC ref: 271,016/IM/216).

MMI

The MMI scenarios were developed to assess different domains of competency, with a focus on non-cognitive skills. The scenarios were determined by academics, the admissions teams and professional/specialist staff within the dental school. Restrospective probing of the members of the team which were involved in scenario selection revealed that the decisions were based largely on clinical experience of the requirements for successful dental practice. A list of the ten stations, the skills these stations were purported to assess and the tasks employed to assess these skills is presented in Table 1.

Table 1 Details of skills and the procedure being assessed by each station
Full size table

The stations were run by dental school staff, including clinical academics and researchers, and current undergraduate dental students from the fourth and fifth year. All staff members and students who took part in the MMIs received extensive training beforehand. The staff had multiple practise runs with simulated students to practise the scoring system (the purpose of this simulation run was to ensure smooth running of the stations and so examiners could familiarise themselves with the scoring system) and they also received a briefing on the days of the interviews.

Procedure

Each circuit took eight students and there were four circuits per session, with each session being half a day. Each station lasted between seven to eight minutes and one minute was given for applicants to make themselves comfortable, be greeted by the examiner and presented with the scenario. The applicants were then given five minutes to perform the task. Candidates had one minute to move between stations. Each station was rated by one or two assessors. The interactive digital stations took around 20 minutes each to complete (ten minutes to explain the task and ten minutes to perform the task). The total MMI time was 104 minutes with approximately 64 candidates being examined per day. The marking criteria for each station are described in Supplementary Material Table 1.

Data analysis

For statistical analysis, we measured performance on all ten items. All the items were tested for normality and sampling adequacy to ensure the data met the requirements for factor analysis. Where data were not normally distributed, a transformation of the outcome variable was performed. A correlation matrix was created to determine the relationship between the variables. A parallel analysis method along with a scree plot were selected to be the extraction methods for determining the number of factors to extract over the eigenvalue rule.27 The parallel analysis was followed by factor rotation to determine the loadings of each item on the factors. All data were analysed using R version 3.3.1.

Results

A factor analysis was conducted on ten items with orthogonal rotation (varimax). The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy verified the sampling adequacy for the analysis KMO = 0.69, and all KMO values for individual items were >0.5. This demonstrated that it was acceptable to proceed with the analysis. Bartlett's test of sphericity, which tests the overall significance of all the correlations within the correlation matrix, was significant (x2 = 189.09, df = 45, p <001), indicating that it was appropriate to use a factor analytic model on this dataset.

All ten items entered the factor analysis together. Using the parameters of this study, the parallel analysis method suggested that two factors be retained. Inspection of the scree plot supported the results of the parallel analysis, suggesting that two factors gave the most interpretable solution. An orthogonal rotation (varimax) was then performed, since the factors were expected to have low correlation, to determine the loading strength of each item to the factor. Inspection of the factor correlation matrix showed non-zero correlation between the proposed factors. For the interpretation of the factors, the pattern matrix was used following the analysis. This analysis revealed that all items loaded significantly on one of the factors. Figure 1 demonstrates the loading strength of each item to the factor.

Fig. 1
figure 1

Factor loadings of the ten items: memory, ethics, presentation, origami, insight, communication, interpretation, tanagram, simodont and CKAT (clinical kinematic assessment test); across the two factors of 'soft skills' and 'sensorimotor skills'

Full size image

The results of the factor analysis of the ten items used in the current study revealed two factors were sufficient to explain the underlying structure of the MMIs. The first factor had an eigenvalue of 1.37 and accounted for 14.6% of the variance in the data. The second factor had an eigenvalue of 0.52 and accounted for a further 6.3% of the variance.

The first factor seems to reflect soft skills as all six items (presentation, memory, ethics, interpretation, and insight) related to the ability to communicate (with the ability to show empathy), analyse and interpret data, describe things, show ethical awareness and reasoning, and give their personal insight into issues.Thus, factor one was labelled as 'soft skills'. The second factor appeared to represent sensorimotor skills as the four items origami, simulator performance, CKAT and tangram loaded most highly on it. All four items related to manual dexterity performance with the ability to follow complex instructions, thus, factor two was labelled as 'sensorimotor'.

Discussion

The present study was based at the dental school at the University of Leeds, where ten selected scenarios were deemed to be useful tasks for identifying the most suitable students for admission. This reflects an approach that has been adopted by many dental schools throughout the UK. While there is a degree of sharing good practice/approaches used across different dental schools, ultimately each dental school has its own MMI structure; that is, each school will use different types and numbers of scenarios and the scoring of performance will differ across institutions.13,16 This situation suggests that there is a need to evaluate the scenarios used and conduct formal statistical tests to ensure that dental schools are using the best possible assessment procedures, with the ultimate goal of establishing an optimal assessment procedure that could be used by all.

An evaluation of the research literature to date suggests that there has been little formal evaluation of MMIs within dental schools to allow a formal evaluation of the individual tests and their psychometric properties, and enable evidence-based improvements in the selection process, despite the nature of MMIs and the wealth of data collected on an annual basis. For example, we found only one study on this topic; that particular study investigated the influence of gender and starting station in the MMI used for dental school entry.28 In medicine, there have been studies that have investigated the MMI test characteristics when station type was manipulated10 and the effect of examiners' systematic differences in the rating pattern for candidates' scores and selection.29 Eva et al. noted that changes to the structure of the stations can yield better outcomes, for example, behavioural interview stations were found to be better than unstructured situational judgement and free-form stations.10 These types of studies indicate the potential for statistical evaluation of the assessment process, with the data then enabling improvements to be implemented on the basis of objective findings. Nevertheless, there is a lack of reported research into the properties of individual tests and the underlying factors and traits that are captured by the MMI stations.

The present study investigated the number of factors that underpinned performance across the MMI stations and examined the statistical relationship between the stations. Our correlation analyses showed low correlations, but the factor analysis revealed two distinct factors that could explain the underlying structure of the MMIs. We labelled these factors as 'soft skills' and 'sensorimotor' ability. If we accept that the design of the MMI had good face validity for the experienced admissions team, then it is possible to conclude that these are two fundamental factors that are essential in prospective dental students; along with academic capability, which is typically assessed via standardised national examinations within the UK. This result tallies well with the general consensus across the dental discipline regarding the critical attributes that are required by dental student. For example, a review paper highlighted the importance of these skills in dental practice and suggested that 'soft skills' increase confidence, professionalism, coordination, friendliness and optimism in an individual.30 The review also suggested that a combination of soft and motor skills are important for patient management, dental practice and business management.

The identification of these two fundamental traits is important because it provides an evidence-based rationale for the factors that MMIs need to capture. In turn, this allows greater efficiencies within MMI design. For example, our data suggests that fewer stations may be required to capture 'soft skills', given that six stations load onto this factor. There are advantages to some redundancy in the stations, for example, a student may perform poorly on an initial station because of nerves; but there are clear economic advantages to having the lowest possible number of tests for each domain of competence as this will help in covering more traits. This will be further decided when mapping these stations with eventual student performance, and thereby a clear view on how these stations could be redesigned by either refining or combining better stations and rejecting poorer ones will be achieved. This mechanism can provide a tool for assessment of these MMI stations to robustly measure broader competency traits and identify the tests that have the best construct validity for these domains. MMIs typically include some form of assessment of motor skills, as manual dexterity is an integral part of dental practice.31,32 Unfortunately, a number of motor skill assessments rely on poorly validated instruments that require subjective evaluations of performance and are intrinsically unreliable.

The results of the current study suggest that it would be highly beneficial for dental schools to adopt and evaluate precise and objective measures of sensorimotor ability. It may also be useful to develop tests that combine the skilled control of the hands together with higher-order cognitive abilities such as decision making, as this reflects the reality of how motor control is implemented within dental clinics. The MMIs within the present study included a virtual reality simulator that required a naturalistic combination of sensorimotor and decision making skills, and this may be a particularly useful station.33 In the future, it will be of interest to determine which of the existing stations provides the best prediction of undergraduate performance, as indexed by performance on the myriad of tests conducted throughout the undergraduate degree. The great advantage of the MMI system is that the usefulness of the stations can be evaluated over time and assessments altered on the basis of this evidence. The present study provides a small but important first step in the statistical evaluation of dentistry MMI stations.

Conclusion

A well-established interview technique for entry to a UK dental school was subjected to factor analysis. The results showed that the interview process captured two fundamental traits across ten assessment stations. Further studies involving these stations and their ability to predict undergraduate performance will allow the iterative and methodical improvement of station design. Thus, such data and analyses will have important implications for the design and refinement of the entry processes for dental schools across the world.