The UKCAT test: developments, research and its use by dental schools in the UK

Key Points

• Informs readers on the use of UKCAT in UK dental schools.

• Notifies readers of some developments in UKCAT since its inception.

• Provides a research update in the use of UKCAT as a research tool.

Abstract

The United Kingdom Clinical Aptitude Test (UKCAT) has now been an active part of UK dental admissions for seven years with the test being used by 11 dental schools within their admissions processes. This paper gives an overview on UKCAT and highlights some of the on-going work in relation to its development. This paper also highlights what UKCAT is and some developments with respect to the UKCAT. It also facilitates the process of keeping dental practitioners informed.

Introduction and background

In the United Kingdom, applications to read medicine and dentistry are made through the Universities and Colleges Admission Service (UCAS). UCAS reports that around 1 in 20 of all applications made through their system exhibit some degree of plagiarism. A research report has been published by the General Medical Council identifying the best practice in the selection of medical students¹ and the paper highlights a number of areas that are also clearly relevant to the selection of dental students. It also identified and discussed the strengths and weaknesses of different student selection methods. It is known that the screening and ranking of applications through academic achievement or academic potential, along with the examination of relevant work experience and personal and supporting statement are unreliable in discriminating between the large numbers of high quality applications received at medical and dental schools. Additionally, staff time pressures (along with constraints on resources) also hinder the admissions process as a whole.²

The traditional interview (even when structured) exhibits poor reliability and validity in student selection to medical school^3,4 and does not translate to how students perform academically;⁵ there is no reason to believe that such a generalisation would not apply to the similar processes used in dental schools.

The United Kingdom Clinical Aptitude Test (UKCAT) was introduced in 2006 to assist in the selection of students to University. For the 2012/13 entry it is estimated that around 10,000 applications were received for the 1,000 or so places on offer to read dentistry in the UK; this equates to between 25 and 30,000 individual applications as each applicant can indicate up to four schools that they would want their application forwarded to. The UKCAT is delivered by Pearson Vue Driving Assessments Limited (Pearson Vue) on behalf of the UKCAT Consortium group (Table 1) and UKCAT Board. In 2012 the test was delivered to a total of 25,431 candidates at 145 centres in the UK and over 210 abroad (Table 2).

Table 1 UKCAT consortium members: ^*denotes dental schools associated with consortium members and who use UKCAT as part of their admissions process

Table 2 UKCAT test delivery sites

The UKCAT is a test that assesses a number of mental abilities identified by medical and dental schools as being important. The test measures aptitude rather than acquired knowledge and includes assessments of verbal, quantitative and abstract reasoning along with decision analysis. The situational judgement section was introduced as a pilot sub-test in 2012 and became a live part of the test in 2013. Table 3 provides further information on each of the sections of the test.

Table 3 Component parts of the UKCAT test

As an admissions screening and selection tool UKCAT is utilised by all but four of the dental schools in the UK⁶ although the way in which it is used by individual dental schools varies. However, its use generally mirrors that of medical schools; by distinguishing between borderline candidates, being used as factor in selection, as a score which has to be reached in order for an application to be considered, and as a means of compensation during the selection process (Tables 4 and 5).⁷ UKCAT strives to ensure objectivity in its use and that fairness and widening participation in under-represented groups remains at the forefront of University selection processes; ultimately UKCAT hopes to assist in the identification of characteristics that will lead to improved patient care in medicine and dentistry.

Table 4 Description of how UKCAT is used by medical and dental schools

Table 5 Summary of how dental schools have utilised the UKCAT scores since its introduction

The UKCAT board acknowledges that candidates wish to prepare fully for the test and to achieve that make a number of specific recommendations having surveyed candidates in 2012. While the notes detailing how candidates can prepare are available on the UKCAT webpages the following summarises some of the key messages:

• UKCAT does not work with any commercial companies who publish books or offer coaching for the UKCAT test and cannot justify any claims these companies may make on their ability to guarantee an applicant's performance in the UKCAT test

• Candidates are advised to be familiar with the test itself, its requirements and the question styles

• UKCAT provides a practice app and official guide that contain practice questions

• Candidates who prepare generally do better than those who do not

• UKCAT provides a UKCATSEN (Special Educational Needs) test for candidates who require additional time consequent to a documented medical condition or disability that would necessitate additional time to complete the UKCAT test; again, further details are freely available on the UKCAT website.

Research and development

UKCAT has engaged in research since its conception and a research group was formally established in 2009; Appendix 1 details the research published to date, however, some key points from this research are summarised.

UKCAT and A level performance

James et al. (2010) compared UKCAT performance to A level achievement by 9,884 applicants for medical and dental students.⁸ The research highlighted that the higher the UKCAT score the likelier it was that the student would achieve the predicted grades.

UKCAT score and interview performance

Research examining UKCAT score and interview performance at one medical school revealed that a low UKCAT score correlated to an increased likelihood of rejection at interview (almost three times), although no relationship between UKCAT score and overall interview performance was noted.⁹ These authors also clearly stated that the UKCAT score should be used as an adjunctive tool in the selection of students and it should be borne in mind that as they showed no correlation between UKCAT score and interview modality that it is entirely feasible that they (UKCAT and interview) measure different attributes and using both tools increases the utilities of diversity and discrimination during selection.

UKCAT and widening access

Research into widening access to UK medical education reported that the use of UKCAT scores could facilitate the entry of some under-represented socioeconomic groups into the medical profession. However, how this could be achieved was very much dependent on how the score was utilised by the admitting institution.¹⁰

UKCAT and predictive validity

While the predictive validity of the UKCAT or its subtest scores as a means of equating undergraduate clinical performance is not yet fully known, there is some evidence to suggest that it may have some predictive validity in some knowledge-based examinations^11,12 and that it can also give an indication of professional behaviour.¹² It should also be noted that there is evidence to the contrary regarding year one medical school performance as shown by research carried out at the Universities of Aberdeen and Dundee.¹³ There is, however, evidence to show that when multi-centre data is collected and analysed that UKCAT has some predictive validity in medical school outcome; particularly among data processed for 'mature students'.¹⁴

The desire of UKCAT to adopt new assessment domains in the test is fuelled by the fact that individual domains have failed to clearly identify any predictive component with respect to study at medical or dental school. In 2012 specifically designed simulated judgements tests (SJTs) were trialled across the candidate population and these were incorporated into the test in 2013. The use of SJTs is not new^15,16,17,18 and these have already been shown an ability to measure professional attributes such as empathy and integrity in the postgraduate dental arena.¹⁵ It is hoped that the introduction of SJTs will increase the predictive validity of the UKCAT itself and demonstrate value as a selection and progression marker for undergraduate and postgraduate performance.

Equally important is the fact that the UKCAT has remained dynamic in its development over the past seven years with modifications to the original processes both proposed and implemented by the groups delegated to carry out work on behalf of the board: test delivery, test development, research panel. While SJTs have already been mentioned it should be noted that UKCAT continues to gather and interpret data relating to progression within medical and dental schools.

The UKCAT research group is currently drawing together information in a number of areas including:

• UKCAT is using its data to explore the value and reliability of established widening participation (WP) markers in use in higher education (HE). The development and identification of reliable markers of WP is something that is welcomed by many consortium members

• A detailed review of mature candidates' performance on the test. It is known that older candidates do not perform as well on the test as school leavers but the association with age and qualifications achieved has not been explored in detail. The longitudinal study suggests that predictive validity of the test for mature candidates is greater and UKCAT is keen to explore why this is the case

• Work around the impact of preparation on test performance. A survey of candidates was undertaken in 2013 to establish how they prepared for the test. This is being analysed alongside their results and a number of demographic factors. It is hoped that this will lead to better guidance and services to candidates regarding preparation

• One longitudinal study has focused on first year performance in medical school, however, since this study started a significant amount of additional progression data has been collected that will allow a fuller study to take place building on the established methodologies in the UKCAT12 study

• The University of Durham are repeating their analysis looking at the impact of the test on widening participation while also looking at the impact of increased use of the test on the candidate pool. Internal construct validity and internal consistency reliability assessments carried out by UKCAT shows that the test functions well.¹⁹ The UKCAT board and its associated research and test development groups ensure that developments are rigorously examined, tested, reported on and actioned upon where necessary.

Box 1: APPENDIX 1 (Reproduced with kind permission of the UKCAT board)

The UKCAT-12 study: Educational attainment, aptitude test performance, demographic and socio economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of twelve UK medical schools.

BMC Medicine 2013; 11: 244 IC McManus, Chris Dewberry, Sandra Nicholson and Jonathan S Dowell

http://www.biomedcentral.com/1741-7015/11/244

This study looked at 4,811 students in 12 UK medical schools who took the UKCAT from 2006 to 2008. Researchers concluded that UKCAT scores and educational attainment measures were significant predictors of outcome. The incremental validity of UKCAT taking educational attainment into account was significant, but small. Medical school performance was also affected by a contextual measure of secondary schooling, students from secondary schools with greater average attainment at A-level performing less well. This collaborative study in 12 medical schools shows the power of large-scale studies of medical education for answering previously unanswerable but important questions about medical student selection, education and training.

Can personal qualities of medical students predict in-course examination success and professional behaviour? An exploratory prospective cohort study.

BMC Medical Education 2012; 12: 69 Jane Adam, Miles Bore, Jean McKendree, Don Munro and David Powis

http://www.biomedcentral.com/1472-6920/12/69/abstract

This study shows numerous significant relationships between both cognitive and non-cognitive test scores, academic examination scores and indicators of professional behaviours in medical students. This suggests that measurement of non-cognitive personal qualities in applicants to medical school could make a useful contribution to selection and admission decisions. Further research is required in larger representative groups, and with more refined predictor measures and behavioural assessment methods, to establish beyond doubt the incremental validity of such measures over conventional cognitive assessments.

The UK Clinical Aptitude Test: Is it a fair test for selecting medical students?

Medical Teacher posted on-line May Paul Lambe, Catherine Waters, David Bristow

http://informahealthcare.com/doi/full/10.3109/0142159X.2012.687482

The aim of this study was to determine if differences in: access to support and advice, in modes of preparation, type of school/college attended, level of achievement in mathematics, gender and age influence candidate performance in the UKCAT and unfairly advantage some candidates over others. The study was carried out using questionnaires of applicants to study on an undergraduate medical degree course. Researchers found that differentials in access to support and advice, in modes of preparation, type of school/college attended, in level of achievement in mathematics, gender and age were found to be associated with candidate performance in the UKCAT. The findings imply that the UKCAT may disadvantage some candidate groups and that this inequity would likely be improved if tutors and career advisors in schools and colleges were more informed about the UKCAT and able to offer appropriate advice on preparation for the test.

Widening access to UK medical education for under-represented socioeconomic groups: modelling the impact of the UKCAT in the 2009 cohort.

BMJ 2012; 344: e1805 Paul A Tiffin, Jonathan S Dowell, John C McLachlan

http://www.bmj.com/content/344/bmj.e1805

The authors set out to determine whether the use of the UK clinical aptitude test (UKCAT) in the medical schools admissions process reduces the relative disadvantage encountered by certain sociodemographic groups. They were able to analyse detailed candidate and admissions data and look at how and how strongly different medical schools used the test. They concluded that the use of the UKCAT may lead to more equitable provision of offers to those applying to medical school from under-represented sociodemographic groups. This may translate into higher numbers of some, but not all, relatively disadvantaged students entering the UK medical profession.

Can the UKCAT select suitable candidates for interview?

Medical Education 2011; 45: 1041–1047 Rebecca Turner, Sandra Nicholson

http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2923.2011.03993.x/abstract

This paper examines current selection practices and questions the role that the UKCAT may take, focussing in particular on whether UKCAT can select suitable candidates for interview. The rejection rate before interview of candidates with low UKCAT scores was 2.7 times that of candidates with high UKCAT scores. However, no relationship between overall UKCAT score and overall interview score existed within a pre-selected cohort of applicants with high UKCAT scores. The paper concluded that the UKCAT can facilitate the independent selection of appropriate candidates for interview but that it is not predictive of success at interview.

A mixed-methods study identifying and exploring medical students' views of the UKCAT

Medical Teacher 2011 Vol. 33, No. 3, Pages 244–249 J.A. Cleland, F.H. French, P.W. Johnston

http://informahealthcare.com/doi/full/10.3109/0142159X.2011.557753

This was a mixed-methods study using a paper-based survey and focus groups with first year medical students in Scotland in 2009–2010. Questionnaire data were analysed using SPSS, focus group data using framework analysis. The UKCAT was viewed unfavourably by first year medical students completing it pre-admission. These negative views seem due to concern as to the use of UKCAT data, and the fairness of the test. The authors conclude that more evidence as to validity and fairness of the UKCAT and how it is used in practice is required.

Predictive validity of the personal qualities assessment for selection of medical students in Scotland.

Medical Teacher 2011, Vol. 33 Jon Dowell, Mary Ann Lumsden, David Powis, Don Munro, Miles Bore, Biokanyo Makubate, Ben Kumwenda

http://informahealthcare.com/doi/abs/10.3109/0142159X.2011.599448?prevSearch=allfield%253A%2528jon%2Bdowell%2529&searchHistoryKey=

The Personal Qualities Assessment (PQA) has been used within the non-cognitive sub-test of the UKCAT, having been developed to enhance medical student selection by measuring a range of noncognitive attributes in applicants to medical school. This study pre-dates the introduction of the UKCAT with applicants to the five Scottish medical schools piloting the PQA in 2001 and 2002. The study aimed to evaluate the predictive validity of PQA through a longitudinal cohort study in which PQA scores were compared with senior year medical school performance. The most significant finding was that students identified by PQA as 'not extreme' on the two personal characteristics scales performed better in an OSCE measure of clinical performance.

Use of UKCAT scores in student selection by UK medical schools, 2006–2010

BMC Medical Education 2011; 11: 98 Jane Adam, Jon Dowell, Rachel Greatrix

http://www.biomedcentral.com/1472-6920/11/98/abstract

This paper summarises the way medical schools have used the test since its inception in 2006,categorises that use and demonstrates trends in use over time. Annual telephone interviews were conducted with UKCAT Consortium medical schools. The paper reports that four ways of using the test results have emerged (Borderline, Factor, Threshold and Rescue methods) with many schools using more than one method. The considerable variation in how medical schools use UKCAT supports the wish of the Consortium that applicants are clearly informed about how the test will be used in order that they can make best use of their limited number of applications.

Comparison of A-level and UKCAT performance in students applying to UK medical and dental schools in 2006: cohort study

BMJ 2010, 340: bmj.c478 David James, Janet Yates, Sandra Nicholson

http://www.bmj.com/content/340/bmj.c478.full

The object of this study was to determine whether the UK Clinical Aptitude Test (UKCAT) adds value to medical and dental school, and in particular whether UKCAT can reduce the socioeconomic bias known to affect A levels. The study suggests that the UKCAT has an inherent favourable bias to men and students from a higher socioeconomic class or independent or grammar schools. At the same time however, it does provide a reasonable proxy for A levels in the selection process.

The value of the UKCAT in predicting pre-clinical performance: a prospective cohort study at Nottingham medical school.

BMC Medical Education 2010, 10: 55 Janet Yates, David James

http://www.biomedcentral.com/1472-6920/10/55

The study aimed to determine whether UKCAT scores predict performance during the first two years of the 5-year undergraduate medical course at Nottingham. This limited study suggested that the predictive value of the UKCAT, particularly the total score, is low although section scores may predict success in specific types of course assessment.

Has the UK Clinical Aptitude Test improved medical student selection?

Medical Education 2010, 44: 1069–1076 Sarah R Wright, Philip M Bradley

http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2923.2010.03792.x/abstract

The study aimed to determine whether the UKCAT had made any improvements to the way medical students are selected. The analysis studied the ability of previous school type and gender to predict UKCAT, personal statement or interview scores in two cohorts of accepted students. The results demonstrated that previous school type was not a significant predictor of either interview or UKCAT scores amongst students who had been accepted onto the programme. UKCAT scores were significant predictors of knowledge examination performance for all but one examination administered in the first two years of Newcastle Medical School whereas interview scores are not.

A comparison of the UKCAT with a traditional admission selection process.

Informahealthcare 2009, Vol. 31, No. 11, Pages 1018–1023 Nishan Fernando, Jennifer Clelland, Kathryn Greaves, Hamish Mckenzie

http://informahealthcare.com/doi/abs/10.3109/01421590802520923?2

The aim of this study was to compare candidate UKCAT performance with medical student selection outcomes at the University of Aberdeen. Results from this study indicate that UKCAT scores show weak correlation with success in Aberdeen's medical admissions process indicating that the UKCAT examines different traits.

Does the UKCAT predict Year 1 performance in medical school?

Medical Education 2009: 43: 1203–1209 Bonnie Lynch, Rhoda MacKenzie, Jon Dowell, Jennifer Clelland, Gordon Prescott

http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2923.2009.03535.x/abstract

This study set out to identify whether UKCAT total score and subtest scores predict Year 1 outcomes at the Universities of Aberdeen and Dundee. UKCAT scores did not predict Year 1 performance at the two medical schools suggesting that studies of UKCAT's ability to predict outcomes in later years of medical school and in subsequent qualification and practice as a doctor are indispensable in determining whether the use of the test as a selection tool is justified.

The UKCAT-12 study: educational attainment, aptitude test performance, demographic and socioeconomic contextual factors as predictors of first year outcome in a collaborative study of twelve UK medical schools.

British Medical Journal 2013 (in press). I C McManus, Chris Dewberry, Sandra Nicholson, Jon Dowell.

This study assessed the validity of educational attainment measures and UKCAT along with socioeconomic background factors, as predictors of medical school performance in twelve UK medical schools. This collaborative study showed the power of large-scale, multi-medical school studies of medical education for answering previously unanswerable but important questions about medical student selection, education and training. UKCAT has predictive validity as a predictor of medical school outcome, particularly in mature applicants to medical school. This study confirmed the validity of using all the existing measures of educational attainment in full at the time of selection decision-making. Contextual measures provide little additional predictive value, with the exception of overall level of secondary school attainment, students from high attaining secondary schools performing less well than those from less well attaining secondary schools, as HESA has previously shown for universities in general

Summary

The use of UKCAT test since its inception had remained a dynamic undergraduate selection tool and has been developed over the years in order to facilitate successful admissions selection procedures. It continues to progress and develop in both the way the test is conducted and how it can facilitate research into successful student selection and progression.