Abstract

Aim

The aim of this study was to evaluate the effect of major curriculum change within a UK dental school on final-year student self-rated confidence levels.

Methods

Final-year dental students graduating in each year between 2007 and 2012 completed the same course evaluation questionnaire, which assessed their confidence in relation to a range of clinical procedures using a Likert-type scale. This period spanned the introduction of a new curriculum and allowed analysis of differences in self-rated confidence between students graduating from the old (2007 and 2008) and new (2009–2012) curricula, across thirty key procedures.

Results

New curriculum students showed significantly higher self-confidence ratings in nineteen of the thirty procedures, compared with those on the old curriculum. For the remaining eleven procedures there was no significant difference between the two curricula. The proportion of students on the outcomes-based curriculum rating themselves as 'confident” was statistically significantly higher in seven out of the thirty procedures, when compared with the traditional curriculum, and unchanged or nonsignificantly increased in the remainder.

Discussion and conclusions

The relationship between specific curricular innovations and student confidence is considered, as is the usefulness of self-rated confidence in curriculum evaluation. Curriculum change appeared to have a positive effect on student confidence across a range of procedures.

Introduction

The advance of outcome- or competency-based education in medicine seems undeniable.1 The same would appear to be true in dentistry, and in the UK this is illustrated by the move towards outcome-based and away from process-based curriculum guidance from the General Dental Council.2 Various other trends in dental education are apparent, including the use of 'outreach' teaching in community settings. In line with a number of these trends and recommendations the school featured in this report introduced a new undergraduate dental curriculum in 2004; the first graduates from this curriculum were in 2009. The previous curriculum had been predominantly subject-based, with relatively little integration. The philosophy of the new curriculum was to:

  • Be outcome-based

  • Facilitate integrated learning

  • Allow for earlier clinical experience

  • Encourage reflective learning

  • Encourage teamwork

  • Provide a significant amount of clinical experience in community settings.

The ultimate aim of any undergraduate dental curriculum should be to produce graduates who are ready to begin to practise safely and with varying degrees of independence depending upon the jurisdiction. A number of authors have addressed the issue of dental graduate preparedness for practice and investigations have frequently utilised measures of self-efficacy, while accepting the limitations of this approach.3,4,5,6,7

Substantial curriculum change, however theoretically well-founded and carefully planned, represents a risk for the institution and its students. The effects of such change should be monitored and evaluated, and in professional programmes this should extend to the preparedness of those about to graduate. This paper describes a survey of final-year undergraduate students conducted every year between 2007 and 2012 by a UK dental school in collaboration with the Scottish Dental Practice Based Research Network (SDPBRN). The format of the survey remained essentially unchanged during this period, consisting of a questionnaire with several sections, the first of which required students to self-assess their confidence in relation to a number of clinical procedures. The data presented spans the introduction of the new curriculum and allows direct comparison of student cohorts who graduated before and after its implementation.

Research aim

To compare final-year student self-rated confidence levels in relation to clinical dental skills before and after implementation of a new undergraduate dental curriculum.

Methods

Final year (BDS Year 5) dental students graduating each year between 2007 and 2012 were invited to complete a course evaluation questionnaire requiring them to assess their confidence in relation to a number of clinical procedures. The questionnaire remained essentially unchanged over this period. The survey was anonymous and students were informed that participation was not compulsory and refusal to take part would not disadvantage them.

The questionnaire was based on one previously developed and used to survey the confidence of vocational trainees in relation to a range of skills required for the delivery of dental care.8 It was adapted for use in the undergraduate setting through discussions involving senior academics from two dental schools, representatives from the SDPBRN, final year dental students and a psychologist. Questionnaire items related to clinical practice, non-clinical practice, professional development, final year experience, clinical scenarios, outreach teaching and career choices. The questionnaire was administered in the second semester of the final BDS year, towards the end of teaching but ahead of the final examination. This report focuses solely on the thirty questions relating to clinical practice. These questions were designed to specifically assess student self-rated confidence levels in relation to clinical skills and used a Likert-type scale of one (not at all confident) to seven (extremely confident). Responses were analysed for significant differences in self-rated confidence between two groups, that is, students graduating from the old curriculum (2007 and 2008) and students graduating from the new curriculum (2009–2012).

The relevant university research ethics committee considered that the proposed analysis did not require formal ethical approval. Consent for use of the data was gained from the dental school management and SDPBRN.

Statistical methods

Anonymised questionnaire response data were entered into the statistical package SPSS version 21 for analysis. As the data was not normally distributed, continuous numerical data was analysed using the Mann-Whitney U test. In addition, the data was split into those who were confident, that is, scoring a five or above, and those who were not. Results were expressed as percentages in each group and a Fisher's exact test was carried out to test for differences between the groups. Due to the number of comparisons being carried out a P value of <0.01 was set to reduce the chance of a type two error.

Results

The survey was completed by 56 students in 2007, 63 in 2008, 61 in 2009, 56 in 2010, 60 in 2011 and 63 in 2012. The participation rate ranged from 71% to 91% across the six years. Data therefore relates to a total of 119 students who had completed the old curriculum and 240 who had completed the new curriculum. Cronbach's alpha was calculated for each survey and this showed excellent reliability and consistency with a range of 0.879 to 0.925 (Table 1).

Table 1 Reliability statistics
Full size table

Table 2 shows that student confidence scores were generally greater in the new curriculum group compared with the old curriculum group. Confidence scores were statistically significantly higher (P <0.01) in the new curriculum in 19 of the 30 clinical procedures. In no case was the confidence score statistically significantly lower at the P <0.01 level but students appeared less confident in carrying out an apicectomy (P = 0.014).

Table 2 Table of mean confidence scores (standard deviations) for each year
Full size table

Table 3 illustrates a different approach to looking at student confidence, that is, the percentage of students in each cohort who described themselves as 'confident' in each procedure (that is, those who scored themselves five, six, or seven on the Likert scale). Fisher's Χ2 test was carried out to test for differences in proportions between the groups. The percentage of students on the new curriculum that rated themselves confident was higher or the same in all but one question when compared with students on the old curriculum, although the difference reached statistical significance in relation to only seven of the 30 procedures. The exception was, again, in relation to confidence in carrying out an apicectomy.

Table 3 Percentage of students who are confident in carrying out procedures before and after curriculum changes (Fisher's exact x2 test)
Full size table

Discussion

In this study, with one exception, student self-rated confidence in relation to clinical procedures either increased or remained unaffected by the introduction of a new curriculum. This discussion will focus on two overriding issues: firstly, the possible impact of curriculum change on levels of confidence; secondly, the interpretation of student confidence in curriculum evaluation.

How might curriculum change impact on student confidence?

It would be speculative to suggest that there was a direct cause and effect relationship between the curriculum changes introduced, either individually or collectively, and the observed improvement in aspects of student self-rated confidence. However, there may be theoretical and empirical support for such speculation and it would be worthwhile considering this.

The new curriculum was designed to be not only outcome-based but to follow the precepts of constructive alignment.9 In a constructively aligned curriculum learning activities are carefully designed to facilitate attainment of the intended learning outcomes, with assessment also aligned. So, for instance, if an outcome stipulates that competence must be attained in an operative skill, learning must take place with the deployment of appropriate resources in an environment that allows such skills to be acquired and developed, from simulation through to practice in an authentic clinical setting, and assessment must feature appropriate testing methods, allowing valid inferences to be drawn about the candidate's competence. According to Biggs and Tang (2007) 'students are 'entrapped' in this web of consistency, optimising the likelihood that they will engage in the appropriate learning activity.'9 If this is true and learning is enhanced by such an approach, one might anticipate that this would be reflected in increased measures of self-efficacy. Unfortunately, compelling data to suggest that outcomes or competency-based approaches affect student attainment is lacking and this has been identified as a research priority.10

Although much has been written on the subject of integration within healthcare curricula11 little is known about its benefits for student learning. One study has demonstrated that both problem-based learning and teacher-centred integrated curricula (biomedical and clinical teaching are integrated but delivery is largely didactic) may be superior to a conventional subject-based curriculum for the development of diagnostic competence in medical students.12 Intuitively, clinical reasoning would seem to be the domain most likely to benefit from integrated learning but this clinical attribute did not feature in the current evaluation.

While early clinical exposure would seem to offer inevitable benefits in terms of increasing the time in undergraduate programmes over which students can develop their skills, again, evidence of benefit is sparse. One study demonstrated that early clinical experience was approved of by dental students, who reported positive effects on readiness for direct patient care,13 and there is some evidence of a correlation between increased clinical experience and improved confidence.14

The approach to the facilitation of reflective learning in the curriculum under consideration has been described15 and is based on the theoretical model of Boud et al. (1985).16 There can be little doubt that the ability to reflect and to learn from reflection is seen as an essential attribute for healthcare professionals.17 Mann et al. (2009)17 reviewed various aspects of reflection in healthcare education and practice and noted that evidence is lacking in relation to a number of key questions, including whether reflection affects clinical behaviour or improves patient care. A recent relatively small scale study suggests that daily reflection can help students to learn from experience but also, importantly, that it may serve to improve student insight into their own competence.18 This last point is intriguing because of the questions that surround the use of self-rated confidence as a measure of preparedness, as discussed below.

Perhaps the most compelling evidence for the effect of a curriculum intervention on student confidence is to be found in relation to outreach teaching. Studies have consistently reported favourable student experience in a variety of outreach (primary care) settings, resulting in improved confidence, growing self-reliance and, in some cases, developing insight.19,20,21,22,23,24 Frequently cited benefits of clinical experience in outreach settings are: the transitional nature of the environment, between dental school and practice; the increased quantity of clinical experience and more appropriate casemix; and the availability of close support. Outreach experience in the curriculum under discussion was very significantly increased from some five to eight days over the course of three clinical years in the old curriculum, to in excess of fifteen weeks in the new curriculum. Outreach, in fact, comprises half of all clinical allocations in the final year of the programme.

It is worth noting that, in the present study, confidence did not increase in relation to every clinical procedure; self-rated confidence in ability to perform an apicectomy, declined from an initially low level. This exception is reassuring since this skill does not constitute one of the learning outcomes of the programme and students should recognise their limitations in this regard. It should also be stated that questionnaire outcomes fed into the annual quality assurance process and that where, in relation to particular clinical skills, confidence appeared unexpectedly low, measures were taken to assess and, if necessary, improve the quality and content of teaching in that area. These interventions might have been expected to result in improved confidence subsequently, as suggested by the minor changes that can be observed between 2007 and 2008 (Table 2). However, such interventions would be dwarfed by the scale of change potentially impacting on student confidence thereafter, with the introduction of an entirely new curriculum.

Are measures of student confidence useful in curriculum evaluation?

Self-rated confidence is often employed in the evaluation of health professions curricula but the credibility of such measures has been debated. There are two inferences to consider: the first is that students are able to accurately self-assess; the second is that an accurate self-assessment will affect confidence, which, in turn, will be a true reflection of competence.

Evidence from a number of studies suggests that clinicians are not able to accurately self-assess and that there are either no, or at best, weak associations between self-assessment and external assessment.25,26,27,28 Davis et al.29 carried out a systematic review entitled, 'Accuracy of physician self-assessment compared with observed measures of competence' and found in a majority of studies there was poor or limited accuracy of self-assessment and this finding was independent of speciality, level of training, domain of self-assessment, or manner of comparison. It should be borne in mind, however, that external measures of competence may themselves be of questionable validity and there is considerable ongoing debate about how competence might best be assessed in dental education.30 Studies that fail to demonstrate a correlation between self-assessment and measures of competence may point as much towards invalid methods of external assessment as they do to inaccurate self-assessment.

So what of confidence and its relation to competence? There is support from the psychology literature for the view that personal efficacy belief has a strong influence on performance.31,32,33 Within the dental literature self-rated confidence has been frequently used as an approach to evaluating preparedness for practice, although often with caveats about the relationship between confidence and competence.3,4,6,7 In some of the studies reviewed by Davis et al.29 the method of 'self-assessment' was self-rated confidence; such conflation may not be justified. Confidence is a complex construct that may relate as much to personality traits and other factors as it does to accurate self-assessment of genuine competence.5,34 Even if clinicians and students are capable of accurately and objectively self-assessing this may not translate simply into confidence.

A key issue is likely to be the professional insight of the students doing the self-assessment and expressing confidence.5 It would seem that certain interventions, such as the daily reflection referred to above, can help students to develop an improved capacity for critical self-assessment.18 Smith et al. (2006)20 report the striking finding that students who had completed an outreach placement recognised that their self-rated confidence before the placement had been over-optimistic and that they had adjusted their internal confidence scales as a result of the outreach experience. These studies suggest that students' insight is at least partly dependent on the nature of the learning experience to which they are exposed. Contextual information, for instance, concerning aspects of curriculum design discussed above, may thus be of considerable importance when interpreting measures of self-rated confidence.

Mindful of the above discussion, the current authors contend that using measures of self-rated confidence, accompanied by sufficient contextual information, can be of value in curriculum evaluation, particularly when used longitudinally to illustrate the impact of interventions, in that they can flag up areas of concern or unexpected directions of change. They provide one means of approaching student perceptions of their own attainment, which may reflect aspects of curriculum design and delivery, and therefore have a role in quality assurance and curriculum refinement and review. A particular concern in an evaluation of this type is that student views will be influenced by their involvement in the intervention in question. In the data presented here some reassurance can be derived from the fact that the changes observed are generally sustained across the four years following introduction of the new curriculum and that increases are seen in many but not all areas. There are clearly other means of evaluating curriculum change; the views of faculty, external examiners, accrediting and regulatory bodies will all provide useful feedback. As with assessment, triangulation of evidence from a number of sources is probably what is required.35 Published evaluation of curriculum change is sparse; it will always be problematic, since most such projects will involve changes in many aspects of curriculum design and delivery and the outcomes, however measured, may prove impossible to relate to specific innovations.

Conclusion

This study presents the outcome of a student evaluation, based on self-rated confidence, during a period of significant curriculum change. The move from a traditional to an outcomes-based, integrated, curriculum involving significant outreach teaching appeared to have a positive effect on student confidence. Attention is drawn to the care that must be employed when interpreting evaluation data derived from self-rated confidence; it is suggested, however, that this is a useful tool when used as part of a broader evaluation strategy.