Background

As the regulatory body for the UK dental professions, the General Dental Council (GDC) assures the quality of the provision of dental education and training. In 2012 the GDC introduced (and subsequently modified in 2015) a framework of Educational Standards that dental education programme providers are expected to demonstrate; failure to do so resulting in a programme being deemed 'insufficient'.1 There are 21 standards described within three domains: 'Protecting patients'; 'Quality evaluation & review of the programme'; and, 'Student assessment'. Within 'Student assessment' standard 17 requires programme providers to engage with patient feedback as part of their assessment framework; 'Assessment must utilise feedback collected from a variety of sources, which should include other members of the dental team, peers, patients and/or customers.'1

The implications of failing to demonstrate any standard are clear, as graduates of a programme deemed to be insufficient by the GDC would not be registerable.

While there is currently no explicit requirement for programme providers in America, Australia, Canada or New Zealand to utilise patient feedback in student assessment, within the UK patient feedback is now perceived to be an important element in many areas of health care training and ongoing professional development.2,3 The General Medical Council include a similar requirement in their educational standards, as described within Promoting Excellence; standards for Medical Education & Training.4 Likewise, patient feedback is currently required as part of the revalidation process for doctors, and is included within the portfolio of dental foundation trainees in England & Wales. Thus, the inclusion of patient feedback within the assessment framework of undergraduate programmes while a requirement of the regulator also has the potential to introduce and promote good future professional practice.

In fully delivering standard 17, providers of programmes face the challenge of gathering and effectively utilising patient feedback. In order to do so there is a need to identify and/or develop an appropriate method, which while manageable and maintainable within current resource, also has real value to the learner.

In response to this, Newcastle University, School of Dental Sciences introduced a new process which provides the opportunity to utilise each and every clinical encounter with a patient to gather feedback and incorporate this into the students' personal electronic portfolio iDentity.5 The patient feedback thus gathered provides a fourth domain of formative assessment of each clinical encounter along with 'subject knowledge', 'professionalism', and 'treatment quality'. The process involves distribution of feedback cards, similar to the 'Friends & Family'6 test used widely within the NHS.

The process is promoted to patients using posters in waiting areas and student clinical bays, and the distribution of cards is initiated by clinical teaching staff. The feedback cards have a simple format asking only three questions, two of which involve a four point Likert response scale. The third question requires a free text response, asking if there was 'Just One Thing' (JOT) that could be done to improve the patient's experience (Fig. 1). The student discusses the completed card with their clinical teacher in order to consider the patient feedback both independently and in the context of the clinical session. A formative 'patient feedback' grade is then awarded by the teacher based on the information on the feedback card.

Figure 1
figure 1

Just One Thing feedback form

Full size image

Aim

The aim of this study was to evaluate the nature of patient responses within the JOT model in order to determine its value in providing undergraduate dental students with feedback on their clinical performance.

In order to achieve this three objectives were described:

  • To quantitatively analyse the number of JOT cards collected in a two week study period

  • To quantitatively analyse the range of responses provided to questions 1 and 2

  • To qualitatively analyse the free text comments received in response to question 3.

Materials and methods

A favourable ethical opinion for this study was gained from the Faculty of Medical Sciences Ethics Committee of Newcastle University (1500/07.7.15).

JOT feedback cards were collected for analysis during a two week study period in April 2016. Students enrolled on two courses were evaluated – Bachelor of Dental Surgery (BDS) and Diploma in Dental Hygiene and Therapy (DDHT). The BDS is a five year undergraduate programme with the majority of students entering university straight from secondary school, while the 27 month DDHT programme has a greater number of students who have entered the programme following qualification and experience working within the health care sector, for example, dental nursing.

The distribution, collection and feedback from the JOT cards followed the procedure previously described, except that at the end of the clinic and following provision of feedback, all cards were placed in an envelope. This was collected on a daily basis by one of the research team. At the end of the study period the data on each card was entered into a spreadsheet by a member of the schools administration team. The anonymised data included; clinic, student course (BDS or DDHT) and year, Likert score for questions 1 and 2, and free text comments to question 3.

Simple descriptive statistics were determined for the number of cards collected for each course/year group, and for the range of responses to questions 1 and 2.

A simple analysis of the themes of the free text comments given in response to question 3 was undertaken by the research team, following the principles of content thematic analysis.7 As a result a number of themes were identified and each comment was coded accordingly.

Results

In the two week study period a total of 332 JOT cards were collected. The greatest number of cards collected were from the year 4 and 5 BDS student clinics, however, the largest pro-rata number were from the DDHT students (Table 1).

Table 1 Number of JOT cards collected by course, year group (DDHT = Diploma in Dental Hygiene & Therapy, BDS = Bachelor of Dental Surgery)
Full size table

The majority of cards were collected on the adult restorative dentistry clinics (75%), with the remainder of departments collecting significantly fewer cards (dental emergency clinic 10%, paediatric dentistry 8%, oral surgery 7%).

The overwhelming majority (89%) of JOT cards scored the students as 'excellent' for both question 1 and 2, with the remaining scoring the student as either 'good' or offering no response to at least one of the two questions. None of the JOT cards were scored as being 'adequate' or 'poor' (Table 2).

Table 2 Distribution of responses to Q1 and Q2 (% of total)
Full size table

In question 3 patients were asked 'Is there just one thing your student could do to improve your experience'. A significant proportion (62%) of patients left a comment in this box.

Following analysis eight main themes were identified. The proportion of comments falling into each theme is provided in Table 3.

Table 3 Percentage of occurrences of a theme within patient comments
Full size table

1. Complimentary by omission

A small number of patients openly stated that they could not identify anything that the student could do to improve their experience, for example:

'None.'

'Can't think of anything.'

2. Nonspecific complimentary relating solely to the student

Almost one third of the free text comments included generic compliments which, while not truly addressing the question asked, would seem to suggest once again that the patient could not think of anything that could be improved upon. Some of the comments explicitly stated this:

'My student is excellent, it would be impossible for him to improve on. Perfection.'

'[Name] could not do any better than he already is, he is a genius dentist in the making.'

Other patients left a single complimentary word or short superlative phrase:

'Brilliant.'

'Always great.'

3. Specific compliment relating solely to the student

Some comments explained patients' experience of excellence of care and two sub-themes emerged:

a. Relating to professional behaviours/attitudes

Comments in this theme often focused on the ability of students to reassure, calm or put patients at ease and the effectiveness of their communication:

'Very calming student who naturally put me at ease at what is usually a traumatic experience.'

'He puts his patients at ease. Includes them in the process, informs them of procedures and asks for help if needed. Shows an ability to think ahead. Professional, would make a good dentist. Takes care to listen to my concerns.'

'He has been very clear about my treatment and supported me with choices I need to make. I have confidence in him and would recommend him to others. Continue to build on his confidence, thank you to an excellent student.'

b. Relating to patients perception of skill

The second sub theme focused on the patient perception of skill or outcome, and a number related to management of pain.

'Very good when giving injections.'

'Thrilled to bits with new denture.'

4. Complimentary about service/institution and teachers

Some patients extended their comments to encapsulate the wider team involved in the delivery of patient care, and mentioned staff either by name or as supervisors, teachers or tutors:

'I am so scared of the dentist – terrible experiences in the past. [Student Name] was Fab! Very thorough and gentle. Dr [Staff Name] – although she looks about 12 – was lovely! Restored my faith in dentists!'

'Excellent with good back up from tutors.'

5. Comparison to previous experience

A small number of patients compared their experience with previous dental care and these often related to previous anxiety or dread of seeing the dentist that had been managed successfully by students:

'Cannot think of anything. [Student Name] is very good at what she does. She has gained my trust, as I dread dentists.'

'[Student Name] has made me overcome my fear of dentists.'

6. Just One Thing?

There were multiple occasions when patients addressed this question. These were often provided alongside a compliment or offered encouragement. Four sub-themes were identified:

a. Confidence of student

'Be more confident in what they do.'

'She's been excellent over the past few months! She is very knowledgeable and she's very passionate. Her communication skills are excellent and she always makes sure that I understand the process and what she is doing. She needs to be a bit more confident because she'll be a great dentist in the future! [Student Name], be proud of yourself!'

b. Timing of care

'Could have been faster but admittedly my tooth was a nightmare.'

'Sometimes seems that he is rushing. Overall happy about treatment.'

'Work a tiny bit quicker, perhaps a little more confidence while handling equipment.'

c. Communication

'It would be better if he could slow down so his client could catch what he says.'

'Remove/drop mask when speaking if hard of hearing patients.'

'To not talk posh, although I have been teaching her some Geordie phrases.'

d. Enhancing experience – specific suggestions

'Put some music on.'

'Offer tissues to wipe mouth – to improve on.'

'Perhaps check more often about any pain but she has a lovely manner!'

'Finish with a cup of tea, otherwise perfect.'

7. Negative

All of the comments bar one were largely positive, with any critique usually being balanced by a compliment or reassurance. The single truly negative comment related to the management of pain, communication issues, confidence and speed.

'[Student Name] needs to listen more if there is pain there is a reason! Also needs to be more confident which will speed up and reduce likelihood of no.1.'

8. Unclear

A small number of comments could not be categorised as it was not clear whether or not they were intended as a compliment or an area for improvement. For example:

'They now work as a team.'

'Keep smiling.'

Discussion

The focus of the published research on patient feedback is within medical practice and in particular the postgraduate arena, however, the evidence base supporting the effectiveness of feedback in enhancing physicians' behaviour is limited and there is a need for future high quality research studies.8 Within a primary dental practice setting attempts to develop patient feedback questionnaires have been successful in producing an instrument with utility for accreditation of practices but lacking specific focus on the individual practitioner. While such questionnaires demonstrate validity, structural integrity and reliability, no attempt has been made to evaluate the value or effectiveness of the instrument in informing professional development.9

Patients may not always be best placed to comment on the quality of care provided by a doctor, dentist or student as they may not be reliable assessors of a practitioner's clinical skills. Moreover they may use this opportunity to comment on peripheral events beyond the control of the practitioner, such as waiting time, and their opinion may be influenced by external factors such as cultural differences between the practitioner and the patient.10

Feedback should perhaps therefore focus more on interpersonal skills, how the patient felt about their experience and the information they were provided with.11

Communication skills are a vital and integral part of success as a dental practitioner. Effective communication is one of the nine principles in the GDC's Standards for the Dental Team document,12 and 'Communication' is one of the four domains of attainment required by the GDC in their document Preparing for Practice which outlines the requirements of UK dental graduates.13 How these skills are 'taught', and the learning opportunities for development in dental schools is therefore important. A systematic research review of communication skills in dental education revealed that many studies reported applied didactic learning and the use of simulated patients in a role-play scenario as a method for skills development.14 The review recommended a greater 'active' input from 'real' patients to communication skills development.14 Such involvement in the reported literature is limited, whether this is a true reflection of actual teaching or only a reflection of the focus in publications is unknown.

There are challenges with using 'real' patient feedback in summative assessments of communication skills due to variability, which would provide challenges for assessment leads in ensuring consistency. The involvement of simulated patients would ensure a level of 'standardisation', but have an impact on value to students. This warrants further consideration. Using patient feedback in a formative sense may also be preferable if engagement with real time responses is desired.

Most UK NHS services now use the 'Friends and Family' Test as a quick tool to assess patient satisfaction with respect to some of these aspects of their clinical experience. At a basic level, this asks 'How likely are you to recommend our practice to “Friends and Family” if they needed similar care or treatment?', followed by one simple follow up free text question. Since April 2015 it has been a contractual requirement for NHS dental practices to obtain this information, however, currently, the results for dental practices are not made widely and publically available as originally intended6 and therefore the value of the data obtained is unclear.

In any open, ongoing formative process of collecting patient feedback, the number of forms collected may vary between individual practitioners due to differing engagement with the process. This may be influenced by the acceptability of the process and the perceived value of the feedback. While gathering patient feedback may be an acceptable and appropriate way of evaluating the practitioner's communication skills, doctors have reported finding it stressful, and indicated they may target specific patients or sessions or they may even alter their behaviour in the knowledge that feedback is being sought.15

In our research this may also have been influenced by each practitioner's enthusiasm for or belief in the process, concerns or fears associated with the collection of feedback and engagement, or their general willingness and ability to comply with prescribed tasks. In this study, although the number of forms collected per student was not attributed, the distribution of cards was initiated by the supervising clinician, so this effect would be minimised.

Within the study period the number of cards collected amounted to an average of slightly more than one per student. Assuming a similar rate of collection throughout a 40 week academic programme, this equates to 20 feedback episodes per student per clinical year. It must however be recognised that the conduct of the study itself may have had a bearing on the number of forms collected, in that an awareness of deviation from the routine process may have either prompted or dissuaded staff and students from engaging. It is impossible to determine the direction of effect. Nonetheless this outcome does demonstrate that it is feasible to collect this volume of feedback, in an ongoing model which is considered to be of greater value in enhancing interpersonal skills than when compared to a single episode.16

Differences were noted in the pro-rata number of cards collected across different programmes (BDS and DDHT), different year of study and different clinical areas. Each of these, together with possible reasons have been considered.

The BDS programme has a larger number of students than the Hygiene and Therapy Diploma. The smaller hygiene and therapy cohort have a smaller number of supervising clinicians, and it is conceivable that these staff may have been more proactive in distributing the feedback cards, possibly due to heightened awareness of the process as there are fewer sessional teachers, and a greater number of staff with substantive roles in programme delivery. Between all staff groups it is conceivable that some clinicians may have greater 'buy-in' and belief in the use and benefit of the feedback cards, or the process, than others.

Clinics with more senior students tended to collect greater numbers of cards. Again, this could be attributed to staff impact, but another factor to consider is the activity occurring during the session, and other competing demands that may be present on clinics with more junior students. The collection period also fell during the time many year 5 students were completing treatment plans and seeing their patients for the last time – this may seem like an obvious and appropriate opportunity to seek feedback.

We expected to observe that clinics where students see patients on a recurring basis, that is, adult restorative clinics, would have a higher number of cards collected than those where the patients were seen on a 'one-off' or 'emergency care' nature, such as oral surgery and a dental emergency clinic. However, our data suggests that the number of cards returned was proportionate to clinical time allocated and patient contacts in each area, rather than the nature of the interaction (for example, third year students have five times the clinical contact time in restorative dentistry compared to paediatric dentistry). There is also substantial consideration required as to the purpose of the clinic, and the priorities in terms of patient care and the nature of the treatment. It is conceivable, appropriately so, that after seeing a patient who has attended in pain for emergency treatment, who may not have eaten or slept well before their appointment and may have a degree of dental anxiety, the clinician may not feel it appropriate to seek feedback. However, our data suggests that patients seen in 'emergency clinics' were as likely to complete cards as comprehensively (that is, complete both questions and comments) as those seen on a continuing care basis (34–41% across the different clinical areas). Therefore, it appears to be reasonable to seek feedback across all clinical areas. At the time that this study was undertaken the JOT cards were not routinely being used in community outreach clinics. It would be useful to repeat this study in this context in order to evaluate whether the different clinical environment alters the nature of feedback provided by patients.

The majority of participants (90%) answered both questions 1 and 2. Seven percent of participants selected to answer either question 1 or question 2. Of the seven participants who did not respond to either question, six provided positive free text comments and one provided no comments. No participants graded any students lower than 'good', with most being graded as 'excellent'. While we would like to suggest this degree of positive feedback is entirely due to excellent student interactions, we must also acknowledge that it may also be due to patients feeling uncomfortable giving anything other than positive feedback. The majority of the cards were collected from the restorative dentistry department, where patients are likely to be attending for multiple appointments. While cards were not handed out or collected by the students themselves, patients will know that the cards can be attributed to their visit and therefore may be unlikely to want to jeopardise future good relations at appointments by providing negative feedback. The positive skew on this data, however, does bring into question the validity of this form of data collection, which was introduced to mimic the friends and family test widely used within the NHS, particularly in primary care. While detailed data from dental practices using the friends and family test locally is not available, perusal of the NHS choices website would seem to suggest that a similar positive bias is not evident in the local population providing feedback on local GP practices. It is possible, however, that a greater sense of anonymity is felt when patients complete friends and family questionnaires compared to our Just One Thing cards. The friends and family test in GP practices can generally be completed online via the practice website, or within the practice, either electronically or by completion of a form. They are unlikely, however, to be traced back to the care provider.

While we have tried to identify a way in which the cards can be distributed and collected to ensure patient anonymity, the resource implications of doing so would be prohibitive. However, more important is the consideration of the loss of opportunity for the student and clinical teacher to discuss the feedback in a timely manner while the clinical episode is still fresh in their minds. This is supported by Bogetz et al.17 who suggest that 'facilitated sessions' are necessary to facilitate full engagement and learning from patients feedback.

Written comments were provided by 62% of participants. Where no comments were added it is tempting to assume that the patient could think of nothing to suggest improvement on, however, other explanations do exist such as a lack of time, a lack of understanding of what they were being asked to do or a desire not to add any further comment.

The vast majority of these free text comments were very positive and insightful, and likely to provide the most helpful feedback to our students. The friendly, caring attitude of our students was frequently alluded to. Some comments were harder to interpret once removed from the context of the situation, but may strike chords with the students involved – such as 'they now work as a team' or 'to smile even more'. Interestingly, both of these comments which may be perceived as negative were accompanied by 'excellent' scores in questions 1 and 2, indicating these patients were generally happy with the care they received.

The focus of the third question was to identify where improvement was necessary, and therefore, given that the majority of cards failed to explicitly answer that question, it could be argued that they fail in their primary purpose. We would suggest that highlighting areas of good or excellent practice on which a student can build is perhaps just as valuable as identifying areas of weakness. Moreover, repeated identification of 'poor performance' over a significant period without appropriate affirmation could undermine confidence and be detrimental to the long term development of the clinician. This is supported by the existence of feedback models in a caring context such as Pendleton, where negative feedback is sandwiched between recognition of good practice.18

The first two questions appear to be less useful in terms of discriminating between students, and consideration could therefore be given to asking only for a single written comment. However, asking the two leading questions before a written question may offer an easy introduction to the questionnaire and may give participants ideas on what to offer comments on.

The aim of this study was to evaluate the nature of patient responses within the JOT model in order to determine its value in providing undergraduate dental students with feedback on their clinical performance. While we feel that we have been successful in achieving the first part of this aim, lack of student involvement at this stage means that we are unable to make a fully informed value judgement on the JOT model. Nonetheless, we have been able to identify that the patients, who completed the JOT cards that were evaluated, used the opportunity to provide feedback in a considered and supportive fashion. We should also note that the patient feedback does not stand alone but rather contributes to a two way dialogue between student and teacher at the end of each treatment episode. We would therefore hope to undertake further work in this area to consider how students and their clinical teachers utilise this type of patient feedback to inform future development.

Conclusion

In a two week study period feedback cards were collected that amounted to one per student. If translated to a full academic year this equates to 20 individual episodes. The majority of cards came from more senior students.

The vast majority of quantitative responses graded students as being excellent or good.

Free text comments were provided in approximately two thirds of cards and while not all gave an explicit response to the question 'Is there JOT your student could do to improve your experience?' the comments were perceived by the researchers to have value to the students' development.