Main

Assessment of medical and dental education has traditionally relied on testing students' ability to acquire and maintain facts. Methods such as written tests, written course work and multiple choice questions have played a major role in the evaluation of clinical education. Recently, however, the attainment of appropriate interpersonal communication skills have been recognised as a crucial part of a health care professional's education. Clinicians who use such skills have been shown to be judged as more capable by their patients,1 attract patients who are loyal to the practitioner2 and are less likely to be sued by their patients.3 Furthermore, patients far prefer to be interviewed by individuals who have been rated highly for their skill in communication.4 There is, however, some debate about how these skills should be taught and assessed. For today's dental schools a further crucial question is how, and at what stage, communication skills' training can best be integrated into an already crowded curriculum. The need to teach communication skills to dental undergraduates is enshrined in the General Dental Council guidance to dental schools5 but is generally taught on a relatively ad hoc basis when compared with other parts of dental undergraduate courses.

There is wide variation into the degree of formality of the teaching and the timing of the teaching of communication skills in dental and medical schools.6 Even more variable is the way in which these essential clinical skills are assessed. It would seem that in some schools, tutors simply allow students to observe them at work and expect undergraduates to learn what they need to know about communication with patients, by direct observation. Students' skills are then often graded by the person they had previously been observing. The problem with this approach is that students will only be assessed as performing well if they show similar traits to those demonstrated by the staff member. This subjectivity can cause wide variability in the skills acquired by students with different tutors and in different institutions. In an attempt to improve objectivity, some schools have attempted to assess students' communication abilities via 'objective structured clinical examinations' (OSCEs). However, to truly measure communication skill using this method, a role player is required to simulate a patient. Whilst the value of such standard patients has been clearly demonstrated, the development, introduction and assessment of such schemes is resource intensive in terms of time and effort.7 It would also seem unlikely that all schools would have convincing actors on their staff who could fulfil this function, and acting professionals are an expensive luxury which few undergraduate training institutions can afford.8

Thus, although the teaching of communication skills is a requirement for UK dental undergraduates, and attainment of the skills vital to graduates' success in practice, no formal, standardised method or criteria for assessing an individual's ability to communicate in a dental setting is available. In fact, a wide review of the literature revealed that there was no standard instrument available for assessment of the communication skills required by dental or medical practitioners. The literature did, however, reveal a number of examples of checklists employed in previous studies, most of which covered some of the issues relevant to a productive dental consultation.9,10,11,12

The aim of the study described was therefore to develop an instrument for the assessment of communication skills which was:

  • Appropriate for use for dental consultations/examinations

  • Feasible for routine use as an assessment tool for undergraduate dental students

  • Easy to use and

  • Reliable (repeatable and valid).

The instrument was also designed to be valid for measuring improvements in skills at early undergraduate level. Its performance in this latter respect will be described in a further paper. The purpose of this paper is to describe how the instrument was developed, review its content and, most importantly, determine its reliability for the assessment of dental students' communication skills.

Method

Development of the Assessment Instrument

In order to ensure content validity (the adequacy of the checklist as a measure of communication), the checklist was developed empirically by observing clinician/patient interactions within an oral medicine clinic. (Oral medicine was considered to be the most appropriate environment as operative dentistry is minimal and communication essential to the diagnostic process). In order to check that the overall communication within a consultation was adequately represented (face validity), consultants and lecturers were observed by a research assistant during patient consultations and examinations within the clinic. Each stage in the consultation and examination was recorded, together with the communication skills observed. A description of this 'process tracking' is shown in Fig. 1.

Figure 1
figure 1

Process tracking of stages of consultation and relevant communication skills

Using the skills revealed by the process tracking phase, plus examination of all currently available checklists, an instrument was developed. For each included item, a seven point Likert scale was used, with the anchors 'least evident' and 'most evident'. Two independent observers then observed ten staff members to assess the staff members' communication skills while using the new instrument. Following each observation, any sources of dissimilar scoring, or difficulties in interpreting or using the assessment schedule were discussed and appropriate alterations made. A final part of the development process measured inter-observer reliability for the whole instrument, and for each item. For this part of the instrument's development, two observers independently observed and recorded the performance of forty-three third year dental students while they treated patients in an oral medicine clinic.

A paired sample t-test was used to examine the inter-rater agreement. Agreement between raters for each item was examined using Cohen's Weighted Kappa. To ensure consensual validity (the extent to which a peer group agree about the instrument's appropriateness), the schedule was also distributed to consultants and lecturers for comment and possible corrections.

Results

The instrument was developed and tested. The Dental Consultation Communication Checklist (DCCC), is shown in Table 1.

Appropriateness

Two items, which had described behaviour demonstrated during the process tracking of clinician/patient interactions, proved not to be relevant to student/patient consultations. Item B5: 'redirects conversation if appropriate' was never observed in the student consultations. Item C3: 'ensures patient dignity, e.g. removal and replacement of dentures' was observed in only six out of forty-three student interactions.

Feasibility and ease of use

The DCCC was found to be feasible to use in an oral medicine setting and no difficulties were encountered when used for student observation.

Reliability

The mean total score for observer 1 was 118.42 (SD 19.9) while the total score for observer 2 was 118.00 (SD 19.9). There was therefore no difference between the observers in the total scores given to the students. There were also no significant differences between the mean scores per student for each observer (p > 0.05).

Table 2 shows the weighted kappa scores for each item on the DCCC when the two examiners scores were compared. The table demonstrates almost perfect agreement between observers for all items, except B4 — summarising and reflecting back (inter-observer kappa 0.77), B10 — making eye contact (inter-observer weighted kappa = 0.60), B12 — showing interest and evidence of listening (inter-observer weighted kappa = 0.79) and E1 — patient freely offering information (inter-observer weighted kappa = 0.75).

Table 2 Table 2

Discussion

The DCCC was specifically designed to be sensitive to communication skills' training given at an early stage in the dental undergraduate course. Whilst its ability to measure the effects of communication skills training is yet to be reported, the testing with a target group — such as third year dental students — who are relatively inexperienced perhaps explains why items such as 'ability to redirect conversation as appropriate' (B5) were demonstrated as inappropriate for inclusion for students. Although B5 is an important part of the consultant/patient interaction, students at this stage are unlikely to control the interaction with a patient to a measurable extent. Interestingly, in further studies subsequent to communication skills training, students were seen to increasingly redirect consultations. This anecdotal observation will be researched further but it would seem that control of consultations is a skill which students do not have early in their clinical careers but can be taught by educational interventions.

While the wide diversity of patients involved in the development process ensured generalisability of the DCCC, specific items such as 'removal and replacement of dentures' were found to be overly prescriptive and inapplicable in many interactions. It may be appropriate to include these items for specific groups of patients but as the instrument was developed with the specific intention of measuring student performance with all types of patient, they were deemed inappropriate.

The DCCC demonstrated almost perfect inter-observer reliability with respect to overall scores. Item-specific inter-observer reliability was also excellent for all but four of the included items. Of these, reliability was still acceptably high, except for the measurement of eye contact. This suggests that for items which are particularly prone to subjective judgement, training of the observer to adhere to strict assessment criteria may be necessary, i.e. one person's judgement of suitable eye contact may be different from another's.

Many of the items included in the DCCC reflected those which had appeared in other interview checklists. This suggests that it will be generalisable and widely applicable. Furthermore, the development of the instrument from process tracking analysis ensures its applicability for the designated purpose (i.e. assessment and training of dental students). The use of Likert scales makes the instrument sensitive to small differences in skills. Simple dichotomous scoring would not have been appropriate as the DCCC would then be relatively insensitive to the effect of communication skills training and would therefore not reveal areas for improvement or skills which need to be 'fine-tuned'. Since the purpose of the DCCC is to highlight students' skill deficiencies, its sensitivity to small improvements in communication is essential. This sensitivity also gives the instrument potential as an audit tool for practices to use from time to time to reinforce the continuing need for good communication between clients and the dental team.

The instrument was found to be easy to use in the test series and was deemed acceptable by the peer group of experienced practitioners to whom the final instrument was distributed for comment. It can therefore be considered to be appropriate, feasible and easy to use. Its applicability as an assessment tool has been ascertained by the process tracking stage of development and the very high inter-observer reliability scores.

This study has demonstrated that it is possible to formally assess communication in an oral medicine and surgery setting in an appropriate and reliable way. Further work is needed to confirm the instrument's construct validity, intra-observer reliability (when one observer uses the instrument twice, will the same score be achieved) and sensitivity to educational training interventions. Research is also under way to determine the relationship between patient satisfaction and students' communication skills score, and also to compare communication scores with students' overall performance in BDS examinations.

Communication is generally accepted as a key process in oral health care provision3 and the interaction between clinician and patient is recognised as providing the foundations for accurate diagnosis and appropriate treatment decision making.13 Furthermore, patient satisfaction with treatment and even clinical outcomes are known to be dependent on the quality of the consultative process.14 However, despite the fact that communication skills are known to be a key factor in a clinician's ability to provide a patient with high quality care, this essential practical skill is often taught and measured in an informal manner. Communication skill training is a requirement of dental undergraduate courses, yet its assessment is not. It seems wrong that these essential skills are not measured as part of formative assessment prior to an individual graduating as a practitioner when all other important clinical skills are subject to strict measurement of standards of competence. The instrument described in this paper offers a means of introducing consistent and reliable evaluation of dentists' interpersonal skills.

Table 1a Table 1a
Table 1b Table 1b
Table 1c Table 1c
Table 1d Table 1d