Introduction

Denplan Ltd runs payment solutions for dentists and their patients. The company's core product is a capitation-based funding system called Denplan Care. The company has diversified to also offer insurance-based products and loans to patients for extensive dental work. Denplan Excel accreditation is a voluntary quality assurance programme for dentists supported by the Patients Association. Accreditation commenced in 2001. By May 2011, 886 dentists were accredited, with more than 100 nearing accreditation.

Denplan Excel accreditation has five key requirements:

  • To use the Denplan Excel oral health score

  • To share information with patients, backed up with plain English information literature

  • To participate every three years in a patient feedback survey

  • To adopt a high standard of record keeping

  • To participate in a facilitated practice assessment every 18 months.

This paper describes the development of a new patient survey instrument for the programme. The objective was to design a concise audit tool for dental practices, not a sophisticated research instrument. It was intended that it would be easy for patients to use, and easy for practices to interpret the patient feedback which they received. It was also anticipated that this feedback would be valuable in informing practice development. Additionally, it was intended that this instrument would help to support accredited practices in gathering evidence for Care Quality Commission registration and General Dental Council revalidation.

The survey previously used in the programme contained 84 questions divided into 28 sections. It was posted to a random sample of patients registered with Denplan. It was considered that a more concise survey, which focused on those issues shown to be of greatest importance to patients, would help to achieve the stated objectives. It was also considered that a concise survey would be less onerous for patients to complete and could therefore encourage higher response rates. Further, it was considered that a survey focusing on the most important aspects of practice success might help practices indentify important areas for development efficiently.

The concept of 'thin slicing'

Gladwell1 made a strong case for the use of focused data in assessing situations. He cited several examples of how 'thin slicing', or restricting the volume of relevant data, can improve management. He claimed that, far from improving decision making, large quantities of, even apparently relevant, data can cause confusion. He stated: 'You need to know very little to find the underlying signature of a complex phenomenon.'

In 1977 Goldman et al.2 developed an algorithm for assessing patients with chest pain based on only four factors:

  1. 1

    The ECG

  2. 2

    Is the pain felt by the patient unstable angina?

  3. 3

    Is there fluid in the patient's lungs?

  4. 4

    Is the patient's systolic blood pressure below 100?

Reilly and Evans3 tested this 'thin sliced' method at Cook County and found that doctors were 70% more likely to accurately predict which patients were having a myocardial infarction than when they used their own diagnostic methods. Their own methods often included 12-15 factors.

In his book The ultimate question, Reichheld4 makes a case for the single question customer satisfaction survey. The question he suggests is: 'How likely are you to recommend us?' The customers were asked to rank their answer on a scale of 1-10 where 10 is most likely. Customers responding with 9 or 10 were considered to be 'promoters' of the business. Those scoring 7 or 8 were considered to be 'passives', while those scoring beneath 7 are ranked as 'detractors'. Reichheld4 then suggested subtracting the percentage of detractors from the percentage of promoters to give the 'net promoter score'. In his experience, most organisations scored between 10 and 20 although he experienced companies who score 80-90. Detractors can be followed up with the question: 'Why don't you recommend us?' The simplicity of this approach is to be applauded. However, only the detractors would be giving any critical feedback. It is possible that satisfied patients could still have important aspects of their care that they would like to see improved.

A common problem with dental satisfaction surveys

Newsome5 highlighted an important problem with dental patient satisfaction surveys; he stated: 'Published studies of dental patient satisfaction nearly always reveal very high levels of satisfaction.' And further: 'The modal response that is, the value that occurs most frequently, is typically the most positive response allowed by the questionnaire.' This could make surveys of limited value in informing practice development. He speculates that these findings could be either:

  1. 1

    Because patients are very happy with their dental practices

  2. 2

    They do not feel confident about their ability to evaluate

  3. 3

    They don't like to criticise.

Gerbert et al.6 found very high levels of patient satisfaction and stated: 'Patients were overwhelmingly satisfied with their current dentist: 87% indicated that they were very satisfied and 98% said that they planned to stay with their dentist.'

A recent example of this problem with the value of patient satisfaction surveys was reported by Howard-Williams.7 He reported running two audit cycles for practitioners on the following topics:

  • Infection control and decontamination

  • Clinical record keeping

  • Quality of radiographs

  • Patient satisfaction

  • Recall intervals based on NICE guidelines

  • Contractual obligations in NHS.

The results were summarised as follows: 'All the audits showed improvement (in the second cycle) with the exception of the patient satisfaction survey where the first audit cycle showed an average patient satisfaction rating of 99% which cannot be improved on.' No copy of the 'instrument' which was used was published, but it is stated that it has been redesigned to try and make it more challenging and informative.

This does indeed seem to be the challenge with patient feedback instruments – how can they be designed to reflect the differences which must exist in perceived performance between different issues in the same practice and between different practices? It is only then that these instruments become valuable in informing practice development.

Developing the new survey questions

Chisholm and Askham8 were critical of the patient survey instruments which they reviewed. They reviewed ten questionnaires being used by regulatory bodies in the UK, the USA and Canada. They suggested that ambiguous questions were common and that 'patient engagement' was usually overlooked or badly covered. They identified five key domains which should be covered in patient questionnaires in order to assess those issues most important to patients:

  • Interpersonal skills

  • Communication of information

  • Patient engagement and enablement

  • Overall satisfaction

  • Technical competence.

On patients giving feedback on 'technical competence' they made the following observation: 'It could be argued that even if patients are not good judges of some aspects of technical competence, the fact that they think a doctor is not competent is something the latter needs to know.' They believed that, although patient perception of lack of competence is not truly objective, it could be indicative. Overall, therefore, they emphasised the importance of patient feedback on communication and competence issues.

Busby9,10,11,12,13 defined dental practice success across four 'dimensions', namely oral health, patient satisfaction, job satisfaction and financial profit. He suggested that a primary focus on patient satisfaction was likely to be rewarded with success in all dimensions. By reviewing the literature he concluded that patient satisfaction with the following were most important to practice success:

  1. 1

    Comfort and freedom from pain in the mouth

  2. 2

    Functionality of the oral structures in relation to eating in particular

  3. 3

    Appearance of the teeth

  4. 4

    Cleanliness and hygiene of the practice and its team

  5. 5

    Communication of the practice team

  6. 6

    Competence of the practice team.

With the help of experts at Electoral Reform Services (ERS), who conducted patient surveys on behalf of Denplan Ltd, the set of ten core questions reproduced in Table 1 was constructed (Fig. 1). These questions were informed principally by the Chisholm and Askham8 and the Busby9,10,11,12,13 reviews summarised above.

Table 1 IVR feasibility study
Full size table
Figure 1
figure 1

A reproduction of the new patient survey

Full size image

Grading of responses

It will be noted that only three grades of response were permitted. Four grades of response are usually considered to be the minimum number for customer surveys.14 However, in view of the fact that the modal result, in most dental satisfaction questionnaires, is the most positive answer offered, it was decided to use only three grades in this project. It does seem pointless to offer four grades when patients are so inclined to use the highest grade. The three grades used in Standards in dentistry15 for clinical audit were used. It was considered that 'ideal' responses would be challenging to achieve.

Feasibility testing with IVR

The feasibility of this instrument was tested using interactive voice response methodology (IVR). Dillman et al.14 describe this method as making use of a pre-recorded script which the invited customer then listens to, once connected via the telephone, and provides their response using the telephone key pad. Dillman et al. comment that IVR can be very useful and cost effective. This mode has been quite widely used in commerce and government administration. However, Dillman et al. warn that this method can be tedious for respondents.14

Using IVR was felt to have advantages over using an internet-based system, which could disenfranchise patients without IT skills. It was also considered that using a telephone might have advantages for patients over completing a paper-based survey. It was considered possible that confidence in confidentiality might be enhanced, and that response rates could consequently be raised.

Electoral Reform Services (ERS)16 were appointed to conduct the feasibility study. They designed the IVR system, which was then tested with seven volunteer practices (20 dentists). Letters inviting patients to participate were distributed by the practices. The letters informed patients of the free-phone number to call and allocated each patient codes to identify practices and to prevent multiple submissions.

About 25% of the patients who received an invitation responded. This figure was lower than that achieved in the existing postal survey (which was around 30% in 2008). Reports were received from practices indicating that some patients did find this method (IVR) of giving feedback inconvenient and tedious. However, the majority of the volunteer dentists did find the survey easy to use and believed that the questions covered issues which were most important to patients. They reported that their results were easy to interpret when presented with a focus on their 'ideal' scores for each question set against the average for all seven practices (see Fig. 2). The observation made by Newsome5 about patients tending to use the highest score available was again observed in this trial since 72% of all patient responses were recorded as 'ideal'. Nevertheless, statistically significant differences were observed in the results when comparing the different participating practices. The percentage of ideal scores received by each practice ranged from 57%-74%.

Figure 2
figure 2

Practice X principal bar chart (telephone interactive survey feasibility study (2009)); all practices PPI = 72%; practice X PPI = 65%

Full size image
Figure 3
figure 3

The highest and lowest percentage 'ideal' responses for each question (qualification- practices received at least 50 responses); highest practice PPI achieved = 84% (qualification at least 50 responses); lowest practice PPI achieved = 60% (qualification at least 50 responses); average net promoter score = 76%

Full size image

Methods

It was decided to maintain these ten core questions but abandon the IVR mode when the new survey was launched as part of the Denplan Excel standard in September 2010. The idea of practices distributing the invitational letters was retained as this allowed all types of patients (Denplan, NHS and private-fee-per-item) to be included easily. The survey was attached to the invitational letter, with patients sending their responses to the ERS in a pre-paid envelope. An option to complete the survey online was also offered. In this format it was a simple matter to add the Reichheld net promoter score, a space for verbatim comments, a request for demographic data, and to retain a general question from the previous survey for comparison. Each dentist was supplied with 200 surveys to distribute, up to a maximum of 600 surveys per practice. Dentists were requested to personally distribute the surveys to consecutive patients attending for care.

Results were presented on a practice (not an individual dentist) basis. The principal results chart sets the percentage of ideal responses achieved for each question for all patients plotted against the average results from other participating practices (called the national reference sample or NRS). An example from the original feasibility study is reproduced in Figure 2. The percentage of ideal responses achieved across all ten questions was shown for each practice. This is called the patient perception index (PPI). The full report extended to seven pages and included a breakdown of results by age, patient contact arrangements and gender.

A statistical analysis was undertaken to investigate, for each question, at an average number of responses (n = 121), the score at which the result becomes statistically significantly (to 95% confidence) below the NRS.

Results

The results from the first three months of operation are presented. During September, October and November 2010 the survey was conducted by 61 practices. A total of 7,381 responses were received by ERS, indicating that, on average, 121 responses were received from each practice. It is not possible to indicate a 'response rate' because it is not known how many invitational letters were distributed by each practice. The average PPI across the 61 practices was 74%. Figure 4 summarises the responses to all ten core questions across all 61 practices. Additionally, the proportion of Denplan, NHS and private-fee-per-item patients (PFPI) surveyed is shown. This sample of 7,381 responses has now been taken as the NRS for those taking part in the survey in 2011.

Figure 4
figure 4

Summary of all responses to the ten core questions (the NRS for 2011); percentage of respondents Denplan patients = 61.0%; percentage of respondents PFPI private = 27.6%; percentage of respondents NHS = 11.4%; average PPI = 74%

Full size image

Figure 5 plots a score below which the result becomes statistically significant against the national reference sample for each of the ten core questions.

Figure 5
figure 5

Scores below these results are statistically significant given an average number of responses

Full size image

Discussion

The results to date suggest that this group of patients perceive favourable outcomes from their dental practices around those issues which the literature suggests are most important to practice success. Despite the theoretically demanding use of the 'ideal' grading, 74% of the responses across these ten core questions received this top grade. Newsome's assertion5 that the modal response is centred on the highest available grade in dental satisfaction surveys holds true for eight out of the ten questions.

The two questions receiving the lowest overall scores were:

Question 3: 'Generally the appearance of my teeth (including any false teeth) is....'

and

Question 9: 'The value for money given by my dental practice is....'

Twenty-seven percent of patients were prepared to grade their dental appearance as 'ideal'. Three percent of patients graded their appearance as 'unacceptable'. Forty-eight percent of patients considered that they received 'ideal' value for money, 50% perceived that they receive 'acceptable' value for money and 2% considered their care 'unacceptable' value for money. This could be considered to be a satisfactory result at this point in the economic cycle of the United Kingdom.

The average net promoter score across the 61 practices was +76. It will be recalled that Reichheld4 maintains that most organisations score between +10 and +20. This result is another indication that these patients seem to have favourable perceptions of their dental practice.

The practices taking part in this survey could not be considered to represent a random sample of all UK practices. They are all participating in a voluntary quality assurance programme. The practices have been independently quality assured against a standard before participating in the patient survey. Several workers have commented on bias which might affect any project conducted using volunteer practitioners. In this respect Gilbert et al.,17 writing about a practice-based research network (PBRN) based in Birmingham, Alabama, stated: 'At some PBRN presentations audience members suggested that dentists who participate in PBRN research might be substantively different from dentists at large. Information from medical PBRNs, suggests that if a practitioner performance is the focus of the research findings may not be representative.'

Because dentists distribute the questionnaires at the practice, the possibility for 'favourable selection' of patients exists. It is possible that the questionnaire (despite the clear instructions in the protocol to distribute to consecutive patients) might not have been given to 'challenging' patients, resulting in a more positive survey outcome for a practice than might be the case if all adult patients attending during the study period had been invited to participate. Dillman et al.,14 in their guidelines for the design and conduct of customer feedback surveys, state in guideline 10.2: 'Develop procedures for ensuring that onsite sampling is carefully executed and will not be affected by personal preference. Dillman et al.14 discuss how corrupting 'selection' can be, particularly if financial rewards are linked to favourable survey results. 'Performance' in the patient survey described in this paper is not linked to any award, or reward. Members are required to show that they act on the feedback. The primary purpose is to inform practice development. Using these methods the practice takes responsibility for the quality of the data they ultimately receive, which in turn is for their developmental benefit alone.

The authors of this paper consider that the provision of the national reference sample for comparison is an important aspect of this audit instrument. This benchmark is central to informing the need for practice development. Without an NRS to refer to, a practice could be forgiven for believing that a score of 78% of patients perceiving that the standard of cleanliness and hygiene in the practice is 'ideal' indicates a good performance. However, when appraised that the NRS on this issue is 94% the practice might well consider how this perception can be improved significantly. It will become apparent from viewing Figure 5 and the discussion below that any score below 91% for this question is of statistical significance with a large enough sample.

It may be considered important to remember, however, when comparing patient perceptions of different practices, that we are not comparing the same group of patients' perceptions in all of the practices (in the manner that 'secret shoppers' in the commercial world may compare services by visiting several different outlets). Each practice is being appraised by its own group of patients. They may have been 'calibrated' only by experiencing one practice. Nevertheless, honest perception is reality for those patients. Perceptions may not always be truly 'fair', and patients almost certainly have differing expectations. Ultimately the relevant 'performance' for a practice is the extent to which they meet the expectations of their patients. The authors would therefore suggest that these comparative results for practices are relevant.

Figure 3 demonstrates the differences between the highest and lowest percentage of 'ideal' grades achieved across all ten questions for all of the practices receiving more than 50 responses. The instrument under investigation appears to be measuring notable differences in perceived performance between the different issues and between different practices. Typically for a question there is a difference of around 35% between the highest and lowest scores. It will be apparent from Figure 5 that this is statistically significant.

The results presented in Figure 5 indicates when a score, based on the average number of responses (n = 121), would be statistically significantly below the NRS. Statistical analysis therefore reveals that for an average number of responses (121) 95% confidence can be expressed at values of between 4 and 9% below the NRS depending on the question. It has been decided that, from September 2011, reports to dental practices will contain a table highlighting any result which is statistically significantly higher or lower than the NRS.

Conclusions

In keeping with international experience of dental patient feedback, this audit suggests high levels of patient satisfaction with their dental service. Nevertheless, by focusing results on the proportion of highest grades received, this instrument is capable of indicating when perceived performance falls significantly below the average. It can therefore inform practice development.