Introduction

The wide variation in the types and availability of out-of-hours dental services in the UK is well documented.1,2 Typically people not registered as NHS dental patients (or who do not receive regular dental care under private terms) may only be able to attend a 'walk-in' emergency dental clinic, if one is provided in their area, and if they know or find out that such a service exists. These often have fixed opening hours, usually on Saturday and/or Sunday morning for two to four hours, and are located in community dental service clinics, dental teaching hospitals or at district general hospitals.1 In contrast, most NHS-registered patients should be able to make telephone contact with a dentist who shares out-of-hours arrangements with their own dentist (usually a locality-based rota involving a number of practices). The emergency dentist on-call, after speaking to the patient, may then invite them to be examined at their surgery or at another designated emergency dental clinic. 'Universal access' services — intended for both registered and unregistered patients — exist in some areas, and these may use either telephone-based or 'walk-in' access arrangements, and see patients in either hospital, NHS community dental service (CDS) or general dental surgery settings.1

Box 1 Out-of-hours emergency dental service characteristics

Despite this diversity in the types of out-of-hours dental care available, and a wealth of studies of different models of GP-provided out-of-hours medical care,3,4,5,6,7,8,9,10,11,12,13,14,15,16 there are no comparative studies that assess the effectiveness of emergency dental care provided in different ways. This study was originally designed to compare the cost-effectiveness of 'universal-access' out-of-hours dental care arrangements with 'conventional' out-of-hours care (that is, arrangements where there are separate service arrangements for registered and unregistered patients). However, since the conventional service alternative is itself a combination of several (here, three) different services, the survey has generated effectiveness data relating to four different service types, which are accessed, operated, staffed, and resourced in a variety of ways.

The survey findings presented in this paper therefore aim to provide a comparison of the level of pain reduction and oral health gain experienced by patients attending the four different service types that existed in two health authorities in South Wales, UK (Bro Taf and Gwent), in early 2000. They included a hospital-based walk-in service (Bro Taf), CDS clinic-based walk-in services (Bro Taf), telephone-access GDP rotas for registered patients (Bro Taf), and telephone-access GDP rotas for both registered and unregistered patients (Gwent's universal access service). The main features of each service — access arrangements, intended clients, geographical coverage etc — are described in the text box. Their effectiveness in terms of patient satisfaction is presented in a companion paper.17

Method

Survey design

For thirteen weekends (defined as 6pm Friday to 8am Monday) in late 1999 and early 2000, all patients attending the four services were asked to complete a questionnaire at attendance (ie before seeing the dentist). The patient was asked to complete the questionnaire by the dentist, or by a receptionist at the walk-in services, and was also handed an information sheet about the survey (in accordance with the research ethics approvals given). All patients who consented to be followed up were sent a follow-up questionnaire four to seven days after their weekend emergency care episode. Telephone reminders were used to encourage non-responders, with new questionnaires sent out when necessary.

Questionnaire content

The attendance questionnaire asked patients to report their self-assessed oral health status, including an assessment of dental pain (and also included a panel for obtaining consent to send a follow-up questionnaire). Changes in self-assessed oral health status were measured using an adapted DS-QoL (Dental Health Status Quality of Life Questionnaire18,19) at both attendance and follow-up. Other more established measures of oral health status — such as the OHIP-14 and OIDP instruments20,21 — were judged to be either too long or otherwise inappropriate for measuring short-term changes in patients with acute dental problems. Although the validity and reliability of the OHIP-14 has been demonstrated amongst walk-in dental patients,22,23 many of the questions/items are clearly inappropriate for patients who are attending with severe dental pain or dental injuries (eg embarrassment smiling, feeling self-conscious or an altered sense of taste). The use of a 'willingness-to-pay' question (and therefore the potential to perform a cost-benefit analysis) was ruled out due to the ethical and practical difficulties it would raise.

As with its general health equivalent, the EQ-5D, the full DS-QoL instrument captures the main pain and functional/social dimensions of quality of life that acute dental problems are known to affect;24 pain/discomfort, eating, speaking, appearance and ability to perform oral hygiene activities. In this paper we present only the scores and score changes from the DS-QoL's visual analogue scale (VAS); shown in Figure 1a. Similarly we present the dental pain VAS scores and score changes for patients attending each service type (Fig. 1b). Lastly, the follow-up questionnaire contained 'transition questions':25 these simply asked the respondent how much they felt better/healthier now, compared with how they felt at some previous time point (eg when they saw the dentist; Fig. 1c).

Figure 1
figure 1

Main outcome questions used: (a) DS-QoL visual analogue scale; (b) dental pain visual analogue scale; and, (c) transition question.

Full size image

Results

Oral health and dental pain score changes

A total of 783 attending patients completed the initial questionnaire, and 423 the follow-up questionnaire. These represent 68.6% of patients seen on study weekends, and 49% of patients sent follow-up questionnaires respectively. The pattern of item non-response for the different outcomes reported here are more complicated and reveal lower response rates (26% to 47%; see footnotes to results tables). Note that response rates to items measured at follow-up (including transition questions and score changes) are reported in terms of the proportion of those who were sent follow-up questionnaires. This is justifiable because patients were not sent a follow-up questionnaire due to administrative issues (not associated with the outcomes of interest), such as incorrect contact details. Consent to receive a follow-up questionnaire was refused by only 5% (49/940) of those contacted, and most (90%) follow-up questionnaires were completed within four weeks of their episode of emergency care, median 15 days after. Table 1 shows the demographic and case-mix characteristics of patients who responded to the follow-up questionnaire in each service.

Table 1 Demographic and case-mix of respondents to follow-up questionnaire
Full size table

Table 2 and Figure 2 show the initial and follow-up dental health status scores, and the mean and median score changes, for each of the four service types. The Kruskal-Wallis tests and multiple comparison procedures reveal that patients attending the rotas for registered had an initial DS-QoL score (median 52) that was significantly higher than that of CDS Sunday service patients (median 43). At follow-up, patients attending the rotas for all had significantly higher DS-QoL scores (median 78) than the CDS Sunday service patients (median 64). The services with the greatest change in median DS-QoL score — the largest improvement in self-assessed oral health status between attendance and follow-up — are the rotas for all. A median DS-QoL score change of 26 amongst these patients was significantly different from the score changes of 8 or 11 reported by patients attending the other three service types.

Table 2 DS-QoL VAS scores at attendance and follow-up, and score changes, by service
Full size table
Figure 2
figure 2

DS-QoL VAS scores at attendance and follow-up, by service type

Full size image

Table 3 and Figure 3 show the pain scores of patients seen in the different services prior to being seen, at follow-up, and the mean and median changes between attendance and follow-up. At attendance, patients seeing dentists in the rotas for all were on average in more pain (median pain score 77) than those attending the other services, but only significantly more pain than those attending the dental hospital (median 69; α = 0.05 significance level). Patients seen at the rotas for all also reported the least pain at follow-up.

Table 3 Pain VAS scores at attendance and follow-up, and score changes, by service
Full size table
Figure 3
figure 3

Pain VAS scores at attendance and follow-up, by service

Full size image

As with the DS-QoL score changes, seeing dentists in the rotas for all created the largest pain reduction between attendance and follow-up (median score change of −72). This partly reflects that patients attending the rotas for all had both the highest levels of self-assessed pain when they sought care, and the lowest levels at follow-up. However, this is only significantly different from the pain score change for dental hospital patients (median −47).

Transition questions

The transition question scores, or the categorical responses derived from the scores (feeling better, the same or worse), were independent of service type (Kruskal Wallis or Chi-square tests respectively, at α = 0.05 significance level). This was despite a higher proportion of dental hospital patients (71%) feeling better after an hour than patients who had attended the other three service types (60% in all three).

As expected, for each of the service types the mean and median perceived improvement increases gradually over time (Table 4). However, most perceived improvement occurs between a day after emergency care and follow-up. This implies that the perceived benefits of emergency dental care are partial or that there is a time-lag in their effect. Interestingly, in all four service types a higher proportion of attenders reported feeling worse the day after seeing the dentist, than an hour after. It is only at follow-up (10 to 17 days after receiving emergency care) that the majority report feeling completely better (with most DS-QoL scores exceeding the 85 mm point). However, by this time most patients will have received further dental care, so any effect cannot be validly attributed to the episode of emergency care.

Table 4 Transitional question VAS scores an hour after seeing the dentist, the day after seeing the dentist, and at follow-up, by service.
Full size table

Discussion

There were no consistent differences in average effectiveness between the four different service types evaluated. When assessed as pain relief, oral health gain, or simply whether people said they felt better, service effectiveness for patients who saw a dentist seems to be equivalent across treatment settings (dental surgeries, CDS clinics, and dental hospital) and following different service access arrangements (walk-in, or telephone access). However, the relatively small sample sizes, particularly in relation to the two walk-in services, together with the low response rates across all services, prevent a more definitive conclusion. Patients attending the universal access rotas experienced both the largest average improvement in oral health status and the largest pain reduction between attendance and follow-up. However, since most (90%) follow-up questionnaires were completed up to four weeks after the initial episode of care, this is likely to reflect the effectiveness of other services attended after the weekend.

What was surprising was the high proportion of patients in all services reporting little or no improvement. An hour after receiving emergency dental care only two-thirds (60%–70%) said they felt better, and by the following day there was an increase in the proportion feeling worse than when they attended. By follow-up about 10% of all dental patients reported feeling no different from when they attended and about 5% reported feeling worse (although, again this may reflect a lack of in-hours dental service availability or patient apathy following the episode of emergency care). Changes between attendance and follow-up in the distribution of oral health VAS scores also implied a modest or partial relief from symptoms. More encouragingly however, by follow-up around four-fifths of emergency attenders had pain VAS scores of 10 or less (ie no pain, or almost no pain).

The apparent low effectiveness of these services for many patients may partly reflect the choice of outcome measures: in particular the implicit focus on relief from symptoms gives no value to the psychological benefits of advice and reassurance. Thus, many of those who reported no or little relief from symptoms may have felt better about their dental problem, or better able to cope with it, but such benefits would not be captured by the measures analysed here. Many patients who seek dental care at weekends seek reassurance about the meaning of their symptoms, as much as treatment and relief from the symptoms themselves.24

The main limitations of this study are the small range of service types evaluated, that they were in only two areas, and that within these services a high response rate was not achieved. However, the two areas themselves were comparable, geographically and demographically, and they also exhibited arrangements that were typical of many other areas in the UK.26 The response rates, while low, may not lead to significant bias:27 a comparison of the case-mix and age structure of all attenders, initial questionnaire responders and follow-up questionnaire responders, revealed no major differences; and a small survey of 77 non-responders (in Gwent only) also revealed no significant differences between responders and non-responders.26

Case-mix differences between services

The only significant differences between services in attending patient characteristics were that the dental hospital saw fewer children, more aged 45–54, more with lost restorations and fewer with abscesses or acute pulpitis (Table 1). Amongst the three other service types, both the dentist-recorded diagnosis and the patient-reported reason for attendance were independent of service type. Nevertheless, the rotas for all appear to see more severe problems (>60% abscesses or acute pulpitis) than the other three services (31%, 32% and 45%) and this may partly explain their greater reported effectiveness (pain and oral health score changes). Larger sample sizes might have allowed a multivariate analysis and revealed more significant and informative differences in case-mix and patient characteristics between the services.

Self-assessed changes in oral health status and pain

Although measuring change in health status (over time) is thought to be most valid if based upon the difference between separate assessments made at two or more points in time,28 for measuring relief from acute dental problems this presented a number of difficulties. Firstly, there are problems of defining pre-care or baseline levels of oral health status, especially because of temporal variability in the severity of acute dental problems.29 For example, asking how much pain they felt when attending the emergency dentist may produce a lower assessment of pain than if they were asked to assess their worst or average level of pain during the previous 24 hours.

Secondly, what is the appropriate follow-up period? A short follow-up period might not be sufficient for the treatment to have an effect (eg antibiotics for swelling). A longer follow-up period — say five days or more — means that the patient is likely to have had more than one episode of care, and any measured 'effect' cannot then be attributed to the initial out-of-hours care episode. There are also practical difficulties in contacting patients very soon after an episode of emergency care. In retrospect, patient 'symptom diaries' would have been a method worth piloting.

Thirdly, a significant proportion of the pain reductions or oral health gains measured might be due to 'regression to the mean': that is, simply due to natural, random variation amongst a group of patients whose baseline health state is near the extreme of a scale. Another statistical difficulty with these scores is that most are non-normally distributed, so only non-parametric statistical tests could be used to detect differences between service types.

Validity and reliability of the effectiveness measures

The reliability of self-assessed health outcome measures for acute problems is difficult to assure. Inter-rater reliability is clearly irrelevant and, since acute health problems change rapidly — typically from hour to hour — ways of assessing test-retest reliability are difficult to devise. However, the test-retest reliability of a similar VAS-based transition question was assessed and found to be good in a study of out-of-hours medical care (although after the acute phase of the presenting problem),7 and the value of transition judgements is acknowledged by a recent review in the dental field.25 Ultimately the validity of transition questions hinges on how accurately people can recall their level of dental pain or health status at some point in the past, and this may be distorted by a number of factors.28

Lastly there is evidence about the validity of using VAS for measuring dental pain30,31 and their suitability for measuring change in acute pain states has been demonstrated in other contexts,32 but the use of visual analogue scales remains controversial. From this study's data, an assurance of reasonable validity is that the pain VAS scores were highly correlated with the Present Pain Intensity score (ordinal scale 0 – 5) of the McGill Pain Questionnaire (Spearman's ρ = 0.79; p < 0.001, for n = 567 subjects), which is a widely used pain score.33

Conclusion

This study provides a useful first step in measuring the effectiveness of services for acute dental problems. Depending on the outcome measure chosen, evidence of the effectiveness of different types of emergency dental service is mixed. Overall, for those people who got to see a dentist out-of-hours, the chances of receiving effective care was similar across services, regardless of the clinic setting or the type of dentist seen. A larger survey amongst a wider range of services might have generated effectiveness results with more discernible patterns, and higher response rates would reduce the current potential for bias in these findings.

Of potentially greater concern are the proportions of patients in all four services who reported little or no relief from symptoms: only two-thirds could say that they felt better an hour after seeing the emergency dentist, and the day after seeing the dentist a tenth said they felt worse. Further analysis of this survey data will explore which types of patient or dental problems are associated with these poor outcomes, and whether the outcomes can be attributed to particular aspects of their service experience.

These findings need to be viewed in the light of two other important considerations. Firstly, while the effectiveness of different models of service delivery for those who get to see a dentist out-of-hours is of undoubted interest, it should ultimately be weighed against the 'effectiveness' for those people (in the same geographical area) who needed but failed to access dental care out-of-hours. Different emergency dental service arrangements, by being more or less publicised, and more or less accessible when they are known to exist, will probably drive different proportions of patients to seek care from the 'wrong services' (eg A & E departments, or GP out-of-hours co-operatives) and lead others to simply wait and suffer until Monday morning. With dental services as fragmented and poorly understood as at present,34 the magnitude of these other patient flows, and the sub-optimal care received, should not be underestimated. Second, for every emergency care-seeking pathway, an ideal measure of service effectiveness should capture both relief from symptoms and the psychological benefits of advice and reassurance.24 Despite the methodological difficulties involved,35,36 only after acknowledging these issues can emergency dental services be designed with a comprehensive understanding of all the costs and benefits involved.