Background

It is being increasingly recognised that clinical care should be based on the best available evidence.1,2 Unfortunately it has been estimated that only 15% of all clinical practice is based on sound research.3 Part of this problem is a well recognised gap between the publication of research evidence and the adoption of this evidence in clinical practice.4 The delay in uptake of a proven technology can be over a decade. The reasons for this delay are multi-factorial and suggested reasons include inertia, information overload, and difficulty in interpreting contradictory messages.5 McGlone et al. identified a number of barriers to change in dentistry: the profession's perception of patients (structural), patient's attitudes to dental health and the cost of dental care, fear of medico-legal action, the treatment-funding system and time lost to practice when attending courses.5

This problem in identifying and adopting evidence-based practice is as much a concern for health service planners and policy makers as it is for clinicians. Therefore together with the current emphasis on recognising the evidence, there is a need to promote best practice by changing clinician's behaviour in line with the evidence identified.5

One method of presenting evidence in an accessible format is clinical guidelines. Clinical guidelines are defined as 'systematically developed statements which assist in decision making about appropriate healthcare for specific clinical conditions'.6 There are reports in the medical literature that clinical guidelines can improve the quality of care.7,8 However, passive dissemination of published guidelines alone is rarely effective in changing the clinical behaviour of practitioners, as many factors influence health professionals' behaviour such as, organisational structure, peer group pressure and individual variation.9,10 There is therefore a need to find effective implementation strategies to optimise the integration of research findings into current practice. Methods of disseminating and implementing research evidence which have been assessed in medical practice include continuing medical education, opinion leaders, audit and feedback, educational outreach, reminders and multi-faceted interventions.8,9,10,11,12 However, it has been recognised that such strategies are not effective under all circumstances7,8,9,10,11,12,13 and few studies have investigated their effectiveness in dentistry.5,14,15,16

Trials in dentistry have tended to look at referral practices. One trial evaluated the effectiveness of orthodontic referral guidelines and found that these did not influence the patient referral behaviour of general practitioners.14 Another trial assessed three differing referral strategies from primary to secondary care for the treatment of impacted third molars and found a clinical algorithm (flowchart), performed better than the then current practice, with a neural network based computer programme (third molar decision support system) performing least well.15

The Scottish Intercollegiate Guidelines Network (SIGN) developed and published a clinical guideline relating to the management of impacted and unerupted third molar teeth in April 2000 (SIGN 42).17 This was identified as an appropriate subject for guideline development for a number of reasons ie variation in practice,15 the procedure is associated with considerable morbidity,18,19,20 the surgical removal of third molar teeth is a common surgical procedure and is therefore associated with significant expenditure.21,22

The aim of this study was to evaluate the effectiveness of different implementation strategies for evidence-based clinical guidelines using SIGN 42 as a model.17 In addition, the study conducted an economic evaluation of the dissemination and implementation strategies and examined behavioural factors mediating the effect of the interventions.

Methods

Ethical approval was obtained from The Multicentre Research Ethics Committee for Scotland (MREC) and the relevant local research ethics committees.

Study design

The study was conducted in dental practices across Scotland; it was a pragmatic, cluster randomised controlled trial employing a 2×2 factorial design.

Sample size calculation

It was estimated that 4–5 patients per practice would have extractions during each data collection period based on data from the Scottish Dental Practice Board.23 A sample size of 60 practices collecting information on 240 patients was required to detect a 20% reduction in inappropriate extractions from 60% to 40%24 assuming 80% power and a 5% significant level. Based on the published trials in medical practice, an intra-class correlation coefficient (ICC) of 0.1 was incorporated into the power calculation to account for the unit of randomisation being dental practices rather than individual practitioners.25

Five hundred and sixty five general dental practices were randomly selected from the Scottish Dental Practice Board list and invited to participate in the trial by mail. In addition 41 practices who had previously participated in a Dental Health Services Research Units (DHSRU) trial were approached.26 Sixty-three practices agreed to participate with written consent. Subsequently, 12 practices withdrew from the trial before randomisation. Therefore 51 of the original 63 volunteer practices were randomised into 4 groups by a statistician independent of the research team through computer generation of a random number sequence. Figure 1 presents the study profile.

Figure 1
figure 1

Trial profile

Full size image

The intervention groups were as follows: Group 1 received a copy of SIGN 42 Guideline and had an opportunity to attend a postgraduate education course (PGEC). In addition to this, Group 2 participated in Audit and Feedback (A and F). Group 3 received a specifically developed computer aided learning (CAL) package. Group 4 participated in A&F and received the CAL package.

Dentists completed a questionnaire based on social cognition models before and after the interventions, the results of which have been reported separately.27 An economic evaluation of the relative cost effectiveness of each of the interventions was also performed.

Description of interventions

Mailing of guidelines and postgraduate courses

In April 2000, all dental practitioners in Scotland including those who were participating in the study received a copy of the guideline direct from SIGN. This mailing consisted of the guideline document detailing the recommendations, together with the references to the evidence on which it was based. This also included a double sided laminated sheet known as the 'Quick Reference Guide' which summarises the findings in an accessible manner. All general dental practitioners (GDPs) recruited into the trial were invited to attend a postgraduate course. The courses ran on identical lines to those run by NHS Education for Scotland (formerly Scottish Council for Postgraduate Medical and Dental Education (SCPMDE)) and a practitioner's expenditure was reimbursed, as it would have been for any other course. It was planned to hold the three courses at regional centres. Attendance was not obligatory, since this was a pragmatic trial and due to lack of interest from the contacted dentists, one of the courses was cancelled.

Computer aided learning with decision support (CAL)

The CAL intervention strategy consisted of a laptop computer-based support tool, with the potential to assist dental practitioners (groups 3 and 4) in deciding on the appropriate treatment of third molars. The software was based solely on the SIGN guideline, the computer delivered this advice in a multimedia format. The package was developed specifically for the trial.

Audit and feedback (A and F)

The participants in this arm of the trial (groups 2 and 4) were divided into 11 groups according to the proximity of their practices to each other. The exact nature of the A and F was decided within each audit group and supported by the researcher with help and advice from the SCPMDE Dental Audit Tutor.

In each A and F group, one member was selected as a facilitator based on their previous experience and knowledge in carrying out an audit. The audit projects were conducted independently of each other on different aspects of clinical practice relating to third molar teeth.

Outcome measurement

The outcome measure was the proportion of patients whose treatment complied with the guideline. This was assessed by two independent researchers and any disagreements were resolved by discussion.

Dental practitioners were asked to identify all 16—24-year-old patients who attended their dental surgery over two, four-month periods in 1999 pre- and 2000 post- intervention. A researcher blind to the intervention groups retrieved these records and transcribed the reason for attendance and treatment on a previously piloted form. Pre-intervention data were collected from 49 practices and post-intervention data from 47 practices.

Statistical analysis

The trial had a factorial design, which provided an opportunity to study and test for interactions between the interventions if they existed.

The level of compliance with the guideline between intervention groups was tested by means of a cluster level analysis using a weighted t-test.28 To control for heterogeneity between patients (case-mix) and practices, multilevel regression analysis was also used. All analyses were on an intention to treat basis.

Results

Practice recruitment

Of the 565 dentists invited to participate, at least one dentist from 63 practices volunteered to participate. Twelve of these practices withdrew before pre-intervention data gathering. The 51 remaining dental practices were randomised to four groups. During the period between pre-intervention and post-intervention data collection, four practices withdrew from the trial, leaving 47 available for post-intervention data collection. The reasons given for withdrawal were too busy, not interested, moving practice, refurbishment of practice and a change of mind. There were no significant differences between the dentists who withdrew from study and those who continued, in terms of their age (t = -1.34, p=0.2); gender (χ2 = 0.15, df=1, p= 0.7); postgraduate qualifications (χ2 = 2.24, df=1, P= 0.1) and their intervention group (χ2 = 4.24, df= 1, P = 0.2).

Of the 51 pre-intervention practices, 47 provided data. Of the 47 post-intervention practices, 46 had data relevant third molar teeth. The numbers of practices or clusters per group are shown in Figure 1. The characteristics of the participating dentists are reported in Table 1.

Table 1 Participants characteristics
Full size table

Of the dentists recruited, 23 (45%) attended one of the two post-graduate courses.

No statistically significant differences between the practices that withdrew from the study and those who continued, in terms of dentists' age, gender, postgraduate qualifications or intervention group were identified.

Pre-intervention, data were collected for 3,342 (M=1,885, F=1,457) patients with a mean age of 21.7 years (SD 2.2, range 14–25.5 yrs) compared with 1,935 (M= 880, F=1,055) patients at the post-intervention stage with a mean age of 21.8 years (SD 2.1, range 16.6–25 yrs) (Table 3). The proportion of patients with a problem with their third molar teeth was 7% before, compared with 22% after intervention (P=0.0001). Female patients were significantly more likely to present with third molar problems than males in both the pre and post-intervention phases (P=0.0001) (Table 2).

Table 3 Weighted mean percentage compliance with guidelines for all groups
Full size table
Table 2 The gender distribution of patients with and without third molar problems before and after intervention
Full size table

Outcome assessment

Overall compliance with the guideline (evidence-based practice) at pre-intervention stage was assessed to be 74% and this increased to 78% at post-intervention.

Comparison of pre-intervention data between the groups indicates that there was no apparent imbalance between groups at baseline (Table 3). The level of adherence to the guideline prior to any discussion, as assessed by the two independent examiners was 74% and 68% pre-intervention and 78% and 75% post-intervention, respectively.

The weighted t-test for A and F versus no A and F post-intervention was not statistically significant (p=0.62) neither was that between CAL versus no CAL (p=0.76).

From the multilevel analysis the odds ratio of compliance with guidelines for dentists who experienced A and F versus those who did not was 1.28 (95% CI 0.62 – 2.63) and this compared with an odds ratio of 0.84 (95% CI 0.88 – 1.74) for the CAL dentists versus no CAL. Neither difference was statistically significant (Table 4).

Table 4 Percentage compliance with guidelines for dentists who experienced A and F versus those who did not and similarly for CAL
Full size table

To account for case-mix 'effect modifiers' in the multilevel model, pericoronitis, caries and pulpal pathology were included. All patients with pulpal pathology were treated in accordance with the guideline. Only one case presenting with caries was not treated in accordance with the guideline. For patients presenting with pericoronitis there was an increase in compliance with guidelines but the effect was not different across groups.

The rate of third molar extractions decreased for 16—24- year-olds after the introduction of the SIGN guideline. This reduction was statistically significant between the pre- and post-intervention phase of this study (37% to 27%, p=0.02) and was consistent across the groups.

Concurrent with the study data, data from the NHS management information and dental accounting system (MIDAS) shows a concomitant reduction in both surgical and non-surgical wisdom tooth extraction rates in Scottish general dental practices during the experimental period (Figures 2 and 3).29

Figure 2
figure 2

NHS Management information and dental accounting system (MIDAS) data for third molar extraction (not requiring division of root or crown) between April 1992 and Aug 2000

Full size image
Figure 3
figure 3

NHS Management information and dental accounting system (MIDAS) data for third molar extraction (not division of root or crown) between April 1992 and Aug 2000

Full size image

Cost effectiveness

With no evidence of an effect, the planned cost effectiveness analysis became a cost-minimisation calculation. After controlling for scale effects, the driver behind cost differences between the groups was CAL. Sensitivity analysis ignoring PC purchase cost and delivery costs attenuated these differences. There were substantial differences in the costs of the interventions with the CAL arm of the trial costing £482.34 per dentist and the A and F £216.51 per dentist. However, a sensitivity analysis ignoring PC purchase and delivery costs attenuated these differences.

Discussion

The design of the study was a clustered randomised controlled trial with an intention to treat analysis. The design provided an opportunity to compare the relative effectiveness of different interventions and balance any modifying effects associated with the context, study population or outcomes.30

The main finding of this trial is that neither A and F nor CAL was more effective than mailing and attendance at a postgraduate course, in increasing the general dental practitioners' compliance with the SIGN guideline for management of impacted and unerupted third molar teeth. This finding is not unique in the primary care setting. A trial published by Eccles and colleagues suggested that A and F does not change behaviour for primary-care radiology referrals.31

A previous evaluation of CAL to develop clinical decision-making skills found no evidence of an effect on dentists' treatment decision-making behaviour.16

An unexpected finding was that there was good adherence with the guideline recommendations at baseline (74%). This level of compliance is higher than would have been predicted from other research evaluating compliance in the UK with the USA's National Institutes of Health (NIH) criteria.25 It may be that this and other publications have influenced knowledge levels and practice since 1996. There has, over recent years, been discussion of the appropriateness of the extraction of impacted third molars in the national and dental media. For example, an 'Effectiveness Matters' publication from the Centre for Reviews and Dissemination at the University of York was published in 1998.21 The findings of the review were based on other reviews and research findings that have been widely reported in the dental press32,33,34,35 and this may have influenced Scottish dentists' behaviour (Figures 2 and 3). If publications relating to the extraction of third molars has influenced care, it is one of the few examples of relatively passive dissemination and implementation being effective in altering clinical practice.

The lack of a difference between the interventions may be a true lack of effect or it may be a reflection of the high level of compliance at baseline which may have produced a ceiling effect, where no greater improvement within the group was possible.

Another significant confounder may be that the volunteer dental practitioners who consented to participate in our trial may not be representative of the Scottish GDP population, possibly having a particular interest in research or the subject of the research. They may be more likely to participate than others and may manage their patients more effectively, rendering the sample less representative of the target population and possibly introducing bias. Current recommendations for the conduct of randomised trials require each participant be provided with enough information to allow for well-informed consent.36 Accordingly, there can be a systematic tendency to recruit a biased group.

A second explanation for the high compliance could be the 'Hawthorne Effect', a social placebo response. Practitioners may have acted differently because they were participating in the trial. Forty-two per cent fewer patients were seen post intervention compared with pre-intervention, with a greater proportion having a third molar problem. The authors do not have an explanation for this finding. An attempt was made to prevent the 'Hawthorne Effect' by keeping the process of data collection as unobtrusive as possible.25 Other explanations for this finding could be that the dentists participating in the trial were more confident in their management of the condition as a result of the guideline, or that there was improvement in case-note keeping as a consequence of their participation in trial.

Unfortunately, a number of practices withdrew from the study, particularly during the first phase. The reasons given for withdrawal from the study indicate that they were principally for reasons other than the demands of the project. The loss of practices seems not to have had a significant effect on the study's power to detect any difference between the interventions, if one was present. The power calculation used to estimate sample size in this study drew its estimates of probable ICC from comparable studies carried out in medical primary care. In practice, the ICC in the study was actually 0.15. This suggested that the study had reasonably high power to detect a 20% difference between the interventions if it was present. This can be used to inform future sample size calculations in studies of this type.

The results of the study examining the mediators of behavioural change demonstrated that dentists' cognition were demonstrably affected by the interventions with those who received the A and F intervention, increasing their third molar-related knowledge significantly more than dentists who did not receive A and F.27 However, the cognitions that were changed did not relate to extraction behaviour.

The dissemination and implementation of guidelines are extremely costly. We compared these very large costs with any benefits that accrued as a result of the interventions. As we were unable to detect any benefit from the interventions, the most cost-effective choice is the least-cost option, in this case mailing, and the opportunity to attend a postgraduate course. The CAL intervention was found to be the most expensive.

As recognised by McGlone et al. our knowledge of altering professional practice in dentistry and particularly as it relates to the effective use of clinical guidelines is somewhat limited. There is an urgent need for further research in the area of altering professional practice in dentistry, which will ensure efficient and effective use of limited resources, with the potential to improve the delivery of care by promoting best practice.5

In March 2000 just prior to the publication of SIGN 42, the National Institute for Clinical Excellence (NICE) in England and Wales published 'Guidance to the NHS on the Removal of Wisdom Teeth'.37 Although there are differences in the development process and presentation of these two guidelines, the SIGN document being more complex, as would be hoped for with recommendations based on best evidence, their conclusions are broadly similar. Therefore there is no reason to believe that if this study had been based on the NICE guidance the results would have been any different.

Conclusion

In an environment where pre-intervention compliance was unexpectedly high, there was no evidence that CAL and A and F increased general dental practitioners' compliance with the SIGN guideline for the management of impacted and unerupted third molar teeth compared with mailing and the opportunity to attend a postgraduate course alone. Related research suggests that these interventions may act as reinforcement of the guideline messages.27