Main

In this paper we intend to report a study that evaluated the effectiveness of referral guidelines for orthodontic treatment.

One area of concern to both general dental practitioners and secondary care orthodontists is the length of waiting lists for orthodontic treatment.1 For example, a recent survey carried out by the consultant orthodontist group revealed that the average wait for a new patient consultation appointment was 4.6 months with a range of 0 to more than 24 months.2 Paradoxically, it has been suggested that up to 45% of orthodontic referrals could be classified as inappropriate and were contributing to the long waiting lists.3

One solution to this problem is developing referral guidelines to provide information for referring dentists about the type of patient that should be referred for treatment.4 However, this step is not easy, because if the referral guideline is to be effective it should be:5

  • Evidence based

  • Developed by those who will be using it

  • Disseminated effectively

  • Evaluated effectively

How can guidelines be drawn up?

It has been suggested that the greatest difficulty in drawing up effective guidelines is ensuring that they are both scientifically valid and relevant to the practitioners who will be using them. This may be approached by using national guidelines which can then be modified by local committees to suit local conditions.6 As a result, the local practitioners may feel that they own the guidelines without having to do extensive background reading.

Dissemination

Dissemination strategies aim at influencing a targeted clinician's awareness, attitudes, knowledge and understanding of a set of guidelines.5 These strategies include publication in professional journals, postal distribution to the relevant groups, incorporation within continuing medical education, educational initiatives that focus on the guidelines and outreach visits.

Methods of dissemination have been subjected to several systematic reviews. The results of these reviews have suggested that methods of dissemination are integral to the effectiveness of guidelines.

When the use of printed educational materials was considered, the reviewers identified 11 suitable studies that involved the evaluation of printed educational materials. These were aimed at improving clinical practice, for example by publication in journals or through targeted mailings. They concluded that the effects of printed educational materials with no additional active intervention appeared small, at best, and of uncertain clinical significance. The distribution of printed material is not costly but the cost-effectiveness of this approach could not be assessed from these studies.7

Another review was concerned with the use of outreach visits as part of guideline utilisation.8 Eighteen trials were included in this review. It was concluded that educational outreach appeared to be a promising approach to modifying professional behaviour, especially prescribing. Importantly, only 5 of the 18 studies concealed allocation of practices or practitioners and after randomisation baseline differences were evident in 6 of the 18 studies. The authors also emphasised that the use of outreach methods was expensive and the cost-effectiveness of such a procedure should be determined.

Finally, the use of local opinion leaders has been studied.9 These are defined as health professionals nominated by their colleagues as being 'educationally influential'. These people have been used to transmit appropriate behaviour to other health care professionals. The authors identified six trials of interventions aimed at improving practice where an opinion leader was used as part of the strategy. The target professionals in these investigations were community-based physicians and hospital based nurses. The reviewers reported that the data set was very sparse and further research was required.

In summary, it appears that the use of printed educational material alone may not be an effective method of changing the behaviour of health professionals. In addition to issuing printed material, use should be made of outreach in the form of visits to individual practices, or holding educational seminars for the target group.

Implementation

Implementation strategies are intended to encourage clinicians to change their own clinical practice in line with guidelines, and they may be divided into those that operate during or outside the doctor-patient consultation. Those within the consultation process include general reminders of the guidelines, feedback specific to individual patients, changes in medical records, and patient specific reminders at the time of consultation. The simplest strategy is to provide clinicians with easily accessible copies of the guidelines, change aspects of medical records, provide a copy of the guidelines in the patient consultation notes, provide flow-sheets and standardised referral letters or embed the guidelines in computer packages.5

The other group of strategies are those operating outside the consultation. These include aggregated feedback on compliance with guidelines and introducing financial incentives.

While there has been a great deal of research into the effectiveness of guidelines for medical care, there has been little published research for dentistry. Nevertheless, in the absence of research clinical guidelines are beginning to be introduced in dental care with limited consideration of methods of development or evaluation. The aims of this investigation were:

  • To develop referral guidelines for orthodontic treatment that may be used practically in the primary dental care sector.

  • To evaluate the effectiveness of the guidelines on the referral process, using randomised controlled trial methodology, thereby testing the null hypothesis that the referral guidelines would have no effect on the proportion of inappropriate referrals.

Methods

Protocol

The site of the study. The investigation was based in Manchester and Stockport. These areas contain a population of 736,000 with a broad range of socio-economic status and dentist:population ratios. Primary dental care is provided by 283 general dental practitioners and 25 community dentists. Secondary orthodontic care is provided by 1 community orthodontist, 5 specialist orthodontic practitioners and 6 consultant orthodontists.

All general dental practitioners in Manchester and Stockport were eligible for the trial. However dentists in practices where another dentist also worked in another practice were excluded.

Outcome measures. The primary outcome measure that we used was the appropriateness of each patient referral for orthodontic treatment. For the purpose of this investigation, a referral was defined as being appropriate if the patient was accepted for treatment or returned to the GDP with a treatment plan. This included the prescription of teeth to be extracted because of caries.

Secondary outcome measures were the reason that the patient was not accepted for treatment and the patient's perceptions of the reason for their referral.

Sample size calculation. The sample size for this investigation was based on data derived from our earlier investigation on the appropriateness of orthodontic referrals.3 We based this on a reduction of inappropriate referral rate in a group of 10–14 year old children from 45% to 30% of all referrals. This difference was selected because we felt that it was clinically significant. A sample size of 176 referrals per group with a significance of 0.05 has a power of 0.8. We then used data already obtained on orthodontic referrals by dentists in the Manchester and Stockport area, and estimated that we needed to randomise around 120 dentists into a guidelines group and a control group to achieve our target number of referrals.

The data were analysed with the chi-squared statistic to evaluate any differences between the groups of dentists who received, or did not, receive the guidelines.

Ethical approval. The study was approved by the relevant local ethical committees.

The development of the referral guidelines

Evidence base. The first stage in the development of the guidelines involved a search of the literature to derive an evidence base. This was carried out primarily by computerised database searches. For orthodontic treatment, there was limited evidence available from randomised controlled trials. Therefore we developed the evidence base of our guidelines from two main sources. These were:

  • The only longitudinal evaluation of the effects of malocclusion.10

  • The results of our earlier investigation into the appropriateness of orthodontic referrals in the Manchester and Stockport Districts.3 This study identified that 46% of referrals for orthodontic treatment were inappropriate. The main reason for this type of referral was poor oral condition.

Involvement of the users of the guidelines. It was important that the guidelines were developed by those who were going to use them. We achieved this by holding a series of meetings to discuss and finally decide the content of the guidelines. The attendees at these meetings were:

  • Six general dental practitioners (nominated by the local dental committees)

  • Three specialist orthodontic practitioners, and

  • Six consultant orthodontists.

The discussion at the meetings was led by members of the investigating team. The meetings had the following structure:

Meeting 1. Prior to this meeting each participant was sent copies of relevant papers concerning the evidence base and methods of developing guidelines. The aim of the first meeting was to introduce the participants to the concept of the development of clinical guidelines and to hold a discussion on the content of the guidelines that we were developing. At the end of the meeting, we drew up a list of the potential guideline contents.

Provisional guidelines and accompanying information for the dentists was then prepared and sent out to the discussion group members.

Meeting 2. This was held 3 weeks later. At this meeting there was further discussion on the guideline content and areas of controversy were debated at length. At the end of this meeting, the final content of the guidelines was agreed upon.

The final guidelines were then sent out for comment to the members of the discussion group. This enabled them to make final comments to the investigating team. Replies were received from two orthodontic consultants, two orthodontic specialists and three GDPs. We then took these comments into account and produced a final version of the guidelines.

The guidelines

The information that was presented in the guidelines was concerned with the following factors:

  • Patient awareness of an orthodontic problem and motivation for treatment

  • Oral hygiene and caries status

  • Need for orthodontic treatment

We also decided that we should produce patient educational material as part of a guideline package. This contained information on the need for orthodontic treatment, requirements of the patient on wearing appliances, care of appliances and commitment to treatment. This was to be given to the patients by the dentists during the consultation prior to making the referral decision.

The presentation of the guidelines and patient information sheet was important to increase the chance that the dentists/patients at least consider them. The appearance of the final version to be sent to the surgeries was planned and manufactured with the assistance of Manchester University Pharmacy Distance Learning Department. This resulted in easy to read, professionally produced material.

Dissemination strategy. We decided that the best pragmatic dissemination strategy was to use two methods:

  • We posted the guidelines to the dentists along with information on the need for guidelines and the method of development that we used. Importantly, we emphasised the fact that the guidelines have been developed in conjunction with the local specialist providers and representative general dental practitioners.

  • Each dentist was invited by their local consultant to attend a seminar at a hotel near to the practice, where there would be the opportunity to discuss the guidelines with members of the development group and local consultants and specialists.

Implementation. It was important that we used several implementation procedures.11 We adopted the following implementation strategies that were practical for the general dental services.

  • The guidelines were prepared in such a way that they could be posted in the practitioner's surgeries.11,12

  • Patient specific feedback was provided by the consultant or the specialist practitioner who informed each dentist of the outcome of the consultation.

  • A questionnaire was sent to each dentist to ascertain their opinion of the guidelines — and to ensure that they had been read.

  • Re-issuing the guidelines 6 months after the first release accompanied by a survey form that collected data on the dentists' perceptions of the usefulness of the guidelines.

Assignment

Most of the dentists in the study area worked single handed, however there were several multi-surgery practices which needed to be taken into account. The unit of randomisation was the dental practice so that all dentists in each multi-dentist practice were in the same experimental group. We excluded practices where dentists also worked in other practices to avoid contamination. A total of 123 dentists were allocated to the following groups by randomly allocating practices using random numbers generated by computer:

  • 62 dentists received orthodontic guidelines,

  • 61 dentists received no guidelines.

The secondary care orthodontists assessed the referred patients unaware of which dentists had been issued with the guidelines.

Collection of referral data

In order to obtain baseline data on the patients that were referred by the dentists we began collecting data on all patients referred for orthodontic treatment from April 1996. Whenever a patient attended for a new patient consultation the following data were collected:

  • Patient and referring dentist details

  • The reason for referral

  • Outcome of consultation

  • Reason for referral being inappropriate

  • Patient perceptions of treatment. This was collected by a questionnaire that was completed by the patients, enquiring about their perceptions of (i) the need for treatment and (ii) the burden of orthodontic treatment.

The guidelines were issued in January 1997 and data were collected until January 1998.

Masking

The general dental practitioners and their referred patients were not aware that they were involved in a study.

When collected, the data were entered into a database. During this process none of the investigating team was aware of the group to which the dentists had been allocated. The identifying code was broken when the data were analysed.

Results

The development of the guidelines

We developed our guidelines by collecting the evidence base and then modifying this for local use by gathering the opinions of the primary and secondary care providers who were going to be the users. By doing this we gave the practitioners a degree of ownership of the guidelines.

This was a surprisingly protracted process that involved a great deal of work by the study team and the dentists and orthodontists who took part. This required two meetings lasting 3 hours with detailed discussion on the content and dissemination of the guidelines. Importantly, there were differences between the opinions of the orthodontists in certain 'key' areas. Some of the group were keen to include information that would advise dentists on the correct age to refer patients, thus reducing the referral of children too early for treatment. Others were happy to accept this group of children and keep them under review. We could not achieve consensus on this issue and information on this type of referral was not included. Similarly, we could not achieve agreement on the inclusion of the Index of Orthodontic Treatment Need. Most of the orthodontists felt that the referral of patients with low need was not a problem. A greater problem was those children who were referred but had a low interest in treatment or had a poor oral condition. Importantly, this was reinforced by our earlier study into referrals by the same group of dentists.3

Finally, the last insurmountable problem was the use of a referral form. Initially, we felt that this was an important step, as it would ensure that the dentists were reminded of the guidelines every time they completed the referral form. However, there were two reasons why this was not appropriate to this study. Firstly there was the possibility of introducing bias into the study, as the secondary providers would know which referrals were from dentists in the guidelines group. Secondly most of the referrals that were made to the specialist orthodontists were by telephone. Consequently, it was felt that we could not include a form.

As a result, we formulated a set of guidelines that were based on national guidelines and evidence from the literature. These were then to reflect local practice that was supported by the previous investigations into the referral of patients in the study area.

Dissemination

As part of our dissemination strategy we intended to hold a series of seminars for the dentists who were issued with the guidelines. Unfortunately, only 16 out of 62 invited dentists were prepared to attend. When we contacted the dentists who did not respond to the invitation many felt that the guidelines were understandable and they did not need to come to the seminars. As a result, the seminars were cancelled.

Dentists' perception of the guidelines

While the dentists were not prepared to come to the seminars they did respond to follow up questionnaires and the second mailing of guidelines. The valid response rate to the questionnaires after three letters and two follow-up phone calls was 62%. The results are shown in Table 1a.

Table 1a Table 1a
Full size table

A second questionnaire was sent with another set of the guidelines material 6 months after the initial posting. The response rate was 66% and again the guidelines were favourably received. The results can be seen in Table 1b.

Table 1b Table 1b
Full size table

In general it appears that the guidelines and the accompanying material was well received by the dentists.

Analysis

Did the guidelines have an effect on appropriateness of referral? Data was collected on 1,179 patients referred by the dentists who were in the guidelines and control groups.

The outcome of the consultation is shown in Table 2. Data analysis revealed that there was no difference in the proportion of appropriate referrals between the dentists that received the guidelines and those that did not.

Table 2 Table 2
Full size table

Possible differences between patients referred by guidelines and control dentists in the factors that influence acceptance for treatment, such as oral hygiene status, need for treatment etc, were evaluated with the chi-squared test. The only difference that was detected was that more of the patients referred by the guidelines dentists did not attend for their consultation appointment. (P = 0.054, χ2(1df) = 3.73)

Patient perceptions and knowledge of orthodontic treatment. Descriptive data from the patient perceptions questionnaire is presented in Table 3. The chi-squared test revealed that for most of the perceptions we evaluated there were no differences in the knowledge of the patients who were referred by the two groups of dentists. However more patients referred by dentists in the guideline group did know why they had been referred (P < 0.05, χ2 (1df) = 7.66) and they knew that they were referred because they had crooked teeth (P < 0.05, χ2 (1df) = 6.85).

Table 3 Table 3
Full size table

Discussion

The effect on orthodontic referrals

It was disappointing to find that the referral guidelines did not have an effect on the appropriateness of referral. The only difference detected was that more of the guideline dentists' patients failed to attend their consultation appointments. This may have occurred because these patients could have been influenced by the patient information leaflet and decided that they did not want treatment. Nevertheless, this was only one small effect of the guidelines and we can, generally, conclude that the guidelines did not have an influence on the dentists. This is not a unique finding in research that has investigated the effectiveness of guidelines in the primary care setting. For example, an investigation into the management of dyspepsia revealed that the guidelines had no effect on the referral rate of patients for secondary care.13 They concluded that this might have occurred because of controversy surrounding the management of dyspepsia. In many ways this study was similar to our investigation. There is controversy in general dental practice about the type of patient that should be referred for orthodontic treatment and this is evident in the high number of inappropriate referrals that are made.2,3 Certainly, the development of the guidelines and methods of dissemination were the same and it is, perhaps, no surprise that the results are also similar. Importantly both these studies were set in the primary care sector and the practitioners were not aware that they were participating in a study.

This problem has been highlighted in a review of the effectiveness of continuing medical education.14 The authors suggested that neither the complexity and method of the educational intervention nor the optimal circumstances provided in teaching centres or wards match the learning and practice circumstances of most physicians in the primary care sector. In addition, many studies experience a ceiling effect in studying volunteer physicians who are often performing at or near optimum levels, and know that they are taking part in a study.15

For example, an investigation into the management of infertility showed a positive effect of guidelines, but the general medical practitioners volunteered to take part in the study.16 The practitioners in our investigation did not have any knowledge of the investigation. The guidelines were presented to them as a local initiative to improve services.

Perceptions of the referred patients

The part of the questionnaire completed by all referred patients evaluated their perceptions of the reason for their referral and the likely requirements of orthodontic treatment. When we analysed the data we found that for most of the responses there were no significant differences between those children attending dentists who should have given them the patient information leaflet and those attending the control dentists. However, more of the patients who attended the guideline group of dentists reported that they knew why their dentist had sent them to the orthodontist and more of them stated that they had crooked teeth. These differences were also clinically significant. This suggests that the group of dentists who had received the guidelines had informed the patients of why they were being referred or the patients had retained information from the patient information leaflet.

This finding is interesting, as the only investigation into perceptions of orthodontic patients revealed that 14% of referred patients did not know why they had been referred.1 Orthodontic treatment is provided mostly for aesthetic problems and uptake of treatment is influenced primarily by demand. It is important that children understand the reasons for their referral to an orthodontist, so that they can make an informed decision on treatment before they attend.

While these data could be interpreted as the guidelines and information leaflet educating the patients the overall effect is limited. There may be several reasons for this finding. Firstly, the patients may not have been issued with the information sheet. Secondly, they may have been issued with an information sheet and not retained the information with which they were presented. As a result, we can suggest that the issuing of an information sheet on orthodontic treatment may be of limited benefit and is unlikely to be worth the cost of printing and distribution.

Dissemination and implementation strategy

The importance of these strategies has been emphasised and the successful introduction of guidelines developed by consensus conference is very dependent on the choice of appropriate dissemination and implementation strategies. This is well illustrated by an investigation into the management of infertility. In the study the guidelines were presented as part of an infertility pack with a structured management sheet in which the guidelines were embedded.16 The management sheet was sent to the secondary care provider instead of the referral letter. We considered following this example in our study by using an orthodontic patient management sheet. However, this could not be achieved because the specialist orthodontic practitioners only took referrals by telephone. Also, this could not be used for the hospital providers, as it would have introduced bias into the study.

One other dissemination strategy that increases effectiveness has been the use of outreach educators (academic detailing) or the holding of seminars for the target group.8 While the use of outreach educators may be an effective method, it is expensive and therefore impractical when we consider the scarce resources allocated to primary and secondary care dentistry.

The only other option was to hold a series of meetings with the practitioners who had been allocated the guidelines. Unfortunately, this was not successful because the dental practitioners were not prepared to attend, despite personal letters being written to them by the local consultants and specialists, followed by a telephone call from the investigating team.

We also used the implementation strategy of sending each dentist a questionnaire on the guidelines when they were first sent out and then re-issuing the guidelines with a different questionnaire 6 months later. The replies that we received from the dentists were mostly positive. They stated that they understood and used the guidelines, however, this was not translated into a change in practice.

We also paid attention to research concerning the presentation of guidelines to both health care providers and patients. We took advice from experts in outreach teaching on the presentation of the guidelines and presented them in a form that allowed the dentists to display them in their surgeries. Information for the patient was similarly prepared.

Our final strategy was to ensure that each dentist received information on the appropriateness of the referral from the secondary care provider. This was provided by post following the consultation. It is worth stressing that this method may have been compromised because of the length of the new patient consultation waiting lists in the study area. These ranged from 12 weeks to 38 weeks. Consequently, a dentist would not remember the details of a patient that had been referred for orthodontic care by the time that they were seen.

Conclusion

This was the first time that a randomised controlled trial into the effectiveness of guidelines in dentistry had been carried out. It was disappointing that the guidelines did not have an effect and this may have arisen because of the pragmatic dissemination and implementation strategy that was used. Importantly, we considered several methods of dissemination and implementation and we ultimately used the only strategies that were appropriate for the general dental service within current resources. As it is extremely unlikely that these resources will be increased, this should be considered to be a 'real world' investigation. Importantly, the results of our study should be considered to have generality.

Finally, even though this study presents a negative finding, it does add to knowledge on the primary/secondary care interface in dentistry. We suggest that even when referral guidelines are produced using optimum methods and then disseminated using the only pragmatic methods available, they do not have an effect. Other strategies should be employed to change general dental practitioner's working practices and these require investigation.

The authors would like to acknowledge the support of the NHS R&D scheme into the primary/secondary care interface and the help of the practitioners and local dental committees who assisted us.