Main

Research has shown large variations occur between dentists in their diagnoses, restorative decisions and treatment plans.1,2,3,4 A previous study by Kay indicated that a key parameter in dentist's decision making process might be the way in which costs and benefits of false negative and false positive decisions are weighed against each other.5 Other work has also suggested that if an individual understands the trade-off between sensitivity and specificity then they make fewer false positive and false negative choices.6,7,7 A previous study, which developed the hypothesis tested in the research reported here, indicated that the accuracy of dentists' treatment decisions could be improved by educating practitioners about the use of probabilistic reasoning when making treatment decisions.8

The aim of this study was to determine whether an educational intervention delivered by a computer aided learning package improved the sensitivity and specificity of dentists' restorative treatment decisions. If the research showed that the new approach was effective in improving the validity of decisions, a teaching package was to be developed and widely disseminated.

The specific objectives of the study were to examine the effect of a previously piloted educative intervention on the sensitivity and specificity and variability of dentists' treatment decisions and to develop the educative intervention into an easily disseminated teaching package.

In order to achieve the aims and objectives of the study, two research hypotheses were tested. These were:

  1. 1

    The intervention will increase the sensitivity and specificity of dentists' restorative treatment decisions.

  2. 2

    The intervention will increase the agreement between dentists in the clinical treatment decisions made by them.

Method

Study material

The study used material constructed and previously reported by the lead author.3,5,7 These were 15 sets of simulated bitewing radiographs. The radiographs were of models which simulated the posterior part of a natural dentition, using extracted teeth mounted in a medium which simulated human bone. One fifth of the teeth included in these simulated dentitions had some visible evidence of caries, ranging from incipient lesions to large cavities. After radiographs of the models had been taken, using a technique precisely equivalent to normal bitewing radiography, the teeth were serially sectioned into 300μm thick longitudinal sections. These sections were then examined for caries using a microscope and each approximal surface was graded on a 5-point scale:

0 – Sound

1 – Lesion limited to outer half of enamel

2 – Lesion more than half way through enamel but not penetrating dentine

3 – Lesion into dentine but not more than half way to pulp

4 – Lesion into dentine and more than half way to pulp

The radiographs were presented to the participating dentists along with an appropriate patient vignette, eg 'this radiograph is of a patient who you will see again within the next two years. His/her diet and oral hygiene are 'average'- that is, better than your worst patient but worse than your best patient.'

The participating dentists were asked to make treatment decisions about the approximal surfaces of the teeth on the radiographic films, grading their decisions on a 6-point scale:

1 – Definitely place a restoration

2 – Probably place a restoration

3 – Might consider placing a restoration

4 – Might consider leaving the tooth unrestored

5 – Probably leave the tooth unrestored

6 – Definitely leave the tooth unrestored

Since each radiograph contains information on 24 surfaces, each dentist made 15 x 24= 360 decisions. The analysis is therefore based on the comparison of more than 30,000 decisions made before or after the educational intervention.

Study design

The study used a Solomon three-group design. After recruitment the participating dentists were randomly assigned to one of three groups as outlined below and described in Table 1.

Table 1 Table 1
Full size table

Group 1 were asked to view the radiographs and made treatment decisions using the scale described above. On the subsequent day, they used the CAL package and then re-read the radiographs.

Group 2 used the CAL package prior to a single reading of the radiographs.

Group 3 read the radiographs and on the subsequent day re-read them. The procedure was the same as for group 1, except they did not utilise the CAL package until after the second viewing of the radiographs.

With O1 to O5 as defined in Table 1, the following effects were compared:

O2 – O1 Effect of intervention and dual reading

O2 – O3 Effect of dual reading and differences in variability between groups

O2 – O5 Effect of intervention and differences in variability between groups

O5 – O4 Effect of dual reading

O4 – O1 Effect of differences in variability between groups

Educational intervention

The intervention used was a computer aided learning (CAL) package, which had the following teaching objectives:

After using the CAL programme, participants would understand:

  1. 1

    The impossibility of perfection in the recognition of pathology.

  2. 2

    The potential effects of uncertainty on the delivery of quality patient care.

  3. 3

    That judgement involves accepting the possibility of making detrimental as well as beneficial decisions.

  4. 4

    That the values and costs associated with any treatment decision relates to: the consumers views, the probability of success of that treatment, the acceptability of both the outcome and the process which occur subsequent to the decision.

Statistical methods

Sensitivity and specificity of the dentist's decisions to restore teeth were calculated at two thresholds of diagnosis, "definitely restore" versus all other ratings (T1), and 'definitely', 'probably', or 'might consider' restoring the tooth versus 'definitely', 'probably', or 'might consider' leaving the tooth unrestored (T2). These calculations used the microscopic criteria of caries penetrating dentine as the gold standard. Area under ROC curves were derived for each dentist in each group at all time points. Comparisons between the different groups were made by independent sample t-tests, and comparisons within the same group made, using paired t-tests. Kappa statistics were computed to assess the level of agreement between dentists within a group.

Results

Thirty-one dentists participated in group 1, 33 in group 2, and 31 in group 3. However, the majority of the data from one dentist in group 3 was unusable. The data from this dentist was, therefore, excluded.

The microscopic examination of the 360 tooth surfaces showed that 244 (68%) were sound and 52 (14%) had caries into enamel and 44 (12%) caries into dentine. Two surfaces (1%) had restorations and data for 18 (5%) were not available owing to the tooth shattering on sectioning.

The mean sensitivity and specificity values for threshold T1 for each study group are shown in Table 2. The initial mean sensitivity value for the dentists in group 1 was 0.318 and this increased to 0.336 after the intervention, the initial mean specificity value of 0.946 increased to 0.950, however these changes were not significant (P = 0.50; P = 0.56).

Table 2 Table 2
Full size table

The mean sensitivity values and specificity values for all the dentists in groups 1, 2 and 3, were similar at all time points, for this threshold T1, with the sensitivity ranging from 0.299 to 0.336, and the specificity ranging from 0.946 to 0.961 (Table 2). All the P-values obtained from the specified comparisons were non significant apart from a significant increase in the mean specificity of the two readings conducted by the dentists in group 3 (Table 2; P = 0.02). The P-values obtained from the specified comparisons are detailed in Table 3 and were all non-significant.

Table 3 Table 3
Full size table

The results for the other threshold T2, were similar. For the dentists in group 1 there was an increase in the initial to post intervention mean sensitivity from 0.519 to 0.520 and in the specificity from 0.852 to 0.876, which were not significant (P = 0.98; P = 0.11). The mean sensitivity values and specificity values for all the dentists were similar at all time points, the sensitivity ranging from 0.493 to 0.539, and the specificity ranging from 0.852 to 0.880 (Table 4).

Table 4 Table 4
Full size table

There was a small non-significant increase in the mean area under the ROC curves for the dentists in group 1, from 0.729 to 0.744 (Table 5). The range of values for the mean areas under the ROC curves for each group of dentist at both time points was small, ranging from 0.720 to 0.752.

Table 5 Table 5
Full size table

The results of calculating kappa for each of the categories separately against a combination of the remaining categories are shown in Table 6. For example, the initial kappa value for the dentists in group 1, comparing 'definitely restore' with the other decision categories is 0.44. There is fair to moderate agreement at this threshold with kappa values ranging across groups and time points from 0.370 to 0.512. Comparing the 'definitely leave unrestored' category to the other categories, the range of kappa values was 0.276 to 0.326, which is considered fair agreement. Although the standard errors of kappa cannot be calculated when the number of readings per dentist varies, there was no evidence that the level of agreement between the dentists improved after the intervention.

Table 6 Table 6
Full size table

Discussion

The project fulfilled all of its objectives. However, the educational intervention was found not to affect the sensitivity and specificity or the agreement between dentist's treatment decisions.

The range of mean sensitivity, specificity, and kappa values which assessed the agreement between dentists, were narrow showing that the results were consistent between the study groups, and for different time points within the study groups. The study had sufficient power to detect small differences in all the indices measured. For example a change of 0.1 in sensitivity, 0.03 in specificity, and 0.03 in the area under the ROC curve would have been detectable with 95% power with the sample size of 30 dentists per group.

The results are surprising since previous evidence had strongly suggested that information about uncertainty enhanced the accuracy and consistency of dentists' restorative treatment decisions.8 In that study, the intervention had been delivered as a seminar and a trend towards improved decision making was clearly shown. However, the power of the previous study was questionable. The results described in this report are more reliable and a definite lack of effect of the intervention was demonstrated. The discrepancy between the results of the two studies can be explained in two ways. The first and most plausible explanation is that the previous study's small sample size allowed any changes in the dentists' treatment decisions after the intervention to be overstated and that the current study has revealed the true lack of impact of education about uncertainty. A second, less plausible, but possibly more interesting explanation might be that delivery of information via a personally delivered interactive seminar is more effective for improving dentists' decision making skills than a computer aided learning package. The educational intervention in the current study was delivered via a CAL package for two reasons. Firstly, it was delivered by computer in order to ensure standardisation across the different experimental groups. Secondly, it was important to have a method of education which could be easily and widely distributed and used, should it prove to be effective.

The study has clearly shown that education about uncertainty delivered by a computer aided learning package had no effect on dentists' decision making. This finding, especially when viewed in the light of the previous study, has very important implications, given the burgeoning use of computer packages in education. The only difference between the two studies was the mode of delivery of the information. The information, study design, materials and methods were exactly the same. Further research is urgently needed to directly compare education delivered by CALs and by personnel. The findings of this study have enormous implications for education.

Conclusions

  1. 1

    Further study of the psychology of treatment decision making is needed in order to determine the key factors influencing dentists' choices.

  2. 2

    Direct comparisons of computer aided learning and traditional education are needed before the ease of distribution of such packages causes potentially ineffective educational methods to overtake traditional ones.