Key Points
-
QLF is a new device to detect very early demineralisation of enamel.
-
QLF can be used in vivo and in vitro.
-
All diagnostic methods suffer from some form of variability.
-
This study determined that, in most cases, QLF does not suffer from significant reliability issues in the analysis stage.
-
Further work on in vivo lesions is required.
-
Novices should be trained before analysing experimental data.
Abstract
Objective To assess the reliability of the analysis stage of quantitative light-induced fluorescence (QLF). The QLF analysis involves subjective input from the user and this study examines the influence of this on the reproducibility of the QLF data.
Method QLF images were taken of 20 human molar teeth that had been previously subjected to a demineralising solution (phosphoric acid 37%) to create artificial white spot lesions on their buccal surfaces. Following examination of the images, 16 were chosen to represent a range of lesion size and severity. Three copies were made of the images and each was allocated a different filename. 10 examiners in three centres were asked to analyse each of the 16 images on three occasions, with at least seven days between each attempt. Simple instructions describing the analysis procedure were supplied and examiners were asked to adhere to these directions. Examiners were asked to rate each of the 16 teeth on their first attempt both quantitatively (5 point scale) and qualitatively in terms of difficulty of analysis. Data reported were the ΔQ at 5% threshold for each tooth on each of three attempts.
Results Using ANOVA and paired t-tests to detect statistical differences, the three attempts of each examiner were used to determine intra-examiner reliability. Only one examiner (a novice at the technique) demonstrated differences between all three attempts and two demonstrated difference between one attempt. When the mean scores were compared to determine the inter-examiner reliability, only one examiner's results were statistically different when compared with two others.
Conclusion This study has demonstrated that the analysis stage of QLF is reliable between examiners and within multiple attempts by the same examiner, when analysing in vitro lesions. Novices at the technique should be trained before analysing experimental data.
Similar content being viewed by others
Main
Quantitative light-induced fluorescence (QLF) is a novel technique for the detection of early carious lesions in human enamel. Based upon the detection, and quantifiable analysis of an increased scattering of fluorescent light associated with demineralised lesions, the QLF process involves two main stages. The first stage is the capture of the tooth image. Following this the resultant image must be analysed using the proprietary software supplied with the device to produce the final result, a measure of mineral loss. The purpose of this study is to investigate the reliability, based upon intra- and inter-examiner agreement, of this stage of QLF analysis.
The QLF device (Inspektor Research Systems, bv, The Netherlands) can be considered as a new measurement device, and, as such, it must be subjected to a variety of tests to ensure that it is both valid and reliable. There are a number of methods of achieving this. First, validity, which can best be described as 'is this device measuring what it claims to measure?' The most usual method of testing this is to compare the new method with the current gold standard within the field. This has been studied by cross-validating QLF with Transverse Micro-Radiography (TMR) that found strong correlations between the two techniques.1,2,3
When assessing the reliability of the instrument, one must consider if there is likely to be any operator bias. The QLF device has the potential for operator bias, as it relies upon a subjective analysis of the stored tooth image. An example of another optical device, the KaVo DIAGNODent,4 does not suffer from such a potential bias, as a value is displayed on a LCD panel without requiring further input from the operator. However, the increased data yield and quantifiable nature of the QLF value, ΔQ, dictates that a more complex analysis is required. The QLF device, rather than merely supplying the arbitrary value characteristic of the DIAGNODent, supplies information on fluorescence loss (DF), lesion size (mm2) and the integration of these two values, ΔQ. The ΔQ value is therefore an indicator of mineral loss with relation to size and severity of the lesion.
The method of QLF analysis is described in detail within the methods and materials section, however the subjectivity of the technique is discussed here. The QLF method relies upon the subjective determination by the operator of several aspects of the QLF image:
a) Determination of sound or unsound enamel.
b) Determination of the presence or absence of a demineralised lesion.
c) The borders of any detected lesion.
d) The morphology of the lesion.
e) Presence of any confounding factors, ie plaque, calculus, moisture, shadowing, enamel defects etc.
Because of these subjective components of QLF an investigation into the reliability of the technique is required to gauge their influence upon the validity of the measurement.
Measurement variability
With any measurement there will be variability. For a measurement of any single factor, repeated measurements of that same factor will vary. Fleiss5 describes this; a true value (T) could be considered as one that is a mean of many repeated measurements. Subsequent individual measurements (X) will vary from this true value for any number of different reasons, including examiner effects, subject effects, imperfect calibration, etc. The true value (T) will vary around a given mean, and the random error (E) of the individual measurement (X) will vary around a mean of zero. This is shown in the following formula:
From this it is possible to determine that a variation of the ΔQ value following repeated measures of the same lesion is inevitable. The purpose of this study is to quantify the intra- and inter-examiner reliability of QLF analyses. Such data is essential to researchers using the device and those taking note of findings from such studies. It is necessary to determine that any differences detected by examiners are the result of a manipulated experimental variable rather than statistical artefacts of an unreliable device. It is also important to determine whether different researchers' results can be directly compared. Reliability of a device also impacts on the number of subjects that are required for a trial, and in clinical situations, the weight that is given to a positive or negative diagnostic finding.
Methods and materials
The QLF method
The basic method for analysing a QLF image is described and shown in Figure 1. The technique is open to a great deal of variation in terms of where the patch is placed, the subjective judgement of the operator as to what is sound enamel and which, if any, borders should be excluded. The inclusion or exclusion of borders can have a profound impact on the reported value. However, it is important to note that there are several guides to successful analysis within the program. One of these is the reconstructed image. Following analysis of the image, the reconstructed image is shown adjacent to the original image. The re-constructed image should be clear of any streaking or dark lines, indicating that the lesion has been successfully 'removed' from the reconstruction. Figure 2 shows both a successful and an unsuccessful reconstruction of the same lesion. Another means by which operators can confirm their analysis technique is the reconstructed lesion that is also shown following analysis. Adjacent to the grayscale reconstructed image, an image of the outline of the lesion is shown. This outline should be compared with the original lesion shown on the QLF image. If the analysis has been successful the isolated lesion will closely match the morphology of that present on the original captured image. An example of this is shown in Figure 3.
Lesion and image production
Twenty previously extracted human third molar teeth were cleaned of all soft-tissue remnants, and their buccal surfaces gently pumiced and abraded with wet-and-dry paper. Each tooth was given two coats of acid-resistant nail varnish (MaxFactor, Proctor and Gamble, UK) leaving an exposed window of enamel on the buccal surface. The exposed windows were of varying sizes and position. The teeth were subjected to varying exposures of 37% phosphoric acid gel (Henry Schein, UK) ranging from 5 seconds to 10 minutes. This produced simulated white-spot lesions of varying severity. Etched enamel lesions appear exactly as other simulated lesions under QLF conditions, for example those created by acidic buffers or gels. Each tooth was examined using QLF. Following this examination, 16 teeth were selected to provide a spread of lesion severity, size, position and hence analysis difficulty (Fig. 4). In studies examining examiner reliability it is desirable to achieve an even spread of easy, moderate and complex cases to ensure a valid result. This was initially assessed subjectively by the investigators and then confirmed by asking the examiners to rate each lesion's analysis complexity.
Final QLF images were taken of each lesion using a laboratory jack platform above which the QLF camera was held in a fixed position. Fine focus was achieved using the vertical control of the jack's platform. Using this method high quality QLF images were obtained. The images were stored on a PC and two further copies made of the image files to ensure exact replicas. Each set of 16 images was placed into a separate folder, and provided with a unique filename to ensure blinding of the examiners.
Examiners
The images were placed on to a CD-ROM and distributed to ten examiners at three different centres: The University of Liverpool, The University of Manchester and The University of Glasgow, all within the UK. Selection of the examiners was based upon achieving a spread of QLF experience. Accompanying the CD-ROM were examiner instructions and a proforma to elicit the following data:
a) A quantitative assessment of each lesion's perceived difficulty of analysis (scale 1–5)
b) A qualitative assessment of each lesion's perceived difficulty of analysis. Examiners were asked to describe why they allocated a particular rating to each lesion
The examiners were asked to complete this assessment on their first attempt only. The instructions included the following analysis 'rules' for examiners:
a) Please use four iso-lines (to control the dimension of the patch).
b) There is no need to use the contour markers (these are used to enable placement of an analysis patch on several images of the same tooth).
c) Place the analysis patch around the lesion, following the shape as closely as possible, but ensuring that the border rests on sound enamel.
d) Exclude a border if you feel that it is crossing 'un-sound' enamel.
e) Following analysis, check to see if the reconstructed image is homogenous in appearance, with no white or black streaks and that the coloured reconstruction matches the outline of the original lesion.
f) Repeat analysis if necessary to achieve the above.
These instructions were illustrated with pictorial examples of the process. The examiners were asked to leave a 'washout' period of 7 days between each of the three attempts. Following completion of the exercise the results were printed directly from the QLF program. The ΔQ value from each tooth and each attempt was recorded for statistical analysis.
Statistical techniques
Numerous statistical techniques for the analysis of reliability exist.5 For the data type presented here, ie continuous data, ANOVA and paired t-tests were used to detect significant differences between the values obtained by the examiners. In order to determine intra-examiner reliability, each of the three attempts was compared with each other using ANOVA and subsequent t-tests. When assessing the inter-examiner data, the mean of these data were used in a crosswise method with ANOVA and t-tests. The significance level was set at 95% (a = 0.05). Raw data were entered into Excel and transferred to SPSS v.10 for statistical analysis.
Results
Quantitative data
All ten examiners completed the study and returned correctly formatted data (100%). An example of an individual examiner's results is shown in Table 1. The results of the ANOVA and t-tests are shown in Table 2 for the intra-examiner assessment, and in Table 3 (for the inter-examiner assessment. The mean difficulty rating for the 16 teeth was 2.5 ± 0.77 (Scale 1–5) demonstrating a good spread of analysis complexity.
Qualitative data
Each examiner was asked to describe the reasons that a particular complexity rating was allocated to each tooth. The comments made were consistent across all the examiners. Lesions classified as simple to analyse (1,2, or 3) were described as simple in shape and surrounded by clearly sound enamel. An example of such a lesion is tooth 14 of Figure 4. Complex ratings (4 or 5) were allocated to teeth that were described as uneven in shape and adjacent to areas of unsound enamel. Proximity to the EDJ or areas of shadow also led to a higher level of difficulty being analysed. When the analysis patches of teeth with ratings of 4 or above were examined, a large proportion had one or more borders excluded. Many examiners mentioned that this was a confusing aspect of the analysis. When looking at repeated analyses of the examiners, consistency of border inclusion/exclusion was low. Despite this, when the complexity of lesion scores were compared with variance scores, no relationship was found (r = 0.4229), suggesting that lesion complexity does not lead to a decrease in reliability.
Discussion
The assessment of reliability is essential for any new scale, measuring device or group of calibrated examiners.6 The QLF can be regarded as a measuring device with two stages. This study has determined that the reliability or reproducibility of the analysis stage of the technique is good, both in terms of intra- and inter-examiner agreement. When assessing the intra-examiner data only one examiner (Examiner 4) was found to have significant differences between all three of their attempts. Two other examiners, (3 and 8) had a statistical difference between their first and last attempts. In total only 5 of 30 comparisons (16%) were significantly different and 3 (10%) of these were from a single, novice, examiner. The inter-examiner assessment demonstrated that only two differences were detected, both of which were between Examiner 4. Of 50 unique comparisons, only 1% created a significant difference as a result of an unreliable analysis. The precision of the device, ie the variability of repeated measures of a single phenomenon was high, resulting in few differences between and within examiner ratings.6
Examiner 4 was responsible for the most differences detected. This particular examiner was a novice at the technique, with very few hours of analysis experience. Many of the other examiners had either extensive experience with QLF or examining carious lesions using computerised analysis techniques such as transverse microradiography (TMR).7 While the TMR software operates in a different manner, familiarity with the computer processes is likely to be of value when transferring skills to QLF. Despite the difference detected between examiners 3 and 4's intra-examiner reliability, when the mean of these examiner's results were entered into the inter-examiner score, agreement was high, with no differences detected. This finding suggests that novices may find that repeating their analyses and reporting mean ΔQ values may be appropriate. It is well accepted that repeated measures could increase reliability and reduce any examiner or technique error. The training of novices by more experienced users is recommended, and, by using examples such as those shown in Figures 2 and 3, the ability to critically assess the analysis demonstrated. The employment of analysis rules will also facilitate new users' abilities to produce reliable results.
Lesions that lie close to un-sound enamel, enamel defects, stain or anatomical features, which are likely to confound the analysis, are perceived as being more difficult to analyse. Despite this, the reliability of measurements of such lesions is still high. The provision of 'rules' within the instructions may permit 'calibration' of examiners in large studies. The authors recommend that QLF examiners should follow basic guidelines to ensure that research reports can be compared. In articles reporting the use of QLF the authors suggest that a description of any analysis is included, for example 'borders were excluded on un-sound enamel; those teeth requiring two or more border exclusions were removed from the trial'.
The QLF software is easy to master and the graphical representations of the original image, the reconstructed image and the lesion all enable users to determine if their analysis is appropriate. When using an analysis technique such as QLF, in which there is an element of subjectivity based upon examiner determinations, these guides are invaluable to both the novice and the expert.
Conclusions
This study has determined that, when using simulated, etched, in vitro, lesions the intra- and inter-examiner agreement of QLF analysis is high. In a multi-centred, multi-examiner trial, high levels of both intra- and inter-examiner agreement were measured. The data suggest that should significant differences be detected between QLF examinations then this can be attributed to the experimental variable rather than any examiner error. Novices in the technique may wish to perform repeated analyses of their images to reduce measurement error. Experts in the technique may wish to compare their results to ensure that their techniques are producing reliable results. The use of analysis rules may help to increase reproducibility among and within examiner groups.
QLF has been shown to be a user-friendly, reproducible and reliable method of quantifying mineral loss from enamel. Further research is required to investigate if such high reliability is seen when analysing in vivo lesions, many of which may be more complex than those used in this trial. Lesions were considered complex when close to areas of un-sound enamel or areas that appeared un-sound on QLF.
References
al-Khateeb S, ten Cate JM, Angmar-Mansson B et al. Quantification of formation and remineralization of artificial enamel lesions with a new portable fluorescence device. Adv Dent Res 1997; 11: 502–506.
Ando M, Hall AF, Eckert GJ, Schemehorn BR, Analoui M, Stookey GK Relative ability of laser fluorescence techniques to quantitate early mineral loss in vitro. Caries Res 1997; 31: 125–31.
Lagerweij M, vander Veen M, Ando M, Lukantsova L, Stookey G The validity and repeatability of three light-induced fluorescence systems: an in vitro study. Caries Res 1999; 33: 220–226.
Ross G Caries diagnosis with the DIAGNOdent laser: a user's product evaluation. Ont Dent 1999; 76: 21–24.
Fleiss JL The design and analysis of clinical experiments New York: John Wiley & Sons 1986.
Brunette DM Critical thinking: understanding and evaluating dental research London: Quintessence Pub. Co 1996.
Anderson P, Elliott JC Scanning X-ray microradiographic study of the formation of caries-like lesions in synthetic apatite aggregates. Caries Res 1985; 19: 403–406.
Author information
Authors and Affiliations
Corresponding author
Additional information
Refereed paper
Rights and permissions
About this article
Cite this article
Pretty, I., Hall, A., Smith, P. et al. The intra- and inter-examiner reliability of quantitative light-induced fluorescence (QLF) analyses. Br Dent J 193, 105–109 (2002). https://doi.org/10.1038/sj.bdj.4801496
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/sj.bdj.4801496
This article is cited by
-
Comparison of Quantitative light-induced fluorescence-digital (QLF-D) images and images of disclosed plaque for planimetric quantification of dental plaque in multibracket appliance patients
Scientific Reports (2020)
-
Quantitative light-induced fluorescence images and digital photographs - Reproducibility of manually marked demineralisations
Journal of Orofacial Orthopedics / Fortschritte der Kieferorthopädie (2017)
-
Assessment of a new infrared laser transillumination technology (808 nm) for the detection of occlusal caries—an in vitro study
Lasers in Medical Science (2015)
-
Demineralization adjacent to orthodontic brackets after application of conventional and self-etching primer systems
Journal of Orofacial Orthopedics / Fortschritte der Kieferorthopädie (2014)
-
Performance of a new fluorescence camera for detection of occlusal caries in vitro
Lasers in Medical Science (2013)