Introduction

Glaucoma is the leading cause of irreversible blindness, with an estimated global prevalence of 3.5% among those 40–80 years of age [1]. As patients with established glaucoma, and many of those at risk or suspected of having the disease, require lifelong monitoring, glaucoma places a considerable burden on eye care providers.

Virtual clinics have been proposed as a way to improve efficiency of glaucoma care [2]. In a virtual clinic, patient data are collected by technicians, nurses or non-specialist optometrists through a series of standardised tests, including assessment of intraocular pressure (IOP), visual fields (VF) and optic disc imaging [3]. Data are then reviewed remotely by an ophthalmologist to decide if the patient is stable or if a change in treatment is required [2]. Using this strategy, a greater number of patients can be reviewed, while the patient’s visit is streamlined meaning less time is spent in the clinic than on a conventional visit [2, 4]. A recent survey of clinical lead ophthalmologists reported 50% were based at a unit where virtual clinics were being used to review patients with glaucoma, with a further 21% planning to establish a virtual clinic in the future [5]. Virtual clinics also seem popular with patients, with surveys showing high levels of satisfaction [6].

The major risk factor for glaucoma progression is IOP and therefore it is essential that measurements of IOP are precise and reproducible. Goldmann applanation tonometry (GAT) is the accepted industry standard for IOP measurement, with studies showing GAT to have excellent reproducibility, with intraobserver intraclass correlation coefficients (ICC) between 0.7 and 0.99, and interobserver ICCs between 0.81 and 0.97 [7,8,9,10,11,12,13]. GAT has several limitations though meaning that it may not be the best tool for use in virtual clinics. The accuracy of GAT depends on the skill of the operator, measurements are somewhat subjective and there is a potential for operator bias [14]. Furthermore, if used in a virtual clinic, there is no direct mechanism whereby the ophthalmologist can verify the quality of measurements obtained. Other tools used in virtual clinics such as automated perimeters and optical coherence tomography have reliability indices or quality scores but there is no equivalent with GAT. GAT is also influenced by the quantity of fluorescein used and properties of the cornea including astigmatism, corneal thickness and biomechanical properties [15]. As GAT requires contact with the cornea, local anaesthetic drops are needed and there is a risk of transmission of infection, particularly of viral pathogens such as adenovirus [16].

The Ocular Response Analyzer (ORA) (Reichert Inc, Buffalo, NY, USA) is an alternative device for IOP assessment. It is an automated, non-contact tonometer that uses infrared to measure the corneal response to deformation by a rapid air pulse [17]. By analysing corneal biomechanical properties, the ORA provides Goldmann-correlated (IOPg) and corneal-compensated (IOPcc) IOPs, corneal hysteresis (CH) and a waveform (quality) score (WS) with each IOP measurement [18]. Studies comparing agreement between IOPg and GAT have reported 95% limits of agreement (LoA) of ±4.55, ±5.35, ±6.95, ±7.4 and ±11.54 mmHg, respectively [13, 19,20,21,22,23]. Tejwani et al. also investigated the reproducibility of IOPg measurements, reporting an ICC of 0.95 [13]. Advantages of the ORA include that it provides a quality score and measures CH; an important risk factor for glaucoma, in addition to being objective and not requiring contact with the cornea.

The aim of this study was to examine the suitability of the ORA for use in virtual glaucoma clinics by comparing IOP measurements obtained by ORA to those obtained by nurses using GAT, reflecting current practice in our virtual clinic. Both were compared with IOP measurements obtained by consultant ophthalmologists using GAT.

Materials and methods

This was a prospective study involving 116 patients with glaucoma, ocular hypertension (OHT) or suspected glaucoma attending the glaucoma clinic at Princess Alexandra Eye Pavilion, Edinburgh, UK. The study methods were approved by the Princess Alexandra Eye Pavilion quality improvement team as a part of the development of virtual clinics for glaucoma. On the day of their appointment, the purpose and nature of the study was explained to patients and informed consent obtained in accordance to the principles of the Declaration of Helsinki [24].

For each patient, IOP was measured in both eyes, with the right eye reported for simplicity, by three methods: an ophthalmologist using GAT, an ophthalmic nurse using GAT and a technician using the ORA G3 (Reichert Inc, Buffalo, NY, USA). The order of testing was random, and ophthalmologists and nurses were masked to the results of all previous tests. Nurses had at least 2 years’ experience of regularly performing GAT and working in glaucoma clinics. ORA measurements were obtained by technicians who had received a short period of training in operation of the device.

The ORA was set to the “multiple measure, quadruple-puff” setting, that is, once the technician pressed the multiple measure icon, it automatically obtained four IOP measurements in a row. The ORA was also set to the “intelligent averaging” option, whereby the final displayed value is an average of all measurements that had a WS quality value within ±1 of the highest WS value achieved for that eye, with measurements outside that quality range automatically excluded [18]. Hence, only the averaged result after four measurements was recorded by the technician for each eye. The technician then repeated the measurement process to record repeat ORA values. For each patient, data were cleared from the ORA machine after assessment to prevent averaging of data across different eyes [18]. Patients also had assessment of central corneal thickness (CCT) using the PachPen (Accutome Inc.).

Statistical analysis

Ophthalmologist GAT, nurse GAT and ORA IOPg samples were compared with the paired-sample t-test or Wilcoxon signed-rank test depending on whether the data were parametric or non-parametric (determined by Shapiro-Wilk). Agreement among these groups as well as among repeat ORA measurements (IOPg1 and IOPg2) were analysed using Bland–Altman plots of difference against mean, with 95% LoA being calculated. LoA were adjusted for trend using a regression model. Linear regression was used to check for proportional bias in the Bland–Altman plots and to investigate CCT, WS and CH as confounding factors. Repeatability of IOPg, GAT (treating the ophthalmologist and nurse IOPs as repeat GAT measurements), IOPcc and CH were assessed by calculating average measures intraclass correlations (ICC; two-way mixed). All tests were two-sided and an α < 0.05 was considered statistically significant. Statistical analyses were performed using SPSS 24 and Stata 14.2 (StataCorp LP, College Station, TX, USA).

Results

The mean (±standard deviation) age of included patients was 70 ± 13 years. Patients had an average CCT of 540 ± 44 μm. Table 1 summarises IOP measurements obtained using GAT and IOP, wave score and CH measurements obtained using ORA. It also outlines the differences between IOPs obtained with the different methods, in addition to the mean differences between repeat IOPg, IOPcc and CH.

Table 1 Mean, standard deviation and range of measurements and of differences between measurements obtained from GAT performed by nurses and ophthalmologists and ORA performed by technicians.
Full size table

The mean GAT IOP obtained by nurses was 15.0 ± 4.0 mmHg compared with 14.6 ± 4.3 mmHg for ophthalmologists (P = 0.056), with an average difference of 0.4 ± 2.9 mmHg (range −12 to 8 mmHg). A Bland–Altman plot comparing GAT IOP measurements obtained by nurses and ophthalmologists is shown in Fig. 1. 95% LoA were ±2.46 × (0.79 + 0.09 × average IOP), equating to ±5.2 mmHg for the sample average IOP of 14.8 mmHg. There was no evidence of proportional bias as indicated by the lack of a significant relationship between the difference and average of IOP between measures (R2 = 0.007, P = 0.377).

Fig. 1: Bland–Altman plot showing the relationship between the average of nurse and ophthalmologist IOP measurements (x axis) and the difference between the nurse and ophthalmologist measurements (y axis).
figure 1

95% limits of agreement were +/– 2.46*(0.79+0.09*average GAT IOP measured by the ophthalmologist and nurse).

Full size image

The mean IOPg was 13.8 ± 4.7 mmHg compared with 14.6 ± 4.3 mmHg for ophthalmologist GAT measurements (P = 0.001). The average difference was −0.8 ± 2.5 mmHg (range −7.5 to 5.1), with 95% LoA of ±2.46 × (0.87 + 0.08 × average IOP), equating to ±4.93 mmHg for the sample average of 14.17 mmHg (Fig. 2). There was no evidence of proportional bias (R2 = 0.03, P = 0.063).

Fig. 2: Bland–Altman plot showing the relationship between the average of ORA IOPg and ophthalmologist IOP measurements (x axis) and the difference between ORA IOPg and ophthalmologist measurements (y axis).
figure 2

95% limits of agreement were +/− 2.46*(0.87+0.08*average of ORA IOPg and GAT IOP measured by the ophthalmologist).

Full size image

The average difference between repeat IOPg measurements was 0.9 ± 1.3 mmHg, with 95% LoA of ±2.46 × (0.76 + 0.02 × average IOP), equating to ±2.52 mmHg at the sample average of 13.28 mmHg (Fig. 3). There was no evidence of proportional bias (R2 = 0.007, P = 0.414).

Fig. 3: Bland–Altman plot showing the relationship between the average of ORA IOPg1 and (repeat) ORA IOPg2 measurements (x axis) and the difference between ORA IOPg1 and ORA IOPg2 measurements (y axis).
figure 3

95% limits of agreement were +/− 2.46*(0.76 + 0.02* average IOPg).

Full size image

Wave score had no effect on agreement between IOPg and ophthalmologist GAT IOP (R2 = 0.001, P = 0.713) or on agreement between IOPg and nurse GAT IOP (R2 = 0.005, P = 0.448) or on agreement between repeat IOPg measurements (R2 = 0.012, P = 0.288). In contrast, in eyes with thinner CCT, IOPg tended to be lower than ophthalmologist GAT IOP, whereas in eyes with a thicker CCT, they tended to be higher (R2 = 0.180, P < 0.001, Fig. 4a). A similar relationship was observed between CCT and nurse GAT IOP measurements (R2 = 0.05, P = 0.025, Fig. 4b). However, CCT had no significant effect on the relationship between repeat IOPg measurements (R2 = 0.003, P = 0.608).

Fig. 4
figure 4

Scatter plots showing the relationship between central corneal thickness (CCT) and the difference between ORA IOPg1 and GAT IOP measured by ophthalmologists (a), between CCT and the difference in ORA IOPg1 and GAT IOP measured by nurses (b), and between corneal hysteresis (CH) and the difference between ORA IOPg1 and GAT IOP measured by ophthalmologists (c).

Full size image

In eyes with lower CH, IOPg tended to be lower than ophthalmologist GAT IOP (R2 = 0.071, P = 0.004, Fig. 4c). However, no significant relationship between CH and differences in IOPg and nurse GAT IOP was observed (R2 = 0.011, P = 0.266), and there was no relationship between CH and differences in repeat IOPg measurements (R2 = 0.003, P = 0.634).

Conducting further linear regression analysis, IOPg tended to increase with CCT and this was statistically significant (R2 = 0.138, P < 0.01). GAT nurse measurements (which are the ones used in virtual clinics) also tended to increase with CCT and this was statistically significant too (R2 = 0.066, P = 0.012). However, there was no statistically significant relationship between CCT and IOPcc (R2 = 0.036, P = 0.069), nor between CCT and GAT ophthalmologist measurements (R2 = 0.026, P = 0.116).

ORA measurements were highly reproducible with ICCs of 0.972, 0.959 and 0.936 for IOPg, IOPcc and CH, respectively. GAT was less reproducible with an ICC of 0.863.

Discussion

This study found only moderate agreement between GAT performed by nurses working in a virtual glaucoma clinic and ophthalmologists, with 95% LoA of ±5.21 mmHg. In other words, 95% of IOP measurements obtained by nurses were within ~5 mmHg of those obtained by an ophthalmologist but 5% differed by a greater margin. Differences of this magnitude are likely to have important clinical implications as management decisions are often influenced by IOP. Furthermore, there was no evidence of proportional bias, meaning a difference of ±5.21 mmHg was present across the range of IOP values.

When IOP is measured in a virtual clinic there is no direct mechanism for the ophthalmologist to verify accuracy of the measurement and therefore it is especially concerning that such disagreement in repeat measures may exist. Although it is not possible to determine whether measurements obtained by nurses or ophthalmologists were more accurate, on average IOP measured by ophthalmologists had slightly better agreement with IOPg with 95% LoA of ±4.93 mmHg compared with ±5.20 mmHg for IOP measured by nurses. This indicates only moderate agreement between repeat GAT and between IOPg and GAT performed by ophthalmologists or nurses. In contrast, there was good agreement between repeat ORA measurements with 95% LoA for IOPg of ±2.52 mmHg. IOPg also had a higher ICC than GAT measuring 0.972 and 0.863, respectively. Repeatability is an important characteristic of a good clinical measure and the results of this study suggest IOPg is more consistent across repeat measures compared with GAT performed by different operators in a normal clinical setting.

Our ±5.21 mmHg LoA between GAT performed by ophthalmologists and nurses is higher than the ±3.7 mmHg observed by Kotecha et al., who also compared nurse/technician and ophthalmologist GAT measurements in glaucoma clinics [25]. Kotecha et al. utilised 100 eyes for this comparison, providing a comparable sample size to ours and a sample size of 100 is considered good for measuring agreement with Bland–Altman plots [26]. An important difference between the studies, however, is that Kotecha et al. used a two-person technique when measuring GAT, which is not typical of normal clinical practice [27]. Furthermore, IOPg still achieved a better LoA than the results of Kotecha et al.’s study.

Our ±4.93 mmHg 95% LoA between IOPg and ophthalmologist GAT measurements fell within the range of ±4.55 to ±11.54 mmHg reported in previous studies [13, 19,20,21,22,23]. It is difficult to explain this large range of LoA in the literature, however, it may be partly due to researchers using different settings on the ORA, such as the averaging mechanism and perhaps reflects variability in GAT rather than IOPg. Our finding of an IOPg ICC of 0.972 was very similar to the ICC of 0.95 found in a previous study by Tejwani et al. [13]. Our GAT ICC of 0.863 was also consistent with previous studies reporting interobserver GAT ICC to range from 0.81 to 0.97 [7,8,9,10,11,12,13].

To interpret the relevance of the 95% LoA, it is crucial to consider what difference in IOP is clinically relevant. The Early Manifest Glaucoma Trial found a 10% reduction in risk of progression for each mmHg lower IOP [28]. Likewise, the OHT Study revealed a 10% increase risk in developing primary angle glaucoma for each mmHg higher IOP [29]. Our finding of ±5.21 mmHg 95% LoA between ophthalmologist and nurse GAT measurements is thus potentially concerning. While 95% agreement limits between IOPg and GAT IOP were better, they still only reached ±4.93 mmHg.

In contrast, agreement between repeat ORA IOPg measurements was good, with a 95% LoA of ±2.52 mmHg. In addition to providing relatively reproducible results, ORA has other potential advantages for use in the virtual clinic environment. Foremost, there is evidence that ORA IOPcc measurements are less affected by corneal properties than GAT [30]. The recent United Kingdom Glaucoma Treatment Study (ISRCTN96423140), a randomised double masked placebo-controlled study, in which newly diagnosed patients with primary open angle glaucoma were randomised to receive a topical prostaglandin analogue or placebo, found IOPcc was the best IOP predictor of VF deterioration [31]. IOPcc performed better than GAT or IOP measured using the Pascal Dynamic Contour Tonometer. In addition, ORA is non-contact, reducing the risk of infection present with GAT, and requires no local anaesthetic drops. It can also be performed after a relatively short period of training: the technicians performing ORA measurements in this study required training for one morning. Lastly, the ORA quality score provides an objective measure of quality that could be recorded in the virtual clinic chart.

Although our primary comparison outcomes were the 95% LoA, isolated differences in measurements falling outside the 95% LoA seen in the Bland–Altman plots are also important to consider as they could have significant clinical implications. For example, the greatest disagreement in GAT IOP measurements between nurses and ophthalmologists was in a patient in whom the nurse measured an IOP of 18 mmHg and the ophthalmologist an IOP of 30 mmHg. The ORA IOPg was 25.9 mmHg and the IOPcc was 27.8 mmHg. Missing an IOP of 30 mmHg could have had significant implications for management decisions for this patient. Furthermore, the greatest difference in IOP measurements between ORA IOPg and ophthalmologists using GAT was 7.5 mmHg in a patient in whom the ophthalmologist obtained an IOP of 14 mmHg, whilst the IOPg reading was only 6.5 mmHg and the IOPcc only 8.2 mmHg (wave score = 7.3). The repeat ORA measurements in this patient were also similarly low (IOPg 6.7 mmHg and IOPcc 8.8 mmHg, wave score = 8.5). The nurse obtained an IOP of 20 mmHg. These differences may relate to true IOP fluctuations or measurement error and highlight limitations of relying on isolated measurements.

There were a number of limitations in this study. First, we did not examine the repeatability of the ophthalmologist or nurse GAT measurements independently but considered GAT performed by nurses and ophthalmologists as repeat measures for calculation of the ICC. We were therefore not able to determine whether nurses or ophthalmologists obtained more reproducible measures. Furthermore, we did not use a two-person masked technique when measuring GAT. However, this meant that the study was more of a reflection of real clinical practice. Finally, we did not collect information on the type of glaucoma or stage of disease, which may have been useful to investigate as confounding factors in agreement between devices.

In summary, a key requirement of clinical measures in virtual clinics is that they are repeatable to provide the remote ophthalmologist with reliable data. This study has shown that under normal clinical conditions measurements from GAT only have moderate agreement when performed by different operators. In contrast, repeat ORA IOPg measurements were more consistent, suggesting, along with its other advantages, that the ORA IOPg may be a more reliable tool for IOP assessment in virtual clinics. For eyes with extremes of corneal thickness on the other hand, ORA IOPcc appears to remain a more appropriate alternative as it was not statistically affected by CCT, whereas both ORA IOPg and nurse GAT measurements increased statistically with corneal thickness.

Summary

What was known before

  • The Goldmann Applanation Tonometer (GAT) is the gold-standard tool for measuring intraocular pressure (IOP).

  • The Ocular Response Analyzer is an alternative, automated tool.

  • Patients with ocular hypertension and glaucoma are increasingly reviewed in virtual clinics where their IOP is measured by an ophthalmic nurse to be reviewed by a remote ophthalmologist.

What this study adds

  • Tatham and co-workers found only moderate agreement between IOP measured by ophthalmologists and ophthalmic nurses using the Goldmann Applanation Tonometer.

  • The Ocular Response Analyzer (ORA) demonstrated greater repeatability suggesting more reliable IOP assessment in virtual clinics.