Background

The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic of 2020 has fuelled a discussion regarding the provision of comprehensive care for macular disease patients whilst limiting face-to-face visits. Globally, most health systems are not yet in a position to conduct a post mortem into the consequences of postponed and cancelled appointments, affecting up to 80% of patients who attend eye clinics [1]. Eye care professionals worldwide have been challenged to risk stratify patients using information available from health records of a patients’ last visit. The key limitation of this forward triage strategy is that the clinical information used for decision making may be incomplete or out-of-date. The use of telemedicine as a strategy to collect more contemporaneous clinical data has seen a sharp rise across all specialties in healthcare with uptake of schemes such as video-consultations and remote monitoring [2,3,4,5]. Although this addresses the priority in which patients should be seen whilst conserving resources in the time of crisis, there remains the question- what is a safe follow-up interval between hospital visits?

In Ophthalmology, the lack of clinic capacity for the monitoring and treatment of chronic eye diseases such as glaucoma, diabetic macular oedema (DMO) and age-related macular degeneration (AMD) is a long-standing issue, particularly with the reliance on imaging assessments requiring specialist equipment such as visual fields analysers and optical coherence tomography (OCT). In recent years, the introduction of Virtual Clinics and new therapeutic approaches, aimed at extending the intervals between treatments looks to address this. Home-monitoring as an adjunct could provide “in-between visit” clinical data that may be used to personalise a patient’s follow-up interval according to need.

Today, valid home monitoring of visual function could be achieved by software with high usability running on patients’ mobile devices [6]. Our current offering to patients for home monitoring of their eye condition consists of the paper-based Amsler grid, despite its well documented limitations [6,7,8]. Namely, technical obstacles such as difficulties with fixation in home monitoring experiments have demonstrated its limited validity in the hands of our patients, leading to a highly variable diagnostic accuracy [9].

Two Food and Drug Administration approved mobile medical apps are currently available for the home monitoring of hyperacuity: myVisiontrack (mVT) [10] and Alleye [11, 12], testing a 3 and 12.7 degree visual field, respectively. It has been shown that the assessment of hyperacuity allows for the early detection of progression in patients with exudative neovascular age-related macular degeneration (nvAMD) and DMO [12]. However, the burden generated through false alarms in the long-term monitoring situation must not be underestimated, for both healthcare professionals and patients.

In this study, we addressed the extent of this false alarm rate and the positive predictive value of regular home monitoring with Alleye in prospective follow-up study of patients currently treated with anti-angiogenic therapy in a pro re nata scheme.

Methods

Ethics, study design and setting

This prospective cohort study was performed with consecutive patient enrolment. All patients attending the injection clinic between March and June 2016 at the Medical Retina Service of the Eye Clinic of the Cantonal Hospital Lucerne were eligible to be involved. The study was approved by the local Ethics Committee. Participating patients were provided with written information, and written informed consent was obtained.

Patient recruitment and enrolment

Patients with exudative nvAMD or DMO in a pro re nata (PRN) treatment regime were provided with written information in advance of their appointment. Subsequently, the clinic research fellow assessed the selection criteria on an individual case by case basis prior to the clinic consultation. In eligible cases, patients were provided with more detailed verbal and written information regarding the study and were asked as to whether they were willing to participate.

Selection criteria

Patients with exudative nvAMD or DMO under treatment with anti-angiogenic therapy (ranibizumab (Lucentis®) or aflibercept (Eylea®)) who had completed treatment induction qualified for inclusion if they presented with a best-corrected visual acuity (BCVA) of at least 60 Early Treatment Diabetic Retinopathy Study (ETDRS) letters. If both eyes were affected by nvAMD or DMO, both eyes were included in the study. Patients with a neurological or physical condition that impeded them from using a mobile device to adequately perform the home monitoring task were excluded.

Examination setting

Relevant clinical characteristics of each patient were recorded. Following ophthalmological routine assessments, standardised and protocol-based instructions for the training of hyperacuity testing were provided by healthcare professionals. After a training period involving repeated testing, patients were asked to perform a baseline assessment in the clinic and home monitoring. Patients performed the test whilst wearing their own glasses with each eye tested individually. Home monitoring was either conducted on an iPod Touch® (Apple Inc.) that was provided by the hospital or on the patients’ mobile device once the application was downloaded from Apple’s App or the Android Google Play store. In addition, all patients received an instruction manual for the application’s usage, and healthcare professionals provided technical assistance during clinical follow-up visits if needed. Patients were asked to conduct home monitoring twice weekly and to return the device at each follow-up visit.

The home monitoring test

The home monitoring test is based on a computerised version of a hyperacuity alignment task (Fig. 1) and has been described in detail elsewhere [12]. Hyper-, or Vernier acuity is the ability to detect a misalignment that deteriorates due to morphological retinal changes causing metamorphopsia [13, 14]. Therefore, measurement of hyperacuity can provide information regarding retinal changes and even enable the quantification of metamorphopsia by presenting stimuli with artificial distortions of different amplitudes [15].

Fig. 1: The task of the Alleye test is to place the middle of three points on the invisible connecting line between the outer points.
figure 1

A total of 12 points must be placed. The Alleye test is performed with one eye. You cover the eye that you are not testing. To test, place both elbows on a table and hold your mobile device approx. 30 cm in front of your face with both hands.

Full size image

Several studies have demonstrated the superiority of active over passive hyperacuity testing (Amsler grid) in the detection of metamorphopsia [13, 16,17,18].

In this active alignment task, three black dots with a one millimetre diameter were presented on a white screen. Alignments were made on four axes; including vertical, horizontal, left and right oblique axes. Patients are tasked with aligning the randomly misaligned dot between two fixed dots to form one straight axis precisely. This is achieved through moving the central dot only. Patients are able to control the middle dot via arrow-buttons on the screen of the touch-based user interface. Once patients were satisfied with the placement of a dot, they confirmed their choice and entered the result. This task was repeated three times for each of the four axes, resulting in a total of 12 inputs per test. The test was performed monocularly after having covered the non-tested eye. Technical specifications are available elsewhere (https://alleye.io/#/product-alleye).

Assessment of disease progression

All study participants were managed in a PRN protocol and routinely returned for follow-up visits between 28 and 35 days. At this visit, patients received ETDRS BCVA testing, biomicroscopy and imaging in the form of an OCT scan. A spectral domain OCT (Spectralis® Heidelberg Engineering GmbH, Heidelberg) was used to obtain four b-scans with a length of six millimetres each. A masked, board-certified senior retinal specialist (MKS) assessed the OCT scan for progression of sub- or intraretinal fluid, and serous or fibrovascular pigment epithelial detachment. Based on this assessment, OCTs were classified as either “progression”; “stable” or “improved” compared to the previous scan. Treatment decisions and date of the intravitreal, anti-angiogenic injection were also recorded.

Statistical analysis

We defined a “monitoring period” as the time interval between two visits at the injection clinic. We excluded measurements with extreme values (changes in the score of >±50 score points from previous measurement), incomplete periods (i.e. periods that did not end with a follow-up visit) and eyes with less than one monitoring period or periods in which patients performed less than one measurement per week (8 periods (3.5%). The mean age and BCVA of patients excluded due to this was similar to the included patient groups. We also excluded measurements with a recorded length of all 12 tasks <40 s as this indicated inappropriate testing. Using these selection criteria, we removed <20% of all data points stored in the database. Continuous variables were summarised with means and standard deviations (SD) or medians and 25th–75th interquartile ranges (IQR). Dichotomous variables were described as rate and percentages.

We defined the outcome progression as “present” versus “absent” by collapsing the “stable” and “improved” category. As patients contributed a number of tests during the “follow up periods” (the time between two follow-up visits), we created a panel dataset in which we specified a user identification per eye and patient, the corresponding period, and the day within the period. The unit of observation was the period per eye. Per period and eye we obtained the score and recorded the occurrence of three sequential red results – defined as a sustained relevant drop of the score as provided by the test, which defined a positive test result or “alarm”.

We correlated the occurrence of a positive test result during a monitoring period with the outcome “progression” at the next follow-up visit. The specificity, defined as the percentage of eyes without progression in the next follow-up visit who had a negative result, the false alarm rate defined as 1-specificity and the positive predictive value, defined as the percentage of positive tests with the outcome progression were calculated. We performed subgroup analyses for patients older than 85 years and patients with DMO. Analyses were performed using the Stata 16.1 statistical software package (StataCorp. 2019. Stata Statistical Software: Release 16. College Station, TX: StataCorp LP.).

Results

Sixty-two patients were enroled in the study. Six patients (10%) were excluded over the course of the study as they either failed to provide home monitoring results or terminated the study early (within the one month minimal observation and measurement period). We analysed 73 eyes of 56 patients. The group of DMO included 13 eyes (17.8%). The mean age was 73.2 years (SD 10.7, range 47–88) and the median number of previous intravitreal injections was 13 (IQR: 5–22). The mean BCVA at study entry was 77.5 letters (SD 7.4, range 62–93) on the ETDRS chart. The DMO group was younger (−15.9 years (95% CI: −21.8 to −10.0; p < 0.001)), but BCVA at study entry was similar (−1.3 letters (95% CI: −6.3 to 3.8; p = 0.619)).

Patients contributed 2258 tests in 222 monitoring periods. Progression was observed in 141 monitoring periods (63.5%).

The average score value across all periods was 58.0 (SD 18.1) and ranged from 3 to 100 points. On average, each patient performed 6.4 tests per period and contributed 2.5 (SD 1.4) observation periods. Per period, patients performed an average of 10.8 (SD 4.8) tests. Surprisingly, 70% of patients continued with home monitoring of their macular function after the official termination of the study.

Specificity, false alarms and positive predictive value

The specificity was 93.8% (95% CI: 86.2–98.0%), the corresponding false alarm rate was 6.1% (95% CI: 2.0–13.8%). The positive predictive value (the probability that a patient with a positive result had disease progression in the next clinical follow-up visit) was 80.0% (95% CI: 59.3–93.2%).

Subgroup analyses

Elderly subjects (age >85 years at study entry) providing 16 eyes (38 observation periods) performed slightly worse. The specificity was 85.7% (95% CI: 57.2–98.2%) and the corresponding false alarm rate was 14.3% (95% CI: 1.8–42.8%). The predicted probability for a progression dropped to 60.0% (95% CI: 14.7–94.7%).

Patients performing >12 tests/period were more likely to show a false alarm (16.7% (95% CI: 3.6–41.4%). The diagnostic performance of eyes with DMO providing 51 observation periods was similar to eyes with AMD (specificity 94.4% (95% CI: 72.7–99.9%), predicted probability of progression 75.0% (95% CI: 19.4–99.4%).

Discussion

Main findings

To the best of our knowledge, this is the first study assessing the performance of smartphone-based hyperacuity assessment through home monitoring in patients with active macular disease requiring intravitreal injections. With the low threshold set of three consecutive red results deemed as a positive test result or “alarm”, home monitoring of hyperacuity showed a reassuringly low false alarm rate. A mere 6 of 100 patients with nvAMD or DMO had a false alarm during a defined monitoring period. If the test was positive, 80% of patients showed signs of anatomical progression at the next follow-up visit. In a screening environment however, a drop of 25 score points or more had a high accuracy for discriminating between disease and age-matched controls [12, 19]. Test performance characteristics were lower in older patients and those that tested more than recommended.

Strengths and limitations

The pragmatic set-up of this study has both strengths and limitations. Given that it is embedded in a real-world clinical setting, the results obtained in this study are more likely to reflect the realistic supplementary nature that home monitoring provides as auxiliary tool for patient management. Conversely, the lack of a strict protocol and the absence of other measures to improve the quality of the testing may have introduced substantial confounders to the data. There is also potential that this study may underestimate the value of home monitoring in patients with less advanced disease through imposing relatively few exclusion criteria. However, by including patients with a BCVA more than 60 letters, we selected a group that usually has a higher performance than patients with a lower visual acuity than this.

Moreover, OCT scans have not been conducted on the day of the alarm, but during the next routine visit. Therefore, we cannot exclude that the alarm signal may have preceded an actionable progression of disease (i.e. SRF or IRF).

Furthermore, OCT scan gradings were not verified by a second assessor. Gradings were performed by a masked retinal specialist as per current guidelines. However, the lack of standardisation and verification may have introduced further uncertainty into the data. These limitations may have distorted the results of this study and there is potential that they may have introduced bias.

The engagement of this elderly patient cohort with mobile technology to perform home testing, coupled with the fact a high proportion of these patients continued with home monitoring following the official cessation of the study indicates that elderly patients with impaired vision are not only highly capable of technological participation in a way that is not always acknowledged, but that they are also motivated to perform self-measurement tests of their macular function independently and to take ownership of their own eye health [9, 10]. As suggested by Thomas et al., a qualification test within the ophthalmic practice could help in selecting patients that qualify for home monitoring [20].

Implications for research

It is imperative that further evaluation of this application occurs by individuals with no affiliation to Alleye working in a multitude of differing clinical settings. In view of the results obtained here, further studies should be designed to quantify clinical efficacy when using home monitoring in the management of patients with macular disease both with and without exudative neovascularisation. Future work involving the evaluation of Alleye and home monitoring in general, particularly in the optimisation of treatment outcomes in a treat and extend management regimen and in the context of drugs with prolonged effects would be invaluable, with the promise of great potential to both patients and healthcare services. It is important to prioritise the evaluation of patient reported outcomes for those receiving care with the adjunct of home monitoring compared to standard care in health service research settings.

Implications for practice

We chose this cohort of patients under anti-angiogenic treatment in a PRN scheme due to the availability of frequent clinical follow-up data. However, in clinical practice, home monitoring may be of particular value in a treat and extend therapy protocol, where it could be used to provide immediate feedback and reassurance to patients regarding their level of macular disease, in addition to the early detection of visual deterioration, if needed.

It is important that home monitoring is viewed as an aid during the intervals between visits as it is not intended to replace clinical judgement; diagnosis and the management of clinical decision-making remain the responsibility of the eye care professional. Clinicians may decide to access individual patients’ home monitoring data via a web application as needed only, or -in high risk situations- they may indeed choose to receive notification alerts when patients generate alarms. Remote access to a patient’s home test values may therefore allow eye care professionals to remain connected with their patients from afar, enabling truly individual patient care plans to be crafted with scheduling and management decisions fine-tuned through a symbiotic relationship between doctor and patient. Furthermore, home monitoring is ideally suited for chronic care models by adequately informing and empowering patients with macular disease to self-care [21]. Improved patient education and awareness of the disease could have a major impact on the prevention of vision impairment [22].

Conclusions

The false alarm rate for the detection of disease progression in macular disease by home monitoring is low. These findings indicate that home monitoring is a useful adjunct to telemedicine in the remote management of patients with nvAMD and DMO.

Summary

What was known before

  • The lack of clinic capacity for the monitoring and treatment of chronic macular diseases such as diabetic macular oedema and age-related macular degeneration is a long-standing issue.

  • It has been shown that the assessment of hyperacuity allows for the early detection of progression in patients with macular disease.

  • Furthermore, this visual function can be monitored at home through software running on patients’ mobile devices.

What this study adds

  • The burden generated through false positives (or false alarms) in the long-term monitoring situation for both healthcare professionals and patients must not be underestimated.

  • This paper therefore reports on the false alarm rate and the positive predictive value of smartphone-based home monitoring in macular disease.

  • We found that a mere 6 of 100 patients had a false alarm during a defined monitoring period and if the test was positive, 80% of patients showed signs of anatomical progression at the next follow-up visit.

  • These findings indicate that home monitoring is a useful adjunct to telemedicine in the remote management of patients with macular disease featuring a minimal burden of false alarms.