Introduction

An increasing number of mobile apps addressing health topics have become available recently. Currently, the public app stores of Apple and Google offer more than one hundred thousand health apps, each with more than four million downloads per day. [1] The plethora of available mobile apps and the euphoric speculations about their future use stand in sharp contrast to the available evidence assessing usefulness, efficacy, and applicability of the technology. In recent survey of the medical literature, the number of reports on clinical efficacy and usability was disappointingly low. [2] While reports on usability stressed the importance of user-friendliness without providing original data for specific health apps [3], the efficacy assessment was limited to a few chronic illnesses such as diabetes [4].

Understanding the users’ needs, capabilities and expectations are of tantamount importance to assure regular use [5, 6]. Ignoring them may affect efficacy (internal validation) and also usefulness of an application (app). Particularly when designing health apps for an elderly population, tools should acknowledge that users face a progressive degeneration of various abilities, i.e., memory functions, spatial abilities or a decrease in flexibility of finger joints and sight [5, 7].

Over a period of several years, we developed and validated the home monitoring device Alleye [8,9,10]. This is a CE (European Conformity) marked and US FDA (Food and Drug Administration) cleared patient self-monitoring test (Alleye, Oculocare medical Inc.) indicated for the detection and characterization of metamorphopsia in those with the capabilities to perform a simple test on their mobile device in the home. The App involves a simple monocular hyperacuity alignment task with individual scores provided at each testing. The task is designed to capture dynamic fluctuations in visual acuity out of hospital, and to identify the need for prompt attention from hospital eye services, or indeed indicate the safety of a later hospital visit.

In a previous analysis, we assessed the impact of changes in the user interface on the system usability scale (SUS) in a small group of patients with wet AMD [11]. The results of that study were promising. To understand the long-term effects of Alleye use, this study evaluated the usability and long-term adherence to Alleye. More specifically, we assessed the overall satisfaction with the App over time and the frequency of testing of individual subjects during follow-up.

Methods

This study received the approval of the Ethics Committee Northwestern and Central Switzerland (EKNZ) (2018–00878) and was performed according to the standards of good clinical practice. All participants gave written consent.

Study design and setting

This study was designed as a prospective follow-up study with consecutive patient enrolment. All patients attending an ophthalmological consultation from September 15, 2018 onwards, at the Retina Centre of the Cantonal Hospital of Lucerne’s Eye Clinic scheduled for anti-VEGF treatment due to retinal pathology were screened for inclusion in this study.

Inclusion and exclusion criteria

To be included in this study, patients needed to have cases of active nvAMD, DME, or another retinal condition requiring anti-VEGF treatment, either newly diagnosed (treatment-naive) or treated with anti–VEGF agents (ranibizumab (Lucentis ®) or aflibercept (Eylea ®)). We excluded patients with a mental or physical illness that impeded from performing the test adequately. Patients scheduled for anti-VEGF injections were approached by the ophthalmologist who checked the inclusion criteria. In the positive case, he or she provided detailed oral and written information about the study and asked whether the patient was willing to participate. Participating patients provided written informed consent to participation and agreed to the publication of data.

Examination setting

Salient clinical characteristics of each patient were secured. The first test with the Alleye was then conducted between the ophthalmological examination and the intravitreal injection. After receiving a careful instruction how to perform the test using a standardised protocol, each patient performed the test on its own. If needed, the study coordinator provided technical assistance. The same study coordinator always provided these assistance services and registered the necessity to do so. All participants performed the test wearing their usual corrected spectacles with un-dilated pupils. The test was a monocular, i.e., eyes were tested separately, with the use of an eye patch. Subsequently, patients either conducted the test at home on an iPod touch fifth generation (Apple Inc.) that was provided by the clinic or on their mobile device after downloading the app from Apple’s App store or Google play. If any problems with the handling of the app or iPod occurred, patients were advised to contact the research assistant by phone or return to the clinic for a tutorial. Approximately one month after inclusion, each participant completed the SUS questionnaire and five additional questions regarding his or her views on the app and frequency of testing (see Appendix). This was considered the baseline assessment.

The Alleye app

The Alleye is a mobile medical software application indicated for the detection and characterization of metamorphopsia, a visual distortion, in patients with retinal conditions and as an aid in the monitoring of the progression of this condition with respect to metamorphopsia. It is intended to be used by patients who have the capability to regularly perform a simple self-test at home. Alleye consists of two different elements: a mobile app for patients and a web interface for eye care professionals. Alleye implements an alignment hyperacuity task that helps patients with retinal conditions to assess their visual function at home. This allows the timely detection of significant changes in vision function, enabling the regular monitoring of the disease progression and/or monitoring of visual function during ongoing treatments (summarized in [8,9,10, 12]).

A usability questionnaire: system usability scale

The SUS was originally developed by John Brooke in 1986 for the evaluation of usability of systems such as, mobile devices, websites as well as hard- and software [13]. The SUS is a valid and reliable tool for measuring usability [13]. It consists of ten items with response options each on a Likert scale ranging from 'one strongly agree’ to 'five strongly disagree' (see Appendix). Participant’s scores have to be converted for each question, added, and multiplied by 2.5. Consequently, original scores are converted from a scale of zero to 40 into values varying from zero to 100. The score may not be seen as percentages—it should be considered in terms of percentile ranking [13]. Sauro and colleagues proposed that a SUS score higher than 68 should be considered above average [14]. Recent research showed that the SUS provides a global measure of system satisfaction as well as additional sub-scales for usability and learnability [15]. The German version of the SUS was developed and validated using state-of-the-art methods [16].

Because the SUS is now over 35 years old and freely available, it has been used in a large number of studies. Bangor and colleagues published the results of studies on many different systems and technologies in their studies [17, 18]. In their analysis of over 3000 SUS measurements, they examined the relationship between SUS results and ratings of systems and products using the terms 'good', 'poor', or 'excellent' [17]. They found a close correlation and therefore suggested using the SUS rating for a given product for grading purposes.

Patients taking part in our clinical study were native German speakers and we, therefore, used the German version of the SUS, adding some specific questions in the end that we took a special interest in. The additional questions were evaluated separately. Participants completed the questionnaires independently in the waiting room after the consultation and then handed it over at the patient counter.

Statistical analysis

We calculated changes of SUS, personal ratings of usefulness, and number of tests performed per month, from baseline and the last follow-up visit at a maximum of 18 months. Using multivariable regression analysis, we investigated the association between baseline characteristics and study retention. Statistics were performed using the Stata 16.1 statistics software package (StataCorp. 2019. Stata Statistical Software: Release 16. College Station, TX: StataCorp LLC.) and a p-value of less than 5 percent was considered statistically significant.

Results

Patients’ characteristics

Between September 2018 and Mai 2019, we enroled 72 patients (95 eyes) at the Cantonal Hospital in Lucerne, Switzerland. The majority of eyes were treated for wet AMD (48/95; 50.5%) and 16 eyes (16.8%) had diabetic macular oedema (DME). A miscellaneous group included 31 eyes (32.6%). At baseline, mean best corrected visual acuity was 74.9 letters (SD 14.8) and did not differ between groups. Mean age was 69.9 (SD 11.4) and 39/72 (54.2%) were female. The group of patients with DME was significantly younger than the group with AMD (−10.7 years) (95%CI: −16.2 to 5.3; p < 0.001). Of included patients, 47/72 (65.2%) reported to use mobile devices daily. Patients who dropped out of the study (n = 19) did not differ in age, gender, BCVA, and SUS at baseline, but stated that they did not use the mobile device daily (Odds Ratio 7.40 (95%CI: 2.32–23.65); p = 0.001).

Findings during follow-up

Clinical characteristics

The overall retention rate until last follow-up was 73.6 % (53/72) and was similar between the clinical groups. The mean BCVA, while under a treat and extend therapy scheme, remained stable in retaining patients. (+0.7 letters (95%CI: −1.2 to 2.6); p = 0.473). Compared to eyes with wet AMD, we saw a weak trend towards higher improvements of BCVA in DME eyes (+2.5 letters (95%CI: −2.4 to 7.4); p = 0.305) among patients remaining on home monitoring until the final follow-up. Age at baseline was not associated with changes of BCVA during the observation period (p = 0.766) and no interaction with macular pathology (AMD: p = 0.768, DME: p = 0.622) was seen.

System Usability Score

The median SUS score at baseline was 90 (interquartile range (IQR) 82.5–95) and 92.5 (IQR 82.5–95) in the last follow-up. Among patients remaining on the program, those with AMD had a slightly lower SUS score at follow-up (-2.2 (95%CI: −6.4 to 2.0); p = 0.298). In patients with DME we observed a slight increase (+1.6 (95%CI: −10.4 to 13.7); p = 0.777) as well as in the miscellaneous group (+4.1 (95%CI: −1.0 to 9.3); p = 0.109).

At baseline, female gender was the only parameter associated with higher SUS values (+6.1 (95%CI: 0.40 to 11.7); p = 0.037) in multivariate analysis assessing age, diagnostic group, and BCVA beside female gender. In the last follow-up, patients with diabetic maculopathy had significantly lower SUS values than patients with wet AMD (−12.4 (95%CI: −21.8 to -3.0); p = 0.010). In patients providing both baseline and follow-up SUS results, 63.6% remained unchanged or reported higher SUS in the follow-up. Changes in the SUS were not associated with female gender (p = 0.189), age (p = 0.394), changes of BCVA from baseline (p = 0.150) or diagnostic group (AMD vs. DME (p = 0.398), AMD vs. miscellaneous (p = 0.116)).

Patients’ preferences

At baseline, 76.4% (55/72) stated that they would recommend the app to a friend and 75.5% confirmed this in the last follow-up. Also, 83.3% (60/72) stated to be very satisfied with the app and 83.0% confirmed this at follow-up. Similarly, 58/72 (80.6%) of respondents said they trusted the Alleye app and 86.8% confirmed this at follow-up.

Additional comments regarding patients’ views were uncommonly reported. If available, they expressed either enthusiasm regarding the usefulness of Alleye particularly vis-à-vis the Amsler grid (n = 4), or were concerned about the clinical meaning of low Alleye scores (n = 3). One participant gave feedback on changes in the user interface (larger arrows for navigating the middle dot) and another participant commented on the navigation on the app (to be able to return to the main menu at any time during the test)

Discussion

Main findings

The majority of users willing to perform home monitoring with the Alleye app are satisfied with the usability and have a positive attitude towards its trustworthiness and usefulness. On average, the usability of the app scored sufficiently high on the validated questionnaire (SUS). Additional questions asking about attitudes and preferences revealed high overall satisfaction at baseline and even some improvements in the follow-up. Additional comments on the interface and navigation were uncommon indicating that the current version of the app is satisfactory.

Results in context of the existing literature

The availability of biologicals delaying the progression and stabilizing the course of nvAMD for a certain period of time has revived the discussion about the relevance of sensitive monitoring of nvAMD. There exists a growing need for technologies, staff, infrastructure, and resources assuring quality of care for chronic diseases such as nvAMD and DME due to the aging baby-boomer generation and the consequential demographic displacement, within developed countries. Consequently, the overburdened healthcare system will not be able to ensure an optimal frequency of retinal monitoring for long. Markun and colleagues, investigating the quality of healthcare in terms of coordination and follow-up of a chronic disease in the elderly, reported that ophthalmologists deliver only suboptimal care [19]. However, the demographic displacement makes it crucial for society to ensure that people remain healthy, and autonomously until high age [20]. In this context, various authors think of Mobile Patient Monitoring Applications (MPMA) as “promising tools” for the elderly to support their autonomy, their quality of life as well as their disease self-management and hope that they will provide a solution to the obstacle of a growing number of patients in need of healthcare delivery meeting a decreasing number of medical professionals [7, 21,22,23].

In this regard, it will be crucial that patients are provided with an affordable user-friendly and mobile tool to support optimal treatment success in chronical retinal conditions. Particularly, Loewenstein et al. showed in a randomized controlled trial, using their nvAMD home monitoring system (ForeseeHome®, Notal Vision Ltd, Tel Aviv, Israel) that visual impairment is lower compared to standard usual care [24,25,26]. However, ForeseeHome® was criticized because of the lack of usability, making the tool less valid in the hands of the patient [1, 27]. Nowadays, the most frequently recommended self-monitoring test in clinical practice is still the Amsler grid, even though its sensitivity and its validity in daily routine has, due to its low usability, also been doubted [28,29,30]. The absence of a valid and simple test for patients will now and even more in the future lead to unnecessary vision loss.

On the other hand, Kaiser and co-workers showed that the generation of elderly people to date is capable, willing and compliant to use technological tools with appropriate guidance and instructions [1]. Further evidence suggests that older people want to interact with contemporary technology in order to stay active but simultaneously feel that it is not yet suitably designed for their needs [5, 6]. A research group showed in 2014 that around eighty percent of people with visual impairment used smartphones and that the majority of them also used apps [25, 31, 32]. However, the lack of a scientific basis, in terms of usability and efficacy for elderly people, makes it impossible for doctors to recommend them to their elderly patients. Interestingly, our study showed a fairly strong relationship between study retention and the frequency of using the mobile device. We found that patients using the mobile device on a daily basis at baseline were about seven times more likely to remain in the study. Other studies examining the benefits of Alleye found that about 4 in 10 patients who would benefit from home monitoring are actually motivated to do it regularly [33, 34]. Accordingly, it must be assumed that the population included in this study represents a selection. The reasons why many patients do not want to do home monitoring remain ill-understood. We suspect, and the data from this study support this, that a minimal routine in the use of digital products is required for patients to be motivated to participate. Accordingly, we suspect that the proportion of home monitoring participants will grow in the coming years. Nevertheless, it is necessary to investigate and address any barriers to use.

Recently, the Health Care Information and Management Systems Society (HIMMS) developed new design guidelines for health apps in general [35]. Unfortunately, these guidelines do not address aging. Gao et al. claimed in 2010 that, while modern information technology products change the lifestyle of younger generations, they have yet much less impact on the current generation of elderly people [36]. Consequently, use of modern mobile devices has been shown to remain relatively constant until the age of 65, but afterward, it has been shown to decline [28]. Wildenbos and co-authors described potential underlying characteristics of elderly people, such as decreased learning aptitude, and cognitive capacity as well as barriers of perception and motivation that modify requirements on user-friendliness [7, 37,38,39].

Concerning a design meeting the needs of elderly people, several authors suggested to prioritize strong back-lights, strong colours, high contrasts and colour schemes, large and clear typefaces on big screens as well as big buttons and redundant user interfaces [6]. 25Lorenz and Oppermann [40] proposed using font sizes between 36ppt and 48ppt and specific button placements at the bottom of the interface in a way that input-hands will not hide screens. Furthermore, Holzinger and colleagues [23] discussed vision-specific changes of aging that may influence the user-app interaction. They listed factors such the visual acuity, accommodation, colour vision, contrast detection, dark adaptation and the susceptibility to glare. They stated already in 2007 that with the correct interface design, the restricted eyesight should not be an obstacle in the use of health apps by patients suffering from nvAMD [23]. Apps will have to meet the needs of the end-user, especially in the elderly where fears of being unable to use modern technology is prevalent. Developers could help them to accept mobile apps without reservations by creating appropriate designs. [26] In agreement with previous authors, we would like to recommend involving the end-users in an iterative developmental process of health apps [41].

Strength and Limitations

The strength of this study is its prospective design involving a typical cohort of patients with a retinal condition. Considering the exploratory nature of this analysis, we did not perform a formal sample size calculation [42]. Another possible drawback is the relatively high percentage of non-adhering patients. These patients either died, had a case of severe illness impeding them from continuing treatment or preferred to stop active treatment. However, other real-world studies reported dropouts of 19% within 12 months and 46% to 57% within 5 years [43,44,45]. Arguably, the sample of patients studied in this paper represents a selection, as they consented to participate in a long-term follow-up study and were willing to perform home monitoring on a regular basis. Finally, while the SUS questionnaire is a validated instrument, the additional questions used in the questionnaire have not been developed and validated in a principled way. Although the constructs addressed in the questions are straightforward, there is a residual risk of misunderstanding and confusion that could have biased the answers. For example, it is unclear whether patients trusted Alleye to measure vision deterioration or whether other features of the app were meant. However, as the same questionnaire was provided at various occasions, we were able to assess intra-individual variability. For questions such as whether or not they would recommend the Alleye app to a friend, the reply patterns remained fairly consistent between the six and the 12 months follow-up assessment. This reassures us that the face validity of those additional questions was sufficient.

Implications for research and practice

The evidence presented in this paper suggests that patients with a macular pathology, who are willing to participate in a hyperacuity home monitoring programme, are satisfied with the usability and usefulness of the app. Interestingly, those patients who were more acquainted to use a mobile device on a daily basis prior to entering the study, were more likely to remain in it. From a clinical perspective, the simple question: “How often do you use a mobile device in daily life?” could be useful to triage those patients to whom the home monitoring should be offered in the first place. Further research could investigate, to what extent training methods, including self-directed protocols, technical assistance, or tutorials available on the app could have to improve the SUS and the adherence to sustained home monitoring. Adopting the role model of patient-centred diabetes care [46] and chronic care models in general [47], we believe that involving patients in the early detection and management of chronic diseases will have a major impact on quality of life and healthcare costs.

Conclusions

The results of this long-term adherence to Alleye in the monitoring of retinal pathology showed high usability on a validated questionnaire. The majority of users willing to perform home monitoring with the Alleye app had a positive attitude towards its trustworthiness and usefulness.

What was known before

  • Currently, the public app stores of Apple and Google offer more than one hundred thousand health apps, each with more than four million downloads per day. The plethora of available mobile apps and the euphoric speculations about their future use stand in sharp contrast to the available evidence assessing usefulness, efficacy, and applicability of the technology

What this study adds

  • The majority of users willing to perform home monitoring with the Alleye app are satisfied with the usability and have a positive attitude towards its trustworthiness and usefulness. On average, the usability of the app scored sufficiently high on the validated questionnaire (SUS). Additional questions asking about attitudes and preferences revealed high overall satisfaction at baseline and even some improvements in the follow-up.