Introduction

Telemedicine refers to the practice of medicine from a distance, whereby interventions, diagnoses, and treatment decisions and recommendations are based on data, documents, and other information transmitted through telecommunication systems [1]. Technology currently allows the medical virtualist to interact with the patient in real time and perform tests and clinical examinations remotely, using tools such as high-resolution video streaming and remotely controlled medical equipment [2]. These tools increase clinician autonomy in the practice of telemedicine; however, as the physician is not physically present with the patient, on-site assistance from another professional is usually necessary to provide care. In this modality of telemedicine, the physician–patient interaction is mediated with the help of a nurse, technician, or other person qualified to collect, send, and receive health information.

There is an extensive published experience with an ophthalmic imaging acquired by allied health professionals, including in large-scale clinical trials such as the Early Treatment Diabetic Retinopathy Study [3] and the Age-Related Eye Disease Study (AREDS) [4]. Professionals responsible for acquiring ophthalmic imaging are ophthalmic photographers, imaging technicians, or even ophthalmologists in the absence of the former professionals. In telemedicine services, however, a nurse or a nurse technician is usually the one physically present with the patient, and he/she is required to play a versatile role in collecting all kinds of clinical data to be transferred to a physician. In addition, telemedicine strategies frequently operate in remote or rural communities, where a specialized professional is not expected to be available. Rather, local, general nurse practitioners are trained to collect data and to serve as an interface between their community and a central, specialized reading center.

On the physician side, at the reading center, high-quality images as expected in order to provide reliable readings. Since, when practicing telemedicine, physicians base their diagnosis on data received from a third party, it is reasonable to question whether the acquisition of such data by specialized personnel would confer greater diagnostic reliability. To the best of our knowledge, no previous study has evaluated whether the images acquired by nurse technicians are of comparable quality to those acquired by a specialist. This is a particularly relevant research question given the emergent use of teleophthalmology and the need for a more robust evidence base on the professionals involved in the various stages of the telemedical workflow. Within this context, the objective of this study is to compare the quality of ophthalmic imaging acquired by an on-site ophthalmologist to those acquired in a telemedicine strategy—namely, by an on-site nurse technician working under the remote supervision of an ophthalmologist.

Methods

A cross-sectional study was performed on a sample of patients participating in the TeleOftalmo project. TeleOftalmo is a telemedicine research and healthcare delivery project that has been ongoing since July 2017 in the state of Rio Grande do Sul (RS), Brazil, and receives major funding from the Brazilian Ministry of Health through the Unified Health System Development Support Program (PROADI-SUS). It consists of a command center headquartered at the Universidade Federal do Rio Grande do Sul Telehealth Center (TelessaúdeRS-UFRGS) and eight remote examination rooms across the state [5].

Primary care physicians refer patients with eye complaints to participate in the project. The inclusion criteria are: low visual acuity (not otherwise specified), refractive error, strabismus, eyelid disorders, conjunctival lesions, cataract without prior indications for surgery, and screening for diabetic retinopathy. Patients who meet the criteria are scheduled to undergo examination at the remote room closest to their home address. On the arranged date, the patient is seen by a specially trained nursing team. The ophthalmologist at the command center has a real-time audio and video link to the remote examination room, and oversees collection of the following data by the local team: visual acuity, automated refraction, air-puff tonometry, slit-lamp photography, and fundus photography.

The team consists of one registered nurse and five nurse technicians. TeleOftalmo training consists of a 1-week theoretical and practical course led by ophthalmologists and the project’s head nurse, followed by a period of work under the supervision of the head nurse. Furthermore, the team received specific additional training on slit-lamp photography, provided by an ophthalmic technologist specialized in ocular photography.

All participants for the present study were seen at the remote examination room of Hospital Restinga e Extremo Sul, located in the city of Porto Alegre. Sampling occurred sequentially and included all patients who attended the data collection site during the study period. The sample size was calculated a priori, considering the objective of estimating the kappa coefficient of agreement for classification of images collected by telemedicine versus by an on-site ophthalmologist. For a statistical power of 80%, concordant proportion of 85%, and expected agreement coefficient of 0.90, the sample size was estimated at 107 images of each type. Accounting for 20% loss, the required number of images was estimated at 128 for each type.

All participants underwent two identical sets of data collection: face-to-face (or “on-site”) and telemedicine. Face-to-face data collection was performed by an on-site ophthalmologist (ADS) trained in standardized TeleOftalmo data collection procedures, which included practical training in equipment operation. Telemedicine data collection was performed by the nurse technicians as usual, in the remote examination room, under the supervision of an off-site ophthalmologist (ALA, PBG, or ALL). The off-site ophthalmologist remotely evaluates the images simultaneously or immediately after collection, and may request recapture of any images deemed unsatisfactory for reading. The TeleOftalmo project had been operational for 4 months when the present study began. Both sets of data collection were performed on the same day, in random order, by applying a randomization table to the schedule of the day’s appointments so that some patients would undergo face-to-face data collection first, while others would undergo remote collection first. The hospital where data collection was performed is equipped with identical remote examination rooms, which allowed both face-to-face and remote image acquisition to be performed simultaneously; the participants simply moved from one room to the other according to the preset randomization plan. The plan consisted of a table with two groups (face-to-face and remote), generated online at www.randomization.com. The randomization table was printed, and appointment schedules were manually added to the table over the course of the study period.

Each data collection set comprised the five standard images acquired during a TeleOftalmo examination: three slit-lamp photos of the anterior segment of the eye (SL 115, Zeiss, Oberkochen, Germany) and two fundus photographs (Visucam 224, Zeiss, Oberkochen, Germany). The slit-lamp photos consist of one view of the anterior chamber with diffuse illumination (SL1), one centered on the pupil (SL2), and one of the anterior chamber periphery (SL3); the fundus photographs consist of one centered on the macula (RetM) and one centered on the optic disc (RetD). Images are described in greater detail in Table 1, and examples of the slit-lamp photos are provided in Fig. 1. All images were stored on a central server by the research team, anonymized by metadata removal, and made available to the evaluators on a cloud server established exclusively for this research project. The images were evaluated by two outside ophthalmologists (ALFAC and TGSM) with no relation to the project, who have extensive experience in eye imaging. They are board-certified ophthalmologists with more than 10 years of practice. They are also PhD candidates in Ophthalmology in Visual Sciences at Federal University of São Paulo, in Brazil. Their PhD work involves studying of eye image used to train artificial intelligence algorithms. For the present study, they were informed in detail by the leading investigator about image characteristics and structures to be evaluated. A small set of images was provided for training, while further instruction was provided as needed during training.

Table 1 Image types, acquisition techniques, and anatomic structures of interest.
Full size table
Fig. 1: Slit-lamp images acquired in the study.
figure 1

Slit-lamp 1 (SL1) is a view of the anterior chamber with diffuse illumination, slit-lamp 2 (SL2) is a photo centered on the pupil, and slit-lamp 3 (SL3) is a photo of the anterior chamber periphery.

Full size image

Both outside evaluators were unaware as to which class of provider (on-site ophthalmologist or telemedicine team) had acquired each image. The evaluation consisted solely of classifying each image as suitable or unsuitable for visualization of the structures of interest, as described in Table 1. Unsuitable images were categorized as “unreadable” when no structure was visible or “reading impaired” when any of the following aspects were present: out of focus, overexposure, underexposure, off-center, color shift, excess artefacts, or others. Image set was divided between graders, so each image was evaluated only once.

Statistical analysis was performed in PASW Statistics, Version 18.0. The chi-square test was used to ascertain the proportion of suitable images in each group. Kappa agreement was calculated with the correction proposed by Gwet [6] due to the high proportion of concordants and heterogeneous marginals.

The research followed the tenets of the Declaration of Helsinki. Informed consent was obtained from the subjects after explanation of the nature and possible consequences of the study. The research was approved by the Hospital de Clínicas de Porto Alegre Research Ethics Committee.

Results

We analyzed 2044 images acquired from 118 patients, with a mean age of 51 ± 19.5 years, of whom 76 (64.4%) were female. Overall, 2048 images were acquired, of which 4 (0.002%) were lost: two forms were returned blank by evaluators (one from each group), and their matched pairs from the corresponding group were excluded from analysis.

The proportion of concordant classification between images acquired by the on-site ophthalmologist and those acquired by the telemedicine team was 94.8%. The corrected kappa agreement for overall assessment was 0.94 (P = 0.01).

Images were classified as suitable in 965/1022 (94.4%) and 958/1022 (93.7%) of cases for the on-site ophthalmologist and telemedicine team, respectively (P = 0.512). Table 2 lists the classification for each type of image. The proportion of images classified as suitable for reading did not differ significantly between groups for SL1, SL3, RetM, and RetD (P = 0.825, P = 0.997, P = 0.194, and P = 0.449, respectively). A greater number of suitable SL2 photos were acquired in the face-to-face group than in the telemedicine group (P = 0.001); this means that one additional unsuitable image was acquired by the technician for every 17 patients examined.

Table 2 Images classified as suitable (by image type) in the on-site and telemedicine groups.
Full size table

Images were classified as unsuitable for reading in 121 cases (0.06%): 109 as “reading impaired” and 12 as “unreadable”. All unreadable images were fundus photographs, either centered on the macula or optic disc, from eyes with dense cataracts (the presence of cataracts was detected from the slit-lamp imaging of the same eye).

The distribution of unsuitable images across the two groups and the factors that impaired readability are described in Table 3. The overall frequency of unsuitable images was similar between groups.

Table 3 Images classified as unsuitable (unreadable or reading impaired) in the on-site and telemedicine groups. Values given as absolute number of images (percentage in parentheses).
Full size table

Discussion

This study demonstrated high agreement and a high overall quality of ophthalmic photographs acquired by trained nurse technicians in a telemedicine strategy. The fact that ~95% of images were of appropriate quality demonstrates a low failure rate, with similar results between ophthalmologists and nurse technicians working under the supervision of a remote specialist. Only for one image subtype did quality differ between the two groups (being lower in the telemedicine group). Nevertheless, suitability of this image type in the telemedicine group was high (above 93%); thus, even if there is a statistical difference, the clinical relevance of this difference should be negligible.

Previous studies have reported a similar proportion of fundus photographs of sufficient quality for analysis. Reported rates of unsuitable images range from 6.2 [7] to 8.7% [8], reaching 14.8% when only nondilated fundus photographs are obtained [8]. Low-quality photos may require repeat examination or referral of the patient elsewhere for face-to-face examination, depending on the workflow established by each facility. Within the TeleOftalmo program, in case of poor-quality images, the remote ophthalmologist may request real-time remote observation of slit-lamp biomicroscopy and, in the case of fundus photography, request sequential image reacquisition and reevaluation, as we use a synchronous telemedicine strategy. If image quality remains unsuitable, the patient is referred for a face-to-face eye exam with an ophthalmologist.

When the quality of each image type was compared in isolation, photos of the lens had the lowest proportion of suitable images in the telemedicine group. The use of standardized slit-lamp photography for cataract grading is a classic method that has long been used in large clinical studies [9, 10]. However, factors that hinder the acquisition of such images include the dexterity and practice necessary for use of the slit lamp, extensive anatomical knowledge of the eye, and ability to observe its structures in depth, as the lens is located behind the iris. The second phase of the AREDS study (AREDS2) abandoned the use of such images altogether and relied on fundus photography alone, given the specificities of slit-lamp image acquisition [11]. Fundus photographs can demonstrate cortical and subcapsular lens opacities, which appear as segmental defects in the red reflex. Nuclear cataracts, in turn, are best observed under slit-lamp illumination [11, 12], which is why we chose to acquire these images for the TeleOftalmo project. Despite the smaller proportion of images suitable for reading in the telemedicine acquisition group, the proportion of suitable images was still 93.8%, which represents a large majority of usable images for reading, grading, and reporting.

Two limitations of this study must be highlighted. The first concerns the limited experience of the nurse technicians with image acquisition. At the time of study initiation, the TeleOftalmo project had only been in operation for 4 months. Therefore, technicians’ performance may have been negatively affected by their relatively brief experience with this activity. The second limitation is that only one ophthalmologist performed on-site image capture. There were no ophthalmic photographers in our setting to participate in this study. The participation of two or more providers might have reduced any biases associated with having this task performed by a single person.

Virtual clinics operated locally by technicians have been used to provide eye check-ups in primary care [13], to screen cases for referral to tertiary ophthalmology care [14, 15], and for follow-up of glaucoma [16]. In describing these strategies, however, authors have rarely reported the qualifications of the technicians involved, the training they received, and the accreditation system used (if any).

Ophthalmic photographers/imagers are allied personnel whose training may follow different routes. Ophthalmic imaging technicians may be trained and certified locally, although national or international training schemes are emerging [17, 18]. In North America, the Joint Commission on Allied Health Personnel in Ophthalmology (JCAHPO) [18] provides reference standards for the training and continuing education of allied ophthalmic personnel, as well as promotes the establishment of formal training programs through the development and implementation of global accreditation standards. In Brazil, there is no specific occupation for allied ophthalmic personnel—a nurse technician who has received on-the-job training usually performs this role. Our study’s results may serve as parameters for the development of a system of continuous quality monitoring or provision of certification.

An additional perspective involves the artificial intelligence tools that are increasingly being used to automate the image quality assessment and decision-making process [19, 20]. A convolutional neural network-trained algorithm for classifying fundus photographs as “Accept” and “Reject” showed 97% agreement with a human evaluator when evaluating image quality for staging diabetic retinopathy [19]. A more detailed technique defined five requirements for retinal image quality assessment (RIQA): sharpness, illumination, homogeneity, field definition, and content. RIQA algorithms demonstrated an area under the ROC curve >0.99 when tested on different public retinal image databases [20]. Automated systems, once widely available, may flag the need for a new capture immediately after image acquisition, while pointing out what needs to be improved on the next shot.

The present study demonstrated a high proportion of suitable images acquired by nurse technicians under the remote supervision of a specialist physician, with adequate quality as compared with images acquired by an on-site ophthalmologist (our reference standard). These findings provide favorable evidence of teleophthalmology programs that are locally operated by technicians in expanding ophthalmic expertise through remote evaluation of images. The scalability of teleophthalmology as a means of care delivery will require the participation of allied health personnel, with an understanding of quality and its regular control. Technicians or nurses should have adequate training to enhance local acquisition of images that are transmitted elsewhere for interpretation and action as necessary. Ophthalmologists reading image and other data can promptly detect and indicate referral of sight-threatening conditions. The success rates achieved herein may serve as a reference for periodic assessments of teleophthalmology services, with a view to ensuring the safe practice of high-quality telemedicine.

Summary

What was known before

  • Image quality is essential for remote diagnosis in teleophthalmology programs.

  • Healthcare professionals that do not have formal education in eye care, such as primary care nurses or health assistants, are frequently assigned to take eye photographs for telemedicine. Although it is an established practice, the quality of these images has not been properly studied.

What this study adds

  • Slit-lamp images and retinographies acquired by trained nurse technicians were found to be adequate for remote reading in more than 90% of our sample.

  • Fundus and anterior segment images presented a similar proportion of adequacy as compared with the same images taken by an ophthalmologist, except for slit-lamp views of the lens.