Introduction

The global burden of diabetes is expected to rise from 285 million in 2010 to well over 439 million in 2030, and thus the burden of diabetic retinopathy (DR) will also increase.1, 2, 3, 4 Currently, 30% (28/93 million) of those with DR have vision-threatening retinopathy (VTDR).4 The burden of diabetes in lower income countries is growing even faster than in higher income countries, and this will increase the need for effective, low-cost screening programs for DR. Between 2000 and 2030, the prevalence of diabetes in India is expected to increase by 150%.2 In Africa, the number of diabetics is projected to double from 12 to 24 million between 2010 and 2030.5

Because there are well-established relatively inexpensive, effective treatments for VTDR, blindness from diabetic eye disease, if detected, can be greatly minimized.6, 7 Limited access to ophthalmologists to provide DR screening and treatment in many parts of the world is a significant hurdle to preventing needless blindness in diabetics. In India, the ophthalmologist to population ratio is 1:107 000. Although 70% of the population lives in rural areas, 70% of ophthalmologists practice in urban areas.8, 9 This discrepancy creates a significant need for outreach programs.

Remote image interpretation using non-mydriatic fundus photographs can provide access to high-quality DR screening. Three 45° images taken with table-top, non-mydriatic fundus cameras have been reported to be between 84–99% sensitive and 72–100% specific in the detection of DR (Supplementary Appendix 1).10, 11, 12 To extend the reach of DR screening to underserved areas both in the United States (US) and throughout the world, non-mydriatic portable digital technologies with WiFi connectivity need to be developed and tested.

The objective of this study is to evaluate the sensitivity and specificity of a handheld 45° non-mydriatic fundus camera (Smartscope, Optomed, Oulu, Finland) to aid in the remote diagnosis of VTDR. The images taken with the handheld fundus camera were compared both to images taken with a 45° table-top fundus camera and to the reference standard of a dilated clinical fundus examination (DFE).

Materials and methods

A single-site (Aravind Eye Care System (AECS), Pondicherry, India), prospective, clinic-based comparative instrument validation study evaluated the agreement among three non-mydriatic 45° images taken with a portable camera (Smartscope), three mydriatic 45° images taken with the same camera, three mydriatic 45° images taken with a table-top fundus camera (Topcon, Tokyo, Japan) and DFE.13 The study was approved by the Institutional Ethics Committee at AECS and the Institutional Review Board at the University of Michigan and adhered to the Standards for Reporting Diagnostic Accuracy Studies (STARD).14 Informed consent was obtained from all participants. The study was performed according to the ICH-GCP guidelines and fulfilled the tenets of the Declaration of Helsinki.

Participants

We recruited a convenience sample of diabetic patients from the retina clinic and a convenience sample of patients without diabetes from the comprehensive eye clinic (1/1/2014–1/31/2015). Subjects without diabetes were included to ensure that the portable imaging technology was adequate to rule out disease. Subjects were recruited from the retina clinic, a case-enriched population, to ensure a sufficient number of cases of VTDR to adequately assess the new portable instrument. Patients underwent complete eye examination and were excluded from the study if the fundus could not be visualized or they had undergone prior vitreoretinal laser or incisional surgery. Patients >21 who met eligibility criteria were invited to return for a study visit (Figure 1).

Figure 1
figure 1

Flow chart of patients/eyes through the study (STARD protocol).

Full size image

Sample size calculation

We conducted sample size calculations based on the desired endpoint of having a sensitivity of at least 80% as recommended by the British Diabetic Association.15 With n=103 eyes that are true positives, for any one modality, we had 80% power to reject (alpha=0.05) the null hypothesis that the sensitivity was 75% in favor of the alternate hypothesis that the sensitivity to detect VTDR was >85% with an expected margin of error of <8%.

Photography protocol

During the return study visit, non-mydriatic photographs were taken in a darkened room with the Smartscope, followed by DFE, and concluding with mydriatic photographs with the Smartscope and a standard table-top fundus camera (Topcon TRC—50DX). The Topcon table-top camera weighs 35 kg (77 pounds) and costs ~US $30 000.00. The Smartscope weighs 400 g (0.88 pounds) and costs ~US $5000.00. Images can be stored within the 8GB internal memory card or transferred to a laptop computer using bluetooth/WiFi technology for review.

A single trained ophthalmic photographer took all photographs. The photographer was provided reference photographs for quality assessment (Supplementary Appendix 2). The photographer obtained consent to practice on 75 patients (150 eyes) before the retina specialist (SS) determined that he was able to obtain sufficiently high-quality images for the study. The photographer imaged each participant until he was satisfied with image quality.

The photographer took three photographs of the following fields in each eye with each camera: (1) the posterior pole centered on the macula (macular image), corresponding to Early Treatment of Diabetic Retinopathy Study (ETDRS) photo area 1; (2) the nasal field, corresponding to ETDRS photo area 2, and (3) the superotemporal arcade, corresponding to ETDRS photo area 4. These three fields are 92% sensitive for detecting any retinopathy and 86% sensitive for detecting proliferative retinopathy compared to all seven standard ETDRS fields.10, 13 This protocol covers the fields of view recommended by the Joslin Vision Network protocol16 that is used in the US Veterans Affairs Health System.17 The photographer selected the best image from each field to use in the study.

We stored all photographs as Joint Photographic Experts Group (JPEG) files after removing all patient identifiers and assigning a unique number identifier linked to the participant’s study ID number. Images from the Smartscope were transmitted and stored in their native state with JPEG 100 quality (1280 × 960 pixels) and images from the Topcon were ~4200 × 2800 pixels post-transmission.

Clinical assessment

A trained ophthalmic technician obtained both presenting and best-corrected visual acuity (BCVA) prior to dilation. The technician recorded the participant’s age, gender, diabetic status (none, type 1 or type 2), and duration of diabetes (years).

All participants then underwent examination including slit-lamp biomicroscopy and indirect ophthalmoscopy using both a +90D and +20D lens by a single retina specialist (SS). The retina specialist noted all anterior segment findings (corneal opacities, iris neovascularization, and lens status) and assessed DR status, that is, mild, moderate, severe non-proliferative DR (NPDR), proliferative DR (PDR), and clinically significant macular edema (CSME). VTDR was defined as per the National Health Service (NHS) as severe NPDR or worse (>R2 level disease) and/or the presence of CSME (Supplementary Appendix 3).18 Cataracts were described according to Lens Opacities Classification System (LOCS) III grading.19 Pseudophakic or aphakic status was recorded.

Remote interpretation of the fundus photographs

Two masked retina specialists (CB, MS) graded the photographs. The retina specialists received batches of 400 de-identified images from the three photographic modalities: (1) non-mydriatic Smartscope; (2) mydriatic Smartscope, and (3) mydriatic Topcon. Consecutive images were from the three fields of the same eye and same imaging modality. We did not include photographs from the same patient taken with different cameras in the same reading batch so as to minimize bias from having seen a potentially ‘better’ picture taken with a different camera. The two readers graded the quality of the photograph (quality reference photographs, Supplementary Appendix 2), reported whether or not VTDR (PDR or CSME) was present, and graded the level of DR (NHS Guidelines, Supplementary Appendix 3) for each eye after examining images from the three fields of view. If the reader rated the macular image as ‘ungradable,’ the image set was considered ungradable as readers would not be able to diagnose CSME. In addition, as the unit of referral is the subject, if one eye of a subject was ungradable, then the subject was considered ungradable. To evaluate intra-observer reliability, the readers re-graded 25% of the images after one month. Stratified randomization of the images chosen was used to ensure that the images that were re-graded contained equal representation from all the three imaging modalities as well as equal representation of images with and without VTDR.

The two graders followed NHS guidelines for reading station quality. A single designated computer in a darkened room was the reading station in each institution (MS, AECS, Pondicherry, Tamil Nadu, India and CB, Kellogg Eye Center, Ann Arbor, Michigan, USA). The computer screen was ≥17 diagonal inches and the resolution of the screen was ≥1600 × 2000 pixels.20 We used standard brightness and contrast settings set by Windows. Graders were not allowed to manipulate the images.

Outcome measures

The primary outcome measure was sensitivity and specificity to detect VTDR using both fundus cameras compared to a reference standard clinical examination. The secondary outcome was the sensitivity and specificity of detecting VTDR using the non-mydriatic Smartscope images compared with the mydriatic Topcon images. The graders’ inter- and intra-observer reliability was also assessed.

Statistical analysis

We determined disease status through a DFE by a retina specialist (SS), which served as the reference standard. Patient demographics and clinical measures of the eye were summarized for the sample with descriptive statistics. We estimated the primary outcome, the sensitivity to detect VTDR for each camera modality compared to the reference standard, and included 95% Wilson confidence intervals. We tested whether we could reject the null hypothesis, H0: sensitivity=0.80 to conclude that the sensitivity to detect VTDR is actually greater (one-sided test). The specificity was also estimated. For each imaging modality type, we calculated inter- and intra-observer agreement for the primary outcome (presence or absence of VTDR) as well as a secondary outcome (quality of the photographs) using a kappa statistic.21 All statistical tests met assumptions of the test. All statistical analyses were performed using SAS software, version 9.4 (SAS Institute, Cary, NC, USA).

Results

Demographics and baseline characteristics

We recruited 155 subjects for the study and 275 (89%) eyes were included in the analysis (Figure 1). Thirty-five eyes were excluded from the analysis because all three fields of view were not obtained for each imaging modality according to the study protocol. Descriptive statistics of the sample are presented in Table 1. Subjects had a mean (±standard deviation (SD)) age of 55.7±9.1 years and 63% were men. 106 subjects (68%) had DM; the average duration of DM was 9.6±7.7 years (median=8 years, range=3 months—30 years). According to the reference standard clinical evaluation, 142 eyes (51.6%) had no DR, 21 eyes (7.7%) had R1 disease, 25 eyes (9.1%) had R2 disease and 87 eyes (31.8%) had R3 disease. Fifty eyes (18.2%) had CSME. A total of 120 eyes (43.6%) had VTDR (representative images available in Supplementary Appendix 4). No adverse events occurred from the DFE or photography protocol.

Table 1 Descriptive statistics of the sample (n=275 eyes of 155 subjects)
Full size table

Sensitivity and specificity to detect VTDR

A cross-tabulation of VTDR diagnosis by reference standard against grader diagnosis from non-dilated Smartscope images, dilated Smartscope images, and dilated Topcon images is reported in Table 2. The sensitivity and specificity to detect VTDR using images from Smartscope and Topcon cameras was compared to the reference standard DFE (Table 3). Graders 1 and 2 had sensitivities of 93% (95% confidence interval (CI): 87– 97%) and 88% (95% CI: 81–93%) and specificities of 84% (95% CI: 77–89%) and 90% (95% CI: 84–94%), respectively, in identifying VTDR using images from the non-mydriatic Smartscope. These estimates assume that all eyes with ungradable images are positive for disease, as an ungradable screening exam would necessitate an in-person examination. If we exclude eyes with ungradable macular images, results were similar (sensitivity 92%, 84%; specificity 99%, 99%). With images from the mydriatic Smartscope, graders 1 and 2 reported similar sensitivities of 94% (95% CI: 88–97%) and 89% (95% CI: 82–94%) and specificities of 85% (95% CI: 78–89%) and 92% (95% CI: 86–95%), respectively, in identifying VTDR. With mydriatic images from the Topcon camera, both graders had slightly higher sensitivities and specificities for identifying VTDR (sensitivity=97% (95% CI: 92–99%) and 95% (95% CI: 90–98%), specificity=89% (95% CI: 83–93%) and 90% (95% CI: 84–94%)). The results were similar for both mydriatic imaging modalities after excluding eyes with ungradable macular images (Table 3). Sensitivity to detect VTDR using the non-mydriatic Smartscope when including all images was significantly higher than 80% (P<0.05 for all comparisons, Table 3). If we next consider the dilated Topcon images as the reference standard, the non-dilated Smartscope still had excellent sensitivity and specificity. Graders 1 and 2 had sensitivity of 88% (95% CI: 81–93%) and 82% (95% CI: 73–88%), and specificity of 99% (95% CI: 96–100%) and 99% (95% CI: 95–100%), respectively, for identifying VTDR.

Table 2 Cross-table of reference standard diagnosis and grader diagnosis for VTDR, stratified by camera modality (n=275 eyes)
Full size table
Table 3 Sensitivity and specificity to detect VTDR from Smartscope (undilated and dilated) and Topcon images vs reference standard clinical exam
Full size table

Inter and intra-observer reliability to detect VTDR

The inter-observer and intra-observer reliability was high in terms of identifying VTDR, κ=0.82–0.91 and 0.82–1.00, respectively (Supplementary Appendix 5). Inter-observer agreement for image quality (excellent, acceptable, or ungradable) was moderate. For the non-mydriatic Smartscope, ĸ ranged from 0.59–0.65 for the three fields of view, compared to ĸ=0.52–0.60 for the mydriatic Smartscope images, and ĸ=0.64–0.74 for the mydriatic Topcon images (Table 4). The macular image was rated ungradable by grader 1 and 2 in 17% and 18% of eyes (26 and 25% of subjects) by non-dilated Smartscope, in 12 and 8% of eyes (17 and 13% of subjects) by dilated Smartscope, and in 5 and 7% of eyes (6 and 10% of subjects) by Topcon, respectively.

Table 4 Comparative analysis of image quality between graders across three fields of view and imaging modalities
Full size table

Discussion

The Smartscope is the first portable, relatively inexpensive, commercially available fundus camera that meets guidelines as an acceptable tool for screening for VTDR. The British Diabetic Association guidelines state that any new imaging device must have a sensitivity ≥80% to be considered for use in DR screening.15 In this study, we found that the portable non-mydriatic Smartscope fundus camera met this criterion with sensitivity between 84 and 93%. The second important criterion for assessing whether a novel technology may be useful in a screening capacity is its specificity, as a low specificity increases the burden on patients who need to seek advanced care for a false positive test. We found that the specificity for detecting VTDR using the non-mydriatic Smartscope camera was high at 84–99%, depending on whether non-gradable images of the macula were excluded from the analysis. When remote readers grade images from non-mydriatic table-top cameras, the range of sensitivities to detect VTDR is between 48 and 98% and the specificities range between 76 and 100% (Supplementary Appendix 1).10–12,22–31 Our results using a non-mydriatic portable fundus camera fall at the top of these ranges.

The two retina specialists grading the non-mydriatic images for VTDR had very high agreement both with each other, κ=0.82, and with themselves when they re-graded the images, κ=0.82 for both graders. This demonstrates that the images produced with this handheld non-mydriatic camera had sufficient quality to enable reliable remote diagnosis of VTDR.

We evaluated whether dilating the pupils would substantially improve remote graders’ ability to accurately diagnose VTDR. There was only a very slight improvement in the sensitivity from 84–92% to 88–94% and the specificity essentially remained unchanged (99% to 97–99% when evaluating gradable images). Therefore, it would likely be reasonable to use the Smartscope without dilating patients in a screening setting.

Although the cost for the camera of ~US $5000 may seem relatively expensive, compared to the costs of lost vision, lost wages and caregiver burden, it is not.32 Our study underscores these findings, as our patients with VTDR were relatively young (mean age 56). Timely screening is an important strategy to prevent blindness from DR.

Currently, there are three main ways in which diabetics are screened for eye disease in the state of Tamil Nadu: primary care offices, eye hospitals or vision centers, and eye camps. Very few primary care physicians have fundus cameras in their offices for screening. The AECS has 11 tertiary care eye hospitals and over 40 vision centers throughout the state. A vision center is a small eye clinic staffed by an ophthalmic technician who can communicate with ophthalmologists at the main eye hospital using web-based conferencing to triage and treat disease that serves a population of 50 000. None of these methods make accessing DR screening as easy as an eye camp, where a team of ophthalmologists and para-professional staff come to remote villages. Portable technology to image the fundus makes screening for DR in these remote, rural settings a distinct possibility (Supplementary Figure 1).

Recent work has found that ultrawide field imaging (200°) identifies ~30% more diabetic lesions than standard 7 field ETDRS photos. These new lesions changed the graders’ diagnosis from no VTDR to VTDR 1.9% of the time.33, 34 Ultimately, this may become the new gold standard for remote DR screening, though the current cameras are not portable and are quite expensive (~US $100 000.00), limiting their widespread use, especially in low-resource settings.

To improve outreach, there is a need for a fundus camera that is commercially available, relatively inexpensive, portable and easy to use while still having high sensitivity and specificity to detect VTDR. The ideal device would be lightweight, non-mydriatic, have no issues with alignment, enable visualization through hazy media, and have easy user-interface and image transfer features. Many new and exciting portable technologies, including smartphone-based technologies, have been recently introduced to the market for DR screening35, 36, 37, 38 but, to the best of our knowledge, none of the existing devices meet all the above-mentioned criteria. The Smartscope fundus camera appears to be one of the first that has many of these ideal characteristics. One of the attributes that facilitates the acquisition of high-quality images is the use of a flexible black silicone eyecup that covers the eye from ambient illumination to maximize physiologic mydriasis.

However, there are still many obstacles to widespread use of the Smartscope. Its portability is currently limited by its dependence on electricity to charge the device as it lasts only a few hours without charging. The Smartscope relies on WiFi for image transfer, so it cannot instantly transfer images through the cellular data networks for remote grading like a smartphone. One of the largest hurdles towards widespread uptake is the considerable training and practice before a photographer could deliver high-quality images (75 patients); it is unlikely that any screening program could employ a single photographer and so the issue of the amount of training needed to obtain high-quality images becomes a potential barrier to implementation.

There were a number of limitations to this study. This study was conducted in a tertiary care eye hospital in India, and a convenience sampling from the retina and comprehensive clinics was used. Therefore, the sample in this study is an enriched case sample with higher rates of VTDR than would be seen in a screening population, so the results may not be directly applicable to other populations. After demonstrating that the non-dilated Smartscope photographs have reasonable sensitivity and specificity to detect VTDR in an ideal setting, one next step will be to test the protocol in a screening eye camp. In terms of sampling, we only recorded the number of patients who returned for a study visit; therefore, we cannot assess the potential impact of the volunteer effect. The 35 subjects who contributed one eye had a larger percentage of diabetics (88.6%) compared to the 120 subjects who contributed both eyes to the study (62.5%; P=0.0035 Fisher’s exact test). This is a potential bias in our sample and could mean that eyes from diabetics are harder to image. In situations where an eye could not be imaged, the subject would have to be referred for full examination, which would increase our reported sensitivity and decrease our reported specificity. We used three images to evaluate for VTDR so we do not yet know whether grading a single image of the posterior pole is a viable option. We used clinical examination as the reference standard for comparison as opposed to 7 field ETDRS stereo-photographs as recommended by the American Telemedicine Association for Category 2 program validation as ETDRS stereo-photographs were beyond the scope of the resources available for this study.

The strengths of our study are the relatively large sample of eyes with VTDR, the use of masked retina specialists as graders, and the use of standard photographs and guidelines for grading quality and severity of disease. To reduce grader bias, the images were arranged such that no two imaging modalities from the same eye were part of the same grading set so that the reader’s diagnosis would not be informed by the potentially ‘better’ image. Employing a single, well trained photographer to take all photos and a single retina specialist to provide the reference standard clinical examination reduced the variability of our results.

Future research is required to replicate our results in the outreach setting. In an outreach camp, a dark environment for imaging may not be available, electricity shortages are common, subjects oftentimes have media opacities and hundreds of people are screened in a single morning. New ways to structure work flow, from image acquisition to image analysis will be needed. New technologies continue to emerge that will positively disrupt the way care is provided for diabetic patients and we need to continue to rigorously evaluate these technologies before deploying them in large scale screening efforts.