In any community, the key to understanding the burden of a specific condition is to conduct an epidemiological study. The deep learning system (DLS) recently showed promising diagnostic performance for diabetic retinopathy (DR). This study aims to use DLS as the grading tool, instead of human assessors, to determine the prevalence and the systemic cardiovascular risk factors for DR on fundus photographs, in patients with diabetes. This is a multi-ethnic (5 races), multi-site (8 datasets from Singapore, USA, Hong Kong, China and Australia), cross-sectional study involving 18,912 patients (n = 93,293 images). We compared these results and the time taken for DR assessment by DLS versus 17 human assessors – 10 retinal specialists/ophthalmologists and 7 professional graders). The estimation of DR prevalence between DLS and human assessors is comparable for any DR, referable DR and vision–threatening DR (VTDR) (Human assessors: 15.9, 6.5% and 4.1%; DLS: 16.1%, 6.4%, 3.7%). Both assessment methods identified similar risk factors (with comparable AUCs), including younger age, longer diabetes duration, increased HbA1c and systolic blood pressure, for any DR, referable DR and VTDR (p > 0.05). The total time taken for DLS to evaluate DR from 93,293 fundus photographs was ~1 month compared to 2 years for human assessors. In conclusion, the prevalence and systemic risk factors for DR in multi-ethnic population could be determined accurately using a DLS, in significantly less time than human assessors. This study highlights the potential use of AI for future epidemiology or clinical trials for DR grading in the global communities.
By 2040, nearly 600 million people will have diabetes worldwide.1 Diabetic retinopathy (DR), a major microvascular complication, is a leading cause of vision impairment.2,3,4 Among people with diabetes, about a third have signs of DR, and up to 10% have more severe levels that require referral (referable DR) or are vision-threatening DR (VTDR).5 Clinical trials have shown that controlling major risk factors such as hyperglycemia and hypertension can reduce the risk of DR progression.6,7,8 Thus, vision loss from DR can be reduced by 50% or more by screening, appropriate referral and treatment.9,10,11,12
Despite these important concepts, there is a lack of understanding of the burden of DR, and thus lack of guidance, priority and resources allocated to tackle this in many countries.13 Epidemiological studies show substantial variation in the prevalence of DR (e.g., 40% in the U.S., 31% in Africa, and 17.6% in India),3,14 and some studies have not been able to confirm the importance of risk factor such as hypertension as a modifiable risk factor.15
In many countries, epidemiological studies are critical to document the burden of DR,16 and to identify the specific role of modifiable risk factors.3,8,17,18 The assessment of DR in such studies, however, has typically relied on an accurate evaluation of retinal photographic images. Such an assessment requires significant resources, including trained manpower, time, and infrastructure. As a result, many countries and regions do not have accurate epidemiological data on DR to establish local strategies and guidelines.19
Deep learning system (DLS) an artificial intelligence (AI)-based machine learning technology.20,21 It has revolutionized the computer vision field and achieved substantial jumps in diagnostic performance for image recognition, speech recognition, and natural language processing.20 In the technical world, DL has been heavily used in autonomous vehicles,22 gaming23,24, and numerous smartphone applications. In medicine, this technique has shown promising diagnostic performance, across specialties including ophthalmology (e.g. detection of diabetic retinopathy [DR], glaucoma, and age-related macular degeneration from fundus photographs and optical coherence tomographs),25,26,27,28,29,30 radiology (e.g. detection of tuberculosis from chest X rays, intracranial hemorrhage from computed tomography of the brain),31,32,33,34 and dermatology (e.g. detection of malignant melanoma from skin photographs)35.
For DR, it has shown promising diagnostic performance using retinal images,21,26,27,30,36,37 when compared to trained human assessors including ophthalmologists. The performance of DLS is comparable to humans in differentiating referable vs non-referable DR.26,27,36 An unanswered question is whether associations between DR (detected by DLS) and risk factors are also similar. Such information will lead to greater acceptance of DLS as a plausible, cost-effective alternative tool compared to traditional human assessment for DR, leading to significant resource savings in epidemiological and clinical studies, including clinical trials.
The objective of this study was to evaluate the ability of the DLS to determine the prevalence and risk factors for DR using a multi-ethnic, multi-site dataset of retinal images from epidemiological and clinical studies of people with diabetes. We compared the performance of the DLS in estimating the prevalence and cardiovascular risk factors of any DR, referable DR and VTDR, as compared to the human assessors. In addition, we estimated the time taken to evaluate the assessment of these outcomes between the two methods.
A total of 18,912 patients (93,293 images) with diabetes were analyzed in this study (Supplementary Figure 1). The participants’ demographics, systemic risk factors, and DR severity levels for the eight datasets are shown in Table 1. The mean values (standard deviation) for age, BMI, diabetes duration, SBP, DBP, HbA1c, total cholesterol, and triglycerides of the 8 cohorts of patients were 62.0 (10.8) years, 27.4 (5.3) kg/m2, 9.0 (7.9) years, 134.7 (19.2) mmHg, 73.8 (10.6) mmHg, 7.4% (1.7) and 5.0 (1.7) mmol/L and 2.1 (2.5) mmol/L, respectively.
For the combined pooled dataset, the AUCs of DLS, with reference to the human assessors’ grading, was 0.863 (95%CI: 0.854, 0.871) for any DR, 0.963 (95% CI: 0.956, 0.969) for referable DR, and 0.950 (95% CI: 0.940, 0.959) for VTDR. The overall prevalence of any DR, referable DR, and VTDR was 15.9, 6.5, and 4.1%, respectively, for human assessors vs 16.1, 6.4, and 3.7% for DLS (Fig. 1).
To analyze 93,293 images, the total time taken for a DLS vs human assessor were 10.4 h vs 1554.8 h (Table 2), with the specific details shown in Supplementary Table 1. For the images ‘deemed’ ungradable by the DLS, the additional time required for manual grading was added onto the total time taken. A total of 7391 images ‘deemed’ ungradable by the DLS underwent a secondary manual grading by human assessors, requiring additional 123.2 h (19.0 man-days), totaling up to 125.4 h (21.1 man-day).
Table 3 shows the relationship of risk factors for the DR outcomes evaluated by DLS vs human assessors. Longer duration of diabetes, increased HbA1c and SBP were significantly associated with any DR, referable DR and VTDR (p < 0.001) for both DLS and human assessors. Supplementary Table 2 shows the analysis for individual dataset. Combining all datasets, the systemic risk factors were comparable between DLS and human assessors to discriminate any DR (0.738 vs 0.743, p = 0.69), referable DR (0.795 vs 0.782, p = 0.40), and VTDR (0.810 vs 0.813, p = 0.85; Supplementary Figure 2), with the specific AUC of each dataset shown in Supplementary Figure 3.
Using forest plot meta-analysis, both grading methods identified similar risk factors, including younger age, longer diabetes duration, increased HbA1c and systolic blood pressure, for any DR (Fig. 2), referable DR (Fig. 3), and VTDR (Fig. 4). In contrast, gender, total cholesterol, and triglycerides were not associated with DR assessed using both methods.
AI using deep learning techniques may potentially revolutionize how medical images are analyzed.25,38 The challenge of AI technology is acceptance by physicians, researchers, and policy makers in terms of robustness and validity of outcomes measured by AI. Besides the obvious potential of using AI in direct clinical care, another immediate application of AI is in research settings, such as in evaluating outcomes in epidemiological studies and clinical trials.
The objective of our study was to evaluate the ability of an AI-based DLS to assess retinal images for DR in population-based epidemiological and hospital-based clinical studies of people with diabetes. We compared results between the DLS and humans in the two key outcomes traditionally measured in such studies (i.e., prevalence and risk factors). We demonstrated comparable outcomes in detecting DR prevalence and risk factor associations between a DLS which was 360 times faster than human assessors. Both the DLS and humans identified a similar prevalence (burden) of DR in the population assessed and longer duration of diabetes, higher HbA1c and higher SBP as risk factors associated with DR. The discriminative ability of these risk factors for DR were comparable between DLS and human assessors. Our study shows while AI technology may need to overcome substantial hurdles, including medico-legal challenges, for application in clinical care,39,40 AI technology is an acceptable research tool for assessing outcomes (in this case DR) in population-based and clinical studies, and is particularly suitable for application in countries without the resources to do full-scale research studies.
Our study showed that DLS is a faster grading tool than human assessors, with immediate availability of the outcome. A particular example is SiMES, which is a population-based study conducted in Singapore.41 We have previously documented the prevalence and risk factors for this cohort, reporting prevalence of any DR to be 25.5%,42 risk factors of longer diabetes duration, higher HbA1c and systolic blood pressure and; protective factors of older age and higher total cholesterol level.41 Using the DLS would have resulted in identical findings (Supplementary Table 2). We estimated that in SiMES, the trained human assessor spent ~2–5 min per image, but with DLS, it requires only 0.4 sec.
In total, given that they have a 6.5-man-day (5 days a week), a human assessor would require 553.8 man-days (>2 years) to complete 93,293 retinal images, without factoring the annual/medical leaves and public holidays. In Singapore, the cost for a human assessor, on average, is budgeted to grade about 9800 patients/year. In other words, a human assessor would require about 2 years to grade ~18,000 patients (100,000 retinal images). For DLS, it correspondingly took about 10 h. Even then, for those images deemed ungradable by DLS (~7.9%), these images will need to be graded secondarily by human assessors and hence, additional time (43.5 man-days) was included in our study. On average, the difference between a DLS (with manual grading) vs a human assessor is ~1 month vs 2 years.
Of the risk factors, HbA1c, duration of diabetes and SBP were the most common risk factors associated with increasing DR severity (p < 0.001) on the forest plot. These risk factors were consistent with published data from cross-sectional and longitudinal diabetic cohorts.43,44,45 Thus, our study shows the robustness of the DLS as an alternative tool for DR grading and could be utilized to analyze thousands or millions of retinal images over a short period of time. For countries, research institutions, community and hospital health care systems worldwide with limited manpower or financial resources, DLS could potentially save significant time and cost
Our study was limited by the DR grading determined based on mostly 2-field retinal photographs instead of the classic standard 7-field stereoscopic Early Treatment DR Study (ETDRS) field, though 7-field photography would take longer and has higher financial implications. In addition, we also did not have the information on the types of diabetes (e.g. Type 1 vs type 2) of the patients. Future studies could evaluate the generalizability of the DLS for diabetic cohorts with different retinal cameras, settings, and imaging modalities such as ultra-wide retinal photography in detection of DR. It will be important to explore the use of multi-modal machine learning approach in combining the clinical data and retinal images to risk stratify patients with diabetes.
AI-based DLS is a potential alternative assessment tool to determine the epidemiology of DR in research settings, and results in robust, comparable prevalence and systemic risk factors for DR. This technology could potentially transform the conduct of large-scale population-based epidemiological studies, including clinical trials.
This study was approved by the Centralized Institutional Review Board (IRB) of SingHealth, Singapore (protocol number SHF/FG648S/2015) and conducted in accordance with the Declaration of Helsinki. Given the retrospective analysis using de-identified images, informed consent was exempted by IRB.
Development and validation of DLS
The clinical, technical details and diagnostic performance of the DLS have been described previously.26 In brief, the DLS was trained using 76,370 retinal images (2-field: optic disc- and macula-centered images), consisting of 88.3% no DR, 6.4% mild non-proliferative DR (NPDR), 3.8% moderate NPDR, and 1.5% VTDR (severe NPDR and proliferative DR). The DR severity level was classified using the International Clinical Diabetic Retinopathy Severity Scale (ICDRSS).46 Any DR was defined as mild NPDR (i.e., only microaneurysms) or worse; referable DR as moderate NPDR (i.e., mild NPDR with scattered retinal hemorrhages and hard exudates) or worse; and VTDR as severe NPDR and PDR. If more than one-third of the photo was obscured, it was considered as “ungradable”. All retinal images used to develop the DLS were obtained from diabetes patients attending Singapore National DR Screening Program (SiDRP) from 2010 to 2013.47
We have previously validated the DLS,26 using 11 separate datasets, with excellent performance, with area under the receiver operating curve (AUC) in detecting referable DR ranging from 0.889 to 0.983.
For this current study, we used 8 multi-ethnic datasets to determine the prevalence and risk factors of DR, including 6 population-based studies: SiDRP with participants from 2014–15,47 Singapore Malay Eye Study (SIMES),42 Singapore Indian Eye Study (SINDI),42 Singapore Chinese Eye Study (SCES),47 Beijing Eye Study (BES),48 and African American Eye Disease Study (AFEDS),49 and two hospital-based studies: Chinese University of Hong Kong (CUHK),50 and Diabetes Management Project (DMP), Melbourne.51 These 8 datasets had risk factors for DR evaluated using similar definitions and methods. We did not include the other 3 datasets (Guangdong, Mexico and University of Hong Kong) due to the absence of systemic information. We standardized the diagnosis of diabetes as a self-reported history of diabetes, current use of diabetic medications, fasting glucose of ≥7 mmol/L, and/or a non-fasting glucose of 11.1 mmol/L or higher at the time of examination.
Details of the different populations have been described previously. SiDRP was started in 2010 as national DR screening program that covers all public primary eye care hospitals in Singapore via a tele-ophthalmology platform.26,47 SiMES, SINDI, and SCES were population-based studies that included participants of three major ethnic groups in Singapore, aged 40–80 years, recruited over an 8-year period: SiMES (Malays, 2004–2006), SINDI (Indians, 2007–2009), and SCES (Chinese, 2009–2011).42 The BES was a population-based study in China that involved participants aged 40 years and beyond.48 Among these population-based studies, we only included those with diabetes in the analyses. AFEDS is a population-based study of African American aged 40 years and older residing in the city of Inglewood, California. Given that the study was still in active recruitment phase, we only included participants with diabetes recruited up till mid-2017 for this analysis. We included two clinic-based studies among patients with diabetes: the CUHK study was a clinic-based cohort for patients with diabetes, recruited in 2016 from a tertiary eye clinic in Hong Kong,50 and the DMP is a clinical-based cohort of patients with diabetes in an eye hospital in Melbourne, Australia.51
Retinal images and DR classification
During DLS training, the input to the neural network was a retinal image, and the individual DR severity levels (0, 1, 2, 3, and 4 for no DR, mild NPDR, moderate NPDR, severe NPDR, and PDR respectively, using ICDRSS classification) were represented by output nodes. The weights of the DLS were adjusted with stochastic gradient descent, to train a classification model for DR. During validation, the DLS model predicted a raw confidence score for each severity level output node, for each image. These node scores were finally linearly weighted to produce a single image-level DR score. An ensemble of two separate models – one trained with the original image, and one with its contest-equalized version – was used. DLS hyperparameters and score thresholds were selected using a set of held-out images.
During validation, for each eye of each patient, the ensembled DR scores of all valid retinal images were averaged to produce an eye-level DR score, for each DR severity level. The predicted DR grade was then obtained by applying the previously-specified score thresholds. For each patient, the grade of the eye with the most severe DR as predicted was used to assess the relationship with systemic risk factors. If one of the two eyes was ungradable, the grade of the other was taken. If both eyes were ungradable, then the patient was classified as ungradable and excluded from the DLS analysis. Based on the training set, we pre-set the optimal operating threshold for any DR, referable DR and VTDR. Ungradable images and eyes with previous retinal laser were not included as part of the analyses.
Retinal photography protocol, classification, and grading of retinal images
All participants in the 8 datasets underwent 2-field (optic disc- and macula-centered) retinal photography. SiDRP, AFEDS, DMP, and CUHK cohorts were imaged using Topcon retinal camera (Tokyo, Japan) while SiMES, SINDI, SCES, and BES used a Canon retinal camera (Tokyo, Japan).42,47,48,49,50,51 For SiDRP, SiMES, SiNDI, SCES, AFEDS, and DMP Melbourne, the images were assessed by the human assessors who were non-ophthalmologists.42,47,49,50,51 The human assessors for BES were a board-certified ophthalmologist and a retinal specialist while CUHK patients were examined by 2 retinal specialists.48
Assessment of systemic risk factors
All datasets consisted of comprehensive patients’ demographics and systemic risk factors (e.g. age, gender, ethnicity, duration of diabetes, HbA1c, systolic and diastolic blood pressure [SBP and DBP], body mass index (BMI), total cholesterol, and triglyceride levels).
Assessment of time taken for image analysis
The grading time of each retinal image was obtained from the individual study center. The SiDRP, AFEDS, and DMP images were graded at the Singapore Eye Research Institute (SERI) and the SiMES, SINDI, and SCES photos at the Blue Mountain Eye Study reading center in Sydney, Australia. Beijing and Hong Kong cohorts were graded by the ophthalmologist and retinal specialists respectively. The average time taken per image for SiDRP assessors was 2 minutes; CUHK: 5 min; and the remaining (SIMES, SINDI, SCES, BES, AFEDS, and DMP Melbourne) were 3 min. The total estimated time taken for human assessor (man-days) = total time taken per image (minutes) × number of retinal images/24/6.5. One man-day is equivalent to 6.5 h/day. For DLS, we recorded the time taken to pre-process and analyze the retinal images using a graphic processing unit (GPU) for 8 datasets. Each retinal image required 0.4 seconds.
First, we calculated the overall area AUC of DLS and level of agreement of DLS in detection of 3 outcomes: any DR, referable DR and VTDR, with reference to human assessors. Level of agreement was assessed using Kappa coefficient: 0–0.2: slight agreement; 0.2–0.4: fair; 0.4–0.6: moderate; 0.6–0.8: good and; 0.8–1.0: excellent. Second, we analyzed the prevalence for any DR, referable DR and VTDR and time taken between the DLS and human assessors. Third, we performed a pooled analysis and used random-effect multivariate logistic regressions across 8 individual datasets on the risk factors for DLS and human-assessed DR outcomes. Then, the strength of the relationship with risk factors, assessed by odds ratios (OR) estimated from the meta-analysis, were compared between DLS and human assessors for statistical difference using Student’s t-tests and forest plots.52 Fourth, we calculated the AUC of the overall model to evaluate the discriminative ability of the combined risk factors for any DR, referable DR and VTDR as determined by DLS and human assessors. All data were expressed as mean (with standard deviation), number (with %) or standardized ORs (with 95% confidence intervals (CI)) with a p-value <0.05 considered to be statistically significant. All statistical analysis was performed using R Statistical Software (version 3.4.3; R Foundation for Statistical Computing, Vienna, Austria). With expected referable DR prevalence, DLS sensitivity and specificity of 5, 90, and 90%, respectively, the sample size required will be 7683 patients with desired precision of 0.03, 95% confidence interval.
The datasets used in this study originated from different principal investigators from different countries. Upon request, the corresponding authors, D.S.W.T and T.Y.W., can send the data request to the individual principal investigator to seek clearance from them.
The AI system described in this study is kept at the Singapore Eye Research Institute (SERI) and National University of Singapore (NUS). The underlying algorithm is copyrighted by SERI, NUS and will not be available to public.
Collaboration, N. C. D. R. F. Worldwide trends in diabetes since 1980: a pooled analysis of 751 population-based studies with 4.4 million participants. Lancet 387, 1513–1530 (2016).
Ting, D. S., Cheung, G. C. & Wong, T. Y. Diabetic retinopathy: global prevalence, major risk factors, screening practices and public health challenges: a review. Clin. Exp. Ophthalmol. 44, 260–277 (2016).
Cheung, N., Mitchell, P. & Wong, T. Y. Diabetic retinopathy. Lancet 376, 124–136 (2010).
Flaxman, S. R. et al. Global causes of blindness and distance vision impairment 1990-2020: a systematic review and meta-analysis. Lancet Glob. Health 5, e1221–e1234 (2017).
Yau, J. W. et al. Global prevalence and major risk factors of diabetic retinopathy. Diabetes Care 35, 556–564 (2012).
Diabetes, C. et al. Effect of intensive diabetes therapy on the progression of diabetic retinopathy in patients with type 1 diabetes: 18 years of follow-up in the DCCT/EDIC. Diabetes 64, 631–642 (2015).
Mohamed, Q., Gillies, M. C. & Wong, T. Y. Management of diabetic retinopathy: a systematic review. JAMA 298, 902–916 (2007).
Group, A. C. et al. Intensive blood glucose control and vascular outcomes in patients with type 2 diabetes. N. Engl. J. Med 358, 2560–2572 (2008).
Photocoagulation treatment of proliferative diabetic retinopathy. Clinical application of Diabetic Retinopathy Study (DRS) findings, DRS Report Number 8. The Diabetic Retinopathy Study Research Group. Ophthalmology 88, 583–600 (1981).
Writing Committee for the Diabetic Retinopathy Clinical Research Network. Panretinal photocoagulation vs intravitreous ranibizumab for proliferative diabetic retinopathy: a randomized clinical trial. JAMA 314, 2137–2146 (2015).
Antonetti, D. A., Klein, R. & Gardner, T. W. Diabetic retinopathy. N. Engl. J. Med 366, 1227–1239 (2012).
The DCCT Research Group. The effect of intensive treatment of diabetes on the development and progression of long-term complications in insulin-dependent diabetes mellitus. N. Engl. J. Med. 329, 977–986 (1993).
Wang, L. Z. et al. Availability and variability in guidelines on diabetic retinopathy screening in Asian countries. Br. J. Ophthalmol. 101, 1352–1360 (2017).
Burgess, P. I. et al. Epidemiology of diabetic retinopathy and maculopathy in Africa: a systematic review. Diabet. Med. 30, 399–412 (2013).
Klein, R. & Klein, B. E. Blood pressure control and diabetic retinopathy. Br. J. Ophthalmol. 86, 365–367 (2002).
Group, D. E. R. et al. Frequency of evidence-based screening for retinopathy in type 1 diabetes. N. Engl. J. Med. 376, 1507–1516 (2017).
Nathan, D. M., Bebu, I. & Lachin, J. M. Frequency of evidence-based screening for diabetic retinopathy. N. Engl. J. Med. 377, 195 (2017).
Group, A. S. et al. Effects of medical therapies on retinopathy progression in type 2 diabetes. N. Engl. J. Med. 363, 233–244 (2010).
Wong, T. Y. et al. Guidelines on diabetic eye care: The International Council of ophthalmology recommendations for screening, follow-up, referral, and treatment based on resource settings. Ophthalmology 125, 1608–1622 (2018).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444, https://doi.org/10.1038/nature14539 (2015).
Wong, T. Y. & Bressler, N. M. Artificial intelligence with deep learning technology looks into diabetic retinopathy screening. JAMA 316, 2366–2367 (2016).
Chen, C., Seff, A., Kornhauser, A. & Xiao, J. DeepDriving: learning affordance for direct perception in autonomous driving. Proc. IEEE International Conference on Computer Vision (ICCV), 2722–2730 (2015).
Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016).
Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 354–359 (2017).
Ting, D. S. W. et al. AI for medical imaging goes deep. Nat. Med 24, 539–540 (2018).
Ting, D. S. W. et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA 318, 2211–2223 (2017).
Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402–2410 (2016).
Ting, D. S. W., Wu, W. C. & Toth, C. Deep learning for retinopathy of prematurity screening. Br. J. Ophthalmol. https://doi.org/10.1136/bjophthalmol-2018-313290 (2018).
Ting, D. S. W. et al. Artificial intelligence and deep learning in ophthalmology. Br. J. Ophthalmol. https://doi.org/10.1136/bjophthalmol-2018-313173 (2018).
De Fauw, J. et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat. Med. 24, 1342–1350 (2018).
Lakhani, P. & Sundaram, B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology 284, 574–582 (2017).
Hwang, E. J. et al. Development and validation of a deep learning-based automatic detection algorithm for active pulmonary tuberculosis on chest radiographs. Clin. Infect. Dis., https://doi.org/10.1093/cid/ciy967 (2018).
Ting, D. S. W., Tan, T. E. & Lim, C. C. T. Development and Validation of a Deep Learning System for Detection of Active Pulmonary Tuberculosis on Chest Radiographs: Clinical and Technical Considerations. Clin. Infect. Dis. https://doi.org/10.1093/cid/ciy969 (2018).
Titano, J. J. et al. Automated deep-neural-network surveillance of cranial images for acute neurologic events. Nat. Med 24, 1337–1341 (2018).
Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).
Gargeya, R. & Leng, T. Automated identification of diabetic retinopathy using deep learning. Ophthalmology 124, 962–969 (2017).
Abramoff, M. D., Lavin, P. T., Birch, M., Shah, N. & Folk, J. C. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digital Med. 39, 1–8 (2018).
Obermeyer, Z. & Lee, T. H. Lost in thought - the limits of the human mind and the future of medicine. N. Engl. J. Med. 377, 1209–1211 (2017).
Char, D. S., Shah, N. H. & Magnus, D. Implementing machine learning in health care - addressing ethical challenges. N. Engl. J. Med 378, 981–983 (2018).
Chen, J. H. & Asch, S. M. Machine learning and prediction in medicine - beyond the peak of inflated expectations. N. Engl. J. Med 376, 2507–2509 (2017).
Wong, T. Y. et al. Prevalence and risk factors for diabetic retinopathy: the Singapore Malay Eye Study. Ophthalmology 115, 1869–1875 (2008).
Tan, G. S. et al. Ethnic differences in the prevalence and risk factors of diabetic retinopathy: the Singapore Epidemiology of Eye Diseases Study. Ophthalmology 125, 529–536 (2018).
Thomas, R. L. et al. Incidence of diabetic retinopathy in people with type 2 diabetes mellitus attending the Diabetic Retinopathy Screening Service for Wales: retrospective analysis. BMJ 344, e874 (2012).
Jones, C. D., Greenwood, R. H., Misra, A. & Bachmann, M. O. Incidence and progression of diabetic retinopathy during 17 years of a population-based screening program in England. Diabetes Care 35, 592–596 (2012).
Xu, J. et al. Ten-year cumulative incidence of diabetic retinopathy. The Beijing Eye Study 2001/2011. PLoS ONE 9, e111320 (2014).
Wilkinson, C. P. et al. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology 110, 1677–1682 (2003).
Nguyen, H. V. et al. Cost-effectiveness of a National Telemedicine Diabetic Retinopathy Screening Program in Singapore. Ophthalmology 123, 2571–2580 (2016).
Jonas, J. B., Xu, L. & Wang, Y. X. The Beijing eye study. Acta Ophthalmol. 87, 247–261 (2009).
Varma, R. African American Eye Disease Study (AFEDS). http://grantome.com/grant/NIH/U10-EY023575-03. Accessed on 6 Jan 2019.
Tang, F. Y. et al. Determinants of quantitative optical coherence tomography angiography metrics in patients with diabetes. Sci. Rep. 7, 2575 (2017).
Lamoureux, E. L. et al. Methodology and early findings of the Diabetes Management Project: a cohort study investigating the barriers to optimal diabetes care in diabetic patients with and without diabetic retinopathy. Clin. Exp. Ophthalmol. 40, 73–82 (2012).
Paternoster, R., Brame, R., Mazerolle, P. & Piquero, A. Using the correct statistical test for the equality of regression coefficient. Criminology 36, 859–866 (1998).
The Singapore Ocular Reading Center (Haslina Hamzah, Jin Y. Ho) has helped coordinated the data transfer and management of the study. This study was supported by the National Medical Research Council Singapore, Ministry of Health, Singapore and Tanoto Foundation. This project received funding from National Medical Research Council (NMRC), Ministry of Health (MOH), Singapore (National Health Innovation Center, Innovation to Develop Grant (NHIC-I2D-1409022); Health Service Research Grant; SingHealth Foundation Research Grant (SHF/FG648S/2015), and the Tanoto Foundation. For Singapore Epidemiology of Eye Diseases (SEED) study, we received funding from NMRC, MOH (grants 0796/2003, IRG07nov013, IRG09nov014, STaR/0003/2008, & STaR/2013; CG/SERI/2010) and Biomedical Research Council (grants 08/1/35/19/550, 09/1/35/19/616). The Singapore Diabetic Retinopathy Program (SiDRP) received funding from the MOH, Singapore (grants AIC/RPDD/SIDRP/SERI/FY2013/0018 & AIC/HPD/FY2016/0912). The Diabetes study in Nephropathy And other Microvascular cOmplications (DYNAMO) received funding from National Medical Research Council (NMRC) Large Collaborative Grant (LCG).
D.T., G.L., M.L.L., W.H., and T.Y.W. are co-inventors of a patent on the deep learning system in this paper; potential conflicts of interests are managed according to institutional policies of the Singapore Health System (SingHealth) and the National University of Singapore (NUS). The remaining authors declare no competing interests.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Ting, D.S.W., Cheung, C.Y., Nguyen, Q. et al. Deep learning in estimating prevalence and systemic risk factors for diabetic retinopathy: a multi-ethnic study. npj Digit. Med. 2, 24 (2019). https://doi.org/10.1038/s41746-019-0097-x
Technical and imaging factors influencing performance of deep learning systems for diabetic retinopathy
npj Digital Medicine (2020)
Different fundus imaging modalities and technical factors in AI screening for diabetic retinopathy: a review
Eye and Vision (2020)
Artificial intelligence for teleophthalmology-based diabetic retinopathy screening in a national programme: an economic analysis modelling study
The Lancet Digital Health (2020)
npj Digital Medicine (2020)
Current Diabetes Reports (2019)