Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Clinically applicable deep learning for diagnosis and referral in retinal disease

Abstract

The volume and complexity of diagnostic imaging is increasing at a pace faster than the availability of human expertise to interpret it. Artificial intelligence has shown great promise in classifying two-dimensional photographs of some common diseases and typically relies on databases of millions of annotated images. Until now, the challenge of reaching the performance of expert clinicians in a real-world clinical pathway with three-dimensional diagnostic scans has remained unsolved. Here, we apply a novel deep learning architecture to a clinically heterogeneous set of three-dimensional optical coherence tomography scans from patients referred to a major eye hospital. We demonstrate performance in making a referral recommendation that reaches or exceeds that of experts on a range of sight-threatening retinal diseases after training on only 14,884 scans. Moreover, we demonstrate that the tissue segmentations produced by our architecture act as a device-independent representation; referral accuracy is maintained when using tissue segmentations from a different type of device. Our work removes previous barriers to wider clinical use without prohibitive training data requirements across multiple pathologies in a real-world setting.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Our proposed AI framework.
Fig. 2: Results of the segmentation network.
Fig. 3: Results on the patient referral decision.
Fig. 4: Generalization to a new scanning device type.
Fig. 5: Visualization of the segmentation results as thickness maps.

Similar content being viewed by others

References

  1. OECD. Computed tomography (CT) exams (indicator). (2017); https://doi.org/10.1787/3c994537-en

  2. OECD. Magnetic resonance imaging (MRI) exams (indicator). (2017). https://doi.org/10.1787/1d89353f-en

  3. Foot, B. & MacEwen, C. Surveillance of sight loss due to delay in ophthalmic treatment or review: frequency, cause and outcome. Eye 31, 771–775 (2017).

    Article  CAS  Google Scholar 

  4. Owen, C. G. et al. The estimated prevalence and incidence of late stage age related macular degeneration in the UK. Br. J. Ophthalmol. 96, 752–756 (2012).

    Article  Google Scholar 

  5. Rudnicka, A. R. et al. Incidence of late-stage age-related macular degeneration in American whites: systematic review and meta-analysis. Am. J. Ophthalmol. 160, 85–93 (2015).

    Article  Google Scholar 

  6. Bourne, R. R. A. et al. Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis. Lancet Glob. Health 5, e888–e897 (2017).

    Article  Google Scholar 

  7. Schmidt-Erfurth, U., Klimscha, S., Waldstein, S. M. & Bogunović, H. A view of the current and future role of optical coherence tomography in the management of age-related macular degeneration. Eye 31, 26–44 (2017).

    Article  CAS  Google Scholar 

  8. Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. J. Am. Med. Assoc. 316, 2402–2410 (2016).

    Article  Google Scholar 

  9. Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115––118 (2017).

    Article  Google Scholar 

  10. Huang, D. et al. Optical coherence tomography. Science 254, 1178–1181 (1991).

    Article  CAS  Google Scholar 

  11. Buchan, J. C. et al. How to defuse a demographic time bomb: the way forward? Eye 31, 1519–1522 (2017).

    Article  CAS  Google Scholar 

  12. Whited, J. D. et al. A modeled economic analysis of a digital teleophthalmology system as used by three federal healthcare agencies for detecting proliferative diabetic retinopathy. Telemed. J. E Health 11, 641–651 (2005).

    Article  Google Scholar 

  13. Ronneberger, O., Fischer, P. & Brox, T. U-Net: convolutional networks for biomedical image segmentation. in Navab N., Hornegger J., Wells W., Frangi A. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. MICCAI 2015. Lecture Notes in Computer Science, vol. 9351 (Springer, Cham, Switzerland, 2015).

  14. Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T. & Ronneberger, O. 3D U-Net: learning dense volumetric segmentation from sparse annotation. in Ourselin, S., Joskowicz, L., Sabuncu, M., Unal, G., Wells, W. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016. MICCAI 2016. Lecture Notes in Computer Science, vol. 9901 (Springer, Cham, Switzerland; 2016).

  15. Muether, P. S., Hermann, M. M., Koch, K. & Fauser, S. Delay between medical indication to anti-VEGF treatment in age-related macular degeneration can result in a loss of visual acuity. Graefes Arch. Clin. Exp. Ophthalmol. 249, 633–637 (2011).

    Article  CAS  Google Scholar 

  16. Arias, L. et al. Delay in treating age-related macular degeneration in Spain is associated with progressive vision loss. Eye 23, 326–333 (2009).

    Article  CAS  Google Scholar 

  17. Karri, S. P. K., Chakraborty, D. & Chatterjee, J. Transfer learning based classification of optical coherence tomography images with diabetic macular edema and dry age-related macular degeneration. Biomed. Opt. Express 8, 579–592 (2017).

    Article  CAS  Google Scholar 

  18. Apostolopoulos, S., Ciller, C., De Zanet, S. I., Wolf, S. & Sznitman, R. RetiNet: automatic AMD identification in OCT volumetric data. Preprint at http://arxiv.org/abs/1610.03628v1 (2016).

  19. Farsiu, S. et al. Quantitative classification of eyes with and without intermediate age-related macular degeneration using optical coherence tomography. Ophthalmology 121, 162–172 (2014).

    Article  Google Scholar 

  20. Srinivasan, P. P. et al. Fully automated detection of diabetic macular edema and dry age-related macular degeneration from optical coherence tomography images. Biomed. Opt. Express 5, 3568–3577 (2014).

    Article  Google Scholar 

  21. Lee, C. S., Baughman, D. M. & Lee, A. Y. Deep learning is effective for classifying normal versus age-related macular degeneration OCT images. Ophthalmol. Retin. 1, 322–327 (2017).

    Article  Google Scholar 

  22. Fang, L. et al. Automatic segmentation of nine retinal layer boundaries in OCT images of non-exudative AMD patients using deep learning and graph search. Biomed. Opt. Express 8, 2732–2744 (2017).

    Article  Google Scholar 

  23. Lee, C. S. et al. Deep-learning based, automated segmentation of macular edema in optical coherence tomography. Biomed. Opt. Express 8, 3440–3448 (2017).

    Article  Google Scholar 

  24. Lu, D. et al. Retinal fluid segmentation and detection in optical coherence tomography images using fully convolutional neural network. Preprint at http://arxiv.org/abs/1710.04778v1 (2017).

  25. Roy, A. G. et al. ReLayNet: retinal layer and fluid segmentation of macular optical coherence tomography using fully convolutional network. Biomed. Opt. Express 8, 3627–3642 (2017).

    Article  Google Scholar 

  26. Castelvecchi, D. Can we open the black box of AI? Nature 538, 20–23 (2016).

    Article  CAS  Google Scholar 

  27. Schmidt-Erfurth, U. et al. Machine learning to analyze the prognostic value of current imaging biomarkers in neovascular age-related macular degeneration. Ophthalmol. Retin. 2, 24–30 (2018).

    Article  Google Scholar 

  28. Schlegl, T. et al. Fully automated detection and quantification of macular fluid in OCT using deep learning. Ophthalmology 125, 549–558 (2018).

    Article  Google Scholar 

  29. Keane, P. A. & Sadda, S. R. Predicting visual outcomes for macular disease using optical coherence tomography. Saudi J. Ophthalmol. 25, 145–158 (2011).

    Article  Google Scholar 

  30. Schaal, K. B., Rosenfeld, P. J., Gregori, G., Yehoshua, Z. & Feuer, W. J. Anatomic clinical trial endpoints for nonexudative age-related macular degeneration. Ophthalmology 123, 1060–1079 (2016).

    Article  Google Scholar 

  31. Schmidt-Erfurth, U. & Waldstein, S. M. A paradigm shift in imaging biomarkers in neovascular age-related macular degeneration. Prog. Retin. Eye Res. 50, 1–24 (2016).

    Article  CAS  Google Scholar 

  32. Villani, E. et al. Decade-long profile of imaging biomarker use in ophthalmic clinical trials. Invest. Ophthalmol. Vis. Sci. 58, BIO76–BIO81 (2017).

    Article  Google Scholar 

  33. Chopra, R., Mulholland, P. J., Dubis, A. M., Anderson, R. S. & Keane, P. A. Human factor and usability testing of a binocular optical coherence tomography system. Transl. Vis. Sci. Technol. 6, 16 (2017).

    Article  Google Scholar 

  34. Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676–682 (2012).

    Article  CAS  Google Scholar 

  35. Keane, P. A. et al. Evaluation of age-related macular degeneration with optical coherence tomography. Surv. Ophthalmol. 57, 389–414 (2012).

    Article  Google Scholar 

  36. Folgar, F. A. et al. Comparison of optical coherence tomography assessments in the comparison of age-related macular degeneration treatments trials. Ophthalmology 121, 1956–1965 (2014).

    Article  Google Scholar 

  37. Duker, J. S., Waheed, N. K. & Goldman, D. Handbook of Retinal OCT: Optical Coherence Tomography E-Book (Elsevier Health Sciences, Oxford, UK; 2013).

  38. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. Rethinking the inception architecture for computer vision. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2818–2826 (2016).

  39. Abadi, M. et al. TensorFlow: large-scale machine learning on heterogeneous systems. Preprint at https://arxiv.org/abs/1603.04467 (2016).

  40. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. in Proceedings of the 3rd International Conference on Learning Representations (ICLR). Preprint at http://arxiv.org/abs/1412.6980 (2015).

  41. Huang, G., Liu, Z., Weinberger, K. Q. & van der Maaten, L. Densely connected convolutional networks. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2261–2269 (2017).

  42. Lakshminarayanan, B., Pritzel, A. & Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Adv. Neural Inf. Process. Syst. 6405–6416 (2017).

  43. De Fauw, J. et al. Automated analysis of retinal imaging using machine learning techniques for computer vision. F1000Res 5, 1573 (2016).

    Article  Google Scholar 

Download references

Acknowledgements

We thank K. Kavukcuoglu, A. Zisserman, M. Jaderberg, K. Simonyan for discussions, A. Cain and M. Cant for work on the visuals, D. Mitchell and M. Johnson for infrastructure and systems administration, J. Morgan and OpenEyes for providing the electronic health record records, T. Peto, P. Blows, A. O’Shea and the NIHR Clinical Research Facility for work on the labeling, T. Heeran, M. Lukic, K. Kortum, K. Fasler, S. Wagner and N. Pontikos for work on the labeling, E. Steele, V. Louw, S. Gill and the rest of Moorfields IT team for work on the data collection and deidentification, S. Al-Abed and N. Smith for Moorfields technical advice at project initiation, R. Wood and D. Corder at Softwire for engineering support at Moorfields, R. Ogbe and the Moorfields Information Governance team for support, M. Hassard for Moorfields research and development support, K. Bonstein and the National Institute for Health Research (NIHR) for support at the Moorfields Biomedical Research Centre (BRC), J. Besley for legal assistance, E. Manna for patient engagement and support, and the rest of the DeepMind team for their support, ideas and encouragement. P.A.K. is supported by an NIHR Clinician Scientist Award (NIHR-CS-2014-14-023). D.A.S., A.T., C.E. and P.T.K. are supported by the NIHR Biomedical Research Centre at Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology and the NIHR Moorfields Clinical Research Facility. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health. R.C. receives studentship support from the College of Optometrists, United Kingdom.

Author information

Authors and Affiliations

Authors

Contributions

P.A.K., M.S., J.C., D.H., P.T.K., T.B. and K.A. initiated the project and the collaboration. O.R., J.D.F., B.R.-P. and S.N. developed the network architectures, training and testing setup. P.A.K., J.R.L. and R.C. designed the clinical setup. P.A.K., J.R.L., J.C., R.C., D.A.S., C.E. and A.T. created the dataset and defined clinical labels. J.D.F., B.R.-P., S.N., N.T., S.Bl., H.A., B.O., D.V., G.v.d.D., O.R. and J.C. contributed to the software engineering. J.R.L., S.Bl. and H.A. created the database. P.A.K., J.R.L., D.K., A.K., C.O.H. and R.R. contributed clinical expertise. O.R., P.A.K., J.D.F., J.R.L., B.R.-P., S.N., N.T. and X.G. analysed the data. T.B., S.Bo., J.C., J.H., F.M. and C.M. managed the project. O.R., P.A.K., J.R.L., J.D.F., B.R.-P., G.R. and H.M. wrote the paper. B.L. contributed to the uncertainty estimation.

Corresponding authors

Correspondence to Pearse A. Keane or Olaf Ronneberger.

Ethics declarations

Competing interests

P.A.K., G.R., H.M. and R.R. are paid contractors of DeepMind. P.A.K. has received speaker fees from Heidelberg Engineering, Topcon, Haag-Streit, Allergan, Novartis and Bayer. P.A.K. has served on advisory boards for Novartis and Bayer, and is an external consultant for DeepMind and Optos. A.T. has served on advisory boards for the following companies: Allergan, Bayer, Genentech, GlaxoSmithKline, Novartis, Roche. C.E. has received speaker fees from Heidelberg Engineering and Haag-Streit UK. P.T.K. has served on advisory boards for Aerie, Allergan, Alcon, Belkin Laser, Novartis and Santen. D.A.S. has received speaker fees from Novartis, Bayer, Allergan, Haag-Streit. The authors have no other competing interests to disclose.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary Text and Figures

Supplementary Figures 1–16 and Supplementary Tables 1–10

Reporting Summary

Supplementary Video 1

OCT viewer | This video demonstrates the interaction with the OCT viewer. The OCT scan belongs to a 72 year old female presented with increasing visual distortion over a 4 month period; the OCT shows loss of RPE consistent with geographic atrophy. The view first goes through the whole volume (128 slices) for a fixed tissue map hypothesis, followed by showing the different tissue map hypotheses for a given slice. Finally, we let the collage cycle through the different hypotheses continually while scrolling through the volume, pausing on several slices briefly to show the variations. The color legend for all segmentation maps is available in Supplementary Table 2.

Supplementary Video 2

Wet AMD | Choroidal neovascularization (CNV) is the pathognomonic feature of the neovascular (“wet”) form of age-related macular degeneration (AMD) and requires urgent treatment to prevent irreversible visual loss. A 72-year old man presented with a history of reduced vision in his left eye. Best corrected visual acuity in the affected eye was 38 Early Treatment Diabetic Retinopathy Study (ETDRS) letters. The model correctly selects the Most Urgent Diagnosis as “CNV”, suggesting referral to an ophthalmologist on an urgent basis. The model segmentation highlights growth of the neovascular tissue in the sub-retinal pigment epithelium (RPE) space – a so-called fibrovascular pigment epithelium detachment (PED). Subretinal fluid can be seen surrounding the inferior margins of the fibrovascular PED indicating the presence of ongoing CNV leakage.

Supplementary Video 3

Normal | Scans are quick and safe to perform and are thus commonly used in the screening of patients without visual symptoms or other ophthalmic findings. A 46-year old man who was referred for retinal specialist review. Best corrected visual acuity was 6/6. The model correctly selects the referral decision as “Observation Only”, suggesting that the OCT findings in isolation do not require referral to an ophthalmologist. The model accurately delineates the neurosensory retina without the presence of any pathologic compartments. It also highlights partial separation of the posterior hyaloid of the vitreous – this is a normal finding as the vitreous gel increasingly liquefies with age.

Supplementary Video 4

Diabetic macular edema | Accumulation of this fluid in the macula – diabetic macular edema (DME) – is the commonest cause of visual impairment in diabetes. A 54-year old man with diabetes was referred to Moorfields for ophthalmologist review with best corrected visual acuity in the affected eye of 45 ETDRS letters. The model correctly detects the presence of macular retinal edema (MRE) and suggests semi-urgent ophthalmology referral. The model highlights intraretinal fluid accumulation, with cystoid spaces in both the inner nuclear and outer plexiform layers, and a mixed petaloid/honeycomb appearance on the en face images. There is also an accompanying significant increase in total retinal thickness.

Supplementary Video 5

Ambiguous case (chronic central serous retinopathy) | In chronic CSR, diagnosis of secondary CNV formation is often challenging due to the frequent presence of shallow irregular pigment epithelium detachments (PEDs). A 60-64 year old woman presented with a history of CSR in her left eye. The model correctly detects the presence of CSR but is far less certain about the presence of CNV. It highlights a gravitational tract of subretinal fluid with a discrete area of fibrovascular PED superior to the fovea.

Supplementary Video 6

Ambiguous case (advanced geographic atrophy) | In advanced forms of AMD, geographic atrophy (GA) may sometimes coexist with CNV formation. In such cases, the CNV component may be clinically silent, and the fundus appearance may be limited to that of GA, making the diagnosis difficult. A 84-year old man was referred to Moorfields. Best corrected visual acuity in the affected eye was 1/60. The ground truth diagnosis was GA and routine referral was recommended. While the model correctly diagnoses the presence of GA and drusen, it suggests urgent referral due to the possible presence of CNV. The presence of subretinal hyperreflective on model segmentation is suggestive of previous CNV formation.

Supplementary Video 7

Difficult case of choroidal neovascularization | A 30 year old male patient, with a known history of CSR, presented with acute visual loss in his left eye and was diagnosed with secondary CNV formation. At this visit, the OCT scans lack many of the prototypical features of CSR, such as subretinal fluid accumulation. The model correctly diagnoses the presence of CNV and suggests the presence of CSR, but with far less certainty.

Supplementary Video 8

Failure case (partial-thickness macular hole) | Ocular media opacities may sometimes cause artefactual reductions in OCT signal strength and this can make accurate image segmentation challenging. Due to localized reduction in OCT signal strength in this case, some of the models erroneously detect the presence of a partial thickness macular hole. As a result, the models are uncertain as to whether the eye is normal or whether routine referral is required.

Supplementary Video 9

Integration with other clinical information | Retinal angiomatous proliferation (RAP) is a variant of choroidal neovascularization (CNV) due to age-related macular degeneration (AMD). A 75-79 year old woman presented with reduced vision in her left eye. The model segmentation highlights the presence of a fibrovascular pigment epithelium detachment (PED) with subretinal hyperreflective material, overlying intraretinal fluid, and surrounding drusen. These findings are highly suggestive of RAP - in its early stages, this can be misdiagnosed as macular retinal edema (MRE), particularly in elderly patients with diabetes. The interpretable representation reduces the risk of misdiagnosis and allows the clinician to easily correlate these findings with other clinical information, e.g., fundus fluorescein angiography.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

De Fauw, J., Ledsam, J.R., Romera-Paredes, B. et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med 24, 1342–1350 (2018). https://doi.org/10.1038/s41591-018-0107-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41591-018-0107-6

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing