Traditionally, medical discoveries are made by observing associations, making hypotheses from them and then designing and running experiments to test the hypotheses. However, with medical images, observing and quantifying associations can often be difficult because of the wide variety of features, patterns, colours, values and shapes that are present in real data. Here, we show that deep learning can extract new knowledge from retinal fundus images. Using deep-learning models trained on data from 284,335 patients and validated on two independent datasets of 12,026 and 999 patients, we predicted cardiovascular risk factors not previously thought to be present or quantifiable in retinal images, such as age (mean absolute error within 3.26 years), gender (area under the receiver operating characteristic curve (AUC) = 0.97), smoking status (AUC = 0.71), systolic blood pressure (mean absolute error within 11.23 mmHg) and major adverse cardiac events (AUC = 0.70). We also show that the trained deep-learning models used anatomical features, such as the optic disc or blood vessels, to generate each prediction.
Access optionsAccess options
Subscribe to Journal
Get full journal access for 1 year
only $8.67 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
WHO The Top 10 Causes of Death (2017).
Stone, N. J. et al. 2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 129, S1–S45 (2014).
Wilson, P. W. et al. Prediction of coronary heart disease using risk factor categories. Circulation 97, 1837–1847 (1998).
National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III). Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) final report. Circulation 106, 3143–3421 (2002).
D’Agostino, R. B. Sr et al. General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation 117, 743–753 (2008).
Conroy, R. M. et al. Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur. Heart J. 24, 987–1003 (2003).
Graham, I. et al. Fourth Joint Task Force of the European Society of Cardiology and Other Societies on Cardiovascular Disease Prevention in Clinical Practice (constituted by representatives of nine societies and by invited experts). Eur. J. Cardiovasc. Prev. Rehabil. 14, E1–E40 (2007).
Yeboah, J. et al. Comparison of novel risk markers for improvement in cardiovascular risk assessment in intermediate-risk individuals. JAMA 308, 788–795 (2012).
Goff, D. C. et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 129, S49–S73 (2014).
Hira, R. S. et al. Frequency and practice-level variation in inappropriate aspirin use for the primary prevention of cardiovascular disease: insights from the National Cardiovascular Disease Registry’s Practice Innovation and Clinical Excellence registry. J. Am. Coll. Cardiol. 65, 111–121 (2015).
Cardiovascular Disease (10-Year Risk) (Framingham Heart Study, accessed 21 June 2017); https://www.framinghamheartstudy.org/risk-functions/cardiovascular-disease/10-year-risk.php
Cooney, M. T. et al. How much does HDL cholesterol add to risk estimation? A report from the SCORE investigators. Eur. J. Cardiovasc. Prev. Rehabil. 16, 304–314 (2009).
Dudina, A. et al. Relationships between body mass index, cardiovascular mortality, and risk factors: a report from the SCORE investigators. Eur. J. Cardiovasc. Prev. Rehabil. 18, 731–742 (2011).
Wang, J. J. et al. Retinal vascular calibre and the risk of coronary heart disease-related death. Heart 92, 1583–1587 (2006).
Wong, T. Y. et al. Quantitative retinal venular caliber and risk of cardiovascular disease in older persons: the cardiovascular health study. Arch. Intern. Med. 166, 2388–2394 (2006).
Seidelmann, S. B. et al. Retinal vessel calibers in predicting long-term cardiovascular outcomes: the Atherosclerosis Risk in Communities Study. Circulation 134, 1328–1338 (2016).
Wong, T. Y. et al. Retinal vascular caliber, cardiovascular risk factors, and inflammation: the multi-ethnic study of atherosclerosis (MESA). Invest. Ophthalmol. Vis. Sci. 47, 2341–2350 (2006).
McGeechan, K. et al. Meta-analysis: retinal vessel caliber and risk for coronary heart disease. Ann. Intern. Med. 151, 404–413 (2009).
McGeechan, K. et al. Prediction of incident stroke events based on retinal vessel caliber: a systematic review and individual-participant meta-analysis. Am. J. Epidemiol. 170, 1323–1332 (2009).
Wong, T. Y. et al. Retinal arteriolar narrowing and risk of coronary heart disease in men and women. The Atherosclerosis Risk in Communities Study. JAMA 287, 1153–1159 (2002).
Witt, N. et al. Abnormalities of retinal microvascular structure and risk of mortality from ischemic heart disease and stroke. Hypertension 47, 975–981 (2006).
Wong, T. Y. et al. Retinal microvascular abnormalities and 10-year cardiovascular mortality: a population-based case–control study. Ophthalmology 110, 933–940 (2003).
Cheung, C. Y.-L. et al. Retinal microvascular changes and risk of stroke: the Singapore Malay Eye Study. Stroke 44, 2402–2408 (2013).
Liew, G. et al. Fractal analysis of retinal microvasculature and coronary heart disease mortality. Eur. Heart J. 32, 422–429 (2011).
Kawasaki, R. et al. Fractal dimension of the retinal vasculature and risk of stroke: a nested case–control study. Neurology 76, 1766–1767 (2011).
Cheung, C. Y. et al. Retinal vascular fractal dimension and its relationship with cardiovascular and ocular risk factors. Am. J. Ophthalmol. 154, 663–674.e1 (2012).
Mookiah, M. R. K. et al. Computer-aided diagnosis of diabetic retinopathy: a review. Comput. Biol. Med. 43, 2136–2155 (2013).
Roychowdhury, S., Koozekanani, D. D. & Parhi, K. K. DREAM: diabetic retinopathy analysis using machine learning. IEEE J. Biomed. Health Inform. 18, 1717–1728 (2014).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).
Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402–2410 (2016).
Ting, D. S. W. et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA 318, 2211–2223 (2017).
Krause, J. et al. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Preprint at https://arxiv.org/abs/1710.01711 (2017).
Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: visualising image classification models and saliency maps. Preprint at https://arxiv.org/abs/1312.6034 (2013).
Wong, T. Y., Klein, R., Klein, B. E. K., Meuer, S. M. & Hubbard, L. D. Retinal vessel diameters and their associations with age and blood pressure. Invest. Ophthalmol. Vis. Sci. 44, 4644–4650 (2003).
Angermueller, C., Pärnamaa, T., Parts, L. & Stegle, O. Deep learning for computational biology. Mol. Syst. Biol. 12, 878 (2016).
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. Rethinking the inception architecture for computer vision. Preprint at https://arxiv.org/abs/1512.00567 (2015).
Dean, J. et al. Large scale distributed deep networks. In Proc. 25th Conference on Advances in Neural Information Processing Systems (eds Pereira, F. et al.) 1223–1231 (Neural Information Processing Systems Foundation, 2012).
Ioffe, S. & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. Preprint at https://arxiv.org/abs/1502.03167 (2015).
Russakovsky, O. et al. ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis. 115, 211–252 (2015).
Caruana, R., Lawrence, S. & Giles, L. Overfitting in neural nets: backpropagation, conjugate gradient, and early stopping. In Proc. 13th Conference on Advances in Neural Information Processing Systems (eds Leen, T. K. et al.) (MIT Press, Cambridge, MA, 2001).
Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Proc. 25th Conference on Advances in Neural Information Processing Systems (eds Pereira, F. et al.) 1097–1105 (Neural Information Processing Systems, 2012).
Xu, K. et al. Show, attend and tell: neural image caption generation with visual attention. Preprint at https://arxiv.org/abs/1502.03044 (2015).
Cho, K., Courville, A. & Bengio, Y. Describing multimedia content using attention-based encoder–decoder networks. IEEE Trans. Multimed. 17, 1875–1886 (2015).
Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. Preprint at https://arxiv.org/abs/1409.0473 (2014).
We thank C. Angermueller, A. Narayanaswamy, A. Rajkomar, A. Taly and P. Nelson from Google Research for their advice and assistance with reviewing the manuscript, and J. Cuadros for access to and clarifications on the EyePACS dataset and for thoughtful discussions. We used the UK Biobank Resource under application number 17643 for this work.
About this article
An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets
Nature Biomedical Engineering (2019)
Nature Medicine (2019)
A multimodal approach to cardiovascular risk stratification in patients with type 2 diabetes incorporating retinal, genomic and clinical features
Scientific Reports (2019)
Scientific Reports (2019)
Nature Machine Intelligence (2019)