Article | Published:

Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning

Nature Biomedical Engineeringvolume 2pages158164 (2018) | Download Citation

Abstract

Traditionally, medical discoveries are made by observing associations, making hypotheses from them and then designing and running experiments to test the hypotheses. However, with medical images, observing and quantifying associations can often be difficult because of the wide variety of features, patterns, colours, values and shapes that are present in real data. Here, we show that deep learning can extract new knowledge from retinal fundus images. Using deep-learning models trained on data from 284,335 patients and validated on two independent datasets of 12,026 and 999 patients, we predicted cardiovascular risk factors not previously thought to be present or quantifiable in retinal images, such as age (mean absolute error within 3.26 years), gender (area under the receiver operating characteristic curve (AUC) = 0.97), smoking status (AUC = 0.71), systolic blood pressure (mean absolute error within 11.23 mmHg) and major adverse cardiac events (AUC = 0.70). We also show that the trained deep-learning models used anatomical features, such as the optic disc or blood vessels, to generate each prediction.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from $8.99

All prices are NET prices.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    WHO The Top 10 Causes of Death (2017).

  2. 2.

    Stone, N. J. et al. 2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 129, S1–S45 (2014).

  3. 3.

    Wilson, P. W. et al. Prediction of coronary heart disease using risk factor categories. Circulation 97, 1837–1847 (1998).

  4. 4.

    National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III). Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) final report. Circulation 106, 3143–3421 (2002).

  5. 5.

    D’Agostino, R. B. Sr et al. General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation 117, 743–753 (2008).

  6. 6.

    Conroy, R. M. et al. Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur. Heart J. 24, 987–1003 (2003).

  7. 7.

    Graham, I. et al. Fourth Joint Task Force of the European Society of Cardiology and Other Societies on Cardiovascular Disease Prevention in Clinical Practice (constituted by representatives of nine societies and by invited experts). Eur. J. Cardiovasc. Prev. Rehabil. 14, E1–E40 (2007).

  8. 8.

    Yeboah, J. et al. Comparison of novel risk markers for improvement in cardiovascular risk assessment in intermediate-risk individuals. JAMA 308, 788–795 (2012).

  9. 9.

    Goff, D. C. et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 129, S49–S73 (2014).

  10. 10.

    Hira, R. S. et al. Frequency and practice-level variation in inappropriate aspirin use for the primary prevention of cardiovascular disease: insights from the National Cardiovascular Disease Registry’s Practice Innovation and Clinical Excellence registry. J. Am. Coll. Cardiol. 65, 111–121 (2015).

  11. 11.

    Cardiovascular Disease (10-Year Risk) (Framingham Heart Study, accessed 21 June 2017); https://www.framinghamheartstudy.org/risk-functions/cardiovascular-disease/10-year-risk.php

  12. 12.

    Cooney, M. T. et al. How much does HDL cholesterol add to risk estimation? A report from the SCORE investigators. Eur. J. Cardiovasc. Prev. Rehabil. 16, 304–314 (2009).

  13. 13.

    Dudina, A. et al. Relationships between body mass index, cardiovascular mortality, and risk factors: a report from the SCORE investigators. Eur. J. Cardiovasc. Prev. Rehabil. 18, 731–742 (2011).

  14. 14.

    Wang, J. J. et al. Retinal vascular calibre and the risk of coronary heart disease-related death. Heart 92, 1583–1587 (2006).

  15. 15.

    Wong, T. Y. et al. Quantitative retinal venular caliber and risk of cardiovascular disease in older persons: the cardiovascular health study. Arch. Intern. Med. 166, 2388–2394 (2006).

  16. 16.

    Seidelmann, S. B. et al. Retinal vessel calibers in predicting long-term cardiovascular outcomes: the Atherosclerosis Risk in Communities Study. Circulation 134, 1328–1338 (2016).

  17. 17.

    Wong, T. Y. et al. Retinal vascular caliber, cardiovascular risk factors, and inflammation: the multi-ethnic study of atherosclerosis (MESA). Invest. Ophthalmol. Vis. Sci. 47, 2341–2350 (2006).

  18. 18.

    McGeechan, K. et al. Meta-analysis: retinal vessel caliber and risk for coronary heart disease. Ann. Intern. Med. 151, 404–413 (2009).

  19. 19.

    McGeechan, K. et al. Prediction of incident stroke events based on retinal vessel caliber: a systematic review and individual-participant meta-analysis. Am. J. Epidemiol. 170, 1323–1332 (2009).

  20. 20.

    Wong, T. Y. et al. Retinal arteriolar narrowing and risk of coronary heart disease in men and women. The Atherosclerosis Risk in Communities Study. JAMA 287, 1153–1159 (2002).

  21. 21.

    Witt, N. et al. Abnormalities of retinal microvascular structure and risk of mortality from ischemic heart disease and stroke. Hypertension 47, 975–981 (2006).

  22. 22.

    Wong, T. Y. et al. Retinal microvascular abnormalities and 10-year cardiovascular mortality: a population-based case–control study. Ophthalmology 110, 933–940 (2003).

  23. 23.

    Cheung, C. Y.-L. et al. Retinal microvascular changes and risk of stroke: the Singapore Malay Eye Study. Stroke 44, 2402–2408 (2013).

  24. 24.

    Liew, G. et al. Fractal analysis of retinal microvasculature and coronary heart disease mortality. Eur. Heart J. 32, 422–429 (2011).

  25. 25.

    Kawasaki, R. et al. Fractal dimension of the retinal vasculature and risk of stroke: a nested case–control study. Neurology 76, 1766–1767 (2011).

  26. 26.

    Cheung, C. Y. et al. Retinal vascular fractal dimension and its relationship with cardiovascular and ocular risk factors. Am. J. Ophthalmol. 154, 663–674.e1 (2012).

  27. 27.

    Mookiah, M. R. K. et al. Computer-aided diagnosis of diabetic retinopathy: a review. Comput. Biol. Med. 43, 2136–2155 (2013).

  28. 28.

    Roychowdhury, S., Koozekanani, D. D. & Parhi, K. K. DREAM: diabetic retinopathy analysis using machine learning. IEEE J. Biomed. Health Inform. 18, 1717–1728 (2014).

  29. 29.

    LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

  30. 30.

    Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

  31. 31.

    Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402–2410 (2016).

  32. 32.

    Ting, D. S. W. et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA 318, 2211–2223 (2017).

  33. 33.

    Krause, J. et al. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Preprint at https://arxiv.org/abs/1710.01711 (2017).

  34. 34.

    Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: visualising image classification models and saliency maps. Preprint at https://arxiv.org/abs/1312.6034 (2013).

  35. 35.

    Wong, T. Y., Klein, R., Klein, B. E. K., Meuer, S. M. & Hubbard, L. D. Retinal vessel diameters and their associations with age and blood pressure. Invest. Ophthalmol. Vis. Sci. 44, 4644–4650 (2003).

  36. 36.

    Angermueller, C., Pärnamaa, T., Parts, L. & Stegle, O. Deep learning for computational biology. Mol. Syst. Biol. 12, 878 (2016).

  37. 37.

    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. Rethinking the inception architecture for computer vision. Preprint at https://arxiv.org/abs/1512.00567 (2015).

  38. 38.

    Dean, J. et al. Large scale distributed deep networks. In Proc. 25th Conference on Advances in Neural Information Processing Systems (eds Pereira, F. et al.) 1223–1231 (Neural Information Processing Systems Foundation, 2012).

  39. 39.

    Ioffe, S. & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. Preprint at https://arxiv.org/abs/1502.03167 (2015).

  40. 40.

    Russakovsky, O. et al. ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis. 115, 211–252 (2015).

  41. 41.

    Caruana, R., Lawrence, S. & Giles, L. Overfitting in neural nets: backpropagation, conjugate gradient, and early stopping. In Proc. 13th Conference on Advances in Neural Information Processing Systems (eds Leen, T. K. et al.) (MIT Press, Cambridge, MA, 2001).

  42. 42.

    Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Proc. 25th Conference on Advances in Neural Information Processing Systems (eds Pereira, F. et al.) 1097–1105 (Neural Information Processing Systems, 2012).

  43. 43.

    Xu, K. et al. Show, attend and tell: neural image caption generation with visual attention. Preprint at https://arxiv.org/abs/1502.03044 (2015).

  44. 44.

    Cho, K., Courville, A. & Bengio, Y. Describing multimedia content using attention-based encoder–decoder networks. IEEE Trans. Multimed. 17, 1875–1886 (2015).

  45. 45.

    Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. Preprint at https://arxiv.org/abs/1409.0473 (2014).

Download references

Acknowledgements

We thank C. Angermueller, A. Narayanaswamy, A. Rajkomar, A. Taly and P. Nelson from Google Research for their advice and assistance with reviewing the manuscript, and J. Cuadros for access to and clarifications on the EyePACS dataset and for thoughtful discussions. We used the UK Biobank Resource under application number 17643 for this work.

Author information

Author notes

  1. These authors contributed equally: Ryan Poplin, Avinash V. Varadarajan, Lily Peng and Dale R. Webster.

Affiliations

  1. Google Research, Google, Mountain View, CA, USA

    • Ryan Poplin
    • , Avinash V. Varadarajan
    • , Katy Blumer
    • , Yun Liu
    • , Greg S. Corrado
    • , Lily Peng
    •  & Dale R. Webster
  2. Verily Life Sciences, South San Francisco, CA, USA

    • Michael V. McConnell
  3. Division of Cardiovascular Medicine, Stanford School of Medicine, Stanford, CA, USA

    • Michael V. McConnell

Authors

  1. Search for Ryan Poplin in:

  2. Search for Avinash V. Varadarajan in:

  3. Search for Katy Blumer in:

  4. Search for Yun Liu in:

  5. Search for Michael V. McConnell in:

  6. Search for Greg S. Corrado in:

  7. Search for Lily Peng in:

  8. Search for Dale R. Webster in:

Contributions

R.P., A.V.V., Y.L., G.S.C., L.P. and D.R.W. designed the research; R.P., A.V.V., K.B., Y.L. and L.P. acquired data and/or executed the research; R.P., A.V.V., K.B., Y.L., M.V.M., L.P. and D.R.W. analysed and/or interpreted the data; R.P., A.V.V., K.B., Y.L., M.V.M., G.S.C., L.P. and D.R.W. prepared the manuscript.

Competing interests

The authors are employees of Google and Verily Life Sciences.

Corresponding author

Correspondence to Lily Peng.

Supplementary information

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/s41551-018-0195-0