Abstract

Here we present deep-learning techniques for healthcare, centering our discussion on deep learning in computer vision, natural language processing, reinforcement learning, and generalized methods. We describe how these computational techniques can impact a few key areas of medicine and explore how to build end-to-end systems. Our discussion of computer vision focuses largely on medical imaging, and we describe the application of natural language processing to domains such as electronic health record data. Similarly, reinforcement learning is discussed in the context of robotic-assisted surgery, and generalized deep-learning methods for genomics are reviewed.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

  2. 2.

    Russakovsky, O.et al. Imagenet large scale visual recognition challenge. Int. J. Compute. Vis. 115, 211–252 (2015).

  3. 3.

    Hirschberg, J. & Manning, C. D. Advances in natural language processing. Science 349, 261–266 2015).

  4. 4.

    Geoffrey Hinton, et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29, 82–97 (2012).

  5. 5.

    Stanford Health. Harnessing the power of data in health. Stanford Medicine 2017 Health Trends Report (2017).

  6. 6.

    Silver, D. et al. Mastering the game of go with deep neural networks and tree search. Nature 529, 484–489 (2016).

  7. 7.

    Yohannes Kassahun, et al. Surgical robotics beyond enhanced dexterity instrumentation: a survey of machine learning techniques and their role in intelligent and autonomous surgical actions. Int. J. Comput. Assist. Radio. Surg. 11, 553–568 (2016).

  8. 8.

    Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017).

  9. 9.

    Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

  10. 10.

    Haenssle, H. A. et al. Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Ann. Oncol. 29, 1836–1842 (2018).

  11. 11.

    Cheng, J.-Z. et al. Computer aided diagnosis with deep learning architecture: applications to breast lesions in us images and pulmonary nodules in CT scans. Sci. Rep. 6, 24454 (2016).

  12. 12.

    Cicero, M. et al. Training and validating a deep convolutional neural network for computer-aided detection and classification of abnormalities on frontal chest radiographs. Invest. Radiol. 52, 281–287 (2017).

  13. 13.

    Kooi, T. et al. Large scale deep learning for computer aided detection of mammographic lesions. Med. Image Anal. 35, 303–312 (2017).

  14. 14.

    Barreira, C. M. et al. Abstract WP61: Automated large artery occlusion detection in st roke imaging-paladin study. Stroke 49, AWP61 (2018).

  15. 15.

    Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402–2410 (2016).

  16. 16.

    Poplin, R. et al. Webster. Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nat. Biomed. Eng. 2, 158–164 (2018).

  17. 17.

    Fauw, J. et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat. Med. 24, 1342 (2018).

  18. 18.

    Cireşan, D. C., Giusti, A., Gambardella, L. M. & Schmidhuber, J. Mitosis detection in breast cancer histology images with deep neural networks. In International Conference on Medical Image Computing and Computer-assisted Intervention 411–418 (Springer, 2013).

  19. 19.

    Liu, Y. et al. Detecting cancer metastases on gigapixel pathology images. Preprint at https://arxiv.org/abs/1703.02442 (2017).

  20. 20.

    Charoentong, P. et al. Pan-cancer immunogenomic analyses reveal genotype–immunophenotype relationships and predictors of response to checkpoint blockade. Cell Rep. 18, 248–262 (2017).

  21. 21.

    Beck, A. H. et al. Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci. Transl. Med. 3, 108ra113 (2011).

  22. 22.

    Yosinski, J., Clune, J., Bengio, Y. and Lipson, L. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 3320–3328 (2014).

  23. 23.

    Jamaludin, A., Kadir, T. and Zisserman, A. Spinenet: automatically pinpointing classification evidence in spinal mris. In International Conference on Medical Image Computing and Computer-Assisted Intervention 166–175 (Springer, 2016).

  24. 24.

    Kermany, D. S. et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172, 1122–1131 (2018).

  25. 25.

    Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (2015).

  26. 26.

    Goodfellow, I. et al. Generative adversarial nets. In Advances in Neural Information Processing Systems 2672–2680 (2014).

  27. 27.

    Sutskever, I., Vinyals, O. & Le, Q. V. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 3104–3112 (2014).

  28. 28.

    Wu, Y. et al. Google’s neural machine translation system: bridging the gap between human and machine translation. Preprint at https://arxiv.org/abs/1609.08144 (2016).

  29. 29.

    Kannan, A. et al. Smart reply: automated response suggestion for email. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (ACM, 2016).

  30. 30.

    Vinyals, O., Toshev, A., Bengio, S. & Erhan, D. Show and tell: a neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 3156–3164 (2015).

  31. 31.

    The Office of the National Coordinator for Health Information Technology. Quick stats: health IT dashboard. https://dashboard.healthit.gov/quickstats/quickstats.php (2017).

  32. 32.

    Shickel, B., Tighe, P. J., Bihorac, A. & Rashidi, P. Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J. Biomed. Health Inform. 22, 1589–1604 (2017).

  33. 33.

    Rajkomar, A. et al. Scalable and accurate deep learning with electronic health records. NPJ Digit. Med. 1, 18 (2018).

  34. 34.

    Miotto, R. et al. Deep patient: an unsupervised representation to predict the future of patients from the electronic health records. Sci. Rep. 6, 26094 (2016).

  35. 35.

    Liu, V., Kipnis, P., Gould, M. K. & Escobar, G. J. Length of stay predictions: improvements through the use of automated laboratory and comorbidity variables. Med. Care 48, 739–744 (2010).

  36. 36.

    Choi, E. et al. Doctor AI: predicting clinical events via recurrent neural networks. In Machine Learning for Healthcare 301–318 (2016).

  37. 37.

    Che, Z. et al. Recurrent neural networks for multivariate time series with missing values. Rep. 8, 1–12 (2018).

  38. 38.

    Suresh, H. et al. Clinical intervention prediction and understanding with deep neural networks. PMLR 68, 322–377 (2017).

  39. 39.

    Johnson, A. E. W.et al. Mimic-iii, a freely accessible critical care database. Sci. Data 3, 160035 (2016).

  40. 40.

    Mao, Q.et al. Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and icu. BMJ Open 8, e017833 (2018).

  41. 41.

    Abbeel, P. & Ng, A. Y. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenty-First International Conference on Machine Learning 1 (ACM, 2004).

  42. 42.

    Ratliff, N. D., Silver, D. & Bagnell, J. A. Learning to search: functional gradient techniques for imitation learning. Autonomous Robots 27, 25–53 (2009).

  43. 43.

    Schulman, J. et al. A case study of trajectory transfer through non-rigid registration for a simplified suturing scenario. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 4111–4117 (IEEE, 2013).

  44. 44.

    Mayer, H.et al. A system for robotic heart surgery that learns to tie knots using recurrent neural networks. Adv. Robot 22, 1521–1537 (2008).

  45. 45.

    Shvets, A., Rakhlin, A., Kalinin, A. A. and Iglovikov, V. Automatic instrument segmentation in robot-assisted surgery using deep learning. Preprint at https://arxiv.org/abs/1803.01207 (2018).

  46. 46.

    Jin, A. et al. Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks. Preprint at https://arxiv.org/abs/1802.08774 (2018).

  47. 47.

    Loh, P.-R. et al. Efficient bayesian mixed-model analysis increases association power in large cohorts. Nat. Genet. 47, 284 (2015).

  48. 48.

    Abadi, M. et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. http://download.tensorflow.org/paper/whitepaper2015.pdf (2015).

  49. 49.

    Tran, D. and Blei, D. M. Implicit causal models for genome-wide association studies. In International Conference on Learning Representations (2018).

  50. 50.

    Lee, S.-I. et al. Learning a prior on regulatory potential from eqtl data. PLoS Genet. 5, e1000358 (2009).

  51. 51.

    Kircher, M. et al. A general framework for estimating the relative pathogenicity of human genetic variants. Nat. Genet. 46, 310–315 (2014).

  52. 52.

    Quang, D., Chen, Y. & Xie, X. Dann: a deep learning approach for annotating the pathogenicity of genetic variants. Bioinformatics 31, 761–3 (2015).

  53. 53.

    Dudley, J. T. et al. Personalized medicine: from genotypes, molecular phenotypes and the quantified self, towards improved medicine. In Pacific Symposium on Biocomputing 342–346 (2014).

  54. 54.

    Leung, M. K. K., Delong, A., Alipanahi, B. & Frey, B. J. Machine learning in genomic medicine: a review of computational problems and data sets. Proc. IEEE 104, 176–197 (2016).

  55. 55.

    Xiong, H. Y. et al. The human splicing code reveals new insights into the genetic determinants of disease. Science 347, 1254806 (2015).

  56. 56.

    Alipanahi, B. et al. Predicting the sequence specificities of dna-and rna-binding proteins by deep learning. Nature Biotechnol. 33, 831–838 (2015).

  57. 57.

    Snyder, T. M., Khush, K. K., Valantine, H. A. & Quake, S. R. Universal noninvasive detection of solid organ transplant rejection. Proc. Natl Acad. Sci. USA 108, 6229–6234 (2011).

  58. 58.

    Abril, M. K. et al. Diagnosis of capnocytophaga canimorsus sepsis by whole-genome next-generation sequencing. In Open Forum Infectious Diseases Vol. 3, ofw144 (Oxford University Press, 2016).

  59. 59.

    Forshew, T. et al. Noninvasive identification and monitoring of cancer mutations by targeted deep sequencing of plasma dna. Sci. Transl. Med. 4, 136ra68–136ra68 (2012).

  60. 60.

    Fan, H. C. et al. Non-invasive prenatal measurement of the fetal genome. Nature 487, 320–324 (2012).

  61. 61.

    Ching, T. et al. Opportunities and obstacles for deep learning in biology and medicine. Preprint at https://doi.org/10.1101/142760 (2017).

  62. 62.

    Angermueller, C., Lee, H. J., Reik, W. & Stegle, O. Deepcpg: accurate prediction of single-cell dna methylation states using deep learning. Genome Biol. 18, 67 (2017).

  63. 63.

    Chen, Y. et al. Gene expression inference with deep learning. Bioinformatics 32, 1832–1839 (2016).

  64. 64.

    Koh, P. W., Pierson, E. & Kundaje, A. Denoising genome-wide histone chip-seq with convolutional neural networks. Bioinformatics 33, i225–i233 (2017).

Download references

Acknowledgements

The authors would like to thank D. Wang, E. Dorfman, and A. Rajkomar for the visual design of the figures in this paper and P. Nejad for insightful conversation and ideas.

Author information

Author notes

  1. These authors contributed equally: Andre Esteva, Alexandre Robicquet.

Affiliations

  1. Stanford University, Stanford, CA, USA

    • Andre Esteva
    • , Alexandre Robicquet
    • , Bharath Ramsundar
    • , Volodymyr Kuleshov
    •  & Sebastian Thrun
  2. Google Research, San Jose, CA, USA

    • Mark DePristo
    • , Katherine Chou
    • , Claire Cui
    • , Greg Corrado
    •  & Jeff Dean

Authors

  1. Search for Andre Esteva in:

  2. Search for Alexandre Robicquet in:

  3. Search for Bharath Ramsundar in:

  4. Search for Volodymyr Kuleshov in:

  5. Search for Mark DePristo in:

  6. Search for Katherine Chou in:

  7. Search for Claire Cui in:

  8. Search for Greg Corrado in:

  9. Search for Sebastian Thrun in:

  10. Search for Jeff Dean in:

Contributions

B.R., V.K., M.D., and K.C. share second authorship. C.C., G.C., and S.T. share third authorship. J.D. is the principal investigator. A.E. and A.R. conceptualized the structure of the review and contributed to the computer vision and reinforcement learning sections. V.K., B.R., and M.D. contributed to the generalized deep learning section. K.C. and J.D. contributed to the natural language processing section. C.C., G.C., S.T., and J.D. oversaw the work. All authors contributed to multiple parts of the review, as well as the style and overall contents.

Competing interests

M.D., C.C., K.C., G.C. and J.D. are employees of Google Inc. This work was internally funded by Google Inc. G.C. is a board member at the Partnership on AI to Benefit People and Society. S.T. is an employee of Udacity, Inc. and the Kitty Hawk Corporation. He is on the faculty of Stanford University and Georgia Institute of Technology. B.R. is a partner of Computable LLC.​

Corresponding author

Correspondence to Andre Esteva.

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/s41591-018-0316-z