Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Automated deep-neural-network surveillance of cranial images for acute neurologic events

Nature Medicine volume 24, pages 1337–1341 (2018)Cite this article

Subjects

Abstract

Rapid diagnosis and treatment of acute neurological illnesses such as stroke, hemorrhage, and hydrocephalus are critical to achieving positive outcomes and preserving neurologic function—‘time is brain’1,2,3,4,5. Although these disorders are often recognizable by their symptoms, the critical means of their diagnosis is rapid imaging6,7,8,9,10. Computer-aided surveillance of acute neurologic events in cranial imaging has the potential to triage radiology workflow, thus decreasing time to treatment and improving outcomes. Substantial clinical work has focused on computer-assisted diagnosis (CAD), whereas technical work in volumetric image analysis has focused primarily on segmentation. 3D convolutional neural networks (3D-CNNs) have primarily been used for supervised classification on 3D modeling and light detection and ranging (LiDAR) data11,12,13,14,15. Here, we demonstrate a 3D-CNN architecture that performs weakly supervised classification to screen head CT images for acute neurologic events. Features were automatically learned from a clinical radiology dataset comprising 37,236 head CTs and were annotated with a semisupervised natural-language processing (NLP) framework16. We demonstrate the effectiveness of our approach to triage radiology workflow and accelerate the time to diagnosis from minutes to seconds through a randomized, double-blinded, prospective trial in a simulated clinical environment.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Data pipeline, CNN triage concept, and analysis of current CT head workflow.
Fig. 2: Systems for encoding images and mapping diseases to urgency.
Fig. 3: Human and algorithm classifier performance and diagnosis heterogeneity.
Fig. 4: Interpretation speed and CNN queue triage.

Similar content being viewed by others

References

  1. Furlan, A. J. Time is brain. Stroke 37, 2863–2864 (2006).

    Article  Google Scholar 

  2. Del Zoppo, G. J., Saver, J. L., Jauch, E. C., Adams, H. P. Jr & American Heart Association Stroke Council. Expansion of the time window for treatment of acute ischemic stroke with intravenous tissue plasminogen activator: a science advisory from the American Heart Association/American Stroke Association. Stroke 40, 2945–2948 (2009).

    Article  Google Scholar 

  3. Jovin, T. G. et al. Thrombectomy within 8h after symptom onset in ischemic stroke. N. Engl. J. Med. 372, 2296–2306 (2015).

    Article  CAS  Google Scholar 

  4. Saver, J. L. Time is brain: quantified. Stroke 37, 263–266 (2006).

    Article  Google Scholar 

  5. Seelig, J. M. et al. Traumatic acute subdural hematoma: major mortality reduction in comatose patients treated within four hours. N. Engl. J. Med. 304, 1511–1518 (1981).

    Article  CAS  Google Scholar 

  6. National Collaborating Centre for Chronic Conditions (UK). Stroke: National Clinical Guideline for Diagnosis and Initial Management of Acute Stroke and Transient Ischaemic Attack (TIA). (Royal College of Physicians, London, 2011).

  7. Broderick, J. P. et al. Guidelines for the management of spontaneous intracerebral hemorrhage: a statement for healthcare professionals from a special writing group of the Stroke Council, American Heart Association. Stroke 30, (905–915 (1999).

    Google Scholar 

  8. Ferro, J. M. et al. Diagnosis of stroke by the nonneurologist: a validation study. Stroke 29, 1106–1109 (1998).

    Article  CAS  Google Scholar 

  9. Mullins, M. E. et al. CT and conventional and diffusion-weighted MR imaging in acute stroke: study in 691 patients at presentation to the emergency department. Radiology 224, 353–360 (2002).

    Article  Google Scholar 

  10. Navi, B. B. et al. The use of neuroimaging studies and neurological consultation to evaluate dizzy patients in the emergency department. Neurohospitalist 3, 7–14 (2013).

    Article  Google Scholar 

  11. Sedaghat, N., Zolfaghari, M. & Brox, T. Orientation-boosted voxel nets for 3D object recognition. Preprint at https://arxiv.org/abs/1604.03351/ (2016).

  12. Maturana, D. & Scherer, S. VoxNet: a 3D convolutional neural network for real-time object recognition. in 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 922–928 (IEEE, Piscataway, NJ, USA, 2015).

  13. Hegde, V. & Zadeh, R. FusionNet: 3D object classification using multiple data representations. Preprint at https://arxiv.org/abs/1607.05695/ (2016).

  14. Sinha, A., Bai, J. & Ramani, K. Deep learning 3D shape surfaces using geometry images. in Computer Vision – ECCV 2016 223–240 (Springer, Cham, Switzerland, 2016).

  15. Brock, A., Lim, T., Ritchie, J. M. & Weston, N. Generative and discriminative voxel modeling with convolutional neural networks. Preprint at https://arxiv.org/abs/1608.04236/ (2016).

  16. Shin, H.-C. et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging 35, 1285–1298 (2016).

    Article  Google Scholar 

  17. Deng, J. et al. ImageNet: a large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248–255 (IEEE, Piscataway, NJ, USA, 2009).

    Chapter  Google Scholar 

  18. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. Preprint at https://arxiv.org/abs/1512.03385/ (2015).

  19. Durand, T., Mordan, T., Thome, N. & Cord, M. Wildcat: weakly supervised learning of deep convnets for image classification, pointwise localization and segmentation. in IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). (IEEE, Piscataway, NJ, USA, 2017).

    Google Scholar 

  20. Chen, P.-H., Botzolakis, E., Mohan, S., Nick Bryan, R. & Cook, T. Feasibility of streamlining an interactive Bayesian-based diagnostic support tool designed for clinical practice. in Medical Imaging2016: PACS and Imaging Informatics: Next Generation and Innovations Vol. 9789, 97890C (International Society for Optics and Photonics, Bellingham, WA, USA, 2016).

  21. Doi, K. Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput. Med. Imaging Graph. 31, 198–211 (2007).

    Article  Google Scholar 

  22. Lehman, C. D. et al. Diagnostic accuracy of digital screening mammography with and without computer-aided detection. JAMA Intern. Med. 175, 1828–1837 (2015).

    Article  Google Scholar 

  23. Wen, W., Wu, C., Wang, Y., Chen, Y. & Li, H. Learning structured sparsity in deep neural networks. Preprint at http://arxiv.org/abs/1608.03665/ (2016).

  24. Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

    Article  CAS  Google Scholar 

  25. Lakhani, P. & Sundaram, B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology 284, 574–582 (2017).

    Article  Google Scholar 

  26. Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. J. Am. Med. Assoc. 316, 2402–2410 (2016).

    Article  Google Scholar 

  27. Liu, Y. et al. Detecting cancer metastases on gigapixel pathology images. Preprint at https://arxiv.org/abs/1703.02442/ (2017).

  28. Oktay, O. et al. Anatomically constrained neural networks (ACNN): application to cardiac image enhancement and segmentation. IEEE Trans. Med. Imaging 37, 384–395 (2017).

    Article  Google Scholar 

  29. Li, X. et al. H-DenseUNet: hybrid densely connected UNet for liver and liver tumor segmentation from CT volumes. Preprint at https://arxiv.org/abs/1709.07330/ (2017).

  30. Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T. & Ronneberger, O. 3D U-Net: learning dense volumetric segmentation from sparse annotation. Preprint at https://arxiv.org/abs/1606.06650/ (2016).

  31. Brosch, T. et al. Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation. IEEE Trans. Med. Imaging 35, 1229–1239 (2016).

    Article  Google Scholar 

  32. Kamnitsas, K. et al. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med. Image Anal. 36, 61–78 (2017).

    Article  Google Scholar 

  33. Agarwal, V. et al. Learning statistical models of phenotypes using noisy labeled training data. J. Am. Med. Inform. Assoc. 23, 1166–1173 (2016).

    Article  Google Scholar 

  34. Halpern, Y., Choi, Y., Horng, S. & Sontag, D. Using anchors to estimate clinical state without labeled data. AMIA Annu. Symp. Proc. 2014, 606–615 (2014).

  35. Merkow, J. et al. DeepRadiologyNet: radiologist level pathology detection in CT head images. Preprint at https://arxiv.org/abs/1711.09313/ (2017).

  36. Riedel, C. H. et al. Assessment of thrombus in acute middle cerebral artery occlusion using thin-slice nonenhanced computed tomography reconstructions. Stroke 41, 1659–1664 (2010).

    Article  Google Scholar 

  37. Kim, E. Y. et al. Detection of thrombus in acute ischemic stroke: value of thin-section noncontrast-computed tomography. Stroke 36, 2745–2747 (2005).

    Article  Google Scholar 

  38. Rubinstein, D., Escott, E. J. & Mestek, M. F. Computed tomographic scans of minimally displaced type II odontoid fractures. J. Trauma 40, 204–210 (1996).

    Article  CAS  Google Scholar 

  39. Bush, K., Huikeshoven, M. & Wong, N. Nasofrontal outflow tract visibility in computed tomography imaging of frontal sinus fractures. Craniomaxillofac. Trauma Reconstr. 6, 237–240 (2013).

    Article  Google Scholar 

  40. Zech, J. et al. Natural language-based machine learning models for the annotation of clinical radiology reports. Radiology 287, 570–580 (2018).

    Article  Google Scholar 

  41. Viertel, V. G. et al. Reporting of critical findings in neuroradiology. AJR Am. J. Roentgenol. 200, 1132–1137 (2013).

    Article  Google Scholar 

  42. He, K., Zhang, X., Ren, S. & Sun, J. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. in 2015 IEEE International Conference on Computer Vision (ICCV) (IEEE, Piscataway, NJ, USA, 2015)

Download references

Acknowledgements

We thank E. Gordon for assistance with data collection for the purposes of building the NLP pipeline. We also thank the National Library of Medicine for making the UMLS Metathesaurus available.

Author information

Author notes

  1. These authors contributed equally: Joseph J. Titano, Marcus Badgeley.

Authors and Affiliations

  1. Department of Radiology, Icahn School of Medicine, New York, NY, USA

    Joseph J. Titano, Javin Schefflein, Andres Su, Michael Cai, Nathaniel Swinburne, John Zech & Burton Drayer

  2. Department of Neurological Surgery, Icahn School of Medicine, New York, NY, USA

    Marcus Badgeley, Margaret Pain, Joshua Bederson, J. Mocco, Samuel Cho, Anthony Costa & Eric K. Oermann

  3. Department of Orthopedic Surgery, Icahn School of Medicine, New York, NY, USA

    Jun Kim & Samuel Cho

  4. Bioengineering and Bioinformatics, Boston University, Boston, MA, USA

    Joseph Lehar

Authors
  1. Joseph J. Titano
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marcus Badgeley
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Javin Schefflein
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Margaret Pain
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Andres Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Michael Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Nathaniel Swinburne
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. John Zech
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jun Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Joshua Bederson
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. J. Mocco
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Burton Drayer
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Joseph Lehar
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Samuel Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Anthony Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Eric K. Oermann
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

A.C. and E.K.O. designed and built the computing environment for this project. J.J.T., J.M., and E.K.O. conceived the study. A.C., J.L., and E.K.O. developed the computer-vision algorithms. J.Z., E.K.O., J.S., and M.P. developed the NLP algorithms. J.J.T., M.P., A.S., J.K., and S.C. built the crowdsourcing platform for obtaining ground-truth labels. J.J.T., J.S., M.C., N.S., and J.Z. generated gold-standard labels and assembled the datasets. M.B. and E.K.O. designed and oversaw the clinical simulation. M.B. performed the statistical analysis and designed the figures. J.J.T., J.B., B.D., and E.K.O. supervised the project. All authors contributed to writing and editing the manuscript.

Corresponding author

Correspondence to Eric K. Oermann.

Ethics declarations

Competing interests

J.L. currently works for Merck in addition to his role as an adjunct professor at Boston University. M.B. currently works for Verily Life Sciences in addition to his role as a medical student at Mount Sinai. All of the present work was performed within the Mount Sinai Health System, and Merck and Verily played no role in the research and have no commercial interest in it.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary Text and Figures

Supplementary Figures 1–4 and Supplementary Tables 2–5

Reporting Summary

Supplementary Table 1

UMLS diagnosis - criticality abstraction

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Titano, J.J., Badgeley, M., Schefflein, J. et al. Automated deep-neural-network surveillance of cranial images for acute neurologic events. Nat Med 24, 1337–1341 (2018). https://doi.org/10.1038/s41591-018-0147-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41591-018-0147-y

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing