Abstract

Rapid diagnosis and treatment of acute neurological illnesses such as stroke, hemorrhage, and hydrocephalus are critical to achieving positive outcomes and preserving neurologic function—‘time is brain’1,2,3,4,5. Although these disorders are often recognizable by their symptoms, the critical means of their diagnosis is rapid imaging6,7,8,9,10. Computer-aided surveillance of acute neurologic events in cranial imaging has the potential to triage radiology workflow, thus decreasing time to treatment and improving outcomes. Substantial clinical work has focused on computer-assisted diagnosis (CAD), whereas technical work in volumetric image analysis has focused primarily on segmentation. 3D convolutional neural networks (3D-CNNs) have primarily been used for supervised classification on 3D modeling and light detection and ranging (LiDAR) data11,12,13,14,15. Here, we demonstrate a 3D-CNN architecture that performs weakly supervised classification to screen head CT images for acute neurologic events. Features were automatically learned from a clinical radiology dataset comprising 37,236 head CTs and were annotated with a semisupervised natural-language processing (NLP) framework16. We demonstrate the effectiveness of our approach to triage radiology workflow and accelerate the time to diagnosis from minutes to seconds through a randomized, double-blinded, prospective trial in a simulated clinical environment.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    Furlan, A. J. Time is brain. Stroke 37, 2863–2864 (2006).

  2. 2.

    Del Zoppo, G. J., Saver, J. L., Jauch, E. C., Adams, H. P. Jr & American Heart Association Stroke Council. Expansion of the time window for treatment of acute ischemic stroke with intravenous tissue plasminogen activator: a science advisory from the American Heart Association/American Stroke Association. Stroke 40, 2945–2948 (2009).

  3. 3.

    Jovin, T. G. et al. Thrombectomy within 8h after symptom onset in ischemic stroke. N. Engl. J. Med. 372, 2296–2306 (2015).

  4. 4.

    Saver, J. L. Time is brain: quantified. Stroke 37, 263–266 (2006).

  5. 5.

    Seelig, J. M. et al. Traumatic acute subdural hematoma: major mortality reduction in comatose patients treated within four hours. N. Engl. J. Med. 304, 1511–1518 (1981).

  6. 6.

    National Collaborating Centre for Chronic Conditions (UK). Stroke: National Clinical Guideline for Diagnosis and Initial Management of Acute Stroke and Transient Ischaemic Attack (TIA). (Royal College of Physicians, London, 2011).

  7. 7.

    Broderick, J. P. et al. Guidelines for the management of spontaneous intracerebral hemorrhage: a statement for healthcare professionals from a special writing group of the Stroke Council, American Heart Association. Stroke 30, (905–915 (1999).

  8. 8.

    Ferro, J. M. et al. Diagnosis of stroke by the nonneurologist: a validation study. Stroke 29, 1106–1109 (1998).

  9. 9.

    Mullins, M. E. et al. CT and conventional and diffusion-weighted MR imaging in acute stroke: study in 691 patients at presentation to the emergency department. Radiology 224, 353–360 (2002).

  10. 10.

    Navi, B. B. et al. The use of neuroimaging studies and neurological consultation to evaluate dizzy patients in the emergency department. Neurohospitalist 3, 7–14 (2013).

  11. 11.

    Sedaghat, N., Zolfaghari, M. & Brox, T. Orientation-boosted voxel nets for 3D object recognition. Preprint at https://arxiv.org/abs/1604.03351/ (2016).

  12. 12.

    Maturana, D. & Scherer, S. VoxNet: a 3D convolutional neural network for real-time object recognition. in 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 922–928 (IEEE, Piscataway, NJ, USA, 2015).

  13. 13.

    Hegde, V. & Zadeh, R. FusionNet: 3D object classification using multiple data representations. Preprint at https://arxiv.org/abs/1607.05695/ (2016).

  14. 14.

    Sinha, A., Bai, J. & Ramani, K. Deep learning 3D shape surfaces using geometry images. in Computer Vision – ECCV 2016 223–240 (Springer, Cham, Switzerland, 2016).

  15. 15.

    Brock, A., Lim, T., Ritchie, J. M. & Weston, N. Generative and discriminative voxel modeling with convolutional neural networks. Preprint at https://arxiv.org/abs/1608.04236/ (2016).

  16. 16.

    Shin, H.-C. et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging 35, 1285–1298 (2016).

  17. 17.

    Deng, J. et al. ImageNet: a large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248–255 (IEEE, Piscataway, NJ, USA, 2009).

  18. 18.

    He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. Preprint at https://arxiv.org/abs/1512.03385/ (2015).

  19. 19.

    Durand, T., Mordan, T., Thome, N. & Cord, M. Wildcat: weakly supervised learning of deep convnets for image classification, pointwise localization and segmentation. in IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). (IEEE, Piscataway, NJ, USA, 2017).

  20. 20.

    Chen, P.-H., Botzolakis, E., Mohan, S., Nick Bryan, R. & Cook, T. Feasibility of streamlining an interactive Bayesian-based diagnostic support tool designed for clinical practice. in Medical Imaging2016: PACS and Imaging Informatics: Next Generation and Innovations Vol. 9789, 97890C (International Society for Optics and Photonics, Bellingham, WA, USA, 2016).

  21. 21.

    Doi, K. Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput. Med. Imaging Graph. 31, 198–211 (2007).

  22. 22.

    Lehman, C. D. et al. Diagnostic accuracy of digital screening mammography with and without computer-aided detection. JAMA Intern. Med. 175, 1828–1837 (2015).

  23. 23.

    Wen, W., Wu, C., Wang, Y., Chen, Y. & Li, H. Learning structured sparsity in deep neural networks. Preprint at http://arxiv.org/abs/1608.03665/ (2016).

  24. 24.

    Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

  25. 25.

    Lakhani, P. & Sundaram, B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology 284, 574–582 (2017).

  26. 26.

    Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. J. Am. Med. Assoc. 316, 2402–2410 (2016).

  27. 27.

    Liu, Y. et al. Detecting cancer metastases on gigapixel pathology images. Preprint at https://arxiv.org/abs/1703.02442/ (2017).

  28. 28.

    Oktay, O. et al. Anatomically constrained neural networks (ACNN): application to cardiac image enhancement and segmentation. IEEE Trans. Med. Imaging 37, 384–395 (2017).

  29. 29.

    Li, X. et al. H-DenseUNet: hybrid densely connected UNet for liver and liver tumor segmentation from CT volumes. Preprint at https://arxiv.org/abs/1709.07330/ (2017).

  30. 30.

    Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T. & Ronneberger, O. 3D U-Net: learning dense volumetric segmentation from sparse annotation. Preprint at https://arxiv.org/abs/1606.06650/ (2016).

  31. 31.

    Brosch, T. et al. Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation. IEEE Trans. Med. Imaging 35, 1229–1239 (2016).

  32. 32.

    Kamnitsas, K. et al. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med. Image Anal. 36, 61–78 (2017).

  33. 33.

    Agarwal, V. et al. Learning statistical models of phenotypes using noisy labeled training data. J. Am. Med. Inform. Assoc. 23, 1166–1173 (2016).

  34. 34.

    Halpern, Y., Choi, Y., Horng, S. & Sontag, D. Using anchors to estimate clinical state without labeled data. AMIA Annu. Symp. Proc. 2014, 606–615 (2014).

  35. 35.

    Merkow, J. et al. DeepRadiologyNet: radiologist level pathology detection in CT head images. Preprint at https://arxiv.org/abs/1711.09313/ (2017).

  36. 36.

    Riedel, C. H. et al. Assessment of thrombus in acute middle cerebral artery occlusion using thin-slice nonenhanced computed tomography reconstructions. Stroke 41, 1659–1664 (2010).

  37. 37.

    Kim, E. Y. et al. Detection of thrombus in acute ischemic stroke: value of thin-section noncontrast-computed tomography. Stroke 36, 2745–2747 (2005).

  38. 38.

    Rubinstein, D., Escott, E. J. & Mestek, M. F. Computed tomographic scans of minimally displaced type II odontoid fractures. J. Trauma 40, 204–210 (1996).

  39. 39.

    Bush, K., Huikeshoven, M. & Wong, N. Nasofrontal outflow tract visibility in computed tomography imaging of frontal sinus fractures. Craniomaxillofac. Trauma Reconstr. 6, 237–240 (2013).

  40. 40.

    Zech, J. et al. Natural language-based machine learning models for the annotation of clinical radiology reports. Radiology 287, 570–580 (2018).

  41. 41.

    Viertel, V. G. et al. Reporting of critical findings in neuroradiology. AJR Am. J. Roentgenol. 200, 1132–1137 (2013).

  42. 42.

    He, K., Zhang, X., Ren, S. & Sun, J. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. in 2015 IEEE International Conference on Computer Vision (ICCV) (IEEE, Piscataway, NJ, USA, 2015)

Download references

Acknowledgements

We thank E. Gordon for assistance with data collection for the purposes of building the NLP pipeline. We also thank the National Library of Medicine for making the UMLS Metathesaurus available.

Author information

Author notes

  1. These authors contributed equally: Joseph J. Titano, Marcus Badgeley.

Affiliations

  1. Department of Radiology, Icahn School of Medicine, New York, NY, USA

    • Joseph J. Titano
    • , Javin Schefflein
    • , Andres Su
    • , Michael Cai
    • , Nathaniel Swinburne
    • , John Zech
    •  & Burton Drayer
  2. Department of Neurological Surgery, Icahn School of Medicine, New York, NY, USA

    • Marcus Badgeley
    • , Margaret Pain
    • , Joshua Bederson
    • , J. Mocco
    • , Samuel Cho
    • , Anthony Costa
    •  & Eric K. Oermann
  3. Department of Orthopedic Surgery, Icahn School of Medicine, New York, NY, USA

    • Jun Kim
    •  & Samuel Cho
  4. Bioengineering and Bioinformatics, Boston University, Boston, MA, USA

    • Joseph Lehar

Authors

  1. Search for Joseph J. Titano in:

  2. Search for Marcus Badgeley in:

  3. Search for Javin Schefflein in:

  4. Search for Margaret Pain in:

  5. Search for Andres Su in:

  6. Search for Michael Cai in:

  7. Search for Nathaniel Swinburne in:

  8. Search for John Zech in:

  9. Search for Jun Kim in:

  10. Search for Joshua Bederson in:

  11. Search for J. Mocco in:

  12. Search for Burton Drayer in:

  13. Search for Joseph Lehar in:

  14. Search for Samuel Cho in:

  15. Search for Anthony Costa in:

  16. Search for Eric K. Oermann in:

Contributions

A.C. and E.K.O. designed and built the computing environment for this project. J.J.T., J.M., and E.K.O. conceived the study. A.C., J.L., and E.K.O. developed the computer-vision algorithms. J.Z., E.K.O., J.S., and M.P. developed the NLP algorithms. J.J.T., M.P., A.S., J.K., and S.C. built the crowdsourcing platform for obtaining ground-truth labels. J.J.T., J.S., M.C., N.S., and J.Z. generated gold-standard labels and assembled the datasets. M.B. and E.K.O. designed and oversaw the clinical simulation. M.B. performed the statistical analysis and designed the figures. J.J.T., J.B., B.D., and E.K.O. supervised the project. All authors contributed to writing and editing the manuscript.

Competing interests

J.L. currently works for Merck in addition to his role as an adjunct professor at Boston University. M.B. currently works for Verily Life Sciences in addition to his role as a medical student at Mount Sinai. All of the present work was performed within the Mount Sinai Health System, and Merck and Verily played no role in the research and have no commercial interest in it.

Corresponding author

Correspondence to Eric K. Oermann.

Supplementary Information

  1. Supplementary Text and Figures

    Supplementary Figures 1–4 and Supplementary Tables 2–5

  2. Reporting Summary

  3. Supplementary Table 1

    UMLS diagnosis - criticality abstraction

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/s41591-018-0147-y

Further reading