Rapid diagnosis and treatment of acute neurological illnesses such as stroke, hemorrhage, and hydrocephalus are critical to achieving positive outcomes and preserving neurologic function—‘time is brain’1,2,3,4,5. Although these disorders are often recognizable by their symptoms, the critical means of their diagnosis is rapid imaging6,7,8,9,10. Computer-aided surveillance of acute neurologic events in cranial imaging has the potential to triage radiology workflow, thus decreasing time to treatment and improving outcomes. Substantial clinical work has focused on computer-assisted diagnosis (CAD), whereas technical work in volumetric image analysis has focused primarily on segmentation. 3D convolutional neural networks (3D-CNNs) have primarily been used for supervised classification on 3D modeling and light detection and ranging (LiDAR) data11,12,13,14,15. Here, we demonstrate a 3D-CNN architecture that performs weakly supervised classification to screen head CT images for acute neurologic events. Features were automatically learned from a clinical radiology dataset comprising 37,236 head CTs and were annotated with a semisupervised natural-language processing (NLP) framework16. We demonstrate the effectiveness of our approach to triage radiology workflow and accelerate the time to diagnosis from minutes to seconds through a randomized, double-blinded, prospective trial in a simulated clinical environment.
Subscribe to Journal
Get full journal access for 1 year
only $18.75 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Furlan, A. J. Time is brain. Stroke 37, 2863–2864 (2006).
Del Zoppo, G. J., Saver, J. L., Jauch, E. C., Adams, H. P. Jr & American Heart Association Stroke Council. Expansion of the time window for treatment of acute ischemic stroke with intravenous tissue plasminogen activator: a science advisory from the American Heart Association/American Stroke Association. Stroke 40, 2945–2948 (2009).
Jovin, T. G. et al. Thrombectomy within 8h after symptom onset in ischemic stroke. N. Engl. J. Med. 372, 2296–2306 (2015).
Saver, J. L. Time is brain: quantified. Stroke 37, 263–266 (2006).
Seelig, J. M. et al. Traumatic acute subdural hematoma: major mortality reduction in comatose patients treated within four hours. N. Engl. J. Med. 304, 1511–1518 (1981).
National Collaborating Centre for Chronic Conditions (UK). Stroke: National Clinical Guideline for Diagnosis and Initial Management of Acute Stroke and Transient Ischaemic Attack (TIA). (Royal College of Physicians, London, 2011).
Broderick, J. P. et al. Guidelines for the management of spontaneous intracerebral hemorrhage: a statement for healthcare professionals from a special writing group of the Stroke Council, American Heart Association. Stroke 30, (905–915 (1999).
Ferro, J. M. et al. Diagnosis of stroke by the nonneurologist: a validation study. Stroke 29, 1106–1109 (1998).
Mullins, M. E. et al. CT and conventional and diffusion-weighted MR imaging in acute stroke: study in 691 patients at presentation to the emergency department. Radiology 224, 353–360 (2002).
Navi, B. B. et al. The use of neuroimaging studies and neurological consultation to evaluate dizzy patients in the emergency department. Neurohospitalist 3, 7–14 (2013).
Sedaghat, N., Zolfaghari, M. & Brox, T. Orientation-boosted voxel nets for 3D object recognition. Preprint at https://arxiv.org/abs/1604.03351/ (2016).
Maturana, D. & Scherer, S. VoxNet: a 3D convolutional neural network for real-time object recognition. in 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 922–928 (IEEE, Piscataway, NJ, USA, 2015).
Hegde, V. & Zadeh, R. FusionNet: 3D object classification using multiple data representations. Preprint at https://arxiv.org/abs/1607.05695/ (2016).
Sinha, A., Bai, J. & Ramani, K. Deep learning 3D shape surfaces using geometry images. in Computer Vision – ECCV 2016 223–240 (Springer, Cham, Switzerland, 2016).
Brock, A., Lim, T., Ritchie, J. M. & Weston, N. Generative and discriminative voxel modeling with convolutional neural networks. Preprint at https://arxiv.org/abs/1608.04236/ (2016).
Shin, H.-C. et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging 35, 1285–1298 (2016).
Deng, J. et al. ImageNet: a large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248–255 (IEEE, Piscataway, NJ, USA, 2009).
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. Preprint at https://arxiv.org/abs/1512.03385/ (2015).
Durand, T., Mordan, T., Thome, N. & Cord, M. Wildcat: weakly supervised learning of deep convnets for image classification, pointwise localization and segmentation. in IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). (IEEE, Piscataway, NJ, USA, 2017).
Chen, P.-H., Botzolakis, E., Mohan, S., Nick Bryan, R. & Cook, T. Feasibility of streamlining an interactive Bayesian-based diagnostic support tool designed for clinical practice. in Medical Imaging2016: PACS and Imaging Informatics: Next Generation and Innovations Vol. 9789, 97890C (International Society for Optics and Photonics, Bellingham, WA, USA, 2016).
Doi, K. Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput. Med. Imaging Graph. 31, 198–211 (2007).
Lehman, C. D. et al. Diagnostic accuracy of digital screening mammography with and without computer-aided detection. JAMA Intern. Med. 175, 1828–1837 (2015).
Wen, W., Wu, C., Wang, Y., Chen, Y. & Li, H. Learning structured sparsity in deep neural networks. Preprint at http://arxiv.org/abs/1608.03665/ (2016).
Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).
Lakhani, P. & Sundaram, B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology 284, 574–582 (2017).
Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. J. Am. Med. Assoc. 316, 2402–2410 (2016).
Liu, Y. et al. Detecting cancer metastases on gigapixel pathology images. Preprint at https://arxiv.org/abs/1703.02442/ (2017).
Oktay, O. et al. Anatomically constrained neural networks (ACNN): application to cardiac image enhancement and segmentation. IEEE Trans. Med. Imaging 37, 384–395 (2017).
Li, X. et al. H-DenseUNet: hybrid densely connected UNet for liver and liver tumor segmentation from CT volumes. Preprint at https://arxiv.org/abs/1709.07330/ (2017).
Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T. & Ronneberger, O. 3D U-Net: learning dense volumetric segmentation from sparse annotation. Preprint at https://arxiv.org/abs/1606.06650/ (2016).
Brosch, T. et al. Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation. IEEE Trans. Med. Imaging 35, 1229–1239 (2016).
Kamnitsas, K. et al. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med. Image Anal. 36, 61–78 (2017).
Agarwal, V. et al. Learning statistical models of phenotypes using noisy labeled training data. J. Am. Med. Inform. Assoc. 23, 1166–1173 (2016).
Halpern, Y., Choi, Y., Horng, S. & Sontag, D. Using anchors to estimate clinical state without labeled data. AMIA Annu. Symp. Proc. 2014, 606–615 (2014).
Merkow, J. et al. DeepRadiologyNet: radiologist level pathology detection in CT head images. Preprint at https://arxiv.org/abs/1711.09313/ (2017).
Riedel, C. H. et al. Assessment of thrombus in acute middle cerebral artery occlusion using thin-slice nonenhanced computed tomography reconstructions. Stroke 41, 1659–1664 (2010).
Kim, E. Y. et al. Detection of thrombus in acute ischemic stroke: value of thin-section noncontrast-computed tomography. Stroke 36, 2745–2747 (2005).
Rubinstein, D., Escott, E. J. & Mestek, M. F. Computed tomographic scans of minimally displaced type II odontoid fractures. J. Trauma 40, 204–210 (1996).
Bush, K., Huikeshoven, M. & Wong, N. Nasofrontal outflow tract visibility in computed tomography imaging of frontal sinus fractures. Craniomaxillofac. Trauma Reconstr. 6, 237–240 (2013).
Zech, J. et al. Natural language-based machine learning models for the annotation of clinical radiology reports. Radiology 287, 570–580 (2018).
Viertel, V. G. et al. Reporting of critical findings in neuroradiology. AJR Am. J. Roentgenol. 200, 1132–1137 (2013).
He, K., Zhang, X., Ren, S. & Sun, J. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. in 2015 IEEE International Conference on Computer Vision (ICCV) (IEEE, Piscataway, NJ, USA, 2015)
We thank E. Gordon for assistance with data collection for the purposes of building the NLP pipeline. We also thank the National Library of Medicine for making the UMLS Metathesaurus available.
J.L. currently works for Merck in addition to his role as an adjunct professor at Boston University. M.B. currently works for Verily Life Sciences in addition to his role as a medical student at Mount Sinai. All of the present work was performed within the Mount Sinai Health System, and Merck and Verily played no role in the research and have no commercial interest in it.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Titano, J.J., Badgeley, M., Schefflein, J. et al. Automated deep-neural-network surveillance of cranial images for acute neurologic events. Nat Med 24, 1337–1341 (2018). https://doi.org/10.1038/s41591-018-0147-y
An automated tissue-to-diagnosis pipeline using intraoperative stimulated Raman histology and deep learning
Molecular & Cellular Oncology (2020)
World Neurosurgery (2020)
Global Spine Journal (2020)
PENet—a scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric CT imaging
npj Digital Medicine (2020)