Abstract
Deep learning is an emerging transformative tool in diagnostic medicine, yet limited access and the interpretability of learned parameters hinders widespread adoption. Here we have generated a diverse repository of 838,644 histopathologic images and used them to optimize and discretize learned representations into 512-dimensional feature vectors. Importantly, we show that individual machine-engineered features correlate with salient human-derived morphologic constructs and ontological relationships. Deciphering the overlap between human and machine reasoning may aid in eliminating biases and improving automation and accountability for artificial intelligence-assisted medicine.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The full image training datasets used to generate the DLFVs are available on Bitbucket (https://bitbucket.org/diamandislabii/faust-feature-vectors-2019) (https://doi.org/10.5281/zenodo.3234829). External testing images generated by the TCGA Research Network (http://cancergenome.nih.gov/) are also publicly available without restrictions.
Code availability
Code and both the general and 1p19q-codeletion trained CNN models are available on Bitbucket (https://bitbucket.org/diamandislabii/faust-feature-vectors-2019 and https://doi.org/10.5281/zenodo.3234829). A stand-alone computing capsule is available on Code Ocean for Feature Vector Extraction and Clustering (https://doi.org/10.24433/CO.3573560.v2) and for feature activation map (FAM) generation (https://doi.org/10.24433/CO.7749421.v1).
Change history
10 December 2019
An amendment to this paper has been published and can be accessed via a link at the top of the paper.
References
Lotz, J. M., Primack, J. & Madau, P. A new nonparametric approach to galaxy morphological classification. Astron. J. 128, 163–182 (2004).
Murphy, W. J. et al. Molecular phylogenetics and the origins of placental mammals. Nature 409, 614–618 (2001).
Louis, D. N. et al. The 2016 World Health Organization classification of tumors of the central nervous system: a summary. Acta Neuropathol. 131, 803–820 (2016).
Nicholson, A. G. et al. Inter-observer variation between pathologists in diffuse parenchymal lung disease. Thorax 59, 500–505 (2004).
Gilles, F. H. et al. Pathologist interobserver variability of histologic features in childhood brain tumors: results from the CCG-945 study. Pediatr. Dev. Pathol. 11, 108–117 (2008).
Allard, F. D. et al. Intraobserver and interobserver variability in the assessment of dysplasia in ampullary mucosal biopsies. Am. J. Surg. Pathol. 42, 1095–1100 (2018).
Naghavi, M. et al. Global, regional and national age-sex specific mortality for 264 causes of death, 1980–2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet 390, 1151–1210 (2017).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Mobadersany, P. et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl Acad. Sci. USA 115, E2970–E2979 (2018).
Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).
Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559–1567 (2018).
Servick, K. Brain scientists dive into deep neural networks. Science 361, 1177 (2018).
Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. Preprint at http://arxiv.org/abs/1409.1556 (2014).
Holzinger, A. et al. Causability and explainabilty of artificial intelligence in medicine. WIRES Data Min. Knowl. Discov. 9, e1312 (2019).
Doshi-Velez, F. & Kim, B. Towards a rigorous science of interpretable machine learning. Preprint at http://arxiv.org/abs/1702.08608(2017).
Samek, W., Wiegand, T. & Müller, K.-R. Explainable artificial intelligence: understanding, visualizing and interpreting deep learning models. Preprint at http://arxiv.org/abs/1708.08296 (2017).
Faust, K. et al. Visualizing histopathologic deep learning classification and anomaly detection using nonlinear feature space dimensionality reduction. BMC Bioinformatics 19, 173 (2018).
Cancer Genome Atlas Research Network et al.Comprehensive, integrative genomic analysis of diffuse lower-grade gliomas. N. Engl. J. Med. 372, 2481–2498 (2015).
Acknowledgements
Funding support for this research and trainees is provided by the Princess Margaret Cancer Foundation, an Adam Coules Research Grant, the Department of Laboratory Medicine and Pathology at the University Health Network, the Brain Tumour Foundation of Canada, the American Society of Clinical Oncology Career Development Award (ASCO-CDA) and The Brain Tumour Charity Expanding Theories Research Grant Program (GN-000560). We thank P. Boutros for critical feedback and suggestions.
Author information
Authors and Affiliations
Contributions
K.F., U.D. and P.D. conceived the idea and approach. K.F. developed the computational workflow. S.B., R.A.Q., A.P., P.D. and R.v.O. reviewed and selected the included cases and annotated and generated the image dataset. K.F., U.D., R.v.O. and P.D. wrote the manuscript, with input from all other authors.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Faust, K., Bala, S., van Ommeren, R. et al. Intelligent feature engineering and ontological mapping of brain tumour histomorphologies by deep learning. Nat Mach Intell 1, 316–321 (2019). https://doi.org/10.1038/s42256-019-0068-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s42256-019-0068-6
This article is cited by
-
A novel feature fusion based deep learning framework for white blood cell classification
Journal of Ambient Intelligence and Humanized Computing (2023)
-
Generation of microbial colonies dataset with deep learning style transfer
Scientific Reports (2022)