Deep learning has triggered the current rise of artificial intelligence and is the workhorse of today’s machine intelligence. Numerous success stories have rapidly spread all over science, industry and society, but its limitations have only recently come into focus. In this Perspective we seek to distil how many of deep learning’s failures can be seen as different symptoms of the same underlying problem: shortcut learning. Shortcuts are decision rules that perform well on standard benchmarks but fail to transfer to more challenging testing conditions, such as real-world scenarios. Related issues are known in comparative psychology, education and linguistics, suggesting that shortcut learning may be a common characteristic of learning systems, biological and artificial alike. Based on these observations, we develop a set of recommendations for model interpretation and benchmarking, highlighting recent advances in machine learning to improve robustness and transferability from the lab to real-world applications.
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
He, K., Zhang, X., Ren, S. & Sun, J. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proc. IEEE Int. Conf. Computer Vision 1026–1034 (ACM, 2015).
Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016).
Moravčík, M. et al. Deepstack: expert-level artificial intelligence in heads-up no-limit poker. Science 356, 508–513 (2017).
Rajpurkar, P. et al. CheXNet: radiologist-level pneumonia detection on chest X-rays with deep learning. Preprint at https://arxiv.org/abs/1711.05225 (2017).
Devlin, J., Chang, M. W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. Annual Conf. North American Chapter of the Association for Computational Linguistics (ACL, 2019).
Rolnick, D. et al. Tackling climate change with machine learning. Preprint at https://arxiv.org/abs/1906.05433 (2019).
Reichstein, M. et al. Deep learning and process understanding for data-driven earth system science. Nature 566, 195–204 (2019).
Szegedy, C. et al. Intriguing properties of neural networks. In Proc. Int. Conf. Learning Representations (ICLR, 2014).
Beery, S., Van Horn, G. & Perona, P. Recognition in terra incognita. In European Conf. Computer Vision 456–473 (Springer, 2018).
Rosenfeld, A., Zemel, R. & Tsotsos, J. K. The elephant in the room. Preprint at https://arxiv.org/abs/1808.03305 (2018).
Heuer, H., Monz, C. & Smeulders, A. W. Generating captions without looking beyond objects. Preprint at https://arxiv.org/abs/1610.03708 (2016).
Buolamwini, J. & Gebru, T. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proc. ACM Fairness Accountability and Transparency 77–91 (PMLR, 2018).
Dastin, J. Amazon scraps secret AI recruiting tool that showed bias against women. Reuters https://reut.rs/2Od9fPr (2018).
Shane, J. Do neural nets dream of electric sheep? AI Wierdness https://aiweirdness.com/post/171451900302/do-neural-nets-dream-of-electric-sheep (2018).
Niven, T. & Kao, H.-Y. Probing neural network comprehension of natural language arguments. In Proc. 57th Annual Meeting of the Association of Computational Linguistics 4658–4664 (2019).
Jia, R. & Liang, P. Adversarial examples for evaluating reading comprehension systems. Preprint at https://arxiv.org/1707.07328 (2017).
Zech, J. R. et al. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS Med. 15, e1002683 (2018).
Bickel, S., Bru¨ckner, M. & Scheffer, T. Discriminative learning under covariate shift. J. Mach. Learn. Res. 10, 2137–2155 (2009).
Schölkopf, B. et al. On causal and anticausal learning. In Proc. Int. Conf. Machine Learning 1255–1262 (ICML, 2012).
Torralba, A. & Efros, A. A. Unbiased look at dataset bias. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (IEEE, 2011).
Branwen, G. The neural net tank urban legend. Gwern.net https://www.gwern.net/Tanks (2011).
Pfungst, O. Clever Hans (The Horse of Mr. Von Osten): A Contribution to Experimental Animal and Human Psychology (Holt, Rinehart and Winston, 1911).
Scouller, K. The influence of assessment method on students’ learning approaches: multiple choice question examination versus assignment essay. Higher Educ. 35, 453–472 (1998).
Wichmann, F. A., Drewes, J., Rosas, P. & Gegenfurtner, K. R. Animal detection in natural scenes: critical features revisited. J. Vis. 10, 6 (2010).
Ribeiro, M. T., Singh, S. & Guestrin, C. “Why should I trust you?”: Explaining the predictions of any classifier. In Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining 1135–1144 (ACM, 2016).
Zhu, Z., Xie, L. & Yuille, A. L. Object recognition with and without objects. In Proc. 26th Int. Joint Conf. Artificial Intelligence 3609–3615 (IJCAI, 2017).
Wang, J. et al. Visual concepts and compositional voting. Ann. Math. Sci. Appl. 3, 151–188 (2018).
Dawson, M., Zisserman, A. & Nellåker, C. From same photo: cheating on visual kinship challenges. In Asian Conf. Computer Vision 654–668 (Springer, 2018).
Biederman, I. On the Semantics of a Glance at a Scene (Erlbaum, 1981).
Biederman, I., Mezzanotte, R. J. & Rabinowitz, J. C. Scene perception: detecting and judging objects undergoing relational violations. Cogn. Psychol. 14, 143–177 (1982).
Oliva, A. & Torralba, A. The role of context in object recognition. Trends Cogn. Sci. 11, 520–527 (2007).
Castelhano, M. S. & Heaven, C. Scene context influences without scene gist: eye movements guided by spatial associations in visual search. Psychon. Bull Rev. 18, 890–896 (2011).
Jo, J. & Bengio, Y. Measuring the tendency of CNNs to learn surface statistical regularities. Preprint at https://arxiv.org/abs/1711.11561 (2017).
Ilyas, A. et al. Adversarial examples are not bugs, they are features. In Proc. Advances NeurIPS 125–136 (NeurIPS, 2019).
Wolpert, D. H. & Macready, W. G. No free lunch theorems for optimization. IEEE T. Evolut. Comput. 1, 67–82 (1997).
Brendel, W. & Bethge, M. Approximating CNNs with bag-of-local-features models works surprisingly well on ImageNet. In Proc. Int. Conf. Learning Representations (ICLR, 2019).
Baker, N., Lu, H., Erlikhman, G. & Kellman, P. J. Deep convolutional networks do not classify based on global object shape. PLoS Comp. Biol. 14, e1006613 (2018).
Geirhos, R. et al. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In Proc. Int. Conf. Learning Representations (ICLR, 2019).
Heinze-Deml, C. & Meinshausen, N. Conditional variance penalties and domain shift robustness. Preprint at https://arxiv.org/abs/1710.11469 (2017).
Malhotra, G. & Bowers, J. What a difference a pixel makes: an empirical examination of features used by CNNs for categorisation. In Proc. Int. Conf. Learning Representations (ICLR, 2019).
Jacobsen, J.-H., Behrmann, J., Zemel, R. & Bethge, M. Excessive invariance causes adversarial vulnerability. In Proc. Int. Conf. Learning Representations (ICLR, 2019).
Kamin, L. J. Predictability, surprise, attention, and conditioning. In Symp. Punishment and Averse Behavior (eds Campbell, B. A. & Church, R. M.) 279–296 (Appleton-Century-Crofts, 1969).
Dickinson, A. Contemporary Animal Learning Theory Vol. 1 (CUP Archive, 1980).
Bouton, M. E. Learning and Behavior: A Contemporary Synthesis (Sinauer Associates, 2007).
Nguyen, A., Yosinski, J. & Clune, J. Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. In Proc. IEEE Conf. Computer Vision and Pattern Recognition 427–436 (IEEE, 2015).
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J. & Song, D. Natural adversarial examples. Preprint at https://arxiv.org/abs/1907.07174 (2019).
Wang, M. & Deng, W. Deep visual domain adaptation: a survey. Neurocomputing 312, 135–153 (2018).
Alcorn, M. A. et al. Strike (with) a pose: neural networks are easily fooled by strange poses of familiar objects. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (IEEE, 2019).
Azulay, A. & Weiss, Y. Why do deep convolutional networks generalize so poorly to small image transformations? J. Mach. Learn. Res. 20, 1–25 (2019).
Dodge, S. & Karam, L. Human and DNN classification performance on images with quality distortions: a comparative study. ACM T. Appl. Perc. 16, 7 (2019).
Russakovsky, O. et al. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2015).
Gururangan, S. et al. Annotation artifacts in natural language inference data. In Proc. Annual Conf. North American Chapter of the Association for Computational Linguistics (ACL, 2018).
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A. & Choi, Y. HellaSwag: can a machine really finish your sentence? In Proc. 57th Annual Meeting Assocciation of Computational Linguistics 4791–4800 (ACL, 2019).
Borowski, J. et al. The notorious difficulty of comparing human and machine perception. In Proc. NeurIPS Shared Representations in Human and Machine Intelligence Workshop (NeurIPS, 2019).
Geirhos, R., Meding, K. & Wichmann, F. A. Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency. Preprint at https://arxiv.org/abs/2006.16736 (2020).
Marr, D. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information (W. H. Freeman and Company, 1982).
Buckner, C. The Comparative Psychology of Artificial Intelligences (PhilSci Archive, 2019); http://philsci-archive.pitt.edu/16034/
Morgan, C. L. Introduction to Comparative Psychology (Scribner, 1903).
Ghahramani, Z. Panel of workshop on advances in approximate Bayesian inference (AABI) 2017. YouTube https://www.youtube.com/watch?v=x1UByHT60mQ (2017).
Marton, F. & Säaljö, R. On qualitative differences in learning—II Outcome as a function of the learner’s conception of the task. Br. J. Educ. Psychol. 46, 115–127 (1976).
Biggs, J. Individual differences in study processes and the quality of learning outcomes. Higher Educ. 8, 381–394 (1979).
Chin, C. & Brown, D. E. Learning in science: a comparison of deep and surface approaches. J. Res. Sci. Teach. 37, 109–138 (2000).
Marcus, G. F. Rethinking eliminative connectionism. Cogn. Psychol. 37, 243–282 (1998).
Kilbertus, N., Parascandolo, G. & Schölkopf, B. Generalization in anti-causal learning. Preprint at https://arxiv.org/abs/1812.00524 (2018).
Marcus, G. Deep learning: a critical appraisal. Preprint at https://arxiv.org/abs/1801.00631 (2018).
Lapuschkin, S. et al. Unmasking Clever Hans predictors and assessing what machines really learn. Nat. Commun. 10, 1096 (2019).
Lake, B. M., Ullman, T. D., Tenenbaum, J. B. & Gershman, S. J. Building machines that learn and think like people. Behav. Brain Sci. 40, e253 (2017).
Chollet, F. The measure of intelligence. Preprint at https://arxiv.org/abs/1911.01547 (2019).
Crosby, M., Beyret, B. & Halina, M. The Animal-AI Olympics. Nat. Mach. Int. 1, 257–257 (2019).
Juliani, A. et al. Obstacle tower: a generalization challenge in vision, control, and planning. In Proc. 28th Int. Joint Conf. Artificial Intelligence (IJCAI, 2019).
Hendrycks, D. & Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. In Proc. Int. Conf. Learning Representations (ICLR, 2019).
Levesque, H., Davis, E. & Morgenstern, L. The Winograd Schema Challenge. In 13th Int. Conf. Principles of Knowledge Representation and Reasoning (KR, 2012).
Trichelair, P., Emami, A., Trischler, A., Suleman, K. & Cheung, J. C. K. How reasonable are common-sense reasoning tasks: a case-study on the Winograd Schema Challenge and SWAG. In Proc. Conf. Empirical Methods in Natural Language Processing and Int. Joint Conf. Natural Language Processing 3373–3378 (ACL, 2019).
Zipf, G. K. Human Behavior and the Principle of Least Effort (Addison-Wesley, 1949).
Ohala, J. J. The phonetics and phonology of aspects of assimilation. Papers Lab. Phono. 1, 258–275 (1990).
Vicentini, A. The economy principle in language. Notes and Observations from early modern English grammars. Mots Palabras Words 3, 37–57 (2003).
Sinz, F. H., Pitkow, X., Reimer, J., Bethge, M. & Tolias, A. S. Engineering a less artificial intelligence. Neuron 103, 967–979 (2019).
Arpit, D. et al. A closer look at memorization in deep networks. In Proc. Int. Conf. Machine Learning (ICML, 2017).
Valle-Perez, G., Camargo, C. Q. & Louis, A. A. Deep learning generalizes because the parameter-function map is biased towards simple functions. In Proc. Int. Conf. Learning Representations (ICLR, 2018).
Shah, H., Tamuly, K., Raghunathan, A., Jain, P. & Netrapalli, P. The pitfalls of simplicity bias in neural networks. Preprint at https://arxiv.org/abs/2006.07710 (2020).
Kalimeris, D. et al. SGD on neural networks learns functions of increasing complexity. In Proc. Advances NeurIPS 3496–3506 (NeurIPS, 2019).
Hermann, K. L. & Lampinen, A. K. What shapes feature representations? exploring datasets, architectures, and training. Preprint at https://arxiv.org/abs/2006.12433 (2020).
Richardson, J. Vectors: Aphorisms & Ten-Second Essays (Ausable, 2001).
Engstrom, L. et al. A discussion of ‘adversarial examples are not bugs, they are features’. Distill https://distill.pub/2019/advex-bugs-discussion/ (2019).
Barbu, A. et al. ObjectNet: a large-scale bias-controlled dataset for pushing the limits of object recognition models. In Proc. Advances NeurIPS 9448–9458 (NeurIPS, 2019).
Li, D., Yang, Y., Song, Y.-Z. & Hospedales, T. M. Deeper, broader and artier domain generalization. In Proc. IEEE Int. Conf. Computer Vision (IEEE, 2017).
Qiu, W. & Yuille, A. UnrealCV: connecting computer vision to unreal engine. In European Conf. Computer Vision 909–916 (Springer, 2016).
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A. & Koltun, V. CARLA: an open urban driving simulator. In Conf. Robot Learning 1–16 (CoRL, 2017).
Creager, E. et al. Flexibly fair representation learning by disentanglement. In Proc. Int. Conf. Machine Learning (ICML, 2019).
Hays, J. & Efros, A. A. Scene completion using millions of photographs. ACM Trans. Graph. 26, 4 (2007).
Hays, J. & Efros, A. A. IM2GPS: estimating geographic information from a single image. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (IEEE, 2008).
Poliak, A., Naradowsky, J., Haldar, A., Rudinger, R. & Van Durme, B. Hypothesis only baselines in natural language inference. In Proc. 7th Joint Conf. Lexical and Computational Semantics 180–191 (ACL, 2018).
Jasani, B., Girdhar, R. & Ramanan, D. Are we asking the right questions in MovieQA? In Proc. IEEE/CVF Int. Conf. Computer Vision Workshop (IEEE, 2019).
Hornik, K., Stinchcombe, M. & White, H. Multilayer feedforward networks are universal approximators. Neural Netw. 2, 359–366 (1989).
d’Ascoli, S., Sagun, L., Bruna, J. & Biroli, G. Finding the needle in the haystack with convolutions: on the benefits of architectural bias. In Proc. Advances NeurIPS (NeurIPS, 2019).
Ulyanov, D., Vedaldi, A. & Lempitsky, V. Deep image prior. In Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition 9446–9454 (IEEE, 2018).
Vaswani, A. et al. Attention is all you need. In Proc. Advances NeurIPS 5998–6008 (NeurIPS, 2017).
Hein, M., Andriushchenko, M. & Bitterwolf, J. Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proc. IEEE Conf. Computer Vision and Pattern Recognition 41–50 (IEEE, 2019).
Lehman, J. et al. The surprising creativity of digital evolution: a collection of anecdotes from the evolutionary computation and artificial life research communities. Art. Life 26, 274–306 (2020).
Madry, A., Makelov, A., Schmidt, L., Tsipras, D. & Vladu, A. Towards deep learning models resistant to adversarial attacks. In Proc. Int. Conf. Learning Representations (ICLR, 2018).
Arjovsky, M., Bottou, L., Gulrajani, I. & Lopez-Paz, D. Invariant risk minimization. Preprint at https://arxiv.org/abs/1907.02893 (2019).
Wu, L., Zhu, Z. & E, W. Towards understanding generalization of deep learning: perspective of loss landscapes. Preprint at https://arxiv.org/abs/1706.10239 (2017).
De Palma, G., Kiani, B. T. & Lloyd, S. Deep neural networks are biased towards simple functions. Preprint at https://arxiv.org/abs/1812.10156 (2018).
Valle-Perez, G., Camargo, C. Q. & Louis, A. A. Deep learning generalizes because the parameter-function map is biased towards simple functions. In Proc. Int. Conf. Learning Representations (ICLR, 2019).
Sun, K. & Nielsen, F. Lightlike neuromanifolds, Occam’s razor and deep learning. Preprint at https://arxiv.org/abs/1905.11027 (2019).
Li, Y., Wei, C. & Ma, T. Towards explaining the regularization effect of initial large learning rate in training neural networks. In Proc. Advances NeurIPS 11674–11685 (NeurIPS, 2019).
Bartlett, P. L., Long, P. M., Lugosi, G. & Tsigler, A. Benign overfitting in linear regression. Proc. Natl Acad Sci. USA https://doi.org/10.1073/pnas.1907378117 (2019).
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting R.G. and C.M.; the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) for supporting C.M. via grant EC 479/1-1; the Collaborative Research Center (Projektnummer 276693517—SFB 1233: Robust Vision) for supporting M.B. and F.A.W.; the German Federal Ministry of Education and Research through the Tübingen AI Center (FKZ 01IS18039A) for supporting W.B. and M.B.; as well as the Natural Sciences and Engineering Research Council of Canada and the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/Interior Business Center (DoI/IBC) contract number D16PC00003 for supporting R.Z. The authors would like to thank J. Borowski, M. Burg, S. Cadena, A. S. Ecker, L. Eisenberg, R. Fleming, I. Fründ, S. Greiner, F. Grießer, S. Keshvari, R. Kessler, D. Klindt, M. Kümmerer, B. Mitzkus, H. Nienborg, J. Rauber, E. Rusak, S. Schneider, L. Schott, T. Sering, Y. Sharma, M. Tangemann, R. Zimmermann and T. Wallis for helpful discussions.
The authors declare no competing interests.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Geirhos, R., Jacobsen, JH., Michaelis, C. et al. Shortcut learning in deep neural networks. Nat Mach Intell 2, 665–673 (2020). https://doi.org/10.1038/s42256-020-00257-z
Sensitivity analysis for interpretation of machine learning based segmentation models in cardiac MRI
BMC Medical Imaging (2021)
Scientific Reports (2021)
Nature Machine Intelligence (2021)
Nature Human Behaviour (2021)
Current limitations to identify covid-19 using artificial intelligence with chest x-ray imaging (part ii). The shortcut learning problem
Health and Technology (2021)