Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

Lessons for artificial intelligence from the study of natural stupidity

Nature Machine Intelligence volume 1, pages 174–180 (2019)Cite this article

Subjects

Abstract

Artificial intelligence and machine learning systems are increasingly replacing human decision makers in commercial, healthcare, educational and government contexts. But rather than eliminate human errors and biases, these algorithms have in some cases been found to reproduce or amplify them. We argue that to better understand how and why these biases develop, and when they can be prevented, machine learning researchers should look to the decades-long literature on biases in human learning and decision-making. We examine three broad causes of bias—small and incomplete datasets, learning from the results of your decisions, and biased inference and evaluation processes. For each, findings from the psychology literature are introduced along with connections to the machine learning literature. We argue that rather than viewing machine systems as being universal improvements over human decision makers, policymakers and the public should acknowledge that these system share many of the same limitations that frequently inhibit human judgement, for many of the same reasons.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: In illusory correlations, an agent mistakenly comes to believe that there is a correlation between a variable of interest and membership to a larger group (or more data-rich group or individual).
Fig. 2: An agent’s beliefs about whether an option is mostly good or mostly bad evolve as the agent experiences a series of positive and negative outcomes, potentially causing the hot stove effect.
Fig. 3: An ‘attentional learning trap’ can emerge with choice-contingent feedback in some environments.
Fig. 4: Reference-dependent risk preferences can be produced by Bayesian prediction.

Similar content being viewed by others

References

  1. Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 354–359 (2017).

    Article  Google Scholar 

  2. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  Google Scholar 

  3. Lazer, D., Kennedy, R., King, G. & Vespignani, A. The parable of Google flu: traps in big data analysis. Science 343, 1203–1206 (2014).

    Article  Google Scholar 

  4. Campolo, A., Sanfilippo, M., Whittaker, M. & Crawford, K. A. I. Now 2017 Report (AI Now Institute, 2017); https://ainowinstitute.org/AI_Now_2017_Report.pdf.

  5. O’Neil, C. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (Broadway Books, 2017).

  6. Barocas, S. & Selbst, A. Big data’s disparate impact. Calif. Law Rev. 104, 671–729 (2016).

    Google Scholar 

  7. Buolamwini, J. & Gebru, T. Gender shades: intersectional accuracy disparities in commercial gender classification. In Proc. Mach. Learn. Res. Vol. 81 (eds Friedler, S. A, & Wilson, C.) 77–91 (PMLR, 2018).

  8. Mohler, G. O. et al. Randomized controlled field trials of predictive policing. J. Am. Stat. Assoc. 110, 1399–1411 (2015).

    Article  MathSciNet  Google Scholar 

  9. Lum, K. & Isaac, W. To predict and serve?. Significance 13, 14–19 (2016).

    Article  Google Scholar 

  10. Ensign, D., Friedler, S. A., Neville, S., Scheidegger, C. & Venkatasubramanian, S. Runaway feedback loops in predictive policing. In Proc. Mach. Learn. Res. Vol. 81 (eds Friedler, S. A. & Wilson, C.) 1–12 (PMLR, 2018).

  11. Denrell, J. Why most people disapprove of me: experience sampling in impression formation. Psychol. Rev. 112, 951–978 (2005).

    Article  Google Scholar 

  12. Crawford, K. The hidden biases in big data. Harvard Business Review https://hbr.org/2013/04/the-hidden-biases-in-big-data (1 April 2013).

  13. Hertwig, R., Barron, G., Weber, E. U. & Erev, I. Decisions from experience and the effect of rare events in risky choice. Psychol. Sci. 15, 534–539 (2004).

    Article  Google Scholar 

  14. Wiener, N. Cybernetics: Or Control and Communication in the Animal and the Machine (MIT Press, 1948).

  15. von Neumann, J. The Computer and the Brain (Yale Univ. Press, 1958).

  16. Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction (Cambridge Univ. Press, 1998).

  17. Rumelhart, D. E., McClelland, J. L. & PDP Research Group. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1: Foundations (MIT Press, 1986).

  18. Marcus, G. Deep learning: a critical appraisal. Preprint at https://arxiv.org/abs/1801.00631 (2018).

  19. Lake, B. M., Ullman, T. D., Tenenbaum, J. B. & Gershman, S. J. Building machines that learn and think like people. Behav. Brain Sci. 40, 1–101 (2016).

    Google Scholar 

  20. Tversky, A. & Kahneman, D. Judgment under uncertainty: heuristics and biases. Science 185, 1124–1131 (1974).

    Article  Google Scholar 

  21. Lake, B. M., Salakhutdinov, R. & Tenenbaum, J. B. Human-level concept learning through probabilistic program induction. Science 350, 1332–1338 (2015).

    Article  MathSciNet  Google Scholar 

  22. Chomsky, N. Aspects of the Theory of Syntax (MIT Press, 1965).

  23. Fox, C. R. & Hadar, L. Decisions from experience’ = sampling error + prospect theory: reconsidering Hertwig, Barron, Weber & Erev (2004). Judgm. Decis. Mak. 1, 159–161 (2006).

    Google Scholar 

  24. Harris, A. J. & Hahn, U. Unrealistic optimism about future life events: a cautionary note. Psychol. Rev. 118, 135–154 (2011).

    Article  Google Scholar 

  25. Chambers, J. R., Windschitl, P. D. & Suls, J. Egocentrism, event frequency, and comparative optimism: when what happens frequently is ‘more likely to happen to me’. Personal. Social. Psychol. Bull. 29, 1343–1356 (2003).

    Article  Google Scholar 

  26. Teodorescu, K. & Erev, I. On the decision to explore new alternatives: the coexistence of under- and over-exploration. J. Behav. Decis. Mak. 27, 109–123 (2014).

    Article  Google Scholar 

  27. Fuller, R. Behavior analysis and unsafe driving: warning—learning trap ahead! J. Appl. Behav. Anal. 24, 73–75 (1991).

    Article  Google Scholar 

  28. Szollisi, A., Liang, G., Konstantinidis, E., Donkin, C. & Newell, B. R. Simultaneous underweighting and overestimation of rare events: unpacking a paradox. J. Exp. Psychol. Gen. (in the press).

  29. Hamilton, D. L. & Gifford, R. K. Illusory correlation in interpersonal perception: a cognitive basis of stereotypic judgments. J. Exp. Social. Psychol. 12, 392–407 (1976).

    Article  Google Scholar 

  30. Mullen, B. & Johnson, C. Distinctiveness-based illusory correlations and stereotyping: a meta-analytic integration. Br. J. Social. Psychol. 29, 11–28 (1990).

    Article  Google Scholar 

  31. Eder, A. B., Fiedler, K. & Hamm-Eder, S. Illusory correlations revisited: the role of pseudocontingencies and working-memory capacity. Q. J. Exp. Psychol. 64, 517–532 (2011).

    Article  Google Scholar 

  32. Kutzner, F., Vogel, T., Freytag, P. & Fiedler, K. A robust classic: illusory correlations are maintained under extended operant learning. Exp. Psychol. 58, 443–453 (2011).

    Article  Google Scholar 

  33. Fiedler, K. & Kutzner, F. in The Wiley Blackwell Handbook of Judgment and Decision Making (eds Keren, G. & Wu, G.) 380–403 (Wiley, 2015).

  34. Fiedler, K., Walther, E., Freytag, P. & Plessner, H. Judgment biases in a simulated classroom — a cognitive – environmental approach. Organ. Behav. Human. Decis. Process. 88, 527–561 (2002).

    Article  Google Scholar 

  35. Halevy, A., Norvig, P. & Pereira, F. The unreasonable effectiveness of data. IEEE Intell. Syst. 24, 8–12 (2009).

    Article  Google Scholar 

  36. Lerman, J. Big data and its exclusions. Stanf. Law Rev. 66, 55–63 (2013).

    Google Scholar 

  37. Tibshirani, R. Regression selection and shrinkage via the lasso. J. R. Stat. Soc. Ser. B 58, 267–288 (1996).

    MATH  Google Scholar 

  38. Kleinberg, J., Mullainathan, S. & Raghavan, M. Inherent trade-offs in the fair determination of risk scores. Preprint at https://arxiv.org/abs/1609.05807 (2016).

  39. Corbett-Davies, S., Pierson, E., Feller, A., Goel, S. & Huq, A. Algorithmic decision making and the cost of fairness. In Proc. 23rd Conf. Knowledge Discovery and Data Mining, 797-806 (2017).

  40. Chen, I., Johansson, F. D. & Sontag, D. Why is my classifier discriminatory? Adv. Neural Inf. Process. Syst. 31, 3543–3554 (2018).

    Google Scholar 

  41. Dawes, R. The robust beauty of improper linear models in decision making. Am. Psychol. 34, 571–582 (1979).

    Article  Google Scholar 

  42. Denrell, J. & March, J. G. Adaptation as information restriction: the hot stove effect. Organ. Sci. 12, 523–538 (2001).

    Article  Google Scholar 

  43. Liu, C., Eubanks, D. L. & Chater, N. The weakness of strong ties: sampling bias, social ties, and nepotism in family business succession. Leadersh. Q. 26, 419–435 (2015).

    Article  Google Scholar 

  44. Le Mens, G., Kareev, Y. & Avrahami, J. The evaluative advantage of novel alternatives: an information-sampling account. Psychol. Sci. 27, 161–168 (2016).

    Article  Google Scholar 

  45. Denrell, J. & Le Mens, G. Seeking positive experiences can produce illusory correlations. Cognition 119, 313–324 (2011).

    Article  Google Scholar 

  46. Niv, Y., Joel, D., Meilijson, I. & Ruppin, E. Evolution of reinforcement learning in uncertain environments: a simple explanation for complex foraging behaviors. Adapt. Behav. 10, 5–24 (2002).

    Article  Google Scholar 

  47. Le Mens, G. & Denrell, J. Rational learning and information sampling: on the ‘naivety’ assumption in sampling explanations of judgment biases. Psychol. Rev. 118, 379–392 (2011).

    Article  Google Scholar 

  48. Fazio, R. H., Eiser, J. R. & Shook, N. J. Attitude formation through exploration: valence asymmetries. J. Personal. Social. Psychol. 87, 293–311 (2004).

    Article  Google Scholar 

  49. Eiser, J. R., Fazio, R. H., Stafford, T. & Prescott, T. J. Connectionist simulation of attitude learning: asymmetries in the acquisition of positive and negative evaluations. Personal. Social. Psychol. Bull. 29, 1221–1235 (2003).

    Article  Google Scholar 

  50. Rich, A. S. & Gureckis, T. M. The limits of learning: exploration, generalization, and the development of learning traps. J. Exp. Psychol. Gen. 147, 1553–1570 (2018).

    Article  Google Scholar 

  51. Shepard, R. N., Hovland, C. L. & Jenkins, H. M. Learning and memorization of classifications. Psychol. Monogr. 75, 1689–1699 (1961).

    Article  Google Scholar 

  52. Jung, J., Concannon, C., Shroff, R., Goel, S. & Goldstein, D. G. Simple rules for complex decisions. Harvard Business Review https://hbr.org/2017/04/creating-simple-rules-for-complex-decisions (19 April 2017).

  53. Jabbari, S., Joseph, M., Kearns, M., Morgenstern, J. & Roth, A. Fairness in reinforcement learning. In Proc. Mach. Learn. Res. Vol. 71 (eds Precup, D. & Whye Teh, Y.) 1617–1626 (PMLR, 2017).

  54. Sculley, D. et al. Hidden technical debt in machine learning systems. Adv. Neural Inf. Process. Syst. 28, 2503–2511 (2015).

    Google Scholar 

  55. Fiedler, K. Beware of samples! A cognitive-ecological sampling approach to judgment biases. Psychol. Rev. 107, 659–676 (2000).

    Article  Google Scholar 

  56. Kahneman, D. & Tversky, A. Prospect theory: an analysis of decision under risk. Économ. J. Econom. Soc. 47, 263–292 (1979).

    Article  Google Scholar 

  57. Denrell, J. Reference-dependent risk sensitivity as rational inference. Psychol. Rev. 122, 461–484 (2015).

    Article  Google Scholar 

  58. Lieder, F., Griffiths, T. L. & Goodman, N. D. Burn-in, bias, and the rationality of anchoring. Adv. Neural Inf. Process. Syst. 25, 2790–2798 (2012).

    Google Scholar 

  59. Lieder, F., Griffiths, T. L. & Hsu, M. Overrepresentation of extreme events in decision making reflects rational use of cognitive resources. Psychol. Rev. 125, 1–32 (2018).

    Article  Google Scholar 

  60. Blei, D. M., Kucukelbir, A. & McAuliffe, J. D. Variational inference: a review for statisticians. J. Am. Stat. Assoc. 112, 859–877 (2017).

    Article  MathSciNet  Google Scholar 

  61. Blei, D. M. & Lafferty, J. D. Dynamic topic models. In Proc. 23rd Int. Conf. Machine Learning 113–120 (ACM, 2006).

  62. Gershman, S. J., Horvitz, E. J. & Tenenbaum, J. B. Computational rationality: a converging paradigm for intelligence in brains, minds, and machines. Science 349, 273–278 (2015).

    Article  MathSciNet  Google Scholar 

  63. Gigerenzer, G. & Brighton, H. Homo heuristicus: why biased minds make better inferences. Top. Cogn. Sci. 1, 107–143 (2009).

    Article  Google Scholar 

  64. Katsikopoulos, K. V. Bounded rationality: the two cultures. J. Econ. Methodol. 21, 361–374 (2014).

    Article  Google Scholar 

  65. Czerlinski, J., Gigerenzer, G. & Goldstein, D. G. How Good Are Simple Heuristics? in Simple Heuristics That Make Us Smart (eds Gigerenzer, G., Todd, P.M., & The ABC Research Group) 97-118 (Oxford Univ. Press, 1999).

  66. Martignon, L. & Hoffrage, U. in Simple Heuristics That Make Us Smart (eds Gigerenzer, G., Todd, P. M., & The ABC Research Group) 119–140 (Oxford Univ. Press, 1999).

  67. Hogarth, R. M. & Karelaia, N. ‘Take-the-best’ and other simple strategies: why and when they work ‘well’ with binary cues. Theory Decis. 61, 205–249 (2006).

    Article  Google Scholar 

  68. Parpart, P., Jones, M. & Love, B. C. Heuristics as Bayesian inference under extreme priors. Cogn. Psychol. 102, 127–144 (2018).

    Article  Google Scholar 

  69. Doshi-Velez, F. & Kim, B. Towards a rigorous science of interpretable machine learning. Preprint at https://arxiv.org/abs/1702.08608 (2017).

  70. Herrnstein, R. J. & Prelec, D. Melioration: a theory of distributed choice. J. Econ. Perspect. 5, 137–156 (1991).

    Article  Google Scholar 

  71. Gureckis, T. M. & Love, B. C. Short-term gains, long-term pains: how cues about state aid learning in dynamic environments. Cognition 113, 293–313 (2009).

    Article  Google Scholar 

  72. Polka, L. & Werker, J. F. Developmental changes in perception of nonnative vowel contrasts. J. Exp. Psychol. Human. Percept. Perform. 20, 421–435 (1994).

    Article  Google Scholar 

  73. Goldstone, R. Influences of categorization on perceptual discrimination. J. Exp. Psychol. Gen. 123, 178–200 (1994).

    Article  Google Scholar 

  74. Levinthal, D. A. & March, J. G. The myopia of learning. Strateg. Manag. J. 14, 95–112 (1993).

    Article  Google Scholar 

  75. Denrell, J. Vicarious learning, undersampling of failure, and the myths of management. Organ. Sci. 14, 227–243 (2003).

    Article  Google Scholar 

  76. Feiler, D. C., Tong, J. D. & Larrick, R. P. Biased judgment in censored environments. Manag. Sci. 59, 573–591 (2013).

    Article  Google Scholar 

  77. Hogarth, R. M., Lejarraga, T. & Soyer, E. The two settings of kind and wicked learning environments. Curr. Dir. Psychol. Sci. 24, 379–385 (2015).

    Article  Google Scholar 

  78. Chapman, L. J. Illusory correlation in observational report. J. Mem. Lang. 6, 151–155 (1967).

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, New York University, New York, NY, USA

    Alexander S. Rich & Todd M. Gureckis

  2. Flatiron Health, New York, NY, USA

    Alexander S. Rich

Authors
  1. Alexander S. Rich
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Todd M. Gureckis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexander S. Rich.

Ethics declarations

Competing interests

A.S.R. is employed by Flatiron Health, an independent subsidiary of Roche.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rich, A.S., Gureckis, T.M. Lessons for artificial intelligence from the study of natural stupidity. Nat Mach Intell 1, 174–180 (2019). https://doi.org/10.1038/s42256-019-0038-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-019-0038-z

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics