Review Article | Published:

Towards a neuroscience of active sampling and curiosity

Nature Reviews Neurosciencevolume 19pages758770 (2018) | Download Citation

Abstract

In natural behaviour, animals actively interrogate their environments using endogenously generated ‘question-and-answer’ strategies. However, in laboratory settings participants typically engage with externally imposed stimuli and tasks, and the mechanisms of active sampling remain poorly understood. We review a nascent neuroscientific literature that examines active-sampling policies and their relation to attention and curiosity. We distinguish between information sampling, in which organisms reduce uncertainty relevant to a familiar task, and information search, in which they investigate in an open-ended fashion to discover new tasks. We review evidence that both sampling and search depend on individual preferences over cognitive states, including attitudes towards uncertainty, learning progress and types of information. We propose that, although these preferences are non-instrumental and can on occasion interfere with external goals, they are important heuristics that allow organisms to cope with the high complexity of both sampling and search, and generate curiosity-driven investigations in large, open environments in which rewards are sparse and ex ante unknown.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Additional information

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    Gottlieb, J., Oudeyer, P. Y., Lopes, M. & Baranes, A. Information seeking, curiosity and attention: computational and empirical mechanisms. Trends Cogn. Sci. 17, 585–593 (2013).

  2. 2.

    Kidd, C. & Hayden, B. Y. The psychology and neuroscience of curiosity. Neuron 88, 449–460 (2015).

  3. 3.

    Gottlieb, J., Hayhoe, M., Hikosaka, O. & Rangel, A. Attention, reward and information seeking. J. Neurosci. 34, 15497–154504 (2014).

  4. 4.

    Rehder, B. & Hoffman, A. B. Eye tracking and selective attention in category learning. Cogn. Psychol. 51, 1–41 (2005).

  5. 5.

    Nelson, J. Finding useful questions: on Bayesian diagnosticity, probability, impact and information gain. Psychol. Rev. 112, 979–999 (2005).

  6. 6.

    Coenen, A., Nelson, J. & Gureckis, T. Asking the right questions about the psychology of human inquiry: nine open challenges. Psychon Bull. Rev. https://doi.org/10.3758/s13423-018-1470-5 (2018).

  7. 7.

    Bossaerts, P. & Murawski, C. Computational complexity and human decision-making. Trends Cogn. Sci. 21, 917–929 (2017).

  8. 8.

    Loewenstein, G. & Molnar, A. The renaissance of belief-based utility in economics. Nat. Hum. Behav. 2, 166–167 (2018).

  9. 9.

    Chater, N. & Loewenstein, G. The under-appreciated drive for sense-making. J. Econ. Behav. Organiz. 126, 137–154 (2016).

  10. 10.

    Wu, C. M., Meder, B., Filimon, F. & Nelson, J. D. Asking better questions: how presentation formats influence information search. J. Exp. Psychol. Learn. Mem. Cogn 43, 1274–1297 (2017).

  11. 11.

    Markant, D. B. & Gureckis, T. M. Is it better to select or to receive? Learning via active and passive hypothesis testing. J. Exp. Psychol. Gen. 143, 94–122 (2014).

  12. 12.

    Berlyne, D. Conflict, Arousal and Curiosity (McGraw-Hill, 1960).

  13. 13.

    Berlyne, D. E. A theory of human curiosity. Br. J. Psychol. 45, 180–191 (1954).

  14. 14.

    Litman, J. A. in Issues in the Psychology of Motivation (ed. Zelick, P. R.) (Nova Science Publishers, 2007).

  15. 15.

    Silvia, P. J. Exploring the Psychology of Interest (Oxford Univ. Press, 2006).

  16. 16.

    Di Domenico, S. I. & Ryan, R. M. The emerging neuroscience of intrinsic motivation: a new frontier in self-determination research. Front. Hum. Neurosci. https://doi.org/10.3389/fnhum.2017.00145 (2017).

  17. 17.

    Kaplan, F. & Oudeyer, P.-Y. In search of the neural circuits of intrinsic motivation. Frontiers Neurosci. 1, 225–225 (2007). This is a clear and succinct review of the concepts and computational models of intrinsic motivation and their importance to artificial intelligence.

  18. 18.

    Gopnik, A. Scientific thinking in young children: theoretical advances, empirical research, and policy implications. Science 337, 1623–1627 (2012).

  19. 19.

    Renninger, K. A. & Hidi, S. E. The Power of Interest for Motivation and Engagement (Routledge, NY, 2016).

  20. 20.

    Begus, K., Gliga, T. & Southgate, V. Infants’ preferences for native speakers are associated with an expectation of information. Proc. Natl Acad. Sci. USA 113, 12397–12402 (2016).

  21. 21.

    Kreps, D. M. & Porteus, E. L. Temporal resolution of uncertainty and dynamic choice theory. Econometrica 46, 185–200 (1978).

  22. 22.

    Caplin, A. & Dean, M. Revealed preference, rational inattention and costly information acquisition. Am. Econ. Rev. 105, 2183–2203 (2015).

  23. 23.

    Caplin, A. & Leahy, J. Psychological expected utility theory and anticipatory feelings. Q. J. Econ. 116, 55–79 (2001).

  24. 24.

    Clark, A. Surfing Uncertainty: Prediction, Action and the Embodied Mind. (Oxford Univ. Press, 2015).

  25. 25.

    Livio, M. Why? What Makes Us Curious?. (Simon and Schuster, 2017).

  26. 26.

    Hayhoe, M. & Ballard, D. Modeling task control of eye movements. Curr. Biol. 24, 622–628 (2014). This paper provides an excellent overview of empirical and modelling studies of eye movement control in natural tasks.

  27. 27.

    Tatler, B. W., Hayhoe, M. N., Land, M. F. & Ballard, D. H. Eye guidance in natural vision: reinterpreting salience. J. Vis. 11, 5–25 (2011).

  28. 28.

    Bach, D. R. & Dolan, R. J. Knowing how much you don’t know: a neural organization of uncertainty estimates. Nat. Rev. Neurosci. 13, 572–586 (2012).

  29. 29.

    Cohen, J. D., McClure, S. M. & Yu, A. J. Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Phil. Trans. R. Soc. B 362, 933–942 (2007).

  30. 30.

    Todd, P. M. & Gigerenzer, G. Précis of simple heuristics that make us smart. Behav. Brain Sci. 23, 727–780 (2000).

  31. 31.

    Reynolds, J. H. & Heeger, D. J. The normalization model of attention. Neuron 61, 168–185 (2009).

  32. 32.

    Thompson, K. G. & Bichot, N. P. A visual salience map in the primate frontal eye field. Prog. Brain Res. 147, 251–262 (2005).

  33. 33.

    Bisley, J. W. & Goldberg, M. E. Attention, intention, and priority in the parietal lobe. Annu. Rev. Neurosci. 33, 1–21 (2010).

  34. 34.

    Hanks, T. D. & Summerfield, C. Perceptual decision making in rodents, monkeys, and humans. Neuron 93, 15–31 (2017).

  35. 35.

    Kable, J. W. & Glimcher, P. W. The neurobiology of decision: consensus and controversy. Neuron 63, 733–745 (2009).

  36. 36.

    Lee, D., Seo, H. & Jung, M. W. Neural basis of reinforcement learning and decision making. Annu. Rev. Neurosci. 35, 287–308 (2012).

  37. 37.

    Krajbich, I., Armel, C. & Rangel, A. Visual fixations and the computation and comparison of value in simple choice. Nat. Neurosci. 13, 1292–1298 (2010).

  38. 38.

    Krajbich, I., Lu, D., Camerer, C. & Rangel, A. The attentional drift-diffusion model extends to simple purchasing decisions. Front. Psychol. 3, 193 (2012).

  39. 39.

    Gottlieb, J. Attention, learning, and the value of information. Neuron 76, 281–295 (2012).

  40. 40.

    Gottlieb, J. Understanding active sampling strategies: empirical approaches and implications for attention and decision reseeaerch. Cortex 102, 150–160 (2018). This is an overview of empirical approaches to information sampling in neurophysiology.

  41. 41.

    Johnson, L., Sullivan, B., Hayhoe, M. & Ballard, D. H. Predicting human visuomotor behavior in a driving task. Phil. Trans. R. Soc. B. 369, 20130044 (2014).

  42. 42.

    Sullivan, B. T., Johnson, L., Rothkopf, C. A., Ballard, D. & Hayhoe, M. The role of uncertainty and reward on eye movements in a virtual driving task. J. Vis. 12, 19 (2012).

  43. 43.

    Leong, Y., Radulescu, A., Daniel, R., DeWoskin, V. & Niv, Y. Dynamic interaction between reinforcement learning and attention in multidimensional environments. Neuron 93, 451–463 (2017).

  44. 44.

    Wilson, R. C. & Niv, Y. Inferring relevance in a changing world. Front. Hum. Neurosci. 5, 189 (2011).

  45. 45.

    Najemnik, J. & Geisler, W. S. Eye movement statistics in humans are consistent with an optimal search strategy. J. Vis 8, 4 (2008).

  46. 46.

    Yang, S. C., Lengyel, M. & Wolpert, D. M. Active sensing in the categorization of visual patterns. eLife 5, e12215 (2016). This paper provides evidence for information-based eye movement strategies using behavioural analysis and Bayesian modelling in humans.

  47. 47.

    Najemnik, J. & Geisler, W. S. Optimal eye movement strategies in visual search. Nature 434, 387–391 (2005).

  48. 48.

    Renninger, L. W., Verghese, P. & Coughlan, J. Where to look next? Eye movements reduce local uncertainty. J. Vis 7, 6 (2007).

  49. 49.

    Vossel, S., Vossel, S., Mathys, C., Stephan, K. E. & Friston, K. J. Cortical coupling reflects bayesian belief updating in the deployment of spatial attention. J. Neurosci. 35, 11532–11542 (2015). This is an analysis of attention in a Bayesian framework using functional MRI in humans.

  50. 50.

    Vossel, S. et al. Spatial attention, precision, and bayesian inference: a study of saccadic response speed. Cereb. Cortex 24, 1436–1450 (2014).

  51. 51.

    Vossel, S., Thiel, C. M. & Fink, G. R. Cue validity modulates the neural correlates of covert endogenous orienting of attention in parietal and frontal cortex. NeuroImage 32, 1257–1264 (2006).

  52. 52.

    Foley, N. C., Kelley, S. P., Mhatre, H., Lopes, M. & Gottlieb, J. Parietal neurons encode expected gains in instrumental information. Proc. Natl Acad. Sci. 114, E3315–E3323 (2017). This paper demonstrates that oculomotor neurons encode expected information gains in monkeys.

  53. 53.

    Ernst, M. O. & Banks, M. S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).

  54. 54.

    Nelson, J., McKenzie, C., Cottrell, G. & Sejnowski, T. Experience matters: information acquisition optimizes probability gain. Psychol. Sci. 21, 960–969 (2010).

  55. 55.

    Zajkowski, W. K., Kossut, M. & Wilson, R. C. A causal role for right frontopolar cortex in directed, but not random, exploration. eLife 6, e27430 (2017).

  56. 56.

    Somerville, L. H. et al. Charting the expansion of strategic exploratory behavior during adolescence. J. Exp. Psychol. Gen. 146, 155–164 (2017).

  57. 57.

    Manohar, S. G. & Husain, M. Attention as foraging for information and value. Front. Hum. Neurosci. 7, 711 (2013).

  58. 58.

    Krishnamurthy, K., Nassar, M. R., Sarode, S. & Gold, J. I. Arousal-related adjustments of perceptual biases optimize perception in dynamic environments. Nat. Hum. Behav. 1, 0107 (2017).

  59. 59.

    Li, V., Herce Castañón, S., Solomon, J. A., Vandormael, H. & Summerfield, C. Robust averaging protects decisions from noise in neural computations. PLOS Comput. Biol. 13, e1005723 (2017).

  60. 60.

    Spitzer, B., Waschke, L. & Summerfield, C. Selective overwiehgting of larger magnitudes during noisy numerical comparison. Nat. Hum. Behav. 1, 0145 (2017).

  61. 61.

    Gold, J. I. & Stocker, A. A. Visual decision-making in an uncertain and dynamic world. Annu. Rev. Vis. Sci. 3, 227–250 (2017).

  62. 62.

    Ebitz, R. B., Albarran, E. & Moore, T. Exploration disrupts choice-predictive signals and alters dynamics in prefrontal cortex. Neuron 97, 450–461 (2018).

  63. 63.

    Gersch, T. M., Foley, N. C., Eisenberg, I. & Gottlieb, J. Neural correlates of temporal credit assignment in the parietal lobe. PLOS ONE 9, e88725 (2014).

  64. 64.

    Rossi, A. F., Pessoa, L., Desimone, R. & Ungerleider, L. G. The prefrontal cortex and the executive control of attention. Exp. Brain Res. 192, 489–497 (2009).

  65. 65.

    Rossi, A. F., Bichot, N. P., Desimone, R. & Ungerleider, L. G. Top down attentional deficits in macaques with lesions of lateral prefrontal cortex. J. Neurosci. 27, 11306–11314 (2007).

  66. 66.

    Morvan, C. & Maloney, L. Human visual search does not maximize the post-saccadic probability of identifying targets. PLOS Comput. Biol. 8, e1002342 (2012). This presents an intriguing demonstration that humans show suboptimal sampling strategies in a task requiring flexible adjustments based on estimates of visibility.

  67. 67.

    Ghahghaei, S. & Verghese, P. Efficient saccade planning requires time and clear choices. Vision Res. 113B, 125–136 (2015).

  68. 68.

    Chong, T. T. et al. Neurocomputational mechanisms underlying subjective valuation of effort costs. PLOS Biol. 15, e1002598 (2017).

  69. 69.

    Shenhav, A. et al. Toward a rational and mechanistic account of mental effort. Annu. Rev. Neurosci. 40, 99–124 (2017).

  70. 70.

    Fan, J. An information theory account of cognitive control. Front. Hum. Neurosci. https://doi.org/10.3389/fnhum.2014.00680 (2014). This paper proposes a reframing of theories of cognitive control from the perspective of informational constraints.

  71. 71.

    Fleming, S. & Daw, N. Self-evaluation of decision-making: a general Bayesian framework for metacognitive computation. Psychol. Rev. 124, 91–114 (2017).

  72. 72.

    Zhang, H., Daw, N. D. & Maloney, L. T. Human representation of visuo-motor uncertainty as mixtures of orthogonal basis distributions. Nat. Neurosci. 18, 1152–1158 (2015).

  73. 73.

    Vasconcelos, M., Monteiro, T. & Kacelnik, A. Irrational choice and the value of information. Sci. Rep. 5, 13874 (2015).

  74. 74.

    Eliaz, K. & Schotter, A. Experimental testing of intrinsic preferences for noninstrumental information. Am. Econ. Rev. 97, 166–169 (2007).

  75. 75.

    Bromberg-Martin, E. S. & Hikosaka, O. Midbrain dopamine neurons signal preference for advance information about upcoming rewards. Neuron 63, 119–126 (2009).

  76. 76.

    Bennett, D., Bode, S., Brydevall, M., Warren, H. & Murawski, C. Intrinsic valuation of information in decision making under uncertainty. PLOS Comp. Biol. 12, e1005020 (2016).

  77. 77.

    Brydevall, M., Bennett, D., Murawski, C. & Bode, S. The neural encoding of information prediction errors during non-instrumental information seeking. Sci. Rep. 8, 6134 (2018).

  78. 78.

    Blanchard, T. C., Hayden, B. Y. & Bromberg-Martin, E. S. Orbitofrontal cortex uses distinct codes for different choice attributes in decisions motivated by curiosity. Neuron 85, 602–614 (2015). This paper demonstrates single-neuron encoding of non-instrumental information value in the monkey orbitofrontal cortex.

  79. 79.

    Golman, R. & Loewenstein, G. Information gaps: a theory of preferences regarding the presence and absence of information. Decision 5, 143–164 (2018).

  80. 80.

    Loewenstein, G. Anticipation and the valuation of delayed consumption. Econ. J. 97, 666–684 (1987).

  81. 81.

    Iigaya, K., Story, G. W., Kurth-Nelson, Z., Dolan, R. J. & Dayan, P. The modulation of savouring by prediction error and its effects on choice. eLife 5, e13747 (2016). This paper presents a reinforcement learning model of non-instrumental information demand, proposing that, in addition to producing learning, dopaminergic reward prediction errors confer value to predictor states.

  82. 82.

    Flagel, S. B. & Robinson, T. E. Neurobiological basis of individual variation in stimulus-reward learning. Curr. Opin. Behav. Sci. 13, 178–185 (2017).

  83. 83.

    Peck, C. J., Jangraw, D. C., Suzuki, M., Efem, R. & Gottlieb, J. Reward modulates attention independently of action value in posterior parietal cortex. J. Neurosci. 29, 11182–11191 (2009).

  84. 84.

    Foley, N. C., Jangraw, D. C., Peck, C. & Gottlieb, J. Novelty enhances visual salience independently of reward in the parietal lobe. J. Neurosci. 34, 7947–7957 (2014).

  85. 85.

    Isoda, M. & Hikosaka, O. A neural correlate of motivational conflict in the superior colliculus of the macaque. J. Neurophysiol. 100, 1332–1342 (2008).

  86. 86.

    Anderson, B. The attention habit: how reward learning shapes attentional selection. Ann. NY Acad. Sci. 1369, 24–39 (2016). This is a comprehensive review of reward-related attention biases and their neural mechanisms and behavioural importance in humans.

  87. 87.

    Hickey, C., Chelazzi, L. & Theeuwes, J. Reward guides vision when it’s your thing: trait reward-seeking in reward-mediated visual priming. PLOS ONE 5, e14087 (2010).

  88. 88.

    Daddaoua, N., Lopes, M. & Gottlieb, J. Intrinsically motivated oculomotor exploration guided by uncertainty reduction and conditioned reinforcement in non-human primates. Sci. Rep. 6, 20202 (2016).

  89. 89.

    Hickey, C., Chelazzi, L. & Theeuwes, J. Reward changes salience in human vision via the anterior cingulate. J. Neurosci. 30, 11096–11103 (2010).

  90. 90.

    Hickey, C. & Peelen, M. V. Neural mechanisms of incentive salience in naturalistic human vision. Neuron 85, 512–518 (2015).

  91. 91.

    Hunt, L. T., Rutledge, R. B., Malalasekera, W. M., Kennerley, S. W. & Dolan, R. J. Approach-induced biases in human information sampling. PLOS Biol. 14, e2000638 (2016).

  92. 92.

    Barbaro, L., Peelen, M. V. & Hickey, C. Valence, not utility, underlies reward-driven prioritization in human vision. J. Neurosci. 37, 10438–10450 (2017). This is among the first empirical demonstrations of reward-based and uncertainty-based modulations of visual representations in the human high-level cortex.

  93. 93.

    San Martín, R., Appelbaum, L. G., Huettel, S. A. & Woldorff, M. G. Cortical brain activity reflecting attentional biasing toward reward-predicting cues covaries with economic decision-making performance. Cereb. Cortex 26, 1–11 (2016).

  94. 94.

    van Lieshout, L. L. F., Vandenbroucke, A. R. E., Müller, N. C. J., Cools, R. & de Lange, F. P. Induction and relief of curiosity elicit parietal and frontal activity. J. Neurosci. 38, 2579–2588 (2018). This is a demonstration of non-instrumental information value and its neural correlates in humans.

  95. 95.

    Loewenstein, G. The psychology of curiosity: a review and reinterpretation. Psychol. Bull. 116, 75–98 (1994).

  96. 96.

    Baldassare, G., Mirolli, M. (eds) Intrinsically Motivated Learning in Natural and Artificial Systems (Springer-Verlag, Berlin, 2013).

  97. 97.

    Gruber, M. J., Gelman, B. D. & Ranganath, C. States of curiosity modulate hippocampus-dependent learning via the dopaminergic circuit. Neuron 84, 486–496 (2014). This paper demonstrates the effects of curiosity on memory and the hippocampus in humans.

  98. 98.

    Kang, M. J. et al. The wick in the candle of learning: epistemic curiosity activates reward circuitry and enhances memory. Psychol. Sci. 20, 963–973 (2009).

  99. 99.

    Baranes, A. F., Oudeyer, P. Y. & Gottlieb, J. Eye movements encode epistemic curiosity in human observers. Vis. Res. 117, 81–90 (2015).

  100. 100.

    Marvin, C. B. & Shohamy, D. Curiosity and reward: valence predicts choice and information prediction errors enhance learning. J. Exp. Psychol. Gen. 145, 266–272 (2016).

  101. 101.

    Jepma, M., Verdonschot, R. G., van Steenbergen, H., Rombouts, S. A. & Nieuwenhuis, S. Neural mechanisms underlying the induction and relief of perceptual curiosity. Front. Behav. Neurosci. https://doi.org/10.3389/fnbeh.2012.00005 (2012). This is a study of perceptual curiosity using functional MRI in humans.

  102. 102.

    Risko, E. F., Anderson, N. C., Lanthier, S. & Kingstone, A. Curious eyes: individual differences in personality predict eye movement behavior in scene-viewing. Cognition 122, 86–90 (2012).

  103. 103.

    Salimpoor, V. N., Zald, D. H., Zatorre, R. J., Dagher, A. & McIntosh, A. R. Predictions and the brain: how musical sounds become rewarding. Trends Cogn. Sci. 19, 86–91 (2015).

  104. 104.

    Huron, D. Sweet Anticipation: Music and the Psychology of Expectation (MIT Press, 2006).

  105. 105.

    Liao, H. I., Yeh, S. L. & Shimojo, S. Novelty versus familiarity principles in preference decisions: task-context of past experience matters. Front. Psychol. 2, 43 (2011).

  106. 106.

    Park, J., Shimojo, E. & Shimojo, S. Roles of familiarity and novelty in visual preference judgments are segregated across object categories. Proc. Natl Acad. Sci. USA 107, 14552–14555 (2010).

  107. 107.

    Güçlütürk, Y., Güçlü, U., van Gerven, M. & van Lier, R. Representations of naturalistic stimulus complexity in early and associative visual and auditory cortices. Sci. Rep. 8, 3439 (2018).

  108. 108.

    Zatorre, R. J. Musical pleasure and reward: mechanisms and dysfunction. Ann. NY Acad. Sci. 1337, 202–211 (2015).

  109. 109.

    Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P. & Pezzulo, G. Active inference: a process theory. Neural Comput. 29, 1–49 (2016).

  110. 110.

    Sutton, R. S. & Barto, A. G. Reinforcement Learning: an Introduction (MIT Press, 1998).

  111. 111.

    Daw, N. D., Gerschman, S. J., Seymour, B., Dayan, P. & Dolan, R. J. Model-based influences on human choices and striatal prediction errors. Neuron 69, 1204–1215 (2011).

  112. 112.

    Friston, K. J. et al. Active inference, curiosity and insight. Neural Comput. 29, 2633–2683 (2017).

  113. 113.

    Morewedge, C. K. & Kahneman, D. Associative processes in intuitive judgment. Trends Cogn. Sci. 14, 435–440 (2010).

  114. 114.

    Buckley, C., Kim, C. S., McGregor, S. & Seth, A. K. The free energy principle for action and perception: a mathematical review. J. Math. Psychol. 81, 55–79 (2017).

  115. 115.

    Gershman, S. J. & Blei, D. M. A tutorial on Bayesian nonparametric models. J. Math. Psychol. 56, 1–12 (2012).

  116. 116.

    Baranes, A. & Oudeyer, P. Y. Active learning of inverse models with intrinsically motivated goal exploration in robots. Rob. Auton. Syst. 61, 49–73 (2013).

  117. 117.

    Oudeyer, P. Y., Kaplan, F. & Hafner, V. V. Instrinsic motivation systems for autonomous mental development. IEEE Trans. Evol. Comput. 11, 265–286 (2007).

  118. 118.

    Forestier, S. & Oudeyer, P. Y. in Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) 3965–3972 (IEEE, 2016).

  119. 119.

    Moulin-Frier, C., Nguyen, S. M. & Oudeyer, P.-Y. Self-organization of early vocal development in infants and machines: the role of intrinsic motivation. Front. Psychol. 4, 1006 (2014).

  120. 120.

    Forestier, S. & Oudeyer, P. Y. in Proc. 39th Annual Meeting of the Cognitive Science Soc. 2013–2018 (Cogsci, 2017).

  121. 121.

    Clement, B., Roy, D., Oudeyer, P. Y. & Lopes, M. Multi-armed bandits for intelligent tutoring systems. J. Educ. Data Mining 7, 2 (2015).

  122. 122.

    Metcalfe, J. Metacognitive judgments and control of study. Curr. Dir. Psychol. Sci. 18, 159–163 (2009).

  123. 123.

    Lopes, M. & Oudeyer, P.-Y. in Proc. IEEE Int. Conf. on Development and Learning and Epigenetic Robotics (ICDL) 1–8 (IEEE, 2012).

  124. 124.

    Son, L. & Sethi, R. Metacognitive control and optimal learning. Cogn. Sci. 30, 759–774 (2006).

  125. 125.

    Baranes, A. F., Oudeyer, P. Y. & Gottlieb, J. The effects of task difficulty, novelty and the size of the search space on intrinsically motivated exploration. Front. Neurosci. 8, 317 (2014). This presents a novel laboratory task for examining intrinsically motivated exploration based on difficulty in humans.

  126. 126.

    Barto, A., Singh, S. & Chenatez, N. in Proc. 3rd Int. Conf. Dvp. Learn 112–119 (San Diego, CA, 2004).

  127. 127.

    Schmidhuber, J. in Proc. Int. Joint Conf. Neural Networks 2, 1458–1463 (IEEE, 1991).

  128. 128.

    Bellemare, M. et al. in Proc. Advances in Neural Information Processing Systems 29 Conf. 1471–1479 (NIPS, 2016).

  129. 129.

    Kulkarni, T. D., Narasimhan, K., Saeedi, A. & Tenenbaum, J. B. in Proc. Advances in Neural Information Processing Systems 29 Conf. 3675–3683 (NIPS, 2016).

  130. 130.

    Pouget, A., Drugowitsch, J. & Kepecs, A. Confidence and certainty: distinct probabilistic quantities for different goals. Nat. Neurosci. 19, 366–374 (2016).

Download references

Acknowledgements

The authors acknowledge support from the Human Frontiers Science Program (Collaborative Research Grant RGP0018/2016 to J.G. and P.-Y.O.), an Inria Neurocuriosity grant (to J.G. and P.-Y.O.), the National Eye Institute (RO1 grant to J.G.) and the National Institute of Mental Health (RO1 grant to J.G.).

Reviewer information

Nature Reviews Neuroscience thanks V. Stuphorn and the other anonymous reviewers for their contribution to the peer review of this work.

Author information

Affiliations

  1. Department of Neuroscience, Columbia University, New York, NY, USA

    • Jacqueline Gottlieb
  2. Kavli Institute for Brain Science, Columbia University, New York, NY, USA

    • Jacqueline Gottlieb
  3. Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA

    • Jacqueline Gottlieb
  4. Inria, Bordeaux, France

    • Pierre-Yves Oudeyer
  5. Ensta ParisTech, Paris, France

    • Pierre-Yves Oudeyer

Authors

  1. Search for Jacqueline Gottlieb in:

  2. Search for Pierre-Yves Oudeyer in:

Contributions

The authors both researched data for the article, provided substantial contributions to discussion of content, wrote the article, and reviewed and edited the manuscript before submission.

Competing interests

The authors declare no competing interests.

Corresponding author

Correspondence to Jacqueline Gottlieb.

Glossary

Agents

Any entities that are capable of learning and decision-making, including humans, other animals and artificial intelligence applications such as robots and self-driving cars.

Instrumental context

A context in which agents are motivated by the desire to obtain a known goal, which is operationalized in the laboratory as maximizing a material reward (such as money, points, food or safety).

About this article

Publication history

Published

DOI

https://doi.org/10.1038/s41583-018-0078-0