Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Towards a neuroscience of active sampling and curiosity

Nature Reviews Neuroscience volume 19, pages 758–770 (2018)Cite this article

Subjects

Abstract

In natural behaviour, animals actively interrogate their environments using endogenously generated ‘question-and-answer’ strategies. However, in laboratory settings participants typically engage with externally imposed stimuli and tasks, and the mechanisms of active sampling remain poorly understood. We review a nascent neuroscientific literature that examines active-sampling policies and their relation to attention and curiosity. We distinguish between information sampling, in which organisms reduce uncertainty relevant to a familiar task, and information search, in which they investigate in an open-ended fashion to discover new tasks. We review evidence that both sampling and search depend on individual preferences over cognitive states, including attitudes towards uncertainty, learning progress and types of information. We propose that, although these preferences are non-instrumental and can on occasion interfere with external goals, they are important heuristics that allow organisms to cope with the high complexity of both sampling and search, and generate curiosity-driven investigations in large, open environments in which rewards are sparse and ex ante unknown.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Proposed architectures of attention and decision-making in current research.
Fig. 2: Evolution of beliefs when sampling information in different contexts.
Fig. 3: Neurons in lateral intraparietal area encode expected gains in information during instrumental sampling.
Fig. 4: Eye movements are impacted by non-instrumental search and curiosity.
Fig. 5: Artificial curiosity based on learning progress and autonomous goal sampling.
Fig. 6: Self-organized play in a laboratory game.

Similar content being viewed by others

References

  1. Gottlieb, J., Oudeyer, P. Y., Lopes, M. & Baranes, A. Information seeking, curiosity and attention: computational and empirical mechanisms. Trends Cogn. Sci. 17, 585–593 (2013).

    Google Scholar 

  2. Kidd, C. & Hayden, B. Y. The psychology and neuroscience of curiosity. Neuron 88, 449–460 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  3. Gottlieb, J., Hayhoe, M., Hikosaka, O. & Rangel, A. Attention, reward and information seeking. J. Neurosci. 34, 15497–154504 (2014).

    CAS  PubMed  PubMed Central  Google Scholar 

  4. Rehder, B. & Hoffman, A. B. Eye tracking and selective attention in category learning. Cogn. Psychol. 51, 1–41 (2005).

    PubMed  Google Scholar 

  5. Nelson, J. Finding useful questions: on Bayesian diagnosticity, probability, impact and information gain. Psychol. Rev. 112, 979–999 (2005).

    PubMed  Google Scholar 

  6. Coenen, A., Nelson, J. & Gureckis, T. Asking the right questions about the psychology of human inquiry: nine open challenges. Psychon Bull. Rev. https://doi.org/10.3758/s13423-018-1470-5 (2018).

    Article  PubMed  Google Scholar 

  7. Bossaerts, P. & Murawski, C. Computational complexity and human decision-making. Trends Cogn. Sci. 21, 917–929 (2017).

    PubMed  Google Scholar 

  8. Loewenstein, G. & Molnar, A. The renaissance of belief-based utility in economics. Nat. Hum. Behav. 2, 166–167 (2018).

    Google Scholar 

  9. Chater, N. & Loewenstein, G. The under-appreciated drive for sense-making. J. Econ. Behav. Organiz. 126, 137–154 (2016).

    Google Scholar 

  10. Wu, C. M., Meder, B., Filimon, F. & Nelson, J. D. Asking better questions: how presentation formats influence information search. J. Exp. Psychol. Learn. Mem. Cogn 43, 1274–1297 (2017).

    PubMed  Google Scholar 

  11. Markant, D. B. & Gureckis, T. M. Is it better to select or to receive? Learning via active and passive hypothesis testing. J. Exp. Psychol. Gen. 143, 94–122 (2014).

    PubMed  Google Scholar 

  12. Berlyne, D. Conflict, Arousal and Curiosity (McGraw-Hill, 1960).

  13. Berlyne, D. E. A theory of human curiosity. Br. J. Psychol. 45, 180–191 (1954).

    CAS  Google Scholar 

  14. Litman, J. A. in Issues in the Psychology of Motivation (ed. Zelick, P. R.) (Nova Science Publishers, 2007).

  15. Silvia, P. J. Exploring the Psychology of Interest (Oxford Univ. Press, 2006).

  16. Di Domenico, S. I. & Ryan, R. M. The emerging neuroscience of intrinsic motivation: a new frontier in self-determination research. Front. Hum. Neurosci. https://doi.org/10.3389/fnhum.2017.00145 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  17. Kaplan, F. & Oudeyer, P.-Y. In search of the neural circuits of intrinsic motivation. Frontiers Neurosci. 1, 225–225 (2007). This is a clear and succinct review of the concepts and computational models of intrinsic motivation and their importance to artificial intelligence.

    Google Scholar 

  18. Gopnik, A. Scientific thinking in young children: theoretical advances, empirical research, and policy implications. Science 337, 1623–1627 (2012).

    CAS  PubMed  Google Scholar 

  19. Renninger, K. A. & Hidi, S. E. The Power of Interest for Motivation and Engagement (Routledge, NY, 2016).

    Google Scholar 

  20. Begus, K., Gliga, T. & Southgate, V. Infants’ preferences for native speakers are associated with an expectation of information. Proc. Natl Acad. Sci. USA 113, 12397–12402 (2016).

    CAS  PubMed  Google Scholar 

  21. Kreps, D. M. & Porteus, E. L. Temporal resolution of uncertainty and dynamic choice theory. Econometrica 46, 185–200 (1978).

    Google Scholar 

  22. Caplin, A. & Dean, M. Revealed preference, rational inattention and costly information acquisition. Am. Econ. Rev. 105, 2183–2203 (2015).

    Google Scholar 

  23. Caplin, A. & Leahy, J. Psychological expected utility theory and anticipatory feelings. Q. J. Econ. 116, 55–79 (2001).

    Google Scholar 

  24. Clark, A. Surfing Uncertainty: Prediction, Action and the Embodied Mind. (Oxford Univ. Press, 2015).

  25. Livio, M. Why? What Makes Us Curious?. (Simon and Schuster, 2017).

  26. Hayhoe, M. & Ballard, D. Modeling task control of eye movements. Curr. Biol. 24, 622–628 (2014). This paper provides an excellent overview of empirical and modelling studies of eye movement control in natural tasks.

    Google Scholar 

  27. Tatler, B. W., Hayhoe, M. N., Land, M. F. & Ballard, D. H. Eye guidance in natural vision: reinterpreting salience. J. Vis. 11, 5–25 (2011).

    PubMed  PubMed Central  Google Scholar 

  28. Bach, D. R. & Dolan, R. J. Knowing how much you don’t know: a neural organization of uncertainty estimates. Nat. Rev. Neurosci. 13, 572–586 (2012).

    CAS  PubMed  Google Scholar 

  29. Cohen, J. D., McClure, S. M. & Yu, A. J. Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Phil. Trans. R. Soc. B 362, 933–942 (2007).

    PubMed  Google Scholar 

  30. Todd, P. M. & Gigerenzer, G. Précis of simple heuristics that make us smart. Behav. Brain Sci. 23, 727–780 (2000).

    CAS  PubMed  Google Scholar 

  31. Reynolds, J. H. & Heeger, D. J. The normalization model of attention. Neuron 61, 168–185 (2009).

    CAS  PubMed  PubMed Central  Google Scholar 

  32. Thompson, K. G. & Bichot, N. P. A visual salience map in the primate frontal eye field. Prog. Brain Res. 147, 251–262 (2005).

    PubMed  Google Scholar 

  33. Bisley, J. W. & Goldberg, M. E. Attention, intention, and priority in the parietal lobe. Annu. Rev. Neurosci. 33, 1–21 (2010).

    CAS  PubMed  PubMed Central  Google Scholar 

  34. Hanks, T. D. & Summerfield, C. Perceptual decision making in rodents, monkeys, and humans. Neuron 93, 15–31 (2017).

    CAS  PubMed  Google Scholar 

  35. Kable, J. W. & Glimcher, P. W. The neurobiology of decision: consensus and controversy. Neuron 63, 733–745 (2009).

    CAS  PubMed  PubMed Central  Google Scholar 

  36. Lee, D., Seo, H. & Jung, M. W. Neural basis of reinforcement learning and decision making. Annu. Rev. Neurosci. 35, 287–308 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  37. Krajbich, I., Armel, C. & Rangel, A. Visual fixations and the computation and comparison of value in simple choice. Nat. Neurosci. 13, 1292–1298 (2010).

    CAS  PubMed  Google Scholar 

  38. Krajbich, I., Lu, D., Camerer, C. & Rangel, A. The attentional drift-diffusion model extends to simple purchasing decisions. Front. Psychol. 3, 193 (2012).

    PubMed  PubMed Central  Google Scholar 

  39. Gottlieb, J. Attention, learning, and the value of information. Neuron 76, 281–295 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  40. Gottlieb, J. Understanding active sampling strategies: empirical approaches and implications for attention and decision reseeaerch. Cortex 102, 150–160 (2018). This is an overview of empirical approaches to information sampling in neurophysiology.

    PubMed  Google Scholar 

  41. Johnson, L., Sullivan, B., Hayhoe, M. & Ballard, D. H. Predicting human visuomotor behavior in a driving task. Phil. Trans. R. Soc. B. 369, 20130044 (2014).

    PubMed  Google Scholar 

  42. Sullivan, B. T., Johnson, L., Rothkopf, C. A., Ballard, D. & Hayhoe, M. The role of uncertainty and reward on eye movements in a virtual driving task. J. Vis. 12, 19 (2012).

    PubMed  PubMed Central  Google Scholar 

  43. Leong, Y., Radulescu, A., Daniel, R., DeWoskin, V. & Niv, Y. Dynamic interaction between reinforcement learning and attention in multidimensional environments. Neuron 93, 451–463 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  44. Wilson, R. C. & Niv, Y. Inferring relevance in a changing world. Front. Hum. Neurosci. 5, 189 (2011).

    PubMed  Google Scholar 

  45. Najemnik, J. & Geisler, W. S. Eye movement statistics in humans are consistent with an optimal search strategy. J. Vis 8, 4 (2008).

    PubMed  PubMed Central  Google Scholar 

  46. Yang, S. C., Lengyel, M. & Wolpert, D. M. Active sensing in the categorization of visual patterns. eLife 5, e12215 (2016). This paper provides evidence for information-based eye movement strategies using behavioural analysis and Bayesian modelling in humans.

    PubMed  PubMed Central  Google Scholar 

  47. Najemnik, J. & Geisler, W. S. Optimal eye movement strategies in visual search. Nature 434, 387–391 (2005).

    CAS  PubMed  Google Scholar 

  48. Renninger, L. W., Verghese, P. & Coughlan, J. Where to look next? Eye movements reduce local uncertainty. J. Vis 7, 6 (2007).

    PubMed  Google Scholar 

  49. Vossel, S., Vossel, S., Mathys, C., Stephan, K. E. & Friston, K. J. Cortical coupling reflects bayesian belief updating in the deployment of spatial attention. J. Neurosci. 35, 11532–11542 (2015). This is an analysis of attention in a Bayesian framework using functional MRI in humans.

    CAS  PubMed  PubMed Central  Google Scholar 

  50. Vossel, S. et al. Spatial attention, precision, and bayesian inference: a study of saccadic response speed. Cereb. Cortex 24, 1436–1450 (2014).

    PubMed  Google Scholar 

  51. Vossel, S., Thiel, C. M. & Fink, G. R. Cue validity modulates the neural correlates of covert endogenous orienting of attention in parietal and frontal cortex. NeuroImage 32, 1257–1264 (2006).

    PubMed  Google Scholar 

  52. Foley, N. C., Kelley, S. P., Mhatre, H., Lopes, M. & Gottlieb, J. Parietal neurons encode expected gains in instrumental information. Proc. Natl Acad. Sci. 114, E3315–E3323 (2017). This paper demonstrates that oculomotor neurons encode expected information gains in monkeys.

    CAS  PubMed  Google Scholar 

  53. Ernst, M. O. & Banks, M. S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).

    CAS  PubMed  Google Scholar 

  54. Nelson, J., McKenzie, C., Cottrell, G. & Sejnowski, T. Experience matters: information acquisition optimizes probability gain. Psychol. Sci. 21, 960–969 (2010).

    PubMed  PubMed Central  Google Scholar 

  55. Zajkowski, W. K., Kossut, M. & Wilson, R. C. A causal role for right frontopolar cortex in directed, but not random, exploration. eLife 6, e27430 (2017).

    PubMed  PubMed Central  Google Scholar 

  56. Somerville, L. H. et al. Charting the expansion of strategic exploratory behavior during adolescence. J. Exp. Psychol. Gen. 146, 155–164 (2017).

    PubMed  Google Scholar 

  57. Manohar, S. G. & Husain, M. Attention as foraging for information and value. Front. Hum. Neurosci. 7, 711 (2013).

    PubMed  PubMed Central  Google Scholar 

  58. Krishnamurthy, K., Nassar, M. R., Sarode, S. & Gold, J. I. Arousal-related adjustments of perceptual biases optimize perception in dynamic environments. Nat. Hum. Behav. 1, 0107 (2017).

    PubMed  PubMed Central  Google Scholar 

  59. Li, V., Herce Castañón, S., Solomon, J. A., Vandormael, H. & Summerfield, C. Robust averaging protects decisions from noise in neural computations. PLOS Comput. Biol. 13, e1005723 (2017).

    PubMed  PubMed Central  Google Scholar 

  60. Spitzer, B., Waschke, L. & Summerfield, C. Selective overwiehgting of larger magnitudes during noisy numerical comparison. Nat. Hum. Behav. 1, 0145 (2017).

    Google Scholar 

  61. Gold, J. I. & Stocker, A. A. Visual decision-making in an uncertain and dynamic world. Annu. Rev. Vis. Sci. 3, 227–250 (2017).

    PubMed  Google Scholar 

  62. Ebitz, R. B., Albarran, E. & Moore, T. Exploration disrupts choice-predictive signals and alters dynamics in prefrontal cortex. Neuron 97, 450–461 (2018).

    CAS  PubMed  Google Scholar 

  63. Gersch, T. M., Foley, N. C., Eisenberg, I. & Gottlieb, J. Neural correlates of temporal credit assignment in the parietal lobe. PLOS ONE 9, e88725 (2014).

    PubMed  PubMed Central  Google Scholar 

  64. Rossi, A. F., Pessoa, L., Desimone, R. & Ungerleider, L. G. The prefrontal cortex and the executive control of attention. Exp. Brain Res. 192, 489–497 (2009).

    Google Scholar 

  65. Rossi, A. F., Bichot, N. P., Desimone, R. & Ungerleider, L. G. Top down attentional deficits in macaques with lesions of lateral prefrontal cortex. J. Neurosci. 27, 11306–11314 (2007).

    CAS  PubMed  Google Scholar 

  66. Morvan, C. & Maloney, L. Human visual search does not maximize the post-saccadic probability of identifying targets. PLOS Comput. Biol. 8, e1002342 (2012). This presents an intriguing demonstration that humans show suboptimal sampling strategies in a task requiring flexible adjustments based on estimates of visibility.

    CAS  PubMed  PubMed Central  Google Scholar 

  67. Ghahghaei, S. & Verghese, P. Efficient saccade planning requires time and clear choices. Vision Res. 113B, 125–136 (2015).

    Google Scholar 

  68. Chong, T. T. et al. Neurocomputational mechanisms underlying subjective valuation of effort costs. PLOS Biol. 15, e1002598 (2017).

    PubMed  PubMed Central  Google Scholar 

  69. Shenhav, A. et al. Toward a rational and mechanistic account of mental effort. Annu. Rev. Neurosci. 40, 99–124 (2017).

    CAS  PubMed  Google Scholar 

  70. Fan, J. An information theory account of cognitive control. Front. Hum. Neurosci. https://doi.org/10.3389/fnhum.2014.00680 (2014). This paper proposes a reframing of theories of cognitive control from the perspective of informational constraints.

    Article  PubMed  PubMed Central  Google Scholar 

  71. Fleming, S. & Daw, N. Self-evaluation of decision-making: a general Bayesian framework for metacognitive computation. Psychol. Rev. 124, 91–114 (2017).

    PubMed  PubMed Central  Google Scholar 

  72. Zhang, H., Daw, N. D. & Maloney, L. T. Human representation of visuo-motor uncertainty as mixtures of orthogonal basis distributions. Nat. Neurosci. 18, 1152–1158 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  73. Vasconcelos, M., Monteiro, T. & Kacelnik, A. Irrational choice and the value of information. Sci. Rep. 5, 13874 (2015).

    Google Scholar 

  74. Eliaz, K. & Schotter, A. Experimental testing of intrinsic preferences for noninstrumental information. Am. Econ. Rev. 97, 166–169 (2007).

    Google Scholar 

  75. Bromberg-Martin, E. S. & Hikosaka, O. Midbrain dopamine neurons signal preference for advance information about upcoming rewards. Neuron 63, 119–126 (2009).

    CAS  PubMed  PubMed Central  Google Scholar 

  76. Bennett, D., Bode, S., Brydevall, M., Warren, H. & Murawski, C. Intrinsic valuation of information in decision making under uncertainty. PLOS Comp. Biol. 12, e1005020 (2016).

    Google Scholar 

  77. Brydevall, M., Bennett, D., Murawski, C. & Bode, S. The neural encoding of information prediction errors during non-instrumental information seeking. Sci. Rep. 8, 6134 (2018).

    PubMed  PubMed Central  Google Scholar 

  78. Blanchard, T. C., Hayden, B. Y. & Bromberg-Martin, E. S. Orbitofrontal cortex uses distinct codes for different choice attributes in decisions motivated by curiosity. Neuron 85, 602–614 (2015). This paper demonstrates single-neuron encoding of non-instrumental information value in the monkey orbitofrontal cortex.

    CAS  PubMed  PubMed Central  Google Scholar 

  79. Golman, R. & Loewenstein, G. Information gaps: a theory of preferences regarding the presence and absence of information. Decision 5, 143–164 (2018).

    Google Scholar 

  80. Loewenstein, G. Anticipation and the valuation of delayed consumption. Econ. J. 97, 666–684 (1987).

    Google Scholar 

  81. Iigaya, K., Story, G. W., Kurth-Nelson, Z., Dolan, R. J. & Dayan, P. The modulation of savouring by prediction error and its effects on choice. eLife 5, e13747 (2016). This paper presents a reinforcement learning model of non-instrumental information demand, proposing that, in addition to producing learning, dopaminergic reward prediction errors confer value to predictor states.

    PubMed  PubMed Central  Google Scholar 

  82. Flagel, S. B. & Robinson, T. E. Neurobiological basis of individual variation in stimulus-reward learning. Curr. Opin. Behav. Sci. 13, 178–185 (2017).

    PubMed  PubMed Central  Google Scholar 

  83. Peck, C. J., Jangraw, D. C., Suzuki, M., Efem, R. & Gottlieb, J. Reward modulates attention independently of action value in posterior parietal cortex. J. Neurosci. 29, 11182–11191 (2009).

    CAS  PubMed  PubMed Central  Google Scholar 

  84. Foley, N. C., Jangraw, D. C., Peck, C. & Gottlieb, J. Novelty enhances visual salience independently of reward in the parietal lobe. J. Neurosci. 34, 7947–7957 (2014).

    CAS  PubMed  PubMed Central  Google Scholar 

  85. Isoda, M. & Hikosaka, O. A neural correlate of motivational conflict in the superior colliculus of the macaque. J. Neurophysiol. 100, 1332–1342 (2008).

    PubMed  PubMed Central  Google Scholar 

  86. Anderson, B. The attention habit: how reward learning shapes attentional selection. Ann. NY Acad. Sci. 1369, 24–39 (2016). This is a comprehensive review of reward-related attention biases and their neural mechanisms and behavioural importance in humans.

    PubMed  Google Scholar 

  87. Hickey, C., Chelazzi, L. & Theeuwes, J. Reward guides vision when it’s your thing: trait reward-seeking in reward-mediated visual priming. PLOS ONE 5, e14087 (2010).

    PubMed  PubMed Central  Google Scholar 

  88. Daddaoua, N., Lopes, M. & Gottlieb, J. Intrinsically motivated oculomotor exploration guided by uncertainty reduction and conditioned reinforcement in non-human primates. Sci. Rep. 6, 20202 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

  89. Hickey, C., Chelazzi, L. & Theeuwes, J. Reward changes salience in human vision via the anterior cingulate. J. Neurosci. 30, 11096–11103 (2010).

    CAS  PubMed  Google Scholar 

  90. Hickey, C. & Peelen, M. V. Neural mechanisms of incentive salience in naturalistic human vision. Neuron 85, 512–518 (2015).

    CAS  PubMed  Google Scholar 

  91. Hunt, L. T., Rutledge, R. B., Malalasekera, W. M., Kennerley, S. W. & Dolan, R. J. Approach-induced biases in human information sampling. PLOS Biol. 14, e2000638 (2016).

    PubMed  PubMed Central  Google Scholar 

  92. Barbaro, L., Peelen, M. V. & Hickey, C. Valence, not utility, underlies reward-driven prioritization in human vision. J. Neurosci. 37, 10438–10450 (2017). This is among the first empirical demonstrations of reward-based and uncertainty-based modulations of visual representations in the human high-level cortex.

    CAS  Google Scholar 

  93. San Martín, R., Appelbaum, L. G., Huettel, S. A. & Woldorff, M. G. Cortical brain activity reflecting attentional biasing toward reward-predicting cues covaries with economic decision-making performance. Cereb. Cortex 26, 1–11 (2016).

    PubMed  Google Scholar 

  94. van Lieshout, L. L. F., Vandenbroucke, A. R. E., Müller, N. C. J., Cools, R. & de Lange, F. P. Induction and relief of curiosity elicit parietal and frontal activity. J. Neurosci. 38, 2579–2588 (2018). This is a demonstration of non-instrumental information value and its neural correlates in humans.

    Google Scholar 

  95. Loewenstein, G. The psychology of curiosity: a review and reinterpretation. Psychol. Bull. 116, 75–98 (1994).

    Google Scholar 

  96. Baldassare, G., Mirolli, M. (eds) Intrinsically Motivated Learning in Natural and Artificial Systems (Springer-Verlag, Berlin, 2013).

    Google Scholar 

  97. Gruber, M. J., Gelman, B. D. & Ranganath, C. States of curiosity modulate hippocampus-dependent learning via the dopaminergic circuit. Neuron 84, 486–496 (2014). This paper demonstrates the effects of curiosity on memory and the hippocampus in humans.

    CAS  PubMed  PubMed Central  Google Scholar 

  98. Kang, M. J. et al. The wick in the candle of learning: epistemic curiosity activates reward circuitry and enhances memory. Psychol. Sci. 20, 963–973 (2009).

    PubMed  Google Scholar 

  99. Baranes, A. F., Oudeyer, P. Y. & Gottlieb, J. Eye movements encode epistemic curiosity in human observers. Vis. Res. 117, 81–90 (2015).

    PubMed  Google Scholar 

  100. Marvin, C. B. & Shohamy, D. Curiosity and reward: valence predicts choice and information prediction errors enhance learning. J. Exp. Psychol. Gen. 145, 266–272 (2016).

    PubMed  Google Scholar 

  101. Jepma, M., Verdonschot, R. G., van Steenbergen, H., Rombouts, S. A. & Nieuwenhuis, S. Neural mechanisms underlying the induction and relief of perceptual curiosity. Front. Behav. Neurosci. https://doi.org/10.3389/fnbeh.2012.00005 (2012). This is a study of perceptual curiosity using functional MRI in humans.

    Article  PubMed  PubMed Central  Google Scholar 

  102. Risko, E. F., Anderson, N. C., Lanthier, S. & Kingstone, A. Curious eyes: individual differences in personality predict eye movement behavior in scene-viewing. Cognition 122, 86–90 (2012).

    PubMed  Google Scholar 

  103. Salimpoor, V. N., Zald, D. H., Zatorre, R. J., Dagher, A. & McIntosh, A. R. Predictions and the brain: how musical sounds become rewarding. Trends Cogn. Sci. 19, 86–91 (2015).

    PubMed  Google Scholar 

  104. Huron, D. Sweet Anticipation: Music and the Psychology of Expectation (MIT Press, 2006).

  105. Liao, H. I., Yeh, S. L. & Shimojo, S. Novelty versus familiarity principles in preference decisions: task-context of past experience matters. Front. Psychol. 2, 43 (2011).

    PubMed  PubMed Central  Google Scholar 

  106. Park, J., Shimojo, E. & Shimojo, S. Roles of familiarity and novelty in visual preference judgments are segregated across object categories. Proc. Natl Acad. Sci. USA 107, 14552–14555 (2010).

    CAS  PubMed  Google Scholar 

  107. Güçlütürk, Y., Güçlü, U., van Gerven, M. & van Lier, R. Representations of naturalistic stimulus complexity in early and associative visual and auditory cortices. Sci. Rep. 8, 3439 (2018).

    PubMed  PubMed Central  Google Scholar 

  108. Zatorre, R. J. Musical pleasure and reward: mechanisms and dysfunction. Ann. NY Acad. Sci. 1337, 202–211 (2015).

    PubMed  Google Scholar 

  109. Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P. & Pezzulo, G. Active inference: a process theory. Neural Comput. 29, 1–49 (2016).

    PubMed  Google Scholar 

  110. Sutton, R. S. & Barto, A. G. Reinforcement Learning: an Introduction (MIT Press, 1998).

  111. Daw, N. D., Gerschman, S. J., Seymour, B., Dayan, P. & Dolan, R. J. Model-based influences on human choices and striatal prediction errors. Neuron 69, 1204–1215 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  112. Friston, K. J. et al. Active inference, curiosity and insight. Neural Comput. 29, 2633–2683 (2017).

    PubMed  Google Scholar 

  113. Morewedge, C. K. & Kahneman, D. Associative processes in intuitive judgment. Trends Cogn. Sci. 14, 435–440 (2010).

    PubMed  PubMed Central  Google Scholar 

  114. Buckley, C., Kim, C. S., McGregor, S. & Seth, A. K. The free energy principle for action and perception: a mathematical review. J. Math. Psychol. 81, 55–79 (2017).

    Google Scholar 

  115. Gershman, S. J. & Blei, D. M. A tutorial on Bayesian nonparametric models. J. Math. Psychol. 56, 1–12 (2012).

    Google Scholar 

  116. Baranes, A. & Oudeyer, P. Y. Active learning of inverse models with intrinsically motivated goal exploration in robots. Rob. Auton. Syst. 61, 49–73 (2013).

    Google Scholar 

  117. Oudeyer, P. Y., Kaplan, F. & Hafner, V. V. Instrinsic motivation systems for autonomous mental development. IEEE Trans. Evol. Comput. 11, 265–286 (2007).

    Google Scholar 

  118. Forestier, S. & Oudeyer, P. Y. in Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) 3965–3972 (IEEE, 2016).

  119. Moulin-Frier, C., Nguyen, S. M. & Oudeyer, P.-Y. Self-organization of early vocal development in infants and machines: the role of intrinsic motivation. Front. Psychol. 4, 1006 (2014).

    Article  PubMed  Google Scholar 

  120. Forestier, S. & Oudeyer, P. Y. in Proc. 39th Annual Meeting of the Cognitive Science Soc. 2013–2018 (Cogsci, 2017).

  121. Clement, B., Roy, D., Oudeyer, P. Y. & Lopes, M. Multi-armed bandits for intelligent tutoring systems. J. Educ. Data Mining 7, 2 (2015).

  122. Metcalfe, J. Metacognitive judgments and control of study. Curr. Dir. Psychol. Sci. 18, 159–163 (2009).

    PubMed  PubMed Central  Google Scholar 

  123. Lopes, M. & Oudeyer, P.-Y. in Proc. IEEE Int. Conf. on Development and Learning and Epigenetic Robotics (ICDL) 1–8 (IEEE, 2012).

  124. Son, L. & Sethi, R. Metacognitive control and optimal learning. Cogn. Sci. 30, 759–774 (2006).

    PubMed  Google Scholar 

  125. Baranes, A. F., Oudeyer, P. Y. & Gottlieb, J. The effects of task difficulty, novelty and the size of the search space on intrinsically motivated exploration. Front. Neurosci. 8, 317 (2014). This presents a novel laboratory task for examining intrinsically motivated exploration based on difficulty in humans.

    PubMed  PubMed Central  Google Scholar 

  126. Barto, A., Singh, S. & Chenatez, N. in Proc. 3rd Int. Conf. Dvp. Learn 112–119 (San Diego, CA, 2004).

  127. Schmidhuber, J. in Proc. Int. Joint Conf. Neural Networks 2, 1458–1463 (IEEE, 1991).

  128. Bellemare, M. et al. in Proc. Advances in Neural Information Processing Systems 29 Conf. 1471–1479 (NIPS, 2016).

  129. Kulkarni, T. D., Narasimhan, K., Saeedi, A. & Tenenbaum, J. B. in Proc. Advances in Neural Information Processing Systems 29 Conf. 3675–3683 (NIPS, 2016).

  130. Pouget, A., Drugowitsch, J. & Kepecs, A. Confidence and certainty: distinct probabilistic quantities for different goals. Nat. Neurosci. 19, 366–374 (2016).

    CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors acknowledge support from the Human Frontiers Science Program (Collaborative Research Grant RGP0018/2016 to J.G. and P.-Y.O.), an Inria Neurocuriosity grant (to J.G. and P.-Y.O.), the National Eye Institute (RO1 grant to J.G.) and the National Institute of Mental Health (RO1 grant to J.G.).

Reviewer information

Nature Reviews Neuroscience thanks V. Stuphorn and the other anonymous reviewers for their contribution to the peer review of this work.

Author information

Authors and Affiliations

  1. Department of Neuroscience, Columbia University, New York, NY, USA

    Jacqueline Gottlieb

  2. Kavli Institute for Brain Science, Columbia University, New York, NY, USA

    Jacqueline Gottlieb

  3. Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA

    Jacqueline Gottlieb

  4. Inria, Bordeaux, France

    Pierre-Yves Oudeyer

  5. Ensta ParisTech, Paris, France

    Pierre-Yves Oudeyer

Authors
  1. Jacqueline Gottlieb
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pierre-Yves Oudeyer
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The authors both researched data for the article, provided substantial contributions to discussion of content, wrote the article, and reviewed and edited the manuscript before submission.

Corresponding author

Correspondence to Jacqueline Gottlieb.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Glossary

Agents

Any entities that are capable of learning and decision-making, including humans, other animals and artificial intelligence applications such as robots and self-driving cars.

Instrumental context

A context in which agents are motivated by the desire to obtain a known goal, which is operationalized in the laboratory as maximizing a material reward (such as money, points, food or safety).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gottlieb, J., Oudeyer, PY. Towards a neuroscience of active sampling and curiosity. Nat Rev Neurosci 19, 758–770 (2018). https://doi.org/10.1038/s41583-018-0078-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41583-018-0078-0

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing