In everyday life, humans face environments that feature uncertain and volatile or changing situations. Efficient adaptive behaviour must take into account uncertainty and volatility. Previous models of adaptive behaviour involve inferences about volatility that rely on complex and often intractable computations. Because such computations are presumably implausible biologically, it is unclear how humans develop efficient adaptive behaviours in such environments. Here, we demonstrate a counterintuitive result: simple, low-level inferences confined to uncertainty can produce near-optimal adaptive behaviour, regardless of the environmental volatility, assuming imprecisions in computation that conform to the psychophysical Weber law. We further show empirically that this Weber-imprecision model explains human behaviour in volatile environments better than optimal adaptive models that rely on high-level inferences about volatility, even when considering biologically plausible approximations of such models, as well as non-inferential models like adaptive reinforcement learning.
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
All the data that support the findings of the present study are available from the corresponding author upon request.
All program codes are freely available at https://github.com/csmfindling/learning_variability_and_volatility
Behrens, T. E., Woolrich, M. W., Walton, M. E. & Rushworth, M. F. Learning the value of information in an uncertain world. Nat. Neurosci. 10, 1214–1221 (2007).
Boorman, E. D., Behrens, T. E., Woolrich, M. W. & Rushworth, M. F. How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 62, 733–743 (2009).
Gershman, S. J., Blei, D. M. & Niv, Y. Context learning, and extinction. Psychological Rev. 117, 1997–1209 (2010).
Tenenbaum, J. B., Kemp, C., Griffiths, T. L. & Goodman, N. D. How to grow a mind: statistics, structure, and abstraction. Science 331, 1279–1285 (2011).
Collins, A. G. & Koechlin, E. Reasoning, learning, and creativity: frontal lobe function and human decision-making. PLoS Biol. 10, e1001293 (2012).
Collins, A. G. & Frank, M. J. Cognitive control over learning: creating, clustering, and generalizing task-set structure. Psychol. Rev. 120, 190–229 (2013).
Donoso, M., Collins, A. G. & Koechlin, E. Foundations of human reasoning in the prefrontal cortex. Science 344, 1481–1486 (2014).
Kolossa, A., Kopp, B. & Fingscheidt, T. A computational analysis of the neural bases of Bayesian inference. Neuroimage 106, 222–237 (2015).
Schuck, N. W., Cai, M. B., Wilson, R. C. & Niv, Y. Human orbitofrontal cortex represents a cognitive map of state space. Neuron 91, 1402–1412 (2016).
Rouault, M., Drugowitsch, J. & Koechlin, E. Prefrontal mechanisms combining rewards and beliefs in human decision-making. Nat. Commun. 10, 301 (2019).
Nassar, M. R., Wilson, R. C., Heasly, B. & Gold, J. I. An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment. J. Neurosci. 30, 12366–12378 (2010).
Payzan-LeNestour, E. & Bossaerts, P. Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Comput. Biol. 7, e1001048 (2011).
Wilson, R. C., Nassar, M. R. & Gold, J. I. A mixture of delta-rules approximation to bayesian inference in change-point problems. PLoS Comput. Biol. 9, e1003150 (2013).
McGuire, J. T., Nassar, M. R., Gold, J. I. & Kable, J. W. Functionally dissociable influences on learning rate in a dynamic environment. Neuron 84, 870–881 (2014).
Bossaerts, P., Yadav, N. & Murawski, C. Uncertainty and computational complexity. Philos. Trans. R. Soc. Lond. B Biol. Sci. 374, 20180138 (2019).
Drugowitsch, J., Wyart, V., Devauchelle, A. D. & Koechlin, E. Computational precision of mental inference as critical source of human choice suboptimality. Neuron 92, 1398–1411 (2016).
Fechner G. T. Elemente der Psychophysik (Breitkopf and Härtel, 1860).
Treisman, M. Noise and Weber’s law: the discrimination of brightness and other dimensions. Psychol. Rev. 71, 314–330 (1964).
Deco, G., Scarano, L. & Soto-Faraco, S. Weber’s law in decision making: integrating behavioral data in humans with a neurophysiological model. J. Neurosci. 27, 11192–11200 (2007).
Wyart, V. & Koechlin, E. Choice variability and suboptimality in uncertain environments. Curr. Opin. Behav. Sci. 11, 109–115 (2016).
Faraji, M., Preuschoff, K. & Gerstner, W. Balancing new against old information: the role of puzzlement surprise in learning. Neural Comput. 30, 34–83 (2018).
Chopin, N., Jacob, P. E. & Papaspiliopoulos, O. SMC2: an efficient algorithm for sequential analysis of state space models. J. R. Stat. Soc. Series B Stat. Methodol. 75, 397–426 (2013).
Doucet, A., Godsill, S. & Andrieu, C. On sequential Monte Carlo sampling methods for Bayesian filtering. Statist. Comput. 10, 197–208 (2000).
Andrieu, C., Doucet, A. & Holenstein, R. Particle Markov chain Monte Carlo methods. J. R. Stat. Soc. Series B Stat. Methodol. 72, 269–342 (2010).
Chopin, N. A sequential particle filter for static models. Biometrika 89, 539–552 (2002).
Shi, L. & Griffiths, T. L. Neural implementation of hierarchical Bayesian inference by importance sampling. Adv. Neural Inf. Process. Syst. 22, 1669–1677 (2009).
Huang, Y. & Rao, R. P. Neurons as Monte Carlo samplers: Bayesian inference and learning in spiking networks. Adv. Neural Inf. Process. Syst. 27, 1943–1951 (2014).
Legenstein, R. & Maass, W. Ensembles of spiking neurons with noise support optimal probabilistic inference in a dynamically changing environment. PLoS Comput. Biol. 10, e1003859 (2014).
Scott, S. L. Bayesian methods for hidden Markov models. J. Am. Stat. Assoc. 97, 337–351 (2002).
Pearce, J. M. & Hall, G. A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev. 87, 532–552 (1980).
Roesch, M., Esber, G. R., Li, J., Daw, N. & Schoenbaum, G. Surprise! Neural correlates of Pearce–Hall and Rescorla–Wagner coexist within the brain. Eur. J. Neurosci. 35, 1190–1200 (2012).
Daw, N. D., O’Doherty, J. P., Dayan, P., Seymour, B. & Dolan, R. J. Cortical substrates for exploratory decisions in humans. Nature 441, 876–879 (2006).
Rigoux, L., Stephan, K. E., Friston, K. J. & Daunizeau, J. Bayesian model selection for group studies - revisited. Neuroimage 84, 971–985 (2014).
Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J. & Friston, K. J. Bayesian model selection for group studies. Neuroimage 46, 1004–1017 (2009).
Palminteri, S., Wyart, V. & Koechlin, E. The importance of falsification in computational cognitive modeling. Trends Cogn. Sci. 21, 425–433 (2017).
Payzan-LeNestour, E. Bayesian Learning in Unstable Settings: Experimental Evidence Based on the Bandit Problem Research Paper No. 10-28 (Swiss Finance Inst., 2010).
Summerfield, C., Behrens, T. E. & Koechlin, E. Perceptual classification in a rapidly changing environment. Neuron 71, 725–736 (2011).
Knight, F. H. Risk, Uncertainty and Profit (Univ. Chicago Press, 1921).
Keynes, J. M. A Treatise on Probability (Macmillan, 1921).
Bogacz, R., Brown, E., Moehlis, J., Holmes, P. & Cohen, J. D. The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychol. Rev. 113, 700–765 (2006).
Wang, X.-J. Neural dynamics and circuit mechnaisms of decision-making. Curr. Opin. Neurobiol. 22, 1039–1046 (2012).
Payzan-LeNestour, E., Dunne, S., Bossaerts, P. & O’Doherty, J. P. The neural representation of unexpected uncertainty during value-based decision making. Neuron 79, 191–201 (2013).
Lebreton, M., Abitbol, R., Daunizeau, J. & Pessiglione, M. Automatic integration of confidence in the brain valuation signal. Nat. Neurosci. 18, 1159–1167 (2015).
Beaumont, M. A. Estimation of population growth or decline in genetically monitored populations. Genetics 164, 1139–1160 (2003).
Niederreiter, H. Random Number Generation and Quasi-Monte Carlo Methods (Society for Industrial and Applied Mathematics, 1992).
Snoek, J., Larochelle, H. & Adams, R. P. Practical Bayesian optimization of machine learning algorithms. Preprint at arXiv http://arxiv.org/abs/1206.2944 (2012).
We thank J. Drevet for her help in collecting human data. Supported by a European Research Council Grant (ERC-2009-AdG #250106) to E.K. and a DGA-INSERM PhD fellowship to C.F. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
The authors declare no competing interests
Peer review information Primary Handling Editor: Marike Schiffer
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Environment volatility follows a bounded gaussian random walk between 0.03 and 0.2 with variance 0.0001 (see Methods).
Extended Data Fig. 2 Relation between the Weber noise component λ in the Weber imprecision model and external volatility.
The figure shows the entropy of posterior beliefs about current combinations (latent state posteriors) for the exact varying-volatility and Weber imprecision model. Each model is simulated N = 50 times in a closed environment (K=2, two-armed bandit), which alternates between high and low-volatility periods. Left, simulations when Weber component λ is set to 0 and constant component μ is large (μ = 0.2). Right, simulations when Weber component λ is large (λ = 1.5) and constant component μ is low (μ = 0.02). Note that the entropies of posterior beliefs are similar between the Weber-imprecision and varying-volatility model only when the Weber component is large enough.
This model is exactly the process generating unstable environments. This generative model assumes that volatility τt follows a bounded random walk with constant variance ν. zt represents the current correct combination. γ represents the probabilities of combination occurrence whenever the correct combination changes. η represents feedback noise. In every trial, observables are stimuli st, actions at and binary feedback rt. See Methods for details.
This model is exactly the process generating changing environments. This generative model assumes that volatility τ is constant. zt represents the current correct combination. γ represents the probabilities of combination occurrence whenever the correct combination changes. η represents feedback noise. In every trial, observables are stimuli st, actions at and binary feedback rt. See Methods for details.
This model is exactly the process generating stable environments. This generative model assumes that volatility is null and that observations are all equally informative. z represents the correct combination. γ represents combinations’ probabilities. η represents feedback noise. In every trial, observables are stimuli st, actions at and binary feedback rt. See Methods for details.
Means (s.e.m) of parameters fitted across participants for exact and forward volatility models.
Means (s.e.m.) of parameters fitted across participants for the Weber-imprecision model.
Means (s.e.m.) of parameters fitted across participants for Reinforcement Learning models.
About this article
Cite this article
Findling, C., Chopin, N. & Koechlin, E. Imprecise neural computations as a source of adaptive behaviour in volatile environments. Nat Hum Behav (2020). https://doi.org/10.1038/s41562-020-00971-z