Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Imprecise neural computations as a source of adaptive behaviour in volatile environments

Nature Human Behaviour volume 5, pages 99–112 (2021)Cite this article

Subjects

Abstract

In everyday life, humans face environments that feature uncertain and volatile or changing situations. Efficient adaptive behaviour must take into account uncertainty and volatility. Previous models of adaptive behaviour involve inferences about volatility that rely on complex and often intractable computations. Because such computations are presumably implausible biologically, it is unclear how humans develop efficient adaptive behaviours in such environments. Here, we demonstrate a counterintuitive result: simple, low-level inferences confined to uncertainty can produce near-optimal adaptive behaviour, regardless of the environmental volatility, assuming imprecisions in computation that conform to the psychophysical Weber law. We further show empirically that this Weber-imprecision model explains human behaviour in volatile environments better than optimal adaptive models that rely on high-level inferences about volatility, even when considering biologically plausible approximations of such models, as well as non-inferential models like adaptive reinforcement learning.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Inferential models of adaptive behaviour.
Fig. 2: Models’ maximal performances in stable, changing and unstable environments.
Fig. 3: Models’ versatility across environments.
Fig. 4: Model fits to human performances in closed, unstable environments.
Fig. 5: Human and model adaptive behaviour following contingency reversals.
Fig. 6: Model fits to human performances in open-ended, changing environments.

Similar content being viewed by others

Data availability

All the data that support the findings of the present study are available from the corresponding author upon request.

Code availability

All program codes are freely available at https://github.com/csmfindling/learning_variability_and_volatility

References

  1. Behrens, T. E., Woolrich, M. W., Walton, M. E. & Rushworth, M. F. Learning the value of information in an uncertain world. Nat. Neurosci. 10, 1214–1221 (2007).

    Article  CAS  Google Scholar 

  2. Boorman, E. D., Behrens, T. E., Woolrich, M. W. & Rushworth, M. F. How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 62, 733–743 (2009).

    Article  CAS  Google Scholar 

  3. Gershman, S. J., Blei, D. M. & Niv, Y. Context learning, and extinction. Psychological Rev. 117, 1997–1209 (2010).

    Article  Google Scholar 

  4. Tenenbaum, J. B., Kemp, C., Griffiths, T. L. & Goodman, N. D. How to grow a mind: statistics, structure, and abstraction. Science 331, 1279–1285 (2011).

    Article  CAS  Google Scholar 

  5. Collins, A. G. & Koechlin, E. Reasoning, learning, and creativity: frontal lobe function and human decision-making. PLoS Biol. 10, e1001293 (2012).

    Article  CAS  Google Scholar 

  6. Collins, A. G. & Frank, M. J. Cognitive control over learning: creating, clustering, and generalizing task-set structure. Psychol. Rev. 120, 190–229 (2013).

    Article  Google Scholar 

  7. Donoso, M., Collins, A. G. & Koechlin, E. Foundations of human reasoning in the prefrontal cortex. Science 344, 1481–1486 (2014).

    Article  CAS  Google Scholar 

  8. Kolossa, A., Kopp, B. & Fingscheidt, T. A computational analysis of the neural bases of Bayesian inference. Neuroimage 106, 222–237 (2015).

    Article  Google Scholar 

  9. Schuck, N. W., Cai, M. B., Wilson, R. C. & Niv, Y. Human orbitofrontal cortex represents a cognitive map of state space. Neuron 91, 1402–1412 (2016).

    Article  CAS  Google Scholar 

  10. Rouault, M., Drugowitsch, J. & Koechlin, E. Prefrontal mechanisms combining rewards and beliefs in human decision-making. Nat. Commun. 10, 301 (2019).

    Article  Google Scholar 

  11. Nassar, M. R., Wilson, R. C., Heasly, B. & Gold, J. I. An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment. J. Neurosci. 30, 12366–12378 (2010).

    Article  CAS  Google Scholar 

  12. Payzan-LeNestour, E. & Bossaerts, P. Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Comput. Biol. 7, e1001048 (2011).

    Article  CAS  Google Scholar 

  13. Wilson, R. C., Nassar, M. R. & Gold, J. I. A mixture of delta-rules approximation to bayesian inference in change-point problems. PLoS Comput. Biol. 9, e1003150 (2013).

    Article  CAS  Google Scholar 

  14. McGuire, J. T., Nassar, M. R., Gold, J. I. & Kable, J. W. Functionally dissociable influences on learning rate in a dynamic environment. Neuron 84, 870–881 (2014).

    Article  CAS  Google Scholar 

  15. Bossaerts, P., Yadav, N. & Murawski, C. Uncertainty and computational complexity. Philos. Trans. R. Soc. Lond. B Biol. Sci. 374, 20180138 (2019).

    Article  Google Scholar 

  16. Drugowitsch, J., Wyart, V., Devauchelle, A. D. & Koechlin, E. Computational precision of mental inference as critical source of human choice suboptimality. Neuron 92, 1398–1411 (2016).

    Article  CAS  Google Scholar 

  17. Fechner G. T. Elemente der Psychophysik (Breitkopf and Härtel, 1860).

  18. Treisman, M. Noise and Weber’s law: the discrimination of brightness and other dimensions. Psychol. Rev. 71, 314–330 (1964).

    Article  CAS  Google Scholar 

  19. Deco, G., Scarano, L. & Soto-Faraco, S. Weber’s law in decision making: integrating behavioral data in humans with a neurophysiological model. J. Neurosci. 27, 11192–11200 (2007).

    Article  CAS  Google Scholar 

  20. Wyart, V. & Koechlin, E. Choice variability and suboptimality in uncertain environments. Curr. Opin. Behav. Sci. 11, 109–115 (2016).

    Article  Google Scholar 

  21. Faraji, M., Preuschoff, K. & Gerstner, W. Balancing new against old information: the role of puzzlement surprise in learning. Neural Comput. 30, 34–83 (2018).

    Article  Google Scholar 

  22. Chopin, N., Jacob, P. E. & Papaspiliopoulos, O. SMC2: an efficient algorithm for sequential analysis of state space models. J. R. Stat. Soc. Series B Stat. Methodol. 75, 397–426 (2013).

    Article  Google Scholar 

  23. Doucet, A., Godsill, S. & Andrieu, C. On sequential Monte Carlo sampling methods for Bayesian filtering. Statist. Comput. 10, 197–208 (2000).

    Article  Google Scholar 

  24. Andrieu, C., Doucet, A. & Holenstein, R. Particle Markov chain Monte Carlo methods. J. R. Stat. Soc. Series B Stat. Methodol. 72, 269–342 (2010).

    Article  Google Scholar 

  25. Chopin, N. A sequential particle filter for static models. Biometrika 89, 539–552 (2002).

    Article  Google Scholar 

  26. Shi, L. & Griffiths, T. L. Neural implementation of hierarchical Bayesian inference by importance sampling. Adv. Neural Inf. Process. Syst. 22, 1669–1677 (2009).

    Google Scholar 

  27. Huang, Y. & Rao, R. P. Neurons as Monte Carlo samplers: Bayesian inference and learning in spiking networks. Adv. Neural Inf. Process. Syst. 27, 1943–1951 (2014).

    Google Scholar 

  28. Legenstein, R. & Maass, W. Ensembles of spiking neurons with noise support optimal probabilistic inference in a dynamically changing environment. PLoS Comput. Biol. 10, e1003859 (2014).

    Article  Google Scholar 

  29. Scott, S. L. Bayesian methods for hidden Markov models. J. Am. Stat. Assoc. 97, 337–351 (2002).

    Article  Google Scholar 

  30. Pearce, J. M. & Hall, G. A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev. 87, 532–552 (1980).

    Article  CAS  Google Scholar 

  31. Roesch, M., Esber, G. R., Li, J., Daw, N. & Schoenbaum, G. Surprise! Neural correlates of Pearce–Hall and Rescorla–Wagner coexist within the brain. Eur. J. Neurosci. 35, 1190–1200 (2012).

    Article  Google Scholar 

  32. Daw, N. D., O’Doherty, J. P., Dayan, P., Seymour, B. & Dolan, R. J. Cortical substrates for exploratory decisions in humans. Nature 441, 876–879 (2006).

    Article  CAS  Google Scholar 

  33. Rigoux, L., Stephan, K. E., Friston, K. J. & Daunizeau, J. Bayesian model selection for group studies - revisited. Neuroimage 84, 971–985 (2014).

    Article  CAS  Google Scholar 

  34. Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J. & Friston, K. J. Bayesian model selection for group studies. Neuroimage 46, 1004–1017 (2009).

    Article  Google Scholar 

  35. Palminteri, S., Wyart, V. & Koechlin, E. The importance of falsification in computational cognitive modeling. Trends Cogn. Sci. 21, 425–433 (2017).

    Article  Google Scholar 

  36. Payzan-LeNestour, E. Bayesian Learning in Unstable Settings: Experimental Evidence Based on the Bandit Problem Research Paper No. 10-28 (Swiss Finance Inst., 2010).

  37. Summerfield, C., Behrens, T. E. & Koechlin, E. Perceptual classification in a rapidly changing environment. Neuron 71, 725–736 (2011).

    Article  CAS  Google Scholar 

  38. Knight, F. H. Risk, Uncertainty and Profit (Univ. Chicago Press, 1921).

  39. Keynes, J. M. A Treatise on Probability (Macmillan, 1921).

  40. Bogacz, R., Brown, E., Moehlis, J., Holmes, P. & Cohen, J. D. The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychol. Rev. 113, 700–765 (2006).

    Article  Google Scholar 

  41. Wang, X.-J. Neural dynamics and circuit mechnaisms of decision-making. Curr. Opin. Neurobiol. 22, 1039–1046 (2012).

    Article  Google Scholar 

  42. Payzan-LeNestour, E., Dunne, S., Bossaerts, P. & O’Doherty, J. P. The neural representation of unexpected uncertainty during value-based decision making. Neuron 79, 191–201 (2013).

    Article  CAS  Google Scholar 

  43. Lebreton, M., Abitbol, R., Daunizeau, J. & Pessiglione, M. Automatic integration of confidence in the brain valuation signal. Nat. Neurosci. 18, 1159–1167 (2015).

    Article  CAS  Google Scholar 

  44. Beaumont, M. A. Estimation of population growth or decline in genetically monitored populations. Genetics 164, 1139–1160 (2003).

    CAS  PubMed  PubMed Central  Google Scholar 

  45. Niederreiter, H. Random Number Generation and Quasi-Monte Carlo Methods (Society for Industrial and Applied Mathematics, 1992).

  46. Snoek, J., Larochelle, H. & Adams, R. P. Practical Bayesian optimization of machine learning algorithms. Preprint at arXiv http://arxiv.org/abs/1206.2944 (2012).

Download references

Acknowledgements

We thank J. Drevet for her help in collecting human data. Supported by a European Research Council Grant (ERC-2009-AdG #250106) to E.K. and a DGA-INSERM PhD fellowship to C.F. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Authors and Affiliations

  1. Ecole Normale Supérieure, PSL Research University, Paris, France

    Charles Findling & Etienne Koechlin

  2. ENSAE ParisTech, Saclay, France

    Charles Findling & Nicolas Chopin

  3. Université Pierre et Marie Curie, Paris, France

    Etienne Koechlin

  4. Institut National de la Santé et de la Recherche Médicale (INSERM), Paris, France

    Etienne Koechlin

Authors
  1. Charles Findling
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicolas Chopin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Etienne Koechlin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

E.K. and C.F. conceived the study and designed the models. C.F. and N.C. developed the models. C.F. programmed the models, performed computer simulations and collected human data. E.K. and C.F. analysed human and simulation data. E.K. and C.F. wrote the paper.

Corresponding author

Correspondence to Etienne Koechlin.

Ethics declarations

Competing interests

The authors declare no competing interests

Additional information

Peer review information Primary Handling Editor: Marike Schiffer

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Examples of five volatility trajectories in unstable environments.

Environment volatility follows a bounded gaussian random walk between 0.03 and 0.2 with variance 0.0001 (see Methods).

Extended Data Fig. 2 Relation between the Weber noise component λ in the Weber imprecision model and external volatility.

The figure shows the entropy of posterior beliefs about current combinations (latent state posteriors) for the exact varying-volatility and Weber imprecision model. Each model is simulated N = 50 times in a closed environment (K=2, two-armed bandit), which alternates between high and low-volatility periods. Left, simulations when Weber component λ is set to 0 and constant component μ is large (μ = 0.2). Right, simulations when Weber component λ is large (λ = 1.5) and constant component μ is low (μ = 0.02). Note that the entropies of posterior beliefs are similar between the Weber-imprecision and varying-volatility model only when the Weber component is large enough.

Extended Data Fig. 3 Full generative model of varying-volatility models.

This model is exactly the process generating unstable environments. This generative model assumes that volatility τt follows a bounded random walk with constant variance ν. zt represents the current correct combination. γ represents the probabilities of combination occurrence whenever the correct combination changes. η represents feedback noise. In every trial, observables are stimuli st, actions at and binary feedback rt. See Methods for details.

Extended Data Fig. 4 Full generative model of constant-volatility models.

This model is exactly the process generating changing environments. This generative model assumes that volatility τ is constant. zt represents the current correct combination. γ represents the probabilities of combination occurrence whenever the correct combination changes. η represents feedback noise. In every trial, observables are stimuli st, actions at and binary feedback rt. See Methods for details.

Extended Data Fig. 5 Full generative model of zero-volatility models.

This model is exactly the process generating stable environments. This generative model assumes that volatility is null and that observations are all equally informative. z represents the correct combination. γ represents combinations’ probabilities. η represents feedback noise. In every trial, observables are stimuli st, actions at and binary feedback rt. See Methods for details.

Extended Data Fig. 6

Means (s.e.m) of parameters fitted across participants for exact and forward volatility models.

Extended Data Fig. 7

Means (s.e.m.) of parameters fitted across participants for the Weber-imprecision model.

Extended Data Fig. 8

Means (s.e.m.) of parameters fitted across participants for Reinforcement Learning models.

Supplementary information

Supplementary Information

Supplementary Discussion, Supplementary Methods, Supplementary Figs. 1–5 and Supplementary Refs. 1–5.

Reporting Summary

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Findling, C., Chopin, N. & Koechlin, E. Imprecise neural computations as a source of adaptive behaviour in volatile environments. Nat Hum Behav 5, 99–112 (2021). https://doi.org/10.1038/s41562-020-00971-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-020-00971-z

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing