Abstract
Anxiety is characterized by altered responses under uncertain conditions, but the precise mechanism by which uncertainty changes the behaviour of anxious individuals is unclear. Here we probe the computational basis of learning under uncertainty in healthy individuals and individuals suffering from a mix of mood and anxiety disorders. Participants were asked to choose between four competing slot machines with fluctuating reward and punishment outcomes during safety and stress. We predicted that anxious individuals under stress would learn faster about punishments and exhibit choices that were more affected by those punishments, thus formalizing our predictions as parameters in reinforcement learning accounts of behaviour. Overall, the data suggest that anxious individuals are quicker to update their behaviour in response to negative outcomes (increased punishment learning rates). When treating anxiety, it may therefore be more fruitful to encourage anxious individuals to integrate information over longer horizons when bad things happen, rather than try to blunt their responses to negative outcomes.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
All data used in this analysis are available on OSF at https://doi.org/10.17605/OSF.IO/UB6J7 (ref. 16).
Code availability
Scripts for model fitting are available on OSF at https://doi.org/10.17605/OSF.IO/UB6J7 (ref. 16) and in the Supplementary Software. For the hBayesDM package, please see https://github.com/CCS-Lab/hBayesDM.
References
GBD Compare Data Visualization (IHME, accessed 17 November 2016).
LeDoux, J. E. & Pine, D. S. Using neuroscience to help understand fear and anxiety: a two-system framework. Am. J. Psychiat. 173, 1083–1093 (2016).
Grupe, D. W. & Nitschke, J. B. Uncertainty and anticipation in anxiety: an integrated neurobiological and psychological perspective. Nat. Rev. Neurosci. 14, 488–501 (2013).
Birrell, J., Meares, K., Wilkinson, A. & Freeston, M. Toward a definition of intolerance of uncertainty: a review of factor analytical studies of the Intolerance of Uncertainty Scale. Clin. Psychol. Rev. 31, 1198–1208 (2011).
Charpentier, C. J., Aylward, J., Roiser, J. P. & Robinson, O. J. Enhanced risk aversion, but not loss aversion, in unmedicated pathological anxiety. Biol. Psychiat. 81, 1014–1022 (2017).
Grillon, C. Models and mechanisms of anxiety: evidence from startle studies. Psychopharmacology 199, 421–437 (2008).
Robinson, O. J., Overstreet, C., Allen, P. S., Pine, D. S. & Grillon, C. Acute tryptophan depletion increases translational indices of anxiety but not fear: serotonergic modulation of the bed nucleus of the stria terminalis? Neuropsychopharmacology 37, 1963–1971 (2012).
Robinson, O. J. et al. The dorsal medial prefrontal (anterior cingulate) cortex–amygdala aversive amplification circuit in unmedicated generalised and social anxiety disorders: an observational study. Lancet Psychiat. 1, 294–302 (2014).
Daw, N. D., O’Doherty, J. P., Dayan, P., Seymour, B. & Dolan, R. J. Cortical substrates for exploratory decisions in humans. Nature 441, 876–879 (2006).
Seymour, B., Daw, N. D., Roiser, J. P., Dayan, P. & Dolan, R. Serotonin selectively modulates reward value in human decision-making. J. Neurosci. 32, 5833–5842 (2012).
Sharp, P. B. & Eldar, E. Computational models of anxiety: nascent efforts and future directions. Curr. Dir. Psychol. Sci. 28, 170–176 (2019).
Robinson, O. J., Vytal, K., Cornwell, B. R. & Grillon, C. The impact of anxiety upon cognition: perspectives from human threat of shock studies. Front. Human Neurosci. 7, 203 (2013).
Mkrtchian, A., Aylward, J., Dayan, P., Roiser, J. P. & Robinson, O. J. Modeling avoidance in mood and anxiety disorders using reinforcement learning. Biol. Psychiat. 82, 532–539 (2017).
Gagne, C., Dayan, P. & Bishop, S. J. When planning to survive goes wrong: predicting the future and replaying the past in anxiety and PTSD. Curr. Opin. Behav. Sci. 24, 89–95 (2018).
Monroe, S. M. & Simons, A. D. Diathesis-stress theories in the context of life stress research: implications for the depressive disorders. Psychol. Bull. 110, 406–425 (1991).
Robinson, O. J. Altered learning under uncertainty in unmedicated mood and anxiety disorders—EU storage. Preprint at OSF https://doi.org/10.17605/OSF.IO/UB6J7 (2018).
Gelman, A. & Rubin, D. B. Inference from iterative simulation using multiple sequences. Stat. Sci. 7, 457–472 (1992).
Bach, D. R. Anxiety-like behavioural inhibition is normative under environmental threat-reward correlations. PLoS Comput. Biol. 11, e1004646 (2015).
Robinson, O. J., Overstreet, C., Charney, D. S., Vytal, K. & Grillon, C. Stress increases aversive prediction-error signal in the ventral striatum. Proc. Natl Acad. Sci. USA 110, 4129–4133 (2013).
Deacon, B. J. & Abramowitz, J. S. Cognitive and behavioral treatments for anxiety disorders: a review of meta‐analytic findings. J. Clin. Psychol. 60, 429–441 (2004).
Wilson, A., Fern, A., Ray, S. & Tadepalli, P. Multi-task reinforcement learning: a hierarchical Bayesian approach. In Proc. 24th International Conference on Machine Learning 1015−1022 (ACM, 2007).
Browning, M., Behrens, T. E., Jocham, G., O’Reilly, J. X. & Bishop, S. J. Anxious individuals have difficulty learning the causal statistics of aversive environments. Nat. Neurosci. 18, 590–596 (2015).
Lissek, S., Pine, D. S. & Grillon, C. The strong situation: a potential impediment to studying the psychobiology and pharmacology of anxiety disorders. Biol. Psychol. 72, 265–270 (2006).
Robinson, O. J., Cools, R., Carlisi, C. O., Sahakian, B. J. & Drevets, W. C. Ventral striatum response during reward and punishment reversal learning in unmedicated major depressive disorder. Am. J. Psychiat. 169, 152–159 (2012).
Maxwell, S. E., Kelley, K. & Rausch, J. R. Sample size planning for statistical power and accuracy in parameter estimation. Annu. Rev. Psychol. 59, 537–563 (2008).
Sheehan, D. et al. The validity of the Mini International Neuropsychiatric Interview (MINI) according to the SCID-P and its reliability. Eur. Psychiat. 12, 232–241 (1997).
Cogent 2000 Team at the FIL and the ICN. Cogent http://www.vislab.ucl.ac.uk/cogent.php (2013).
Carlisi, C. O. & Robinson, O. J.. The role of prefrontal–subcortical circuitry in negative bias in anxiety: translational, developmental and treatment perspectives. Brain Neurosci. Adv. https://doi.org/10.1177/2398212818774223 (2018).
Mkrtchian, A., Roiser, J. P. & Robinson, O. J. Threat of shock and aversive inhibition: induced anxiety modulates Pavlovian-instrumental interactions. J. Exp. Psychol. Gen. 146, 1694–1704 (2017).
JASP Team. JASP (Version 0.7. 5.5) Google Sch. 765, 766 (2016).
Ahn, W.-Y., Haines, N. & Zhang, L. Revealing neurocomputational mechanisms of reinforcement learning and decision-making with the hBayesDM package. Comp. Psychiat. 1, 24–57 (2017).
Stan Development Team. RStan: the R interface to Stan. R package version 2.17.3, http://mc-stan.org/ (2018).
Ahn, W.-Y., Krawitz, A., Kim, W., Busemeyer, J. R. & Brown, J. W. A model-based fMRI analysis with hierarchical Bayesian parameter estimation. J. Neurosci. Psychol. Econ. 4, 95–110 (2011).
Guitart-Masip, M. et al. Go and no-go learning in reward and punishment: interactions between affect and effect. Neuroimage 62, 154–166 (2012).
Huys, Q. J., Pizzagalli, D. A., Bogdan, R. & Dayan, P. Mapping anhedonia onto reinforcement learning: a behavioural meta-analysis. Biol. Mood Anxiety Disord. 3, 12 (2013).
Niv, Y. et al. Reinforcement learning in multidimensional environments relies on attention mechanisms. J. Neurosci. 35, 8145–8157 (2015).
Ahn, W. Y., Busemeyer, J. R., Wagenmakers, E. J. & Stout, J. C. Comparison of decision learning models using the generalization criterion method. Cogn. Sci. 32, 1376–1402 (2008).
Ahn, W.-Y. et al. Decision-making in stimulant and opiate addicts in protracted abstinence: evidence from computational modeling with pure users. Front. Psychol. 5, 849 (2014).
Kruschke, J. Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan (Academic, 2014).
Allen, M., et al. Raincloud plots: a multi-platform tool for robust data visualization [version 1; peer review: 2 approved]. Wellcome Open Res. 4, 63 (2019).
Acknowledgements
This research was funded by a Medical Research Foundation Equipment Competition grant (no. C0497; principal investigator O.J.R.) and a Medical Research Council Career Development Award to O.J.R. (no. MR/K024280/1). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
Author information
Authors and Affiliations
Contributions
O.J.R., J.A. and R.L.B. conceived and designed the study and acquired the data. O.J.R., J.A., V.V., J.P.R., P.D. and W.-Y.A. analysed and interpreted the data. W.-Y.A. and O.J.R. contributed to the creation of new software used in this work. All authors drafted the Article or substantively revised it and all authors approved the Article and are individually accountable for their own contributions.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Peer review information: Primary Handling Editor: Mary Elizabeth Sutherland.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Supplementary Information
Supplementary Methods 1, Supplementary Results 1 and 2, Supplementary Tables 1−3 and Supplementary References.
Supplementary Software
A PDF version of the script used to fit the models reported in this paper. For the downloadable and editable script, as well as a link to the required hBayesDM package for R, please see the link in the Code availability statement.
Rights and permissions
About this article
Cite this article
Aylward, J., Valton, V., Ahn, WY. et al. Altered learning under uncertainty in unmedicated mood and anxiety disorders. Nat Hum Behav 3, 1116–1123 (2019). https://doi.org/10.1038/s41562-019-0628-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41562-019-0628-0
This article is cited by
-
Anxiety associated with perceived uncontrollable stress enhances expectations of environmental volatility and impairs reward learning
Scientific Reports (2023)
-
Anterior cingulate and medial prefrontal cortex oscillations underlie learning alterations in trait anxiety in humans
Communications Biology (2023)
-
A neural and behavioral trade-off between value and uncertainty underlies exploratory decisions in normative anxiety
Molecular Psychiatry (2022)
-
Stress-induced avoidance in mood disorders
Nature Human Behaviour (2022)
-
Trait somatic anxiety is associated with reduced directed exploration and underestimation of uncertainty
Nature Human Behaviour (2022)