Anxiety is characterized by altered responses under uncertain conditions, but the precise mechanism by which uncertainty changes the behaviour of anxious individuals is unclear. Here we probe the computational basis of learning under uncertainty in healthy individuals and individuals suffering from a mix of mood and anxiety disorders. Participants were asked to choose between four competing slot machines with fluctuating reward and punishment outcomes during safety and stress. We predicted that anxious individuals under stress would learn faster about punishments and exhibit choices that were more affected by those punishments, thus formalizing our predictions as parameters in reinforcement learning accounts of behaviour. Overall, the data suggest that anxious individuals are quicker to update their behaviour in response to negative outcomes (increased punishment learning rates). When treating anxiety, it may therefore be more fruitful to encourage anxious individuals to integrate information over longer horizons when bad things happen, rather than try to blunt their responses to negative outcomes.
Access optionsAccess options
Subscribe to Journal
Get full journal access for 1 year
only $8.67 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
GBD Compare Data Visualization (IHME, accessed 17 November 2016).
LeDoux, J. E. & Pine, D. S. Using neuroscience to help understand fear and anxiety: a two-system framework. Am. J. Psychiat. 173, 1083–1093 (2016).
Grupe, D. W. & Nitschke, J. B. Uncertainty and anticipation in anxiety: an integrated neurobiological and psychological perspective. Nat. Rev. Neurosci. 14, 488–501 (2013).
Birrell, J., Meares, K., Wilkinson, A. & Freeston, M. Toward a definition of intolerance of uncertainty: a review of factor analytical studies of the Intolerance of Uncertainty Scale. Clin. Psychol. Rev. 31, 1198–1208 (2011).
Charpentier, C. J., Aylward, J., Roiser, J. P. & Robinson, O. J. Enhanced risk aversion, but not loss aversion, in unmedicated pathological anxiety. Biol. Psychiat. 81, 1014–1022 (2017).
Grillon, C. Models and mechanisms of anxiety: evidence from startle studies. Psychopharmacology 199, 421–437 (2008).
Robinson, O. J., Overstreet, C., Allen, P. S., Pine, D. S. & Grillon, C. Acute tryptophan depletion increases translational indices of anxiety but not fear: serotonergic modulation of the bed nucleus of the stria terminalis? Neuropsychopharmacology 37, 1963–1971 (2012).
Robinson, O. J. et al. The dorsal medial prefrontal (anterior cingulate) cortex–amygdala aversive amplification circuit in unmedicated generalised and social anxiety disorders: an observational study. Lancet Psychiat. 1, 294–302 (2014).
Daw, N. D., O’Doherty, J. P., Dayan, P., Seymour, B. & Dolan, R. J. Cortical substrates for exploratory decisions in humans. Nature 441, 876–879 (2006).
Seymour, B., Daw, N. D., Roiser, J. P., Dayan, P. & Dolan, R. Serotonin selectively modulates reward value in human decision-making. J. Neurosci. 32, 5833–5842 (2012).
Sharp, P. B. & Eldar, E. Computational models of anxiety: nascent efforts and future directions. Curr. Dir. Psychol. Sci. 28, 170–176 (2019).
Robinson, O. J., Vytal, K., Cornwell, B. R. & Grillon, C. The impact of anxiety upon cognition: perspectives from human threat of shock studies. Front. Human Neurosci. 7, 203 (2013).
Mkrtchian, A., Aylward, J., Dayan, P., Roiser, J. P. & Robinson, O. J. Modeling avoidance in mood and anxiety disorders using reinforcement learning. Biol. Psychiat. 82, 532–539 (2017).
Gagne, C., Dayan, P. & Bishop, S. J. When planning to survive goes wrong: predicting the future and replaying the past in anxiety and PTSD. Curr. Opin. Behav. Sci. 24, 89–95 (2018).
Monroe, S. M. & Simons, A. D. Diathesis-stress theories in the context of life stress research: implications for the depressive disorders. Psychol. Bull. 110, 406–425 (1991).
Robinson, O. J. Altered learning under uncertainty in unmedicated mood and anxiety disorders—EU storage. Preprint at OSF https://doi.org/10.17605/OSF.IO/UB6J7 (2018).
Gelman, A. & Rubin, D. B. Inference from iterative simulation using multiple sequences. Stat. Sci. 7, 457–472 (1992).
Bach, D. R. Anxiety-like behavioural inhibition is normative under environmental threat-reward correlations. PLoS Comput. Biol. 11, e1004646 (2015).
Robinson, O. J., Overstreet, C., Charney, D. S., Vytal, K. & Grillon, C. Stress increases aversive prediction-error signal in the ventral striatum. Proc. Natl Acad. Sci. USA 110, 4129–4133 (2013).
Deacon, B. J. & Abramowitz, J. S. Cognitive and behavioral treatments for anxiety disorders: a review of meta‐analytic findings. J. Clin. Psychol. 60, 429–441 (2004).
Wilson, A., Fern, A., Ray, S. & Tadepalli, P. Multi-task reinforcement learning: a hierarchical Bayesian approach. In Proc. 24th International Conference on Machine Learning 1015−1022 (ACM, 2007).
Browning, M., Behrens, T. E., Jocham, G., O’Reilly, J. X. & Bishop, S. J. Anxious individuals have difficulty learning the causal statistics of aversive environments. Nat. Neurosci. 18, 590–596 (2015).
Lissek, S., Pine, D. S. & Grillon, C. The strong situation: a potential impediment to studying the psychobiology and pharmacology of anxiety disorders. Biol. Psychol. 72, 265–270 (2006).
Robinson, O. J., Cools, R., Carlisi, C. O., Sahakian, B. J. & Drevets, W. C. Ventral striatum response during reward and punishment reversal learning in unmedicated major depressive disorder. Am. J. Psychiat. 169, 152–159 (2012).
Maxwell, S. E., Kelley, K. & Rausch, J. R. Sample size planning for statistical power and accuracy in parameter estimation. Annu. Rev. Psychol. 59, 537–563 (2008).
Sheehan, D. et al. The validity of the Mini International Neuropsychiatric Interview (MINI) according to the SCID-P and its reliability. Eur. Psychiat. 12, 232–241 (1997).
Cogent 2000 Team at the FIL and the ICN. Cogent http://www.vislab.ucl.ac.uk/cogent.php (2013).
Carlisi, C. O. & Robinson, O. J.. The role of prefrontal–subcortical circuitry in negative bias in anxiety: translational, developmental and treatment perspectives. Brain Neurosci. Adv. https://doi.org/10.1177/2398212818774223 (2018).
Mkrtchian, A., Roiser, J. P. & Robinson, O. J. Threat of shock and aversive inhibition: induced anxiety modulates Pavlovian-instrumental interactions. J. Exp. Psychol. Gen. 146, 1694–1704 (2017).
JASP Team. JASP (Version 0.7. 5.5) Google Sch. 765, 766 (2016).
Ahn, W.-Y., Haines, N. & Zhang, L. Revealing neurocomputational mechanisms of reinforcement learning and decision-making with the hBayesDM package. Comp. Psychiat. 1, 24–57 (2017).
Stan Development Team. RStan: the R interface to Stan. R package version 2.17.3, http://mc-stan.org/ (2018).
Ahn, W.-Y., Krawitz, A., Kim, W., Busemeyer, J. R. & Brown, J. W. A model-based fMRI analysis with hierarchical Bayesian parameter estimation. J. Neurosci. Psychol. Econ. 4, 95–110 (2011).
Guitart-Masip, M. et al. Go and no-go learning in reward and punishment: interactions between affect and effect. Neuroimage 62, 154–166 (2012).
Huys, Q. J., Pizzagalli, D. A., Bogdan, R. & Dayan, P. Mapping anhedonia onto reinforcement learning: a behavioural meta-analysis. Biol. Mood Anxiety Disord. 3, 12 (2013).
Niv, Y. et al. Reinforcement learning in multidimensional environments relies on attention mechanisms. J. Neurosci. 35, 8145–8157 (2015).
Ahn, W. Y., Busemeyer, J. R., Wagenmakers, E. J. & Stout, J. C. Comparison of decision learning models using the generalization criterion method. Cogn. Sci. 32, 1376–1402 (2008).
Ahn, W.-Y. et al. Decision-making in stimulant and opiate addicts in protracted abstinence: evidence from computational modeling with pure users. Front. Psychol. 5, 849 (2014).
Kruschke, J. Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan (Academic, 2014).
Allen, M., et al. Raincloud plots: a multi-platform tool for robust data visualization [version 1; peer review: 2 approved]. Wellcome Open Res. 4, 63 (2019).
This research was funded by a Medical Research Foundation Equipment Competition grant (no. C0497; principal investigator O.J.R.) and a Medical Research Council Career Development Award to O.J.R. (no. MR/K024280/1). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
The authors declare no competing interests.
Peer review information: Primary Handling Editor: Mary Elizabeth Sutherland.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Methods 1, Supplementary Results 1 and 2, Supplementary Tables 1−3 and Supplementary References.
A PDF version of the script used to fit the models reported in this paper. For the downloadable and editable script, as well as a link to the required hBayesDM package for R, please see the link in the Code availability statement.