Abstract
Exploration reduces uncertainty about the environment and improves the quality of future decisions, but at the cost of provisional uncertain and suboptimal outcomes. Although anxiety promotes intolerance to uncertainty, it remains unclear whether and by which mechanisms anxiety relates to exploratory decision-making. We use a dynamic three-armed-bandit task and find that higher trait-anxiety is associated with increased exploration, which in turn harms overall performance. We identify two distinct behavioral sources: first, decisions made by anxious individuals are guided toward reduction of uncertainty; and second, decisions are less guided by immediate value gains. These findings are similar in both loss and gain domains, and further demonstrate that an affective trait relates to exploration and results in an inverse-U-shaped relationship between anxiety and overall performance. Additional imaging data (fMRI) suggests that normative anxiety correlates negatively with the representation of expected-value in the dorsal-anterior-cingulate-cortex, and in contrast, positively with the representation of uncertainty in the anterior-insula. We conclude that a trade-off between value-gains and uncertainty-reduction entails maladaptive decision-making in individuals with higher normal-range anxiety.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$259.00 per year
only $21.58 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The data used for data analyses can be provided following reasonable request.
Code availability
The code used for data analyses can be provided following reasonable request.
References
Cohen JD, McClure SM, Yu AJ. Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philos Trans R Soc Lond Ser B, Biol Sci. 2007;362:933–42.
Addicott MA, Pearson JM, Sweitzer MM, Barack DL, Platt ML. A Primer on Foraging and the Explore/Exploit Trade-Off for Psychiatry Research. Neuropsychopharmacol : Off Publ Am Coll Neuropsychopharmacol. 2017;42:1931–9.
Mehlhorn K, Newell BR, Todd PM, Lee MD, Morgan K, Braithwaite VA, et al. Unpacking the exploration-exploitation tradeoff: a synthesis of human and animal literatures. Decision. 2015;2:191–215.
Scholl J, Klein-Flugge M. Understanding psychiatric disorder by capturing ecologically relevant features of learning and decision-making. Behav Brain Res. 2018;355:56–75.
Hartley CA, Phelps EA. Anxiety and decision-making. Biol Psychiatry. 2012;72:113–8.
Buhr K, Dugas MJ. The role of fear of anxiety and intolerance of uncertainty in worry: an experimental manipulation. Behav Res Ther. 2009;47:215–23.
Grupe DW, Nitschke JB. Uncertainty and anticipation in anxiety: an integrated neurobiological and psychological perspective. Nat Rev Neurosci. 2013;14:488–501.
Maner JK, Schmidt NB. The role of risk avoidance in anxiety. Behav Ther. 2006;37:181–9.
Maner JK, Richey JA, Cromer K, Mallott M, Lejuez CW, Joiner TE, et al. Dispositional anxiety and risk-avoidant decision-making. Pers Indiv Differ. 2007;42:665–75.
Charpentier CJ, Aylward J, Roiser JP, Robinson OJ. Enhanced Risk Aversion, But Not Loss Aversion, in Unmedicated Pathological Anxiety. Biol Psychiatry. 2017;81:1014–22.
Frank MJ, Doll BB, Oas-Terpstra J, Moreno F. Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat Neurosci. 2009;12:1062–8.
Gershman SJ. Deconstructing the human algorithms for exploration. Cognition. 2018;173:34–42.
Gershman SJ, Tzovaras BG. Dopaminergic genes are associated with both directed and random exploration. Neuropsychologia. 2018;120:97–104.
Wilson RC, Geana A, White JM, Ludvig EA, Cohen JD. Humans Use Directed and Random Exploration to Solve the Explore-Exploit Dilemma. J Exp Psychol Gen. 2014;143:2074–81.
Badre D, Doll BB, Long NM, Frank MJ. Rostrolateral prefrontal cortex and individual differences in uncertainty-driven exploration. Neuron. 2012;73:595–607.
Cavanagh JF, Figueroa CM, Cohen MX, Frank MJ. Frontal theta reflects uncertainty and unexpectedness during exploration and exploitation. Cereb Cortex. 2012;22:2575–86.
Kayser AS, Mitchell JM, Weinstein D, Frank MJ. Dopamine, locus of control, and the exploration-exploitation tradeoff. Neuropsychopharmacol : Off Publ Am Coll Neuropsychopharmacol. 2015;40:454–62.
Tomov MS, Truong V, Hundia RA, Gershman SJ. Dissociable neural correlates of uncertainty underlie different exploration strategies. Nat Commun. 2020;11:1–12.
Daw ND, O'Doherty JP, Dayan P, Seymour B, Dolan RJ. Cortical substrates for exploratory decisions in humans. Nature. 2006;441:876–9.
Laureiro-Martinez D, Brusoni S, Canessa N, Zollo M. Understanding the exploration-exploitation dilemma: an fmri study of attention control and decision-making performance. Strategic Manag J. 2015;36:319–38.
Blanchard TC, Gershman SJ. Pure correlates of exploration and exploitation in the human brain. Cogn Affect Behav Neurosci. 2018;18:117–26.
Chakroun K, Mathar D, Wiehler A, Ganzer F, Peters J. Dopaminergic modulation of the exploration/exploitation trade-off in human decision-making. Elife. 2020;9:1–44.
Paulus MP, Stein MB. An insular view of anxiety. Biol Psychiatry. 2006;60:383–7.
Etkin A, Wager TD. Functional neuroimaging of anxiety: a meta-analysis of emotional processing in PTSD, social anxiety disorder, and specific phobia. Am J Psychiatry. 2007;164:1476–88.
Bishop SJ. Neurocognitive mechanisms of anxiety: an integrative account. Trends Cogn Sci. 2007;11:307–16.
Aylward J, Valton V, Ahn WY, Bond RL, Dayan P, Roiser JP, et al. Altered learning under uncertainty in unmedicated mood and anxiety disorders. Nat Hum Behav. 2019;3:1116–23.
Browning M, Behrens TE, Jocham G, O'Reilly JX, Bishop SJ. Anxious individuals have difficulty learning the causal statistics of aversive environments. Nat Neurosci. 2015;18:590. +
Lamba A, Frank MJ, FeldmanHall O. Anxiety Impedes Adaptive Social Learning Under Uncertainty. Psychol Sci. 2020;31:592–603.
Bijsterbosch J, Smith S, Bishop SJ. Functional Connectivity under Anticipation of Shock: Correlates of Trait Anxious Affect versus Induced Anxiety. J Cogn Neurosci. 2015;27:1840–53.
Xu P, Gu R, Broster LS, Wu R, Van Dam NT, Jiang Y, et al. Neural basis of emotional decision making in trait anxiety. J Neurosci. 2013;33:18641–53.
Fung BJ, Qi S, Hassabis D, Daw N, Mobbs D. Slow escape decisions are swayed by trait anxiety. Nat Hum Behav. 2019;3:702–8.
Spielberger CD, Gorsuch RL, Lushene R, Vagg PR, Jacobs GA. Manual for the State-Trait Anxiety Inventory. Palo Alto, CA: Consulting Psychologists Press; 1983.
De Martino B, Kumaran D, Seymour B, Dolan RJ. Frames, biases, and rational decision-making in the human brain. Science. 2006;313:684–7.
Jepma M, Lopez-Sola M. Anxiety and framing effects on decision making: insights from neuroimaging. J Neurosci. 2014;34:3455–6.
Tzovara A, Murray MM, Bourdaud N, Chavarriaga R, Millan Jdel R, De, et al. The timing of exploratory decision-making revealed by single-trial topographic EEGanalyses. NeuroImage. 2012;60:1959–69.
Gershman SJ, Pesaran B, Daw ND. Human reinforcement learning subdivides structured action spaces by learning effector-specific values. J Neurosci. 2009;29:13524–31.
Bishop CM. Pattern recognition and machine learning. New York: Springer; 2006.
Stephan KE, Penny WD, Daunizeau J, Moran RJ, Friston KJ. Bayesian model selection for group studies. NeuroImage. 2009;46:1004–17.
Wilson RC, Collins AG. Ten simple rules for the computational modeling of behavioral data. Elife. 2019;8:1–33.
Hayes AF, Rockwood NJ. Regression-based statistical mediation and moderation analysis in clinical research: Observations, recommendations, and implementation. Behav Res Ther. 2017;98:39–57.
Maldjian JA, Laurienti PJ, Kraft RA, Burdette JH. An automated method for neuroanatomic and cytoarchitectonic atlas-based interrogation of fMRI data sets. NeuroImage. 2003;19:1233–9.
Cascio CN, Konrath SH, Falk EB. Narcissists' social pain seen only in the brain. Soc Cogn Affect Neurosci. 2015;10:335–41.
Wang F, Peng K, Bai Y, Li R, Zhu Y, Sun P, et al. The Dorsal Anterior Cingulate Cortex Modulates Dialectical Self-Thinking. Front Psychol. 2016;7:152.
Eickhoff SB, Stephan KE, Mohlberg H, Grefkes C, Fink GR, Amunts K, et al. A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. NeuroImage. 2005;25:1325–35.
Bludau S, Eickhoff SB, Mohlberg H, Caspers S, Laird AR, Fox PT, et al. Cytoarchitecture, probability maps and functions of the human frontal pole. NeuroImage. 2014;93:260–75.
Levy DJ, Glimcher PW. The root of all value: a neural common currency for choice. Curr Opin Neurobiol. 2012;22:1027–38.
Aberg KC, Doell KC, Schwartz S. The left hemisphere learns what is right: Hemispatial reward learning depends on reinforcement learning processes in the contralateral hemisphere. Neuropsychologia. 2016;89:1–13.
Cohen MX. Individual differences and the neural representations of reward expectation and reward prediction error. Soc Cogn Affect Neurosci. 2007;2:20–30.
Baliki MN, Mansour A, Baria AT, Huang L, Berger SE, Fields HL, et al. Parceling human accumbens into putative core and shell dissociates encoding of values for reward and pain. J Neurosci. 2013;33:16383–93.
Aberg KC, Kramer EE, Schwartz S. Neurocomputational correlates of learned irrelevance in humans. NeuroImage. 2020;213:116719.
Vul E, Harris C, Winkielman P, Pashler H. Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition. Perspect Psychological Sci : J Assoc Psychological Sci. 2009;4:274–90.
Aston-Jones G, Cohen JD. An integrative theory of locus coeruleus-norepinephrine function: Adaptive gain and optimal performance. Annu Rev Neurosci. 2005;28:403–50.
Aberg KC, Muller J, Schwartz S. Trial-by-Trial Modulation of Associative Memory Formation by Reward Prediction Error and Reward Anticipationas Revealed by a Biologically Plausible Computational Model. Front Human Neurosci. 2017;11:1–15.
Burnham KP, Anderson DR. Multimodel inference—understanding AIC and BIC in model selection. Socio Method Res. 2004;33:261–304.
Palminteri S, Wyart V, Koechlin E. The Importance of Falsification in Computational Cognitive Modeling. Trends Cogn Sci. 2017;21:425–33.
Domenech P, Koechlin E. Executive control and decision-making in the prefrontal cortex. Curr Opin Behav Sci. 2015;1:101–6.
Rauch SL, Shin LM, Wright CI. Neuroimaging studies of amygdala function in anxiety disorders. Ann N. Y Acad Sci. 2003;985:389–410.
Shackman AJ, Fox AS. Contributions of the Central Extended Amygdala to Fear and Anxiety. J Neurosci. 2016;36:8050–63.
Costa VD, Mitz AR, Averbeck BB. Subcortical Substrates of Explore-Exploit Decisions in Primates. Neuron. 2019;103:533–45. e535
Knox WB, Otto AR, Stone P, Love BC. The nature of belief-directed exploratory choice in human decision-making. Front Psychol. 2011;2:398.
Gagne C, Zika O, Dayan P, Bishop SJ. Impaired adaptation of learning to contingency volatility in internalizing psychopathology. Elife. 2020;9:1–51.
Pulcu E, Browning M. The Misestimation of Uncertainty in Affective Disorders. Trends Cogn Sci. 2019;23:865–75.
Payzan-LeNestour E, Bossaerts P. Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Comput Biol. 2011;7:e1001048.
Collins AGE, Frank MJ. How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. Eur J Neurosci. 2012;35:1024–35.
Dar KA, Iqbal N, Mushtaq A. Intolerance of uncertainty, depression, and anxiety: examining the indirect and moderating effects of worry. Asian J Psychiatry. 2017;29:129–33.
Carleton RN. Into the unknown: a review and synthesis of contemporary models involving uncertainty. J Anxiety Disord. 2016;39:30–43.
Einstein DA. Extension of the Transdiagnostic Model to Focus on Intolerance of Uncertainty: a Review of the Literature and Implications for Treatment. Clin Psychol : Publ Div Clin Psychol Am Psychological Assoc. 2014;21:280–300.
Pepperdine E, Lomax C, Freeston MH. Disentangling intolerance of uncertainty and threat appraisal in everyday situations. J Anxiety Disord. 2018;57:31–8.
Oglesby ME, Allan NP, Schmidt NB. Randomized control trial investigating the efficacy of a computer-based intolerance of uncertainty intervention. Behav Res Ther. 2017;95:50–7.
Kolling N, Behrens TE, Mars RB, Rushworth MF. Neural mechanisms of foraging. Science. 2012;336:95–8.
Shenhav A, Straccia MA, Cohen JD, Botvinick MM. Anterior cingulate engagement in a foraging context reflects choice difficulty, not foraging value. Nat Neurosci. 2014;17:1249–54.
Hayden BY, Pearson JM, Platt ML. Neuronal basis of sequential foraging decisions in a patchy environment. Nat Neurosci. 2011;14:933–9.
Charnov EL. Optimal foraging, the marginal value theorem. Theor Popul Biol. 1976;9:129–36.
Findling C, Skvortsova V, Dromnelle R, Palminteri S, Wyart V. Computational noise in reward-guided learning drives behavioral variability in volatile environments. Nat Neurosci. 2019;22:2066. +
Simmons A, Matthews SC, Paulus MP, Stein MB. Intolerance of uncertainty correlates with insula activation during affective ambiguity. Neurosci Lett. 2008;430:92–97.
Gogolla N. The insular cortex. Curr Biol. 2017;27:R580–R586.
Alvarez RP, Kirlic N, Misaki M, Bodurka J, Rhudy JL, Paulus MP, et al. Increased anterior insula activity in anxious individuals is linked to diminished perceived control. Transl Psychiatry. 2015;5:e591.
Ploghaus A, Tracey I, Gati JS, Clare S, Menon RS, Matthews PM, et al. Dissociating pain from its anticipation in the human brain. Science. 1999;284:1979–81.
Wright P, He G, Shapira NA, Goodman WK, Liu Y. Disgust and the insula: fMRI responses to pictures of mutilation and contamination. Neuroreport. 2004;15:2347–51.
Dunsmoor JE, Paz R. Fear generalization and anxiety: behavioral and neural mechanisms. Biol Psychiatry. 2015;78:336–43.
Lissek S. Toward an Account of Clinical Anxiety Predicated on Basic, Neurally Mapped Mechanisms of Pavlovian Fear-Learning: The Case for Conditioned Overgeneralization. Depression Anxiety. 2012;29:257–63.
Laufer O, Paz R. Monetary loss alters perceptual thresholds and compromises future decisions via amygdala and prefrontal networks. J Neurosci. 2012;32:6304–11.
Shalev L, Paz R, Avidan G. Visual Aversive Learning Compromises Sensory Discrimination. J Neurosci. 2018;38:2766–79.
Resnik J, Sobel N, Paz R. Auditory aversive learning increases discrimination thresholds. Nat Neurosci. 2011;14:791–6.
Laufer O, Israeli D, Paz R. Behavioral and Neural Mechanisms of Overgeneralization in Anxiety. Curr Biol. 2016;26:713–22.
Eysenck MW, Derakshan N, Santos R, Calvo MG. Anxiety and cognitive performance: attentional control theory. Emotion. 2007;7:336–53.
Bar-Haim Y, Lamy D, Pergamin L, Bakermans-Kranenburg MJ, van IMH. Threat-related attentional bias in anxious and nonanxious individuals: a meta-analytic study. Psychological Bull. 2007;133:1–24.
Schonbrodt FD, Perugini M. At what sample size do correlations stabilize? J Res Pers. 2013;47:609–12.
Open Science C. PSYCHOLOGY. Estimating the reproducibility of psychological science. Science. 2015;349:aac4716.
Grady CL, Rieck JR, Nichol D, Rodrigue KM, Kennedy KM. Influence of sample size and analytic approach on stability and interpretation of brain-behavior correlations in task-relatedfMRIdata. Hum Brain Mapp. 2021;42:204–19.
Coricelli G, Rustichini A. Counterfactual thinking and emotions: regret and envy learning. Philos T R Soc B. 2010;365:241–7.
Bandura A. Self-Efficacy Mechanism in Human Agency. Am Psychol. 1982;37:122–47.
Roese NJ, Epstude K, Fessel F, Morrison M, Smallman R, Summerville A, et al. Repetitive Regret, Depression, and Anxiety: Findings from a Nationally Representative Survey. J Soc Clin Psychol. 2009;28:671–88.
Ehring T, Watkins ER. Repetitive Negative Thinking as a Transdiagnostic Process. Int J Cogn Ther. 2008;1:192–205.
Muris P. Relationships between self-efficacy and symptoms of anxiety disorders and depression in a normal adolescent sample. Pers Indiv Differ. 2002;32:337–48.
Zumberg KM, Chang C, Sanna LJ. Does problem orientation involve more than generalized self-efficacy? Predicting psychological and physical functioning in college students. Pers Indiv Differ. 2008;45:328–32.
Luszczynska A, Gutierrez-Dona B, Schwarzer R. General self-efficacy in various domains of human functioning: Evidence from five countries. Int J Psychol. 2005;40:80–9.
Chambers JA, Power KG, Durham RC. The relationship between trait vulnerability and anxiety and depressive diagnoses at long-term follow-up of Generalized Anxiety Disorder. J Anxiety Disord. 2004;18:587–607.
Weger M, Sandi C. High anxiety trait: A vulnerable phenotype for stress-induced depression. Neurosci Biobehav Rev. 2018;87:27–37.
Mineka S, Oehlberg K. The relevance of recent developments in classical conditioning to understanding the etiology and maintenance of anxiety disorders. Acta Psychol. 2008;127:567–80.
Paulus MP, Yu AJ. Emotion and decision-making: affect-driven belief systems in anxiety and depression. Trends Cogn Sci. 2012;16:476–83.
Lee D. Decision making: from neuroscience to psychiatry. Neuron. 2013;78:233–48.
Rosen JB, Schulkin J. From normal fear to pathological anxiety. Psychological Rev. 1998;105:325–50.
Grupe DW. Decision-Making in Anxiety and Its Disorders. In: Dreher JC, Tremblay L, (eds). Decision Neuroscience - An Integrative Approach. Academic Press; 2017. p. 327–38.
Balsamo M, Romanelli R, Innamorati M, Ciccarese G, Carlucci L, Saggino A. The State-Trait Anxiety Inventory: shadows and Lights on its Construct Validity. J Psychopathol Behav. 2013;35:475–86.
Kotov R, Gamez W, Schmidt F, Watson D. Linking "Big" Personality Traits to Anxiety, Depressive, and Substance Use Disorders: A Meta-Analysis. Psychological Bull. 2010;136:768–821.
Stanton K, Watson D. Positive and Negative Affective Dysfunction in Psychopathology. Soc Personal Psychol Compass. 2014;8:555–67.
Clark LA, Watson D, Mineka S. Temperament, Personality, and the Mood and Anxiety Disorders. J Abnorm Psychol. 1994;103:103–16.
Acknowledgements
We thank Dr. Edna Furman-Haran and Fanny Attar for MRI procedures. KCA is incumbent of The Sam and Frances Belzberg Research Fellow Chair in Memory and Learning, and was also supported by a Swiss Society of Friends of the Weizmann Institute Postdoctoral Fellowship grant. The work was supported by a Joy-Ventures grant, an ISF #2352/19, and an ERC-2016-CoG #724910 grants to RP.
Author information
Authors and Affiliations
Contributions
KCA and RP designed the study. KCA collected the data. KCA and IT analyzed the data. KCA, IT, and RP wrote and edited the paper.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
About this article
Cite this article
Aberg, K.C., Toren, I. & Paz, R. A neural and behavioral trade-off between value and uncertainty underlies exploratory decisions in normative anxiety. Mol Psychiatry 27, 1573–1587 (2022). https://doi.org/10.1038/s41380-021-01363-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41380-021-01363-z
This article is cited by
-
Heterogeneity in strategy use during arbitration between experiential and observational learning
Nature Communications (2024)
-
Value-free random exploration is linked to impulsivity
Nature Communications (2022)
-
Stress-induced avoidance in mood disorders
Nature Human Behaviour (2022)
-
Trait somatic anxiety is associated with reduced directed exploration and underestimation of uncertainty
Nature Human Behaviour (2022)