Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Anxious individuals have difficulty learning the causal statistics of aversive environments

Nature Neuroscience volume 18, pages 590–596 (2015)Cite this article

Subjects

Abstract

Statistical regularities in the causal structure of the environment enable us to predict the probable outcomes of our actions. Environments differ in the extent to which action-outcome contingencies are stable or volatile. Difficulty in being able to use this information to optimally update outcome predictions might contribute to the decision-making difficulties seen in anxiety. We tested this using an aversive learning task manipulating environmental volatility. Human participants low in trait anxiety matched updating of their outcome predictions to the volatility of the current environment, as predicted by a Bayesian model. Individuals with high trait anxiety showed less ability to adjust updating of outcome expectancies between stable and volatile environments. This was linked to reduced sensitivity of the pupil dilatory response to volatility, potentially indicative of altered norepinephrinergic responsivity to changes in this aspect of environmental information.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Task structure.
Figure 2: Estimates of participants' learning rates.
Figure 3: Post-outcome pupil dilation tracks both environmental volatility and outcome surprise.
Figure 4: The relationship between trait anxiety and post-outcome pupil dilation as a function of trial-wise estimates of volatility and surprise.

Similar content being viewed by others

References

  1. Butler, G. & Mathews, A. Cognitive processes in anxiety. Adv. Behav. Res. Ther. 5, 51–62 (1983).

    Article  Google Scholar 

  2. Hartley, C.A. & Phelps, E.A. Anxiety and decision-making. Biol. Psychiatry 72, 113–118 (2012).

    Article  Google Scholar 

  3. Paulus, M.P. & Yu, A.J. Emotion and decision-making: affect-driven belief systems in anxiety and depression. Trends Cogn. Sci. 16, 476–483 (2012).

    Article  Google Scholar 

  4. Birrell, J., Meares, K., Wilkinson, A. & Freeston, M. Toward a definition of intolerance of uncertainty: a review of factor analytical studies of the Intolerance of Uncertainty Scale. Clin. Psychol. Rev. 31, 1198–1208 (2011).

    Article  Google Scholar 

  5. Carleton, R.N., Norton, M.A. & Asmundson, G.J. Fearing the unknown: a short version of the Intolerance of Uncertainty Scale. J. Anxiety Disord. 21, 105–117 (2007).

    Article  Google Scholar 

  6. Yu, A.J. & Dayan, P. Uncertainty, neuromodulation, and attention. Neuron 46, 681–692 (2005).

    Article  CAS  Google Scholar 

  7. Payzan-LeNestour, E., Dunne, S., Bossaerts, P. & O'Doherty, J.P. The neural representation of unexpected uncertainty during value-based decision making. Neuron 79, 191–201 (2013).

    Article  CAS  Google Scholar 

  8. Behrens, T.E., Woolrich, M.W., Walton, M.E. & Rushworth, M.F. Learning the value of information in an uncertain world. Nat. Neurosci. 10, 1214–1221 (2007).

    Article  CAS  Google Scholar 

  9. Sutton, R. & Barto, A.G. Reinforcement Learning (MIT Press, 1998).

  10. Rescorla, R.A. & Wagner, A.R. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. in Classical Conditioning II: Current Research and Theory (eds. A.H. Black & W.F. Prokasy) 64–99 (Appleton-Centuary-Crofts, 1972).

  11. Behrens, T.E., Hunt, L.T., Woolrich, M.W. & Rushworth, M.F. Associative learning of social value. Nature 456, 245–249 (2008).

    Article  CAS  Google Scholar 

  12. Indovina, I., Robbins, T.W., Nunez-Elizalde, A.O., Dunn, B.D. & Bishop, S.J. Fear-conditioning mechanisms associated with trait vulnerability to anxiety in humans. Neuron 69, 563–571 (2011).

    Article  CAS  Google Scholar 

  13. Preuschoff, K., 't Hart, B.M. & Einhauser, W. Pupil dilation signals surprise: evidence for noradrenaline's role in decision making. Front. Neurosci. 5, 115 (2011).

    Article  Google Scholar 

  14. Nassar, M.R. et al. Rational regulation of learning dynamics by pupil-linked arousal systems. Nat. Neurosci. 15, 1040–1046 (2012).

    Article  CAS  Google Scholar 

  15. Eldar, E., Cohen, J.D. & Niv, Y. The effects of neural gain on attention and learning. Nat. Neurosci. 16, 1146–1153 (2013).

    Article  CAS  Google Scholar 

  16. Gilzenrat, M.S., Nieuwenhuis, S., Jepma, M. & Cohen, J.D. Pupil diameter tracks changes in control state predicted by the adaptive gain theory of locus coeruleus function. Cogn. Affect. Behav. Neurosci. 10, 252–269 (2010).

    Article  Google Scholar 

  17. Aston-Jones, G. & Cohen, J.D. An integrative theory of locus coeruleus–norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 28, 403–450 (2005).

    Article  CAS  Google Scholar 

  18. Gorman, J.M., Hirschfeld, R.M. & Ninan, P.T. New developments in the neurobiological basis of anxiety disorders. Psychopharmacol. Bull. 36 (suppl. 2), 49–67 (2002).

    PubMed  Google Scholar 

  19. Charney, D.S. & Drevets, W.C. Neurobiological basis of anxiety disorders. in Neuropsychopharmacology: the Fifth Generation of Progress (eds. Davis, K.L., Charney, D.S., Coyle, J.T. & Nemeroff, C.) 901–930 (American College of Neuropsychopharmacology, 2002).

  20. Spielberger, C.D., Gorsuch, R.L. & Lushene, R.D. Manual for the State-Trait Anxiety Inventory (STAI) (Consulting Psychologists Press, 1983).

  21. Bestmann, S. et al. Influence of uncertainty and surprise on human corticospinal excitability during preparation for action. Curr. Biol. 18, 775–780 (2008).

    Article  CAS  Google Scholar 

  22. MacKay, D.J.C. Information Theory, Inference and Learning Algorithms (Cambridge University Press, 2003).

  23. O'Reilly, J.X. et al. Dissociable effects of surprise and model update in parietal and anterior cingulate cortex. Proc. Natl. Acad. Sci. USA 110, 3660–3669 (2013).

    Article  Google Scholar 

  24. Casey, B.J. et al. DSM-5 and RDoC: progress in psychiatry research? Nat. Rev. Neurosci. 14, 810–814 (2013).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

The authors would like to thank S. Hicks and C. Kennard for use of the eye-tracking system. This research was supported by European Research Council grant GA 260932 and US National Institutes of Health grant R01MH091848.

Author information

Authors and Affiliations

  1. Nuffield Department of Clinical Neurosciences, Functional MRI of the Brain Centre, John Radcliffe Hospital, Oxford, UK

    Michael Browning, Timothy E Behrens, Gerhard Jocham, Jill X O'Reilly & Sonia J Bishop

  2. Center for Behavioral Brain Sciences, University of Magdeburg, Magdeburg, Germany

    Gerhard Jocham

  3. Department of Psychology and Helen Wills Neuroscience Institute, University of California, Berkeley, California, USA

    Sonia J Bishop

Authors
  1. Michael Browning
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Timothy E Behrens
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gerhard Jocham
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jill X O'Reilly
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sonia J Bishop
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

M.B. and S.J.B. wrote the manuscript. M.B., S.J.B. and T.E.B. designed the task. M.B. collected the data. All of the authors contributed to data analysis. T.E.B. developed the Bayesian model. All of the authors commented on the manuscript.

Corresponding author

Correspondence to Sonia J Bishop.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Integrated supplementary information

Supplementary Figure 1 Calibration of Electrical Shocks.

Before each participant completed the aversive learning task, subjective pain scores (left hand axis) were collected for different objective strengths of electrical shock (x axis), see online methods. A 10 point scale was used for subjective pain where 1=minimal pain, 10=worst possible pain. No shock that exceeded a subjective level of 7 was applied. A sigmoid curve fitted to this data was used to equate the subjective level of pain experienced during the learning task across participants. The graph shows data points (circles) and the fitted sigmoid (black line) from one participant. In the task, the shock magnitude associated with each shape was varied on a trial-to-trial basis from 1-99, where 1 equated to a subjective pain level of 1, and 99 to a subjective pain level of 7. Magnitude values between these extremes were calculated using the sigmoidal fit. This is illustrated here by the dashed line, which shows how the objective strength of the electrical stimulus (x axis) corresponding to a desired ‘magnitude’ of 80 (right hand axis) would be calculated for this participant.

Supplementary Figure 2 Effects of volatility upon participants’ learning rates in a structurally equivalent reward task.

In addition to the aversive learning task, reported within the main manuscript, participants also completed a structurally equivalent reward learning task. Here the two shapes chosen between had different probabilities of leading to winning points, with the magnitude of reward points being unique for each shape and varying randomly between 1 and 99 across trials. Following the task the points participants had won were converted into a monetary reward (either £5 or £10). (a) Participants’ choices during the stable and volatile blocks of the reward task were fitted with the same Rescorla Wagner learning model used in the aversive task. Estimates of individual participants’ learning rates are displayed (circles), using a logarithmic scale, separately for the stable and volatile blocks for the two task schedules (Schedule 1 = stable task block first, n = 16, Schedule 2 = volatile task block first, n = 15). Black lines display mean (+–SEM) of participants’ estimated learning rates, grey dotted lines link the learning rates in volatile and stable blocks for each participant. Participants showed higher learning rates in the volatile versus stable blocks regardless of the order in which they were completed, F(1,29) = 15.3, p = 0.001, replicating previous results8. (b) The relative log learning rate for the volatile versus the stable blocks (i.e. log(LR in volatile block) – log(LR in stable block)) was not significantly correlated with participant trait anxiety, r(29) = –0.16, p = 0.4. It should be noted that the difference in the effect of anxiety on learning rate for this reward learning task and the aversive learning task reported in the main manuscript was also not significant, F(1,28) = 0.57, p = 0.46. This hence limits the conclusions that can be drawn regarding the specificity of the anxiety-related deficit in adjusting learning rate to cases where outcomes are aversive as opposed to reward-related. Error bars represent the standard deviation of the estimated parameters from the behavioral model for each participant.

Supplementary Figure 3 Influence of potential shock magnitude and prior outcome history on actual and simulated behavior choice as a function of block volatility (a,b) and participant anxiety (c-f).

Regression analyses were conducted on a participant by participant basis to examine the extent to which current shock magnitude linked to each shape and the outcomes of the five previous trials predicted choice of shape A versus shape B (e.g. circle versus square) on a given trial. The dependent variable was participant choice (a,c,d) or model choice (b,e,f) on trial ‘n’ (coded as 1 if the subject/model chose shape A (e.g. circle) and 0 if shape B (e.g. square)). For the model choice regressions, the coupled Rescorla Wagner (RW) predictor and sofmax action selector learning model was used to simulate choice behavior on a trial to trial basis, using the blockwise estimates of the free parameters (for learning rate, decision temperature and inverse temperature) previously calculated for each participant (see online methods). In each regression analysis, predictor variables comprised shock magnitude for shape A, shock magnitude for shape B, and outcome for trial n-1 to n-5 (coded as 1 if shape A was associated with the shock). As the distribution of parameter estimates across participants was non-normal, median values (horizontal black lines) and interquartile ranges (boxes) are presented. (a) Influence of the predictor variables on subject choice, estimated separately for trials in the stable (blue boxes) and volatile (red boxes) blocks. The impact of previous outcomes decreases across time and - consistent with the higher learning rate used by participants in volatile blocks - the impact of prior outcomes on shape choice is greater for more recent trial outcomes (n-1, n-2) in the volatile versus the stable block. (b) A parallel analysis was run on the model-derived choices. The similar pattern in (b) as for (a) is as expected given the good fit of the RW & softmax action selector model to the behavior data (see Supplementary Modeling Note). Plots c and d illustrate how the influence of prior outcomes on behavioral choice as a function of block volatility differs between low and high trait anxious individuals (as defined using a median split, for ease of graphical illustration). As expected given the inverse relationship between trait anxiety and change in learning rate between stable and volatile blocks (Fig 2b), a greater difference between blocks in the effect of recent prior outcomes on choice behavior is observed for low (c) than for high (d) anxious participants. Plots e and f illustrate that a parallel pattern is observed when using simulated data from the RW and softmax action selector model. In summary, panels a, c and d illustrate, in a model-free manner, the key behavior of participants assessed by the learning rate analyses reported within the main manuscript. Additionally, the plots shown in panels b, e and f indicate that the coupled RW predictor and softmax action selector is able to reliably capture these behaviors.

Supplementary Figure 4 Relationship between trait anxiety and Bayesian volatility when multiple parameters, including a decay function, are allowed to compete for influence over a dynamic learning rate.

As described in the Supplementary Modeling Note, a more complex model, designed to capture a variety of potential influences on learning rate, was additionally fitted to choice data at the level of the individual participant. On each trial, learning rate was estimated as a weighted sum of three terms: the mean learning rate across trials (αmean), the trial-wise demeaned Bayesian volatility (subject specific weight =ωBayes) and the trial-wise demeaned exponential decay function (subject specific weight =ωExp). These parameters along with the slope of the exponential function(λ) and the decision temperature and risk preference terms from the selector model were fitted to participant behavior across the entire task. The resulting parameter estimates were then correlated, across participants, against trait anxiety. Of the six parameters fitted to participants’ behavior, only the weight of the volatility term (ωBayes) was significantly correlated with trait anxiety, illustrated above r(28) = –0.37, p = 0.04, with none of the other parameters showing a significant relationship with trait anxiety, ps>0.2. Consistent with the findings of the main paper, this result indicates that, even when a variety of factors were allowed to influence a dynamic learning rate, trait anxiety was uniquely associated with reduced influence of environmental volatility upon learning rate. Error bars represent the standard deviation of the parameter estimates for each subject.

Supplementary Figure 5 Shock receipt on difficult trials (where the two options were close in expected value) was positively correlated with trait anxiety (a), and negatively correlated with both change in learning rate between stable and volatile blocks (b) and modulation of post-outcome pupil dilation by trial volatility (c).

The advantage of flexible adaptation of learning rate to changes in environmental volatility is that it allows learning to occur at an optimal rate for a given environment. As subjects with higher levels of trait anxiety were less able to adjust their learning between stable and volatile blocks, they would be predicted to more often incorrectly judge the action most likely to result in shock. This in turn would be most likely to have an observable effect on trials where the two options were close in expected value. Following Behrens et al8, we defined ‘difficult trials’ as those in which the Pascalian value of the two shapes differed by less than 5. (a) Trait anxiety was positively correlated with the proportion of ‘difficult’ trials on which shock was received. (b) We further confirmed that this ‘proportion of difficult trials where shock was received’ index of performance was negatively related to adjustment of learning rate between stable and volatile task blocks, r(28) = –0.31, p = 0.05, 1-tailed (b), as well as to the pupil response to trial volatility, r(26) = –0.41, p = 0.01, 1-tailed (c).

Supplementary Figure 6 No significant association was observed between Trait Anxiety and Risk Preference or Inverse Decision Temperature.

The softmax action selector contained two free parameters: risk preference (the tendency to minimize shock probability versus shock magnitude) and inverse decision temperature (the degree to which a participant used expected values, i.e. estimated shock probability x shock magnitude, for each shape to guide choices). These parameters were estimated from participants’ choices in the stable and volatile blocks of the aversive learning task (see Online Methods). Panels a-d show the relationship between the log transformed parameter estimates and trait anxiety. There was no significant relationship between trait anxiety and risk preference as calculated for either the stable (a; r(28) = –0.23, p = 0.2) or volatile (b; r(28) = –0.09, p = 0.6) task blocks. Trait anxiety was also not associated with the mean value of this parameter across blocks or with the difference score between blocks (ps >.1). There was also no relationship between trait anxiety and inverse decision temperature as estimated for the stable (c; r(28) < 0.01, p = 0.99) or volatile (d; r(28) = 0.24, p = 0.2) blocks. Trait anxiety was also not associated with mean decision temperature across blocks, or with the difference in decision temperature between blocks (ps >.1). Error bars represent the standard deviation of the parameter estimates for each subject.

Supplementary Figure 7 Analyses of reaction time and pupil dilation data using non-Bayesian estimates of trial volatility and outcome surprise

Analyses of the pupil dilation and reaction time data reported in this paper utilized estimates of trial volatility and outcome surprise derived from a Bayesian learner (see Supplementary Modeling Note). Here, we replicated these analyses using non-Bayesian estimates of volatility and surprise. Volatility was simply coded as ‘1’ for all trials in the volatile block and ‘0’ for all trials in the stable block. In addition, a non-Bayesian surprise regressor was created by coding all trials in which the less predictive stimulus was associated with the shock as being high surprise (‘1’) and the other trials as low surprise (‘0’). These measures of volatility and surprise were entered into the regression models predicting reaction time slowing (a) and pupil response (b,c) together with the other, control, predictor variables previously used. As in the main analyses, the beta estimates for these regressors were correlated against trait anxiety. The results of these additional analyses replicated those reported in the paper: trait anxiety did not influence the degree to which participants slowed their response following a surprising outcome, r(28) = –0.02, p = 0.9 (a). Further trait anxiety was associated with a significantly reduced pupil response during high volatility trials, r(26) = –0.51, p = 0.005 (b), but showed no significant relationship with the pupil response to surprising outcomes, r(26) = 0.07, p = 0.7 (c). In summary, the relationship reported between trait anxiety and effects of volatility and outcome surprise held regardless of the manner in which these indices were calculated. Error bars represent the standard deviations of the parameter estimates from the behavioral model (a) and the regression coefficients (beta weights) from the pupil analysis (b, c) for each subject.

Supplementary Figure 8 Baseline corrected, z-transformed eyetracker traces from the aversive learning task.

(a). Mean pupil dilation across participants separated into trials in which shock was administered or no shock was administered. (b). Mean pupil dilation across trials shown separately for high and low trait anxious participants. Effect of volatility on pupil dilation shown separately for low (c) and high (d) trait anxious participants. Effect of surprise on pupil dilation shown separately for low (e) and high (f) trait anxious participants. NB for illustration, participants have been separated into high and low anxious groups based on a median split on trait anxiety scores. Similarly trials were classified as high/low volatility or surprise based on a median split on these variables. Lines show mean value (across participants) of z-transformed, baseline corrected pupil diameter. Shaded regions represent SEM.

Supplementary Figure 9 The relationship between trait anxiety and post outcome pupil dilation as a function of trial-wise estimates of volatility, controlling for trial number within each block.

This figure illustrates the same analyses as those described in Figure 4 of the main manuscript with the exception that regressors coding for the trial number within each task block have been included in the participant-wise regression analyses of the effect of volatility on post outcome pupil dilation. The results illustrated in Figure 4 remain robust with the inclusion of these extra control parameters. Specifically, a strong negative correlation between trait anxiety and the effect of volatility upon pupil dilation post outcome is still observed (a), r(26) = –0.5, p = 0.004. As before, this is driven by low but not high trait anxious participants showing a modulatory effect of volatility on the post outcome pupil response (b), a median split on trait anxiety levels is used here as in Figure 4. Asterisks indicate 1s time bins in which Bonferonni corrected t-tests differed between the groups at p <.05 corrected, 2-tailed. Error bars in panel a represent the standard deviations of the regression coefficients (beta weights) from the pupil analysis for each subject. Shaded regions in panel b represent the standard error of the mean.

Supplementary Figure 10 Results from additional pupil analyses investigating the interaction between outcome (shock received versus no shock received), and volatility (a) and surprise (b).

The delivery of a shock causes a large dilation of participants’ pupils (Figure S8a). In order to determine whether effects of volatility and surprise were modulated by outcome (shock received versus no shock received), we reran the pupil regression analyses descripted in the main manuscript with additional regressors encoding the interaction between volatility and outcome and the interaction between surprise and outcome. Neither interaction term had a significant influence on pupil dilation (t-tests of mean regressor values against 0, p > 0.1 for all time bins). Consistent with this, adding these terms to the model reduced the goodness of fit of the model by an adjusted R2 of 0.0002. Shaded areas represent the standard error of the mean.

Supplementary Figure 11 Interaction of estimated trialwise volatility and surprise on pupil dilation post outcome.

The pupil analysis was rerun with the inclusion of a surprise x volatility interaction term. There was no significant interaction of surprise x volatility upon pupil dilation in any of the 1s post-outcome time bins (t-test of mean regressor value against 0, p > 0.3 for all time bins). While the post-outcome main effect of surprise on pupil dilation precedes that of volatility (Figure 3), trialwise volatility changes very slowly (see Supplementary Modeling Note) – i.e. the volatility of a given trial n and that of the following trial n+1 is highly correlated (r >.9). Hence this regression analysis should capture any effect of volatility on trial n on the response to surprise in trial n+1. Shaded areas represent standard error of the mean.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–11 and Supplementary Modeling Note (PDF 2953 kb)

Source data

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Browning, M., Behrens, T., Jocham, G. et al. Anxious individuals have difficulty learning the causal statistics of aversive environments. Nat Neurosci 18, 590–596 (2015). https://doi.org/10.1038/nn.3961

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn.3961

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing