Estimating the controllability of the environment enables agents to better predict upcoming events and decide when to engage controlled action selection. How does the human brain estimate controllability? Trial-by-trial analysis of choices, decision times and neural activity in an explore-and-predict task demonstrate that humans solve this problem by comparing the predictions of an ‘actor’ model with those of a reduced ‘spectator’ model of their environment. Neural blood oxygen level-dependent responses within striatal and medial prefrontal areas tracked the instantaneous difference in the prediction errors generated by these two statistical learning models. Blood oxygen level-dependent activity in the posterior cingulate, temporoparietal and prefrontal cortices covaried with changes in estimated controllability. Exposure to inescapable stressors biased controllability estimates downward and increased reliance on the spectator model in an anxiety-dependent fashion. Taken together, these findings provide a mechanistic account of controllability inference and its distortion by stress exposure.
This is a preview of subscription content, access via your institution
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 per month
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Rent or buy this article
Get just this article for as long as you need it
Prices may be subject to local taxes which are calculated during checkout
The experimental paradigm and the code used to generate the figures are available at the following address: https://github.com/romainligneul/NHBcontrollability. Second-level SPM images are available at https://neurovault.org/collections/8810/. Anonymized data can be accessed using ORCID identification at https://data.donders.ru.nl/collections/di/dccn/DSC_3017049.01_905.
The scripts used to collect and analyse data are available upon publication at https://github.com/romainligneul/NHBcontrollability.
Bastos, A. M. et al. Canonical microcircuits for predictive coding. Neuron 76, 695–711 (2012).
Gläscher, J., Daw, N., Dayan, P. & O’Doherty, J. P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585–595 (2010).
Rao, R. P. N. & Ballard, D. H. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 2, 79–87 (1999).
Niv, Y. Learning task-state representations. Nat. Neurosci. 22, 1544–1553 (2019).
Kim, D., Park, G. Y., O′Doherty, J. P. & Lee, S. W. Task complexity interacts with state-space uncertainty in the arbitration between model-based and model-free learning. Nat. Commun. 10, 5738 (2019).
Hamid, A. A., Frank, M. J. & Moore, C. I. Wave-like dopamine dynamics as a mechanism for spatiotemporal credit assignment. Cell 184, 2733–2749 (2021).
Moscarello, J. M. & Hartley, C. A. Agency and the calibration of motivated behavior. Trends Cogn. Sci. 21, 725–735 (2017).
Maier, S. F. & Seligman, M. E. P. Learned helplessness at fifty: insights from neuroscience. Psychol. Rev. 123, 349–367 (2016).
Cheng, C., Cheung, S. F., Chio, J. H. & Chan, M.-P. S. Cultural meaning of perceived control: a meta-analysis of locus of control and psychological symptoms across 18 cultural regions. Psychol. Bull. 139, 152–188 (2013).
Hammack, S. E., Cooper, M. A. & Lezak, K. R. Overlapping neurobiology of learned helplessness and conditioned defeat: implications for PTSD and mood disorders. Neuropharmacology 62, 565–575 (2012).
Harrow, M., Hansford, B. G. & Astrachan-Fletcher, E. B. Locus of control: relation to schizophrenia, to recovery, and to depression and psychosis — A 15-year longitudinal study. Psychiatry Res. 168, 186–192 (2009).
Gillan, C. M. et al. Obsessive–compulsive disorder patients have a reduced sense of control on the illusion of control task. Front. Psychol. 5, 204 (2014).
Diener, C., Kuehner, C., Brusniak, W., Struve, M. & Flor, H. Effects of stressor controllability on psychophysiological, cognitive and behavioural responses in patients with major depression and dysthymia. Psychol. Med. 39, 77–86 (2009).
Amat, J. et al. Medial prefrontal cortex determines how stressor controllability affects behavior and dorsal raphe nucleus. Nat. Neurosci. 8, 365–371 (2005).
Bland, S. T. et al. Stressor controllability modulates stress-induced dopamine and serotonin efflux and morphine-induced serotonin efflux in the medial prefrontal cortex. Neuropsychopharmacology 28, 1589–1596 (2003).
Challis, C. et al. Raphe GABAergic neurons mediate the acquisition of avoidance after social defeat. J. Neurosci. 33, 13978–13988 (2013).
Kerr, D. L., McLaren, D. G., Mathy, R. M. & Nitschke, J. B. Controllability modulates the anticipatory response in the human ventromedial prefrontal cortex. Front. Psychol. 3, 557 (2012).
Wood, K. H. et al. Controllability modulates the neural response to predictable but not unpredictable threat in humans. NeuroImage 119, 371–381 (2015).
Alvarez, R. P. et al. Increased anterior insula activity in anxious individuals is linked to diminished perceived control. Transl. Psychiatry 5, e591–e591 (2015).
Bräscher, A.-K., Becker, S., Hoeppli, M.-E. & Schweinhardt, P. Different brain circuitries mediating controllable and uncontrollable pain. J. Neurosci. 36, 5013–5025 (2016).
Haggard, P. Sense of agency in the human brain. Nat. Rev. Neurosci. 18, 196–207 (2017).
Kühn, S., Brass, M. & Haggard, P. Feeling in control: neural correlates of experience of agency. Cortex 49, 1935–1942 (2013).
Spengler, S., von Cramon, D. Y. & Brass, M. Was it me or was it you? How the sense of agency originates from ideomotor learning revealed by fMRI. NeuroImage 46, 290–298 (2009).
Liljeholm, M., Wang, S., Zhang, J. & O’Doherty, J. P. Neural correlates of the divergence of instrumental probability distributions. J. Neurosci. 33, 12519–12527 (2013).
Vejmelka, M. & Paluš, M. Inferring the directionality of coupling with conditional mutual information. Phys. Rev. E 77, 026214 (2008).
Barnett, L., Barrett, A. B. & Seth, A. K. Granger causality and transfer entropy are equivalent for Gaussian variables. Phys. Rev. Lett. 103, 238701 (2009).
Pearl, J. Interpretation and identification of causal mediation. Psychol. Methods 19, 459–481 (2014).
Dorfman, H. M. & Gershman, S. J. Controllability governs the balance between Pavlovian and instrumental action selection. Nat. Commun. 10, 5826 (2019).
Ly, V., Wang, K. S., Bhanji, J. & Delgado, M. R. A reward-based framework of perceived control. Front. Neurosci. 13, 65 (2019).
Huys, Q. J. M. & Dayan, P. A. Bayesian formulation of behavioral control. Cognition 113, 314–328 (2009).
Lee, S. W., Shimojo, S. & O’Doherty, J. P. Neural computations underlying arbitration between model-based and model-free learning. Neuron 81, 687–699 (2014).
Otto, A. R., Raio, C. M., Chiang, A., Phelps, E. A. & Daw, N. D. Working-memory capacity protects model-based learning from stress. Proc. Natl Acad. Sci. USA 110, 20941–20946 (2013).
Shahar, N. et al. Improving the reliability of model-based decision-making estimates in the two-stage decision task with reaction-times and drift-diffusion modeling. PLoS Comput. Biol. 15, e1006803 (2019).
Shahar, N. et al. Credit assignment to state-independent task representations and its relationship with model-based decision making. Proc. Natl Acad. Sci. USA 116, 15871–15876 (2019).
Voss, M., Chambon, V., Wenke, D., Kühn, S. & Haggard, P. In and out of control: brain mechanisms linking fluency of action selection to self-agency in patients with schizophrenia. Brain 140, 2226–2239 (2017).
Maier, S. F. & Seligman, M. E. Learned helplessness: theory and evidence. J. Exp. Psychol. Gen. 105, 3–46 (1976).
Miller, K. J., Botvinick, M. M. & Brody, C. D. Dorsal hippocampus contributes to model-based planning. Nat. Neurosci. 20, 1269–1276 (2017).
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P. & Dolan, R. J. Model-based influences on humans’ choices and striatal prediction errors. Neuron 69, 1204–1215 (2011).
Westbrook, A. et al. Dopamine promotes cognitive effort by biasing the benefits versus costs of cognitive work. Science 367, 1362–1366 (2020).
Hahn, A. et al. Reconfiguration of functional brain networks and metabolic cost converge during task performance. eLife 9, e52443 (2020).
O’Doherty, J. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452–454 (2004).
Garrison, J., Erdeniz, B. & Done, J. Prediction error in reinforcement learning: a meta-analysis of neuroimaging studies. Neurosci. Biobehav. Rev. 37, 1297–1310 (2013).
Grogan, J. P., Sandhu, T. R., Hu, M. T. & Manohar, S. G. Dopamine promotes instrumental motivation, but reduces reward-related vigour. eLife 9, e58321 (2020).
Weiss, A., Chambon, V., Lee, J. K., Drugowitsch, J. & Wyart, V. Interacting with volatile environments stabilizes hidden-state inference and its brain signatures. Nat. Commun. 12, 2228 (2019).
O’Callaghan, C., Vaghi, M. M., Brummerloh, B., Cardinal, R. N. & Robbins, T. W. Impaired awareness of action-outcome contingency and causality during healthy ageing and following ventromedial prefrontal cortex lesions. Neuropsychologia 128, 282–289 (2019).
Kim, D., Park, G. J., O’Doherty, J.-D. & Lee, S. W. Task complexity interacts with state-space uncertainty in the arbitration between model-based and model-free learning. Nat. Commun. 10, 5738 (2019).
Boureau, Y.-L., Sokol-Hessner, P. & Daw, N. D. Deciding how to decide: self-control and meta-decision making. Trends Cogn. Sci. 19, 700–710 (2015).
Piray, P., Toni, I. & Cools, R. Human choice strategy varies with anatomical projections from ventromedial prefrontal cortex to medial striatum. J. Neurosci. 36, 2857–2867 (2016).
Wanke, N. & Schwabe, L. Subjective uncontrollability over aversive events reduces working memory performance and related large-scale network interactions. Cereb. Cortex 30, 3116–3129 (2019).
Preuschoff, K., Quartz, S. R. & Bossaerts, P. Human insula activation reflects risk prediction errors as well as risk. J. Neurosci. 28, 2745–2752 (2008).
Behrens, T. E. J., Woolrich, M. W., Walton, M. E. & Rushworth, M. F. S. Learning the value of information in an uncertain world. Nat. Neurosci. 10, 1214–1221 (2007).
Albares, M. et al. The dorsal medial frontal cortex mediates automatic motor inhibition in uncertain contexts: evidence from combined fMRI and EEG studies. Hum. Brain Mapp. 35, 5517–5531 (2014).
van Belle, J., Vink, M., Durston, S. & Zandbelt, B. B. Common and unique neural networks for proactive and reactive response inhibition revealed by independent component analysis of functional MRI data. NeuroImage 103, 65–74 (2014).
Shenhav, A., Cohen, J. D. & Botvinick, M. M. Dorsal anterior cingulate cortex and the value of control. Nat. Neurosci. 19, 1286–1291 (2016).
Leibfried, F., Pascual-Diaz, S. & Grau-Moya, J. A unified Bellman optimality principle combining reward maximization and empowerment. Preprint at https://arxiv.org/abs/1907.12392 (2020).
Mohamed, S. & Jimenez Rezende, D. Variational information maximisation for intrinsically motivated reinforcement learning. Adv. Neural Inf. Process. Syst. 28, 2125–2133 (2015).
Dickerson, S. S. & Kemeny, M. E. Acute stressors and cortisol responses: a theoretical integration and synthesis of laboratory research. Psychol. Bull. 130, 355–391 (2004).
Daunizeau, J., Adam, V. & Rigoux, L. VBA: a probabilistic treatment of nonlinear models for neurobiological and behavioural data. PLoS Comput. Biol. 10, e1003441 (2014).
Hebart, M. N., Görgen, K. & Haynes, J.-D. The Decoding Toolbox (TDT): a versatile software package for multivariate analyses of functional imaging data. Front. Neuroinform. 8, 88 (2015).
We thank M. Frank for his constructive comments on the manuscript and computational models. We thank P. Gaalman for help with fMRI data acquisition. We thank K. Elefteriadou, F. Dianellou, K. Koelbel and J. Breen and SOLO labsupport Leiden University for their help and support with data acquisition for the stress experiment. This work was supported by grants from the Fyssen Foundation and the Behaviour and Brain Research Foundation awarded to R.L. (Young Investigator 2017) and a Vici award from the Netherlands Organisation for Scientific Research to R.C. (NWO 453-14-005). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
The authors declare no competing financial interests.
Peer review information
Nature Human Behaviour thanks Sang Wan Lee and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary notes, methods, Figs. 1–5 and Tables 1–6.
Rights and permissions
About this article
Cite this article
Ligneul, R., Mainen, Z.F., Ly, V. et al. Stress-sensitive inference of task controllability. Nat Hum Behav 6, 812–822 (2022). https://doi.org/10.1038/s41562-022-01306-w
This article is cited by
The computational psychopathology of emotion