Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Advances in the computational understanding of mental illness

Abstract

Computational psychiatry is a rapidly growing field attempting to translate advances in computational neuroscience and machine learning into improved outcomes for patients suffering from mental illness. It encompasses both data-driven and theory-driven efforts. Here, recent advances in theory-driven work are reviewed. We argue that the brain is a computational organ. As such, an understanding of the illnesses arising from it will require a computational framework. The review divides work up into three theoretical approaches that have deep mathematical connections: dynamical systems, Bayesian inference and reinforcement learning. We discuss both general and specific challenges for the field, and suggest ways forward.

This is a preview of subscription content

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Count of publications listed on pubmed and referring to “computational psychiatry” in title, abstract or keywords.
Fig. 2: Dynamical Systems.
Fig. 3: Dynamical system applications.
Fig. 4: Learning rates.
Fig. 5: Working memory in reinforcement learning.
Fig. 6: Challenges for task-based measurements.

References

  1. 1.

    Kendler KS. Toward a philosophical structure for psychiatry. Am J Psychiatry. 2005;162:433–40.

    Google Scholar 

  2. 2.

    Kendler KS. Explanatory models for psychiatric illness. Am J Psychiatry. 2008;165:695–702.

    Google Scholar 

  3. 3.

    Dagher A, Robbins TW. Personality, addiction, dopamine: Insights from parkinson’s disease. Neuron. 2009;61:502–10.

    CAS  Google Scholar 

  4. 4.

    Marr D. Vision. New York, NY, USA: Freeman; 1982.

    Google Scholar 

  5. 5.

    Kendler KS. David skae and his nineteenth century etiologic psychiatric diagnostic system: looking forward by looking back. Mol Psychiatry. 2017;22:802–7.

    CAS  Google Scholar 

  6. 6.

    Friston K, Moran R, Seth AK. Analysing connectivity with granger causality and dynamic causal modelling. Curr Opin Neurobiol. 2013;23:172–8.

    CAS  Google Scholar 

  7. 7.

    Huys QJM. Bayesian approaches to learning and decision-making. In Anticevic, A and Murray, J, editors, Computational psychiatry: mathematical modelling of mental illness. Elsevier; 2017.

  8. 8.

    Stephan KE, Mathys C. Computational approaches to psychiatry. Curr Opin Neurobiol. 2014;25:85–92.

    CAS  Google Scholar 

  9. 9.

    Stephan KE, Schlagenhauf F, Huys QJM, Raman S, Aponte EA, Brodersen KH, et al. Computational neuroimaging strategies for single patient predictions. NeuroImage. 2017;145:180–99.

    CAS  Google Scholar 

  10. 10.

    Itani S, Rossignol M, Lecron F, Fortemps P. Towards interpretable machine learning models for diagnosis aid: a case study on attention deficit/hyperactivity disorder. PLoS ONE. 2019;14:e0215720.

    CAS  Google Scholar 

  11. 11.

    Liu Y, Admon R, Mellem MS, Belleau EL, Kaiser RH, Clegg R, et al. Machine learning identifies large-scale reward-related activity modulated by dopaminergic enhancement in major depression. Biol Psychiatry: Cogn Neurosci Neuroimaging. 2020;5:163–72.

    Google Scholar 

  12. 12.

    Woo C-W, Chang LJ, Lindquist MA, Wager TD. Building better biomarkers: brain models in translational neuroimaging. Nat Neurosci. 2017;20:365–77.

    CAS  Google Scholar 

  13. 13.

    Adams RA, Huys QJM, Roiser JP. Computational psychiatry: towards a mathematically informed understanding of mental illness. J Neurol, Neurosurg, Psychiatry. 2016;87:53–63.

    Google Scholar 

  14. 14.

    Corlett PR, Fletcher PC. Computational psychiatry: a rosetta stone linking the brain to mental illness. lancet Psychiatry. 2014;1:399–402.

    Google Scholar 

  15. 15.

    Huys QJM, Maia TV, Frank MJ. Computational psychiatry as a bridge from neuroscience to clinical applications. Nat Neurosci. 2016;19:404–13.

    CAS  Google Scholar 

  16. 16.

    Maia TV, Frank MJ. From reinforcement learning models to psychiatric and neurological disorders. Nat Neurosci. 2011;14:154–62.

    CAS  Google Scholar 

  17. 17.

    Montague PR. Neuroeconomics: a view from neuroscience. Funct Neurol. 2007;22:219–34.

    Google Scholar 

  18. 18.

    Montague PR, Dolan RJ, Friston KJ, Dayan P. Computational psychiatry. Trends Cogn Sci. 2012;16:72–80.

    Google Scholar 

  19. 19.

    Rutledge RB, Chekroud AM, Huys QJ. Machine learning and big data in psychiatry: toward clinical applications. Curr Opin Neurobiol. 2019;55:152–9.

    CAS  Google Scholar 

  20. 20.

    Steele JD, Paulus MP. Pragmatic neuroscience for clinical psychiatry. Br J Psychiatry. 2019;215:404–8.

    Google Scholar 

  21. 21.

    Wang X-J, Krystal JH. Computational psychiatry. Neuron. 2014;84:638–54.

    CAS  Google Scholar 

  22. 22.

    Wiecki TV, Poland JS, Frank MJ. Model-based cognitive neuroscience approaches to computational psychiatry: clustering and classification. Clin Psychol Sci. 2015;3:378–99.

    Google Scholar 

  23. 23.

    Maia TV, Huys QJM, Frank MJ. Theory-based computational psychiatry. Biol Psychiatry. 2017;82:382–4.

    Google Scholar 

  24. 24.

    Bertsekas DP and Tsitsiklis JN. Neuro-Dynamic Programming. Athena Scientific; 1996.

  25. 25.

    Murphy K, Weiss Y, Jordan MI. Loopy belief propagation for approximate inference: an empirical study. ArXiv, 2013. http://arxiv.org/abs/1301.6725v1.

  26. 26.

    Kalman RE. A new approach to linear filtering and prediction problem. Trans ASME. 1960;82:35–45.

    Google Scholar 

  27. 27.

    Todorov E. General duality between optimal control and estimation. In 2008 47th IEEE Conference on Decision and Control. IEEE; 2008. https://doi.org/10.1109/cdc.2008.4739438.

  28. 28.

    Strogatz SH. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering. studies in nonlinearity. 2nd edn, Westview Press; 2015.

  29. 29.

    Wang X-J. Synaptic basis of cortical persistent activity: the importance of nmda receptors to working memory. J Neurosci. 1999;19:9587–603.

    CAS  Google Scholar 

  30. 30.

    Cramer AOJ, van Borkulo CD, Giltay EJ, van der Maas HLJ, Kendler KS, Scheffer M, et al. Major depression as a complex dynamic system. PloS one. 2016;11:e0167490.

    Google Scholar 

  31. 31.

    Robinaugh DJ, Hoekstra RHA, Toner ER, Borsboom D. The network approach to psychopathology: a review of the literature 2008–18 and an agenda for future research. Psychol Med. 2019;50:353–66.

    Google Scholar 

  32. 32.

    Strawinska-Zanko, U and Liebovitch, LS, (eds) Mathematical modeling of social relationships. Springer International Publishing; 2018. https://doi.org/10.1007/978-3-319-76765-9.

  33. 33.

    Breakspear M. Dynamic models of large-scale brain activity. Nat Neurosci. 2017;20:340–52.

    CAS  Google Scholar 

  34. 34.

    Bystritsky A, Nierenberg AA, Feusner JD, Rabinovich M. Computational non-linear dynamical psychiatry: a new methodological paradigm for diagnosis and course of illness. J Psychiatr Res. 2012;46:428–35.

    CAS  Google Scholar 

  35. 35.

    Durstewitz D, Huys QJ, Koppe G. Psychiatric illnesses as disorders of network dynamics. Biological Psychiatry CNNI. 2020. Advance Online Publiation.

  36. 36.

    Amit DJ, Brunel N. Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex. Cereb Cortex. 1997;7:237–52.

    CAS  Google Scholar 

  37. 37.

    Lisman JE, Fellous J-M, Wang X-J. A role for NMDA-receptor channels in working memory. Nat Neurosci. 1998;1:273–5.

    CAS  Google Scholar 

  38. 38.

    Wang M, Yang Y, Wang C-J, Gamo NJ, Jin LE, Mazer JA, et al. NMDA receptors subserve persistent neuronal firing during working memory in dorsolateral prefrontal cortex. Neuron. 2013;77:736–49.

    CAS  Google Scholar 

  39. 39.

    Compte A, Brunel N, Goldman-Rakic PS, Wang X-J. Synaptic mechanisms and network dynamics underlying spatial working memory in a cortical network model. Cereb Cortex. 2000;10:910–23.

    CAS  Google Scholar 

  40. 40.

    Durstewitz D, Seamans JK, Sejnowski TJ. Neurocomputational models of working memory. Nat Neurosci. 2000;3:1184–91.

    CAS  Google Scholar 

  41. 41.

    Cano-Colino M, Almeida R, Compte A. Serotonergic modulation of spatial working memory: predictions from a computational network model. Front Integr Neurosci. 2013;7:71.

    Google Scholar 

  42. 42.

    Cano-Colino M, Almeida R, Gomez-Cabrero D, Artigas F, Compte A. Serotonin regulates performance nonmonotonically in a spatial working memory network. Cereb Cortex (N. Y, N. Y: 1991). 2014;24:2449–63.

    Google Scholar 

  43. 43.

    Maia TV, Cano-Colino M. The role of serotonin in orbitofrontal function and obsessivecompulsive disorder. Clin Psychol Sci. 2015;3:460–82.

    Google Scholar 

  44. 44.

    Cano-Colino M, Compte A. A computational model for spatial working memory deficits in schizophrenia. Pharmacopsychiatry. 2012;45:S49–S56.

    Google Scholar 

  45. 45.

    Murray JD, Anticevic A, Gancsos M, Ichinose M, Corlett PR, Krystal JH, et al. Linking microcircuit dysfunction to cognitive impairment: effects of disinhibition associated with schizophrenia in a cortical working memory model. Cereb Cortex. 2014;24:859–72.

    Google Scholar 

  46. 46.

    Starc M, Murray JD, Santamauro N, Savic A, Diehl C, Cho YT, et al. Schizophrenia is associated with a pattern of spatial working memory deficits consistent with cortical disinhibition. Schizophrenia Res. 2017;181:107–16.

    Google Scholar 

  47. 47.

    Hamm JP, Peterka DS, Gogos JA, Yuste R. Altered cortical ensembles in mouse models of schizophrenia. Neuron. 2017;94:153–67.e8.

    CAS  Google Scholar 

  48. 48.

    Stein, H, Barbosa, J, Rosa-Justicia, M, Prades, L, Morató, A, Galan, A, et al. (2019). Disrupted serial dependence suggests deficits in synaptic potentiation in anti-NMDAR encephalitis and schizophrenia. 2019, https://www.biorxiv.org/content/10.1101/830471v1.

  49. 49.

    Adams RA, Napier G, Roiser JP, Mathys C, Gilleen J. Attractor-like dynamics in belief updating in schizophrenia. J Neurosci. 2018;38:9471–85.

    CAS  Google Scholar 

  50. 50.

    Jardri R, Duverne S, Litvinova A, Deneve S. Experimental evidence for circular inference in schizophrenia. Nat Commun. 2017;8:14218. https://doi.org/10.1038/ncomms14218.

    CAS  Article  Google Scholar 

  51. 51.

    Hopfield J. Neural networks and physical systems with emergent collective computational abilities. Proc Natl Acad Sci USA. 1982;79:2554.

    CAS  Google Scholar 

  52. 52.

    Wills TJ, Lever C, Cacucci F, Burgess N, O’Keefe J. Attractor dynamics in the hippocampal representation of the local environment. Science. 2005;308:873–6.

    CAS  Google Scholar 

  53. 53.

    Echeveste R, Aitchison L, Hennequin G, and Lengyel M. Cortical-like dynamics in recurrent circuits optimized for sampling-based probabilistic inference. 2019, https://www.biorxiv.org/content/10.1101/696088v1.

  54. 54.

    Lengyel M, Kwag J, Paulsen O, Dayan P. Matching storage and recall: hippocampal spike timing-dependent plasticity and phase response curves. Nat Neurosci. 2005;8:1677–83.

    CAS  Google Scholar 

  55. 55.

    Deneve S, Latham PE, Pouget A. E cient computation and cue integration with noisy population codes. Nat Neurosci 2001;4:826–31.

    CAS  Google Scholar 

  56. 56.

    Bogacz R, Brown E, Moehlis J, Holmes P, Cohen JD. The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychol Rev. 2006;113:700–65.

    Google Scholar 

  57. 57.

    Foss-Feig JH, Adkinson BD, Ji JL, Yang G, Srihari VH, McPartland JC, et al. Searching for cross-diagnostic convergence: neural mechanisms governing excitation and inhibition balance in schizophrenia and autism spectrum disorders. Biol Psychiatry. 2017;81:848–61.

    Google Scholar 

  58. 58.

    Carandini M, Heeger DJ. Normalization as a canonical neural computation. Nat Rev Neurosci. 2011;13:51–62.

    Google Scholar 

  59. 59.

    Heeger DJ. Normalization of cell responses in cat striate cortex. Vis Neurosci. 1992;9:181–97.

    CAS  Google Scholar 

  60. 60.

    De Martino B, Harrison NA, Knafo S, Bird G, Dolan RJ. Explaining enhanced logical consistency during decision making in autism. J Neurosci: Offcial J Soc Neurosci. 2008;28:10746–50.

    Google Scholar 

  61. 61.

    Lawson RP, Aylward J, White S, Rees G. A striking reduction of simple loudness adaptation in autism. Sci Rep. 2015;5:16157.

    CAS  Google Scholar 

  62. 62.

    Louie K, Khaw MW, Glimcher PW. Normalization is a general neural mechanism for context-dependent decision making. Proc Natl Acad Sci USA. 2013;110:6139–44.

    CAS  Google Scholar 

  63. 63.

    Rosenberg A, Patterson JS, Angelaki DE. A computational perspective on autism. Proc Natl Acad Sci. 2015;112:9158–65.

    CAS  Google Scholar 

  64. 64.

    Vattikuti S, Chow CC. A computational model for cerebral cortical dysfunction in autism spectrum disorders. Biol Psychiatry. 2010;67:672–8.

    Google Scholar 

  65. 65.

    Beck JM, Latham PE, Pouget A. Marginalization in neural circuits with divisive normalization. J Neurosci: Offcial J Soc Neurosci. 2011;31:15310–9.

    CAS  Google Scholar 

  66. 66.

    Ramirez-Mahaluf JP, Compte A. Serotonergic Modulation of Cognition in Prefrontal Cortical Circuits in Major Depression. In: Anticevic, A. & Murray, J. (eds) Computational Psychiatry, Elsevier; 2018. p. 27–46.

  67. 67.

    Moran RJ, Symmonds M, Stephan KE, Friston KJ, Dolan RJ. An in vivo assay of synaptic function mediating human cognition. Curr Biol: CB. 2011;21:1320–5.

    CAS  Google Scholar 

  68. 68.

    Symmonds M, Moran CH, Leite MI, Buckley C, Irani SR, Stephan KE, et al. Ion channels in eeg: isolating channel dysfunction in nmda receptor antibody encephalitis. Brain. 2018;141:1691–702.

    Google Scholar 

  69. 69.

    Friston KJ, Harrison L, Penny W. Dynamic causal modelling. Neuroimage. 2003;19:1273–302.

    CAS  Google Scholar 

  70. 70.

    Seth AK, Barrett AB, Barnett L. Granger causality analysis in neuroscience and neuroimaging. J Neurosci. 2015;35:3293–7.

    CAS  Google Scholar 

  71. 71.

    Dima D, Dietrich DE, Dillo W, Emrich HM. Impaired top-down processes in schizophrenia: a dcm study of erps. NeuroImage. 2010;52:824–32.

    Google Scholar 

  72. 72.

    Dima D, Roiser JP, Dietrich DE, Bonnemann C, Lanfermann H, Emrich HM, et al. Understanding why patients with schizophrenia do not perceive the hollow-mask illusion using dynamic causal modelling. NeuroImage. 2009;46:1180–6.

    Google Scholar 

  73. 73.

    Brodersen KH, Deserno L, Schlagenhauf F, Lin Z, Penny WD, Buhmann JM, et al. Dissecting psychiatric spectrum disorders by generative embedding. Neuroimage Clin. 2014;4:98–111.

    Google Scholar 

  74. 74.

    Brodersen KH, Schofield TM, Leff AP, Ong CS, Lomakina EI, Buhmann JM, et al. Generative embedding for model-based classification of fmri data. PLoS Comput Biol. 2011;7:e1002079.

    CAS  Google Scholar 

  75. 75.

    Frässle S, Marquand AF, Schmaal L, Dinga R, Veltman DJ, van der Wee NJA, et al. Predicting individual clinical trajectories of depression with generative embedding. NeuroImage Clin. 2020;26:102213.

    Google Scholar 

  76. 76.

    Frässle S, Lomakina EI, Kasper L, Manjaly ZM, Leff A, Pruessmann KP, et al. A generative model of whole-brain effective connectivity. NeuroImage. 2018;179:505–29.

    Google Scholar 

  77. 77.

    Frässle S, Lomakina EI, Razi A, Friston KJ, Buhmann JM, Stephan KE. Regression dcm for fmri. NeuroImage. 2017;155:406–21.

    Google Scholar 

  78. 78.

    Braun U, Schaefer A, Betzel RF, Tost H, Meyer-Lindenberg A, Bassett DS. From maps to multi-dimensional network mechanisms of mental disorders. Neuron. 2018;97:14–31.

    CAS  Google Scholar 

  79. 79.

    Gu S, Pasqualetti F, Cieslak M, Telesford QK, Yu AB, Kahn AE, et al. Controllability of structural brain networks. Nat Commun. 2015;6. https://doi.org/10.1038/ncomms9414.

  80. 80.

    Perry A, Roberts G, Mitchell PB, Breakspear M. Connectomics of bipolar disorder: a critical review, and evidence for dynamic instabilities within interoceptive networks. Mol Psychiatry. 2018;24:1296–318.

    Google Scholar 

  81. 81.

    Durstewitz D. A state space approach for piecewise-linear recurrent neural networks for identifying computational dynamics from neural measurements. PLoS Comput Biol. 2017;13:e1005542.

    Google Scholar 

  82. 82.

    Koppe G, Toutounji H, Kirsch P, Lis S, Durstewitz D. Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fmri. PLoS Comput Biol. 2019;15:e1007263.

    CAS  Google Scholar 

  83. 83.

    Piccirillo ML, Rodebaugh TL. Foundations of idiographic methods in psychology and applications for psychotherapy. Clin Psychol Rev. 2019;71:90–100.

    Google Scholar 

  84. 84.

    Borsboom D, Kievit RA, Cervone D, and Hood SB. The two disciplines of scientific psychology, or: The disunity of psychology as a working hypothesis. In: Valsiner J, Molenaar PCM, Lyra MCDP, and Chaudhary N, editors. Dynamic process methodology in the social and developmental sciences. Springer Science + Business Media. 2009. https://doi.org/10.1007/978-0-387-95922-1_4.

  85. 85.

    Molenaar PC, Campbell CG. The new person-specific paradigm in psychology. Curr Directions Psychol Sci. 2009;18:112–7.

    Google Scholar 

  86. 86.

    Borsboom D, Cramer AOJ, Schmittmann VD, Epskamp S, Waldorp LJ. The small world of psychopathology. PLoS One. 2011;6:e27407.

    CAS  Google Scholar 

  87. 87.

    Fried EI, van Borkulo CD, Cramer AOJ, Boschloo L, Schoevers RA, Borsboom D. Mental disorders as networks of problems: a review of recent insights. Soc Psychiatry Psychiatr Epidemiol. 2016;52:1–10.

    Google Scholar 

  88. 88.

    American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM5 R). American Psychiatric Pub; 2013.

  89. 89.

    World Health Organization. International classification of diseases. World Health Organization Press; 1990.

  90. 90.

    Newson JJ, Hunter D, Thiagarajan TC. The heterogeneity of mental health assessment. Front Psychiatry. 2020;11:76.

    Google Scholar 

  91. 91.

    van de Leemput IA, Wichers M, Cramer AOJ, Borsboom D, Tuerlinckx F, Kuppens P, et al. Critical slowing down as early warning for the onset and termination of depression. Proc Natl Acad Sci USA. 2014;111:87–92.

    Google Scholar 

  92. 92.

    van Borkulo C, Boschloo L, Borsboom D, Penninx BWJH, Waldorp LJ, Schoevers RA. Association of symptom network structure with the course of longitudinal depression. JAMA Psychiatry. 2015;72:1219–26.

    Google Scholar 

  93. 93.

    Bringmann LF, Ferrer E, Hamaker EL, Borsboom D, Tuerlinckx F. Modeling nonstationary emotion dynamics in dyads using a time-varying vector-autoregressive model. Multivar Behav Res. 2018;53:293–314.

    Google Scholar 

  94. 94.

    Bringmann LF, Vissers N, Wichers M, Geschwind N, Kuppens P, Peeters F, et al. A network approach to psychopathology: new insights into clinical longitudinal data. PLoS One. 2013;8:e60188.

    CAS  Google Scholar 

  95. 95.

    Lodewyckx T, Tuerlinckx F, Kuppens P, Allen NB, Sheeber L. A hierarchical state space approach to affective dynamics. J Math Psychol. 2011;55:68–83.

    Google Scholar 

  96. 96.

    Loossens T, Mestdagh M, Dejonckheere E, Kuppens P, Tuerlinckx F, Verdonck S. The affective ising model: a computational account of human affect dynamics. PsyArXiv, 2019. https://doi.org/10.31234/osf.io/ky23d.

  97. 97.

    van Borkulo CD, Borsboom D, Epskamp S, Blanken TF, Boschloo L, Schoevers RA, et al. A new method for constructing networks from binary data. Sci Rep. 2014;4:5918.

    Google Scholar 

  98. 98.

    Dejonckheere E, Mestdagh M, Houben M, Rutten I, Sels L, Kuppens P, et al. Complex affect dynamics add limited information to the prediction of psychological well-being. Nat Human Behav. 2019;3:478–91.

    Google Scholar 

  99. 99.

    Molenaar PC. Dynamic assessment and adaptive optimization of the psychotherapeutic process. Behav Assess. 1987;9:389–416.

    Google Scholar 

  100. 100.

    Henry TR, Robinaugh D, Fried EI. On the control of psychological networks. PsyArXiv, 2020. https://doi.org/10.31234/osf.io/7vpz2.

  101. 101.

    Bach DR, Dolan RJ. Knowing how much you don’t know: a neural organization of uncertainty estimates. Nat Rev Neurosci. 2012;13:572–86.

    CAS  Google Scholar 

  102. 102.

    Doya K, Ishii S, Pouget A, Rao R, editors. Bayesian brain: Probabilistic approaches to neural coding. Cambridge, MA: MIT Press; 2007.

    Google Scholar 

  103. 103.

    Pulcu E, Browning M. The misestimation of uncertainty in affective disorders. Trends Cogn Sci. 2019;23:865–75.

    Google Scholar 

  104. 104.

    Houlsby NMT, Huszár F, Ghassemi MM, Orbán G, Wolpert DM, Lengyel M. Cognitive tomography reveals complex, task-independent mental representations. Curr Biol. 2013;23:2169–75.

    CAS  Google Scholar 

  105. 105.

    Huys QJM, Guitart-Masip M, Dolan RJ, Dayan P. Decision-theoretic psychiatry. Clin Psychol Sci. 2015b;3:400–21.

    Google Scholar 

  106. 106.

    Stankevicius A, Huys QJM, Kalra A, Seri’es P. Optimism as a prior belief about the probability of future reward. PLoS Comput Biol. 2014;10:e1003605.

    Google Scholar 

  107. 107.

    Rupprechter S, Stankevicius A, Huys QJM, Steele JD, Seri’es P. Major depression impairs the use of reward values for decision-making. Sci Rep. 2018;8:13798.

    Google Scholar 

  108. 108.

    Rupprechter S, Stankevicius A, Huys QJM, Series P, Steele JD. Abnormal reward valuation and event-related connectivity in unmedicated major depressive disorder. Psychol Med. 2020. Advance online publication.

  109. 109.

    Aylward J, Hales C, Robinson E, Robinson OJ. Translating a rodent measure of negative bias into humans: the impact of induced anxiety and unmedicated mood and anxiety disorders. Psychological Med. 2020;50:237–46.

    Google Scholar 

  110. 110.

    Kim M, Kim S, Lee K-U, and Jeong B. Pessimistically biased perception in panic disorder during risk learning. Depression Anxiety. 2020.

  111. 111.

    Gray J, Feldon J, Rawlins J, Hemsley D, Smith A. The neuropsychology of schizophrenia. Behav Brain Sci. 1991;14:1–20.

    Google Scholar 

  112. 112.

    Hemsley DR, Garety PA. The formation of maintenance of delusions: a bayesian analysis. Br J Psychiatry. 1986;149:51–6.

    CAS  Google Scholar 

  113. 113.

    Sterzer P, Adams RA, Fletcher P, Frith C, Lawrie SM, Muckli L, et al. The predictive coding account of psychosis. Biol Psychiatry. 2018;84:634–43.

    Google Scholar 

  114. 114.

    Karvelis P, Seitz AR, Lawrie SM, Seri’es P. Autistic traits, but not schizotypy, predict increased weighting of sensory information in bayesian visual integration. eLife. 2018;7. https://doi.org/10.7554/elife.34115.

  115. 115.

    Teufel C, Subramaniam N, Dobler V, Perez J, Finnemann J, Mehta PR, et al. Shift toward prior knowledge confers a perceptual advantage in early psychosis and psychosis-prone healthy individuals. Proc Natl Acad Sci. 2015;112:13401–6.

    CAS  Google Scholar 

  116. 116.

    Powers AR, Mathys C, Corlett PR. Pavlovian conditioning-induced hallucinations result from overweighting of perceptual priors. Science. 2017;357:596–600.

    CAS  Google Scholar 

  117. 117.

    Stuke H, Weilnhammer VA, Sterzer P, Schmack K. Delusion proneness is linked to a reduced usage of prior beliefs in perceptual decisions. Schizophrenia Bull. 2019;45:80–6.

    Google Scholar 

  118. 118.

    Schmack K, G’omez-Carrillo de Castro A, Rothkirch M, Sekutowicz M, Rössler H, Haynes J-D, et al. Delusions and the role of beliefs in perceptual inference. J Neurosci: Offcial J Soc Neurosci. 2013;33:13701–12.

    CAS  Google Scholar 

  119. 119.

    Schmack K, Schnack A, Priller J, Sterzer P. Perceptual instability in schizophrenia: probing predictive coding accounts of delusions with ambiguous stimuli. Schizophrenia Res Cognition. 2015;2:72–77.

    Google Scholar 

  120. 120.

    Nour MM, Dahoun T, Schwartenbeck P, Adams RA, FitzGerald THB, Coello C, et al. Dopaminergic basis for signaling belief updates, but not surprise, and the link to paranoia. Proc Natl Acad Sci. 2018;115:E10167–E10176.

    CAS  Google Scholar 

  121. 121.

    Ross RM, McKay R, Coltheart M, Langdon R. Jumping to conclusions about the beads task? a meta-analysis of delusional ideation and data-gathering. Schizophrenia Bull. 2015;41:1183–91.

    Google Scholar 

  122. 122.

    Baker SC, Konova AB, Daw ND, Horga G. A distinct inferential mechanism for delusions in schizophrenia. Brain. 2019;142:1797–812.

    Google Scholar 

  123. 123.

    Ermakova AO, Gileadi N, Knolle F, Justicia A, Anderson R, Fletcher PC, et al. Cost evaluation during decision-making in patients at early stages of psychosis. Comput Psychiatry. 2019;3:18–39.

    Google Scholar 

  124. 124.

    Moutoussis M, Bentall RP, El-Deredy W, Dayan P. Bayesian modelling of jumping-toconclusions bias in delusional patients. Cogn Neuropsychiatry. 2011;16:422–47.

    Google Scholar 

  125. 125.

    Rescorla R and Wagner A. A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In Black, A and Prokasy, W, editors, Classiacal Conditioning II:Current research and theory. New York :Appleton-Centuary-Crofts; 1972. p. 64–99.

  126. 126.

    Roweis S, Ghahramani Z. A unifying review of linear gaussian models. Neural Comput. 1999;11:305–45.

    CAS  Google Scholar 

  127. 127.

    Behrens TEJ, Woolrich MW, Walton ME, Rushworth MFS. Learning the value of information in an uncertain world. Nat Neurosci. 2007;10:1214–21.

    CAS  Google Scholar 

  128. 128.

    Mathys CD, Lomakina EI, Daunizeau J, Iglesias S, Brodersen KH, Friston KJ, et al. Uncertainty in perception and the Hierarchical Gaussian Filter. Frontiers in Human. Front Human Neurosci. 2014;8:825.

    Google Scholar 

  129. 129.

    McGuire JT, Nassar MR, Gold JI, Kable JW. Functionally dissociable influences on learning rate in a dynamic environment. Neuron. 2014;84:870–81.

    CAS  Google Scholar 

  130. 130.

    Nassar MR, Bruckner R, Frank MJ. Statistical context dictates the relationship between feedback-related EEG signals and learning. eLife. 2019;8. https://doi.org/10.7554/eLife.46975.

  131. 131.

    Nassar MR, Bruckner R, Gold JI, Li S-C, Heekeren HR, Eppinger B. Age differences in learning emerge from an insu cient representation of uncertainty in older adults. Nat Commun. 2016;7:11609.

    CAS  Google Scholar 

  132. 132.

    Nassar MR, Wilson RC, Heasly B, Gold JI. An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment. J Neurosci: Offcial J Soc Neurosci. 2010;30:12366–78.

    CAS  Google Scholar 

  133. 133.

    Yu AJ, Dayan P. Uncertainty, neuromodulation, and attention. Neuron. 2005;46:681–92.

    CAS  Google Scholar 

  134. 134.

    Browning M, Behrens TE, Jocham G, O’Reilly JX, Bishop SJ. Anxious individuals have di culty learning the causal statistics of aversive environments. Nat Neurosci. 2015;18:590–6.

    CAS  Google Scholar 

  135. 135.

    Huang H, Thompson W, Paulus MP. Computational dysfunctions in anxiety: failure to differentiate signal from noise. Biol Psychiatry. 2017;82:440–6.

    Google Scholar 

  136. 136.

    Aylward J, Valton V, Ahn W-Y, Bond RL, Dayan P, Roiser JP et al. Altered learning under uncertainty in unmedicated mood and anxiety disorders. Nat. Hum. Behav. 2019.

  137. 137.

    Lamba A, Frank MJ, and FeldmanHall O (2020). Anxiety impedes adaptive social learning under uncertainty. Psychol Sci. 2020; 32343637.

  138. 138.

    Lawson RP, Mathys C, Rees G. Adults with autism overestimate the volatility of the sensory environment. Nat Neurosci. 2017;20:1293–9.

    CAS  Google Scholar 

  139. 139.

    Hernaus D, Xu Z, Brown EC, Ruiz R, Frank MJ, Gold JM, et al. Motivational deficits in schizophrenia relate to abnormalities in cortical learning rate signals. Cogn Affective Behav Neurosci. 2018b;18:1338–51.

    CAS  Google Scholar 

  140. 140.

    Pulcu E and Browning M. Affective bias as a rational response to the statistics of rewards and punishments. eLife. 2017.

  141. 141.

    Korn CW, Sharot T, Walter H, Heekeren HR, Dolan RJ. Depression is related to an absence of optimistically biased belief updating about future life events. Psychol Med. 2014;44:579–92.

    CAS  Google Scholar 

  142. 142.

    Mathews A, MacLeod C. Cognitive vulnerability to emotional disorders. Annu Rev Clin Psychol. 2005;1:167–95.

    Google Scholar 

  143. 143.

    Rouhani N, Niv Y. Depressive symptoms bias the prediction-error enhancement of memory towards negative events in reinforcement learning. Psychopharmacology. 2019;236:2425–35.

    CAS  Google Scholar 

  144. 144.

    Gershman SJ, Niv Y. Novelty and Inductive Generalization in Human Reinforcement Learning. Top Cogn Sci. 2015;7:391–415.

    Google Scholar 

  145. 145.

    Gittins J, Kevin G, and Richard W. Multi-armed Bandit Allocation Indices. 2nd ed. Hoboken, New Jersey: Wiley; 2011. Library Catalog: www.wiley.com.

  146. 146.

    Schulz E, Gershman SJ. The algorithmic architecture of exploration in the human brain. Curr Opin Neurobiol. 2019;55:7–14.

    CAS  Google Scholar 

  147. 147.

    Charpentier CJ, Aylward J, Roiser JP, Robinson OJ. Enhanced risk aversion, but not loss aversion, in unmedicated pathological anxiety. Biol Psychiatry. 2017;81:1014–22.

    Google Scholar 

  148. 148.

    Konova AB, Lopez-Guzman S, Urmanche A, Ross S, Louie K, Rotrosen J et al. Computational markers of risky decision-making for identification of temporal windows of vulnerability to opioid use in a real-world clinical setting. JAMA Psychiatry. 2019;77:368–77.

    Google Scholar 

  149. 149.

    Sutton RS, Barto AG. Reinforcement learning: an introduction. 2nd edn. Cambridge, MA: MIT Press; 2017.

    Google Scholar 

  150. 150.

    Schultz W, Dayan P, Montague PR. A neural substrate of prediction and reward. Science. 1997;275:1593–9.

    CAS  Google Scholar 

  151. 151.

    Eshel N, Roiser JP. Reward and punishment processing in depression. Biol Psychiatry. 2010;68:118–24.

    Google Scholar 

  152. 152.

    Pizzagalli DA, Iosifescu D, Hallett LA, Ratner KG, Fava M. Reduced hedonic capacity in major depressive disorder: evidence from a probabilistic reward task. J Psychiatr Res. 2008;43:76–87.

    Google Scholar 

  153. 153.

    Pizzagalli DA, Jahn AL, O’Shea JP. Toward an objective characterization of an anhedonic phenotype: a signal-detection approach. Biol Psychiatry. 2005;57:319–27.

    Google Scholar 

  154. 154.

    Chase HW, Frank MJ, Michael A, Bullmore ET, Sahakian BJ, Robbins TW. Approach and avoidance learning in patients with major depression and healthy controls: relation to anhedonia. Psychol Med. 2009;40:433–40.

    Google Scholar 

  155. 155.

    Kumar P, Waiter G, Ahearn T, Milders M, Reid I, Steele JD. Abnormal temporal difference reward-learning signals in major depression. Brain. 2008;131(Pt 8):2084–93.

    CAS  Google Scholar 

  156. 156.

    Must A, Szabo Z, Bodi N, Szasz A, Janka Z, Keri S. Sensitivity to reward and punishment and the prefrontal cortex in major depression. J Affective Disord. 2006;90:209–15.

    Google Scholar 

  157. 157.

    Cavanagh JF, Bismark AW, Frank MJ, Allen JJB. Multiple dissociations between comorbid depression and anxiety on reward and punishment processing: Evidence from computationally informed EEG. Computational Psychiatry. 2019;3:1–17.

    Google Scholar 

  158. 158.

    Huys QJM, Pizzagalli DA, Bogdan R, Dayan P. Mapping anhedonia onto reinforcement learning: a behavioural meta-analysis. Biol Mood Anxiety Disord. 2013;3:12.

    Google Scholar 

  159. 159.

    Lawlor VM, Webb CA, Wiecki TV, Frank MJ, Trivedi M, Pizzagalli DA et al. Dissecting the impact of depression on decision-making. Psychol Med. 2019;1–10.

  160. 160.

    Webb CA, Dillon DG, Pechtel P, Goer FK, Murray L, Huys QJM, et al. Neural correlates of three promising endophenotypes of depression: evidence from the embarc study. Neuropsychopharmacology. 2016;41:454–63.

    CAS  Google Scholar 

  161. 161.

    Linke JO, Koppe G, Scholz V, Kanske P, Durstewitz D, Wessa M. Aberrant probabilistic reinforcement learning in first-degree relatives of individuals with bipolar disorder. J Affective Disord. 2020;264:400–6.

    Google Scholar 

  162. 162.

    Bylsma LM, Morris BH, Rottenberg J. A meta-analysis of emotional reactivity in major depressive disorder. Clin Psychol Rev. 2008;28:676–91.

    Google Scholar 

  163. 163.

    Huys QJM, Dayan P, Daw. Depression: a decision-theoretic account. Ann Rev Neurosci 2015a;38:1–23.

    CAS  Google Scholar 

  164. 164.

    Rutledge RB, Moutoussis M, Smittenaar P, Zeidman P, Taylor T, Hrynkiewicz L, et al. Association of neural and emotional impacts of reward prediction errors with major depression. JAMA Psychiatry. 2017;74:790–7.

    Google Scholar 

  165. 165.

    Eldar E, Rutledge RB, Dolan RJ, Niv Y. Mood as representation of momentum. Trends Cogn Sci. 2016;20:15–24.

    Google Scholar 

  166. 166.

    Konova AB, Louie K, Glimcher PW. The computational form of craving is a selective multiplication of economic value. Proc Natl Acad Sci. 2018;115:4122–7.

    CAS  Google Scholar 

  167. 167.

    Neville V, Dayan P, Gilchrist ID, Paul ES, Mendl M. Dissecting the links between reward and loss, decision-making, and self-reported affect using a computational approach. PsyArXiv. 2020. https://doi.org/10.31234/osf.io/ndc7h.

  168. 168.

    Eldar E, Niv Y. Interaction between emotional state and learning underlies mood instability. Nat Commun. 2015;6:6149.

    CAS  Google Scholar 

  169. 169.

    Mason L, Eldar E, Rutledge RB. Mood instability and reward dysregulation—a neurocomputational model of bipolar disorder. JAMA Psychiatry. 2017;74:1275.

    Google Scholar 

  170. 170.

    Salamone JD, Pardo M, Yohn SE, López-Cruz L, SanMiguel N, Correa M. Mesolimbic dopamine and the regulation of motivated behavior. Curr Top Behav Neurosci. 2016;27:231–57.

    Google Scholar 

  171. 171.

    Collins AGE, Frank MJ. Opponent actor learning (opal): modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive. Psychol Rev. 2014;121:337–66.

    Google Scholar 

  172. 172.

    Niv Y, Daw ND, Joel D, Dayan P. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacol (Berl). 2007;191:507–20.

    CAS  Google Scholar 

  173. 173.

    Westbrook A, Braver TS. Dopamine does double duty in motivating cognitive effort. Neuron. 2016;91:708.

    CAS  Google Scholar 

  174. 174.

    Westbrook JA, van den Bosch R, Maatta JI, Hofmans L, Papadopetraki D, Cools R, et al. Dopamine promotes cognitive effort by biasing the benefits versus costs of cognitive work. *co-senior authors. Science. 2020;367:1362–6.

    CAS  Google Scholar 

  175. 175.

    Berwian IM, Wenzel JG, Collins AGE, Seifritz E, Stephan KE, Walter H et al. Computational mechanisms of effort and reward decisions in patients with depression and their association with relapse after antidepressant discontinuation. JAMA Psychiatry. 2020.

  176. 176.

    Gold JM, Waltz JW, Frank MJ. Effort cost computation in schizophrenia: a commentary on the recent literature. Biol Psychiatry. 2015;78:747–53.

    Google Scholar 

  177. 177.

    Treadway MT, Bossaller NA, Shelton RC, Zald DH. Effort-based decision-making in major depressive disorder: a translational model of motivational anhedonia. J Abnorm Psychol. 2012;121:553–8.

    Google Scholar 

  178. 178.

    Meyniel F, Goodwin GM, Deakin JW, Klinge C, MacFadyen C, Milligan H, et al. A specific role for serotonin in overcoming effort cost. eLife. 2016;5.

  179. 179.

    Patzelt EH, Kool W, Millner AJ, Gershman SJ. Incentives boost model-based control across a range of severity on several psychiatric constructs. Biol Psychiatry. 2019a;85:425–33.

    Google Scholar 

  180. 180.

    Patzelt EH, Kool W, Millner AJ, Gershman SJ. The transdiagnostic structure of mental effort avoidance. Sci Rep. 2019b;9:1689.

    Google Scholar 

  181. 181.

    Maier S, Seligman M. Learned helplessness: theory and evidence. J Exp Psychol: Gen. 1976;105:3–46.

    Google Scholar 

  182. 182.

    Huys QJM, Dayan P. A Bayesian formulation of behavioral control. Cognition. 2009;113:314–28.

    Google Scholar 

  183. 183.

    O’Reilly RC, Rudy JW. Conjunctive representations in learning and memory: principles of cortical and hippocampal function. Psychol Rev. 2001;108:311–45.

    Google Scholar 

  184. 184.

    Davidow JY, Foerde K, Galván A, Shohamy D. An upside to reward sensitivity: the hippocampus supports enhanced reinforcement learning in adolescence. Neuron. 2016;92:93–9.

    CAS  Google Scholar 

  185. 185.

    Jang AI, Nassar MN, Dillon DG, Frank MJ. Positive reward prediction errors during decision making strengthen memory encoding. Nat Hum Behav. 2019;3:719–32.

    Google Scholar 

  186. 186.

    Dabney W, Kurth-Nelson Z, Uchida N, Starkweather CK, Hassabis D, Munos R, et al. A distributional code for value in dopamine-based reinforcement learning. Nature. 2020;577:671–5.

    CAS  Google Scholar 

  187. 187.

    Ehlers A, Clark DM. A cognitive model of posttraumatic stress disorder. Behav Res Ther. 2000;38:319–45.

    CAS  Google Scholar 

  188. 188.

    Gershman SJ, Daw ND. Reinforcement learning and episodic memory in humans and animals: an integrative framework. Annu Rev Psychol. 2017;68:101–28.

    Google Scholar 

  189. 189.

    Bornstein AM, Norman KA. Reinstated episodic context guides sampling-based decisions for reward. Nat Neurosci. 2017;20:997–1003.

    CAS  Google Scholar 

  190. 190.

    Mattar MG, Daw ND. Prioritized memory access explains planning and hippocampal replay. Nat Neurosci. 2018;21:1609–17.

    CAS  Google Scholar 

  191. 191.

    Scimeca JM, Badre D. Striatal contributions to declarative memory retrieval. Neuron. 2012;75:380–92.

    CAS  Google Scholar 

  192. 192.

    Scimeca JM, Katzman PL, Badre D. Striatal prediction errors support dynamic control of declarative memory decisions. Nat Commun. 2016;7:1–15.

    Google Scholar 

  193. 193.

    Miller EK, Cohen JD. An integrative theory of prefrontal cortex function. Annu Rev Neurosci. 2001;24:167–202.

    CAS  Google Scholar 

  194. 194.

    Collins AGE, Frank MJ. How much of reinforcement learning is working memory, not reinforcement learning? a behavioral, computational, and neurogenetic analysis. Eur J Neurosci. 2012;35:1024–35.

    Google Scholar 

  195. 195.

    Daw ND, Niv Y, Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci. 2005;8:1704–11.

    CAS  Google Scholar 

  196. 196.

    Schlagenhauf F, Huys QJM, Deserno L, Rapp MA, Beck A, Heinze H-J, et al. Striatal dysfunction during reversal learning in unmedicated schizophrenia patients. Neuroimage. 2014;89:171–80.

    Google Scholar 

  197. 197.

    Waltz JA, Gold JM. Probabilistic reversal learning impairments in schizophrenia: further evidence of orbitofrontal dysfunction. Schizophr Res. 2007;93:296–303.

    Google Scholar 

  198. 198.

    Collins AGE, Albrecht MA, Waltz JA, Gold JM, Frank MJ. Interactions among working memory, reinforcement learning, and effort in value-based choice: A new paradigm and selective deficits in schizophrenia. Biol psychiatry. 2017a;82:431–9.

    Google Scholar 

  199. 199.

    Collins AGE, Brown. J, Gold J, Waltz J, Frank MJ. Working memory contributions to reinforcement learning in schizophrenia. J Neurosci. 2014;34:13747–56.

    CAS  Google Scholar 

  200. 200.

    Dowd EC, Frank MJ, Collins AGE, Gold JM, Barch DM. Probabilistic reinforcement learning in patients with schizophrenia: Relationships to anhedonia and avolition. Biol Psychiatry: Cogn Neurosci Neuroimaging. 2016;1:460–73.

    Google Scholar 

  201. 201.

    Collins AGE, Ciullo B, Frank MJ, Badre D. Working memory load strengthens reward prediction errors. J Neurosci. 2017b;37:4332–42.

    CAS  Google Scholar 

  202. 202.

    Collins AGE, Frank MJ. Within- and across-trial dynamics of human eeg reveal cooperative interplay between reinforcement learning and working memory. Proc Natl Acad Sci. 2018;115:2502–7.

    CAS  Google Scholar 

  203. 203.

    Collins AGE, Frank MJ. Neural signature of hierarchically structured expectations predicts clustering and transfer of rule sets in reinforcement learning. Cognition. 2016;152:160–9.

    Google Scholar 

  204. 204.

    Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ. Model-based influences on humans’ choices and striatal prediction errors. Neuron. 2011;69:1204–15.

    CAS  Google Scholar 

  205. 205.

    Hernaus D, Gold JM, Waltz JA, Frank MJ. Impaired expected value computations coupled with overreliance on stimulus-response learning in schizophrenia. Biol Psychiatry: Cogn Neurosci neuroimaging. 2018a;3:916–26.

    Google Scholar 

  206. 206.

    Schwabe L, Wolf OT. Stress prompts habit behavior in humans. J Neurosci. 2009;29:7191–8.

    CAS  Google Scholar 

  207. 207.

    Lloyd K, Becker N, Jones M, Bogacz R. Learning to use working memory: a reinforcement learning gating model of rule acquisition in rats. Front Comput Neurosci. 2012;6:87.

    Google Scholar 

  208. 208.

    O’Reilly RC, Frank MJ. Making working memory work: a computational model of learning in the frontal cortex and basal ganglia. Neural Comput. 2006;18:283–328.

    Google Scholar 

  209. 209.

    Nassar MR, Helmers J, Frank MJ. Chunking as a rational strategy for lossy data compression in visual working memory. Psychol Rev. 2018;125:486–511.

    Google Scholar 

  210. 210.

    Lieder F, Gri ths TL. Resource-rational analysis: understanding human cognition as the optimal use of limited computational resources. Behav Brain Sci. 2020;43:e1.

    Google Scholar 

  211. 211.

    Cools R. Chemistry of the adaptive mind: lessons from dopamine. Neuron. 2019;104:113–31.

    CAS  Google Scholar 

  212. 212.

    Doll BB, Duncan KD, Simon DA, Shohamy D, Daw ND. Model-based choices involve prospective neural activity. Nat Neurosci. 2015;18:767–72.

    CAS  Google Scholar 

  213. 213.

    Huys QJM, Tobler PN, Hasler G, Flagel SB. The role of learning-related dopamine signals in addiction vulnerability. Prog Brain Res. 2014;211:31–77.

    CAS  Google Scholar 

  214. 214.

    McClure SM, Daw ND, Montague PR. A computational substrate for incentive salience. TINS. 2003;26:423–8.

    CAS  Google Scholar 

  215. 215.

    Schad DJ, Rapp MA, Garbusow M, Nebe S, Sebold M, Obst E, et al. Dissociating neural learning signals in human sign- and goal-trackers. Nat Hum Behav. 2020;4:201–14.

    Google Scholar 

  216. 216.

    Robbins TW, Gillan CM, Smith DG, de Wit S, Ersche KD. Neurocognitive endophenotypes of impulsivity and compulsivity: towards dimensional psychiatry. Trends Cogn Sci. 2012;16:81–91.

    Google Scholar 

  217. 217.

    Gillan CM, Apergis-Schoute AM, Morein-Zamir S, Urcelay GP, Sule A, Fineberg NA, et al. Functional neuroimaging of avoidance habits in obsessivecompulsive disorder. Am J Psychiatry. 2015;172:284–93.

    Google Scholar 

  218. 218.

    Gillan CM, Morein-Zamir S, Urcelay GP, Sule A, Voon V, Apergis-Schoute AM, et al. Enhanced avoidance habits in obsessive-compulsive disorder. Biol Psychiatry. 2014;75:631–8.

    Google Scholar 

  219. 219.

    Gillan CM, Papmeyer M, Morein-Zamir S, Sahakian BJ, Fineberg NA, Robbins TW, et al. Disruption in the balance between goal-directed behavior and habit learning in obsessivecompulsive disorder. Am J Psychiatry. 2011;168:718–26.

    Google Scholar 

  220. 220.

    Voon V, Derbyshire K, Rück C, Irvine MA, Worbe Y, Enander J, et al. Disorders of compulsivity: a common bias towards learning habits. Mol Psychiatry. 2015;20:345–52.

    CAS  Google Scholar 

  221. 221.

    Ziegler G, Hauser TU, Moutoussis M, Bullmore ET, Goodyer IM, Fonagy P, et al. Compulsivity and impulsivity traits linked to attenuated developmental frontostriatal myelination trajectories. Nat Neurosci. 2019;22:992–9.

    CAS  Google Scholar 

  222. 222.

    Culbreth AJ, Westbrook A, Daw ND, Botvinick M, Barch DM. Reduced model-based decision-making in schizophrenia. J Abnorm Psychol. 2016;125:777–87.

    Google Scholar 

  223. 223.

    Nebe S, Kroemer NB, Schad DJ, Bernhardt N, Sebold M, Müller DK, et al. No association of goal-directed and habitual control with alcohol consumption in young adults. Addiction Biol. 2018;23:379–93.

    Google Scholar 

  224. 224.

    Gillan CM, Kalanthroff E, Evans M, Weingarden HM, Jacoby RJ, Gershkovich M, et al. Comparison of the association between goal-directed planning and self-reported compulsivity vs obsessive-compulsive disorder diagnosis. JAMA Psychiatry. 2019;1–10.

  225. 225.

    Gillan CM, Kosinski M, Whelan R, Phelps EA, Daw ND. Characterizing a psychiatric symptom dimension related to deficits in goal-directed control. Elife. 2016;5.

  226. 226.

    Rouault M, Seow T, Gillan CM, Fleming SM. Psychiatric symptom dimensions are associated with dissociable shifts in metacognition but not task performance. Biol Psychiatry. 2018;84:443–51.

    Google Scholar 

  227. 227.

    Wheaton MG, Gillan CM, Simpson HB. Does cognitive-behavioral therapy affect goal-directed planning in obsessive-compulsive disorder? Psychiatry Res. 2019;273:94–99.

    Google Scholar 

  228. 228.

    Otto AR, Gershman SJ, Markman AB, Daw ND. The curse of planning: dissecting multiple reinforcement-learning systems by taxing the central executive. Psychol Sci. 2013a;24:751–61.

    Google Scholar 

  229. 229.

    Otto AR, Raio CM, Chiang A, Phelps EA, Daw ND. Working-memory capacity protects model-based learning from stress. Proc Natl Acad Sci USA. 2013b;110:20941–6.

    CAS  Google Scholar 

  230. 230.

    Schad DJ, Jünger E, Sebold M, Garbusow M, Bernhardt N, Javadi AH, et al. Processing speed enhances model-based over model-free reinforcement learning in the presence of high working memory functioning. Front Psychol. 2014;5:1450.

    Google Scholar 

  231. 231.

    Huys QJM, Eshel N, O’Nions E, Sheridan L, Dayan P, Roiser JP. Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Comput Biol. 2012;8:e1002410.

    CAS  Google Scholar 

  232. 232.

    Huys QJM, Lally N, Faulkner P, Eshel N, Seifritz E, Gershman SJ, et al. Interplay of approximate planning strategies. Proc Natl Acad Sci USA. 2015c;112:3098–103.

    CAS  Google Scholar 

  233. 233.

    Lally N, Huys QJM, Eshel N, Faulkner P, Dayan P, Roiser JP. The neural basis of aversive pavlovian guidance during planning. J Neurosci: Offcial J Soc Neurosci. 2017;37:10215–29.

    CAS  Google Scholar 

  234. 234.

    Huys QJM, Renz D. A formal valuation framework for emotions and their control. Biol Psychiatry. 2017;82:413–20.

    Google Scholar 

  235. 235.

    Amlung M, Marsden E, Holshausen K, Morris V, Patel H, Vedelago L, et al. Delay discounting as a transdiagnostic process in psychiatric disorders. JAMA Psychiatry. 2019;76:1176.

    Google Scholar 

  236. 236.

    Story GW, Moutoussis M, and Dolan RJ. A computational analysis of aberrant delay discounting in psychiatric disorders. Front Psychol. 2016;6.

  237. 237.

    Hakimi S, Hare TA. Enhanced neural responses to imagined primary rewards predict reduced monetary temporal discounting. J Neurosci. 2015;35:13103–9.

    CAS  Google Scholar 

  238. 238.

    Kurth-Nelson Z, Bickel W, Redish AD. A theoretical account of cognitive effects in delay discounting. Eur J Neurosci. 2012;35:1052–64.

    Google Scholar 

  239. 239.

    Zorowitz S, Momennejad I, Daw ND. Anxiety, avoidance, and sequential evaluation. Comput Psychiatry. 2020;4:1–17.

    Google Scholar 

  240. 240.

    Korn CW, Bach DR. Minimizing threat via heuristic and optimal policies recruits hippocampus and medial prefrontal cortex. Nat Hum Behav. 2019;3:733–45.

    Google Scholar 

  241. 241.

    Hauser TU, Allen M, Purg N, Moutoussis M, Rees G, and Dolan RJ. Noradrenaline blockade specifically enhances metacognitive performance. eLife. 2017;6.

  242. 242.

    Daw ND and Dayan P. The algorithmic anatomy of model-based evaluation. Philos Trans R Soc Lond B Biol Sci. 2014; 369(1655).

  243. 243.

    Deserno L and Hauser TU. Beyond a cognitive dichotomy: can multiple decision systems prove useful to distinguish compulsive and impulsive symptom dimensions? Biological Psychiatry. 2020.

  244. 244.

    Dezfouli A, Balleine BW. Habits, action sequences and reinforcement learning. Eur J Neurosci. 2012;35:1036–51.

    Google Scholar 

  245. 245.

    Dezfouli A, Balleine BW. Actions, action sequences and habits: evidence that goal-directed and habitual action control are hierarchically organized. PLoS Comput Biol. 2013;9:e1003364.

    Google Scholar 

  246. 246.

    Shahar N, Moran R, Hauser TU, Kievit RA, McNamee D, Moutoussis M, et al. Credit assignment to state-independent task representations and its relationship with model-based decision making. Proc Natl Acad Sci. 2019b;116:15871–6.

    CAS  Google Scholar 

  247. 247.

    Rajendran G, Mitchell P. Cognitive theories of autism. Dev Rev. 2007;27:224–60.

    Google Scholar 

  248. 248.

    Franklin NT, Frank MJ. Compositional clustering in task structure learning. PLOS Comput Biol. 2018;14:e1006116.

    Google Scholar 

  249. 249.

    Wingate D, Diuk C, Donnell T, Tenenbaum J, Gershman S. Compositional policy priors. MIT CSAIL Technical Report 2013-007. 2013.

  250. 250.

    Franklin NT, Frank MJ. Generalizing to generalize: humans flexibly switch between compositional and conjunctive structures during reinforcement learning. PLoS Comput. Biol. 2020;16:e1007720.

    CAS  Google Scholar 

  251. 251.

    Behrens TE, Muller TH, Whittington JC, Mark S, Baram AB, Stachenfeld KL, et al. What is a cognitive map? organizing knowledge for flexible behavior. Neuron. 2018;100:490–509.

    CAS  Google Scholar 

  252. 252.

    Whittington JC, Muller TH, Mark S, Chen G, Barry C, Burgess N, et al. The Tolman-Eichenbaum machine: unifying space and relational memory through generalisation in the hippocampal formation. 2019. https://www.biorxiv.org/content/10.1101/770495v1.

  253. 253.

    Barreto A, Dabney W, Munos R, Hunt JJ, Schaul T, van Hasselt HP. et al. Successor Features for Transfer in Reinforcement Learning. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R, (eds). Advances in Neural Information Processing Systems 30, Curran Associates, Inc.; 2017. p. 4055–65.

  254. 254.

    Lehnert L and Littman ML. Successor features combine elements of model-free and model-based reinforcement learning. arXiv. 2019. doi: 1901.11437.

  255. 255.

    Lehnert L, Littman ML, Frank MJ. Reward-predictive representations generalize across tasks in reinforcement learning. bioRxiv. 2020. https://doi.org/10.1101/653493v2.

    Article  Google Scholar 

  256. 256.

    Momennejad I, Russek EM, Cheong JH, Botvinick MM, Daw ND, Gershman SJ. The successor representation in human reinforcement learning. Nat Hum Behav. 2017;1:680–92.

    CAS  Google Scholar 

  257. 257.

    Russek EM, Momennejad I, Botvinick MM, Gershman SJ, Daw ND. Predictive representations can link model-based reinforcement learning to model-free mechanisms. PLoS Comput Biol. 2017;13:e1005768.

    Google Scholar 

  258. 258.

    Stachenfeld KL, Botvinick MM, Gershman SJ. The hippocampus as a predictive map. Nat Neurosci. 2017;20:1643.

    CAS  Google Scholar 

  259. 259.

    Guitart-Masip M, Huys QJM, Fuentemilla L, Dayan P, Duzel E, Dolan RJ. Go and no-go learning in reward and punishment: interactions between affect and effect. Neuroimage. 2012;62:154–66.

    Google Scholar 

  260. 260.

    Huys QJM, Cools R, Gölzer M, Friedel E, Heinz A, Dolan RJ, Dayan P. Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding. PLoS Comput Biol. 2011;7:e1002028.

    CAS  Google Scholar 

  261. 261.

    Boureau Y-L, Dayan P. Opponency revisited: competition and cooperation between dopamine and serotonin. Neuropsychopharmacology. 2011;36:74–97.

    CAS  Google Scholar 

  262. 262.

    Dayan P, Niv Y, Seymour B, Daw ND. The misbehavior of value and the discipline of the will. Neural Netw. 2006;19:1153–60.

    Google Scholar 

  263. 263.

    Cartoni E, Puglisi-Allegra S, Baldassarre G. The three principles of action: a pavlovianinstrumental transfer hypothesis. Front Behav Neurosci. 2013;7:153.

    Google Scholar 

  264. 264.

    Dorfman HM and Gershman SJ. Controllability governs the balance between pavlovian and instrumental action selection. Nat Commun. 2019;10.

  265. 265.

    Swart JC, Froböse MI, Cook JL, Geurts DE, Frank MJ, Cools R et al. Catecholaminergic challenge uncovers distinct pavlovian and instrumental mechanisms of motivated (in)action. eLife. 2017;6.

  266. 266.

    Garbusow M, Nebe S, Sommer C, Kuitunen-Paul S, Sebold M, Schad DJ, et al. Pavlovian-to-instrumental transfer and alcohol consumption in young male social drinkers: behavioral, neural and polygenic correlates. J Clin Med. 2019;8. https://doi.org/10.3390/jcm8081188.

  267. 267.

    Millner AJ, den Ouden HEM, Gershman SJ, Glenn CR, Kearns JC, Bornstein AM, et al. Suicidal thoughts and behaviors are associated with an increased decision-making bias for active responses to escape aversive states. J Abnorm Psychol. 2019;128:106–18.

    Google Scholar 

  268. 268.

    Millner AJ, Gershman SJ, Nock MK, den Ouden HEM. Pavlovian control of escape and avoidance. J Cogn Neurosci. 2018;30:1379–90.

    Google Scholar 

  269. 269.

    Mkrtchian A, Aylward J, Dayan P, Roiser JP, Robinson OJ. Modeling avoidance in mood and anxiety disorders using reinforcement learning. Biol Psychiatry. 2017;82:532–9.

    Google Scholar 

  270. 270.

    Hall LS, Adams MJ, Arnau-Soler A, Clarke TK, Howard DM, Zeng Y, et al. Genome-wide meta-analyses of stratified depression in generation scotland and uk biobank. Transl Psychiatry. 2018;8:9.

    Google Scholar 

  271. 271.

    Smith DJ, Escott-Price V, Davies G, Bailey ME, Colodro-Conde L, Ward J, et al. Genome-wide analysis of over 106 000 individuals identifies 9 neuroticism-associated loci. Mol Psychiatry. 2016;21:749–57.

    CAS  Google Scholar 

  272. 272.

    Wolfers T, Beckmann CF, Hoogman M, Buitelaar JK, Franke B, and Marquand AF. Individual differences v. the average patient: mapping the heterogeneity in adhd using normative models. Psychol Med. 2019; 1–10.

  273. 273.

    Wolfers T, Doan NT, Kaufmann T, Alnaes D, Moberget T, Agartz I, et al. Mapping the heterogeneous phenotype of schizophrenia and bipolar disorder using normative models. JAMA Psychiatry. 2018;75:1146–55.

    Google Scholar 

  274. 274.

    Wainschtein P, Jain DP, Yengo L, Zheng Z, Cupples LA, Shadyab AH, et al. Recovery of trait heritability from whole genome sequence data. 2019, https://www.biorxiv.org/content/10.1101/588020v1.

  275. 275.

    Ross CA, Aylward EH, Wild EJ, Langbehn DR, Long JD, Warner JH, et al. Huntington disease: natural history, biomarkers and prospects for therapeutics. Nat Rev Neurol. 2014;10:204–16.

    CAS  Google Scholar 

  276. 276.

    Barch DM, Carter CS, Committee CE. Measurement issues in the use of cognitive neuroscience tasks in drug development for impaired cognition in schizophrenia: a report of the second consensus building conference of the cntrics initiative. Schiz Bull 2008;34:613–8.

    Google Scholar 

  277. 277.

    Gignac GE, Szodorai ET. Effect size guidelines for individual differences researchers. Personal Individ Differences. 2016;102:74–8.

    Google Scholar 

  278. 278.

    Savitz JB, Rauch SL, Drevets WC. Clinical application of brain imaging for the diagnosis of mood disorders: the current state of play. Mol Psychiatry. 2013;18:528–39.

    CAS  Google Scholar 

  279. 279.

    Wager TD, Atlas LY, Lindquist MA, Roy M, Woo C-W, Kross E. An fmri-based neurologic signature of physical pain. N. Engl J Med. 2013;368:1388–97.

    CAS  Google Scholar 

  280. 280.

    Hedge C, Powell G, Sumner P. The reliability paradox: why robust cognitive tasks do not produce reliable individual differences. Behav Res Methods. 2018;50:1166–86.

    Google Scholar 

  281. 281.

    Enkavi AZ, Eisenberg IW, Bissett PG, Mazza GL, MacKinnon DP, Marsch LA, et al. Large-scale analysis of test-retest reliabilities of self-regulation measures. Proc Nat Acad Sci USA. 2019;116:5472–7.

    CAS  Google Scholar 

  282. 282.

    Huys QJM. Computational cognitive methods for precision psychiatry. In Williams, L., editor, Neuroscience-informed precision psychiatry. APA; 2020.

  283. 283.

    Rouder JN, Haaf JM. A psychometrics of individual differences in experimental tasks. Psychonomic Bull Rev. 2019;26:452–67.

    Google Scholar 

  284. 284.

    Brown VM, Chen J, Gillan CM, and Price RB. Improving the reliability of computational analyses: model-based planning and its relationship with compulsivity. Biological Psychiatry CNNI. 2020.

  285. 285.

    Shahar N, Hauser TU, Moutoussis M, Moran R, Keramati M, consortium N, et al. Improving the reliability of model-based decision-making estimates in the two-stage decision task with reaction-times and drift-diffusion modeling. PLoS Comput Biol 2019a;15:e1006803.

    CAS  Google Scholar 

  286. 286.

    Paulus MP, Huys QJ, Maia TV. A roadmap for the development of applied computational psychiatry. Biol Psychiatry Cogn Neurosci Neuroimaging. 2016;1:386–92.

    Google Scholar 

  287. 287.

    Browning M, Carter CS, Chatham C, Den Ouden H, Gillan CM, Baker JT, et al. Realizing the Clinical Potential of Computational Psychiatry: Report From the Banbury Center Meeting, February 2019. Biol Psychiatry. 2020;88:e5-e10.

    Google Scholar 

  288. 288.

    Eisenberg IW, Bissett PG, Enkavi AZ, Li J, MacKinnon DP, Marsch LA, et al. Uncovering the structure of self-regulation through data-driven ontology discovery. Nat Commun. 2019;10.

Download references

Author information

Affiliations

Authors

Contributions

All authors reviewed literature and jointly wrote the paper.

Corresponding author

Correspondence to Quentin J. M. Huys.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Huys, Q.J.M., Browning, M., Paulus, M.P. et al. Advances in the computational understanding of mental illness. Neuropsychopharmacol. 46, 3–19 (2021). https://doi.org/10.1038/s41386-020-0746-4

Download citation

Further reading

Search

Quick links