Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Stress, genotype and norepinephrine in the prediction of mouse behavior using reinforcement learning

This article has been updated

Abstract

Individual behavioral performance during learning is known to be affected by modulatory factors, such as stress and motivation, and by genetic predispositions that influence sensitivity to these factors. Despite numerous studies, no integrative framework is available that could predict how a given animal would perform a certain learning task in a realistic situation. We found that a simple reinforcement learning model can predict mouse behavior in a hole-box conditioning task if model metaparameters are dynamically controlled on the basis of the mouse's genotype and phenotype, stress conditions, recent performance feedback and pharmacological manipulations of adrenergic alpha-2 receptors. We find that stress and motivation affect behavioral performance by altering the exploration-exploitation balance in a genotype-dependent manner. Our results also provide computational insights into how an inverted U–shape relation between stress/arousal/norepinephrine levels and behavioral performance could be explained through changes in task performance accuracy and future reward discounting.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Experiment and behavioral results.
Figure 2: The hole-box model and its simulations.
Figure 3: Behavioral performance measures can be reproduced and de-noised by the model.
Figure 4: Genetic strain, stress and norepinephrine manipulations influence daily estimated metaparameters.
Figure 5: Multilinear regression analyses and simulations of the trained ANN reveal interactions between modulatory factors and their effects on model metaparameters.

Similar content being viewed by others

Change history

  • 26 August 2009

    In the version of this article originally posted online, the range for β in the Online Methods section ‘Metaparameter estimation based on fit to behavioral data’ was given as [10–1.0, 10–1.5]. It should have been [10–1.0, 101.5]. The error has been corrected in the PDF and HTML versions of this article.

References

  1. Kabbaj, M., Devine, D.P., Savage, V.R. & Akil, H. Neurobiological correlates of individual differences in novelty-seeking behavior in the rat: differential expression of stress-related molecules. J. Neurosci. 20, 6983–6988 (2000).

    Article  CAS  Google Scholar 

  2. Joëls, M., Pu, Z., Wiegert, O., Oitzl, M.S. & Krugers, H.J. Learning under stress: how does it work? Trends Cogn. Sci. 10, 152–158 (2006).

    Article  Google Scholar 

  3. Sandi, C. & Pinelo-Nava, M.T. Stress and memory: behavioral effects and neurobiological mechanisms. Neural Plast. doi:10.1155/2007/78970 (15 April 2007).

  4. Sandi, C., Loscertales, M. & Guaza, C. Experience-dependent facilitating effect of corticosterone on spatial memory formation in the water maze. Eur. J. Neurosci. 9, 637–642 (1997).

    Article  CAS  Google Scholar 

  5. Kim, J.J. & Yoon, K.S. Stress: metaplastic effects in the hippocampus. Trends Neurosci. 21, 505–509 (1998).

    Article  CAS  Google Scholar 

  6. Kim, J.J. & Diamond, D.M. The stressed hippocampus, synaptic plasticity and lost memories. Nat. Rev. Neurosci. 3, 453–462 (2002).

    Article  CAS  Google Scholar 

  7. Sara, S.J The locus coeruleus and noradrenergic modulation of cognition. Nat. Rev. Neurosci. 10, 211–223 (2009).

    Article  CAS  Google Scholar 

  8. Holmes, A., Wrenn, C.C., Harris, A.P., Thayer, K.E. & Crawley, J.N. Behavioral profiles of inbred strains on novel olfactory, spatial and emotional tests for reference memory in mice. Genes Brain Behav. 1, 55–69 (2002).

    Article  CAS  Google Scholar 

  9. Herrero, A.I., Sandi, C. & Venero, C. Individual differences in anxiety trait are related to spatial learning abilities and hippocampal expression of mineralocorticoid receptors. Neurobiol. Learn. Mem. 86, 150–159 (2006).

    Article  CAS  Google Scholar 

  10. Dayan, P. & Balleine, B.W. Reward, motivation and reinforcement learning. Neuron 36, 285–298 (2002).

    Article  CAS  Google Scholar 

  11. Botvinick, M.M., Braver, T.S., Carter, C.S., Barch, D.M. & Cohen, J.D. Conflict monitoring and cognitive control. Psychol. Rev. 108, 624–652 (2001).

    Article  CAS  Google Scholar 

  12. Aston-Jones, G. & Cohen, J.D. An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 28, 403–450 (2005).

    Article  CAS  Google Scholar 

  13. Yu, A.J. & Dayan, P. Uncertainty, neuromodulation and attention. Neuron 46, 681–692 (2005).

    Article  CAS  Google Scholar 

  14. Sutton, R.S. & Barto, A.G. Reinforcement Learning: an Introduction (MIT Press, Cambridge, Massachusetts, USA, 1998).

    Google Scholar 

  15. Tanaka, S.C. et al. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat. Neurosci. 7, 887–893 (2004).

    Article  CAS  Google Scholar 

  16. Redish, A.D. Addiction as a computational process gone awry. Science 306, 1944–1947 (2004).

    Article  CAS  Google Scholar 

  17. Schultz, W., Dayan, P. & Montague, P.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

    Article  CAS  Google Scholar 

  18. Wickens, J.R., Begg, A.J. & Arbuthnott, G.W. Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex in vitro. Neuroscience 70, 1–5 (1996).

    Article  CAS  Google Scholar 

  19. Samejima, K., Ueda, Y., Doya, K. & Kimura, M. Representation of action-specific reward values in the striatum. Science 310, 1337–1340 (2005).

    Article  CAS  Google Scholar 

  20. Doya, K. Metalearning and neuromodulation. Neural Netw. 15, 495–506 (2002).

    Article  Google Scholar 

  21. Doya, K. Modulators of decision making. Nat. Neurosci. 11, 410–416 (2008).

    Article  CAS  Google Scholar 

  22. Bailey, C.H., Giustetto, M., Huang, Y.Y., Hawkins, R.D. & Kandel, E.R. Is heterosynaptic modulation essential for stabilizing Hebbian plasticity and memory? Nat. Rev. Neurosci. 1, 11–20 (2000).

    Article  CAS  Google Scholar 

  23. Moscarello, J.M., Ben-Shahar, O. & Ettenberg, A. Dynamic interaction between medial prefrontal cortex and nucleus accumbens as a function of both motivational state and reinforcer magnitude: a c-Fos immunocytochemistry study. Brain Res. 1169, 69–76 (2007).

    Article  CAS  Google Scholar 

  24. Behrens, T.E.J., Woolrich, M.W., Walton, M.E. & Rushworth, M.F.S. Learning the value of information in an uncertain world. Nat. Neurosci. 10, 1214–1221 (2007).

    Article  CAS  Google Scholar 

  25. Frank, M.J., Moustafa, A.A., Haughey, H.M., Curran, T. & Hutchison, K.E. Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc. Natl. Acad. Sci. USA 104, 16311–16316 (2007).

    Article  CAS  Google Scholar 

  26. Schweighofer, N. et al. Low serotonin levels increase delayed reward discounting in humans. J. Neurosci. 28, 4528–4532 (2008).

    Article  CAS  Google Scholar 

  27. Aston-Jones, G., Rajkowski, J. & Cohen, J. Locus coeruleus and regulation of behavioral flexibility and attention. Prog. Brain Res. 126, 165–182 (2000).

    Article  CAS  Google Scholar 

  28. Homayoun, H., Khavandgar, S. & Zarrindast, M.R. Morphine state-dependent learning: interactions with alpha2-adrenoceptors and acute stress. Behav. Pharmacol. 14, 41–48 (2003).

    Article  CAS  Google Scholar 

  29. Clement, Y. et al. Anxiety in mice: a principal component analysis study. Neural Plast, doi 10.1155/2007/35457 (21 March 2007).

  30. Pattij, T. & Vanderschuren, L.J. The neuropharmacology of impulsive behaviour. Trends Pharmacol. Sci. 29, 192–199 (2008).

    Article  CAS  Google Scholar 

  31. Ciaranello, R.D., Barchas, R., Kessler, S. & Barchas, J.D. Catecholamines: strain differences in biosynthetic enzyme activity in mice. Life Sci. I. 11, 565–572 (1972).

    Article  CAS  Google Scholar 

  32. David, D.J., Renard, C.E., Jolliet, P., Hascoet, M. & Bourin, M. Antidepressant-like effects in various mice strains in the forced swimming test. Psychopharmacology (Berl.) 166, 373–382 (2003).

    Article  CAS  Google Scholar 

  33. Amat, J. et al. Medial prefrontal cortex determines how stressor controllability affects behavior and dorsal raphe nucleus. Nat. Neurosci. 8, 365–371 (2005).

    Article  CAS  Google Scholar 

  34. Dalley, J.W. et al. Nucleus accumbens D2/3 receptors predict trait impulsivity and cocaine reinforcement. Science 315, 1267–1270 (2007).

    Article  CAS  Google Scholar 

  35. Cabib, S., Puglisi-Allegra, S. & Ventura, R. The contribution of comparative studies in inbred strains of mice to the understanding of the hyperactive phenotype. Behav. Brain Res. 130, 103–109 (2002).

    Article  CAS  Google Scholar 

  36. Briand, L.A., Gritton, H., Howe, W.M., Young, D.A. & Sarter, M. Modulators in concert for cognition: modulator interactions in the prefrontal cortex. Prog. Neurobiol. 83, 69–91 (2007).

    Article  CAS  Google Scholar 

  37. Luksys, G., Knuesel, J., Sheynikhovich, D., Sandi, C. & Gerstner, W. Effects of stress and genotype on metaparameter dynamics in reinforcement learning. Adv. Neural Inf. Process. Syst. 19, 937–944 (2007).

    Google Scholar 

  38. Samini, M., Kardan, A. & Mehr, S.E. Alpha-2 agonists decrease expression of morphine-induced conditioned place preference. Pharmacol. Biochem. Behav. 88, 403–406 (2008).

    Article  CAS  Google Scholar 

  39. Press, W.H., Flannery, B.P., Teukolsky, S.A. & Vetterling, W.T. Numerical Recipes in C: The Art of Scientific Computing (Cambridge University Press, Cambridge, UK, 1992).

    Google Scholar 

  40. Marquardt, D. An algorithm for least squares estimation of nonlinear parameters. SIAM J. Appl. Math. 11, 431–441 (1963).

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank C. Rossetti and E. Gantelet for their help with experiments and E. Vasilaki for her useful comments on the manuscript. This work was supported by a grant from the Swiss National Science Foundation to C.S. by funds from École Polytechnique Fédérale de Lausanne to W.G. and by a collaborative Swiss National Science Foundation Sinergia Project to W.G. and C.S.

Author information

Authors and Affiliations

Authors

Contributions

All three authors were involved in designing the study and writing the manuscript. G.L. performed the experiments, model simulations and data analyses. C.S. and W.G. supervised the project.

Corresponding author

Correspondence to Gediminas Luksys.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–10, Supplementary Tables 1–3, Supplementary Methods and Supplementary Discussion (PDF 1332 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Luksys, G., Gerstner, W. & Sandi, C. Stress, genotype and norepinephrine in the prediction of mouse behavior using reinforcement learning. Nat Neurosci 12, 1180–1186 (2009). https://doi.org/10.1038/nn.2374

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn.2374

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing