Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Dissociating neural learning signals in human sign- and goal-trackers

Abstract

Individuals differ in how they learn from experience. In Pavlovian conditioning models, where cues predict reinforcer delivery at a different goal location, some animals—called sign-trackers—come to approach the cue, whereas others, called goal-trackers, approach the goal. In sign-trackers, model-free phasic dopaminergic reward-prediction errors underlie learning, which renders stimuli ‘wanted’. Goal-trackers do not rely on dopamine for learning and are thought to use model-based learning. We demonstrate this double dissociation in 129 male humans using eye-tracking, pupillometry and functional magnetic resonance imaging informed by computational models of sign- and goal-tracking. We show that sign-trackers exhibit a neural reward prediction error signal that is not detectable in goal-trackers. Model-free value only guides gaze and pupil dilation in sign-trackers. Goal-trackers instead exhibit a stronger model-based neural state prediction error signal. This model-based construct determines gaze and pupil dilation more in goal-trackers.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Assessment of sign- and goal-trackers via eye-tracking.
Fig. 2: Pupil dilation during Pavlovian conditioning in sign-trackers and goal-trackers.
Fig. 3: PIT in sign-trackers versus goal-trackers.
Fig. 4: NAc BOLD response in sign-trackers versus goal-trackers.
Fig. 5: Neural appetitive RPE signals in sign-trackers versus goal-trackers.
Fig. 6: Neural SPE learning signals in sign-trackers versus goal-trackers.

Data availability

Source data are available for Figs. 1–6 and Supplementary Figs. 2–12. Data sharing will be based on acceptance by the study team that: (1) a valid and timely scientific question, based on a written protocol, has been posed by those seeking to access the data; and (2) the role of the original study team will be fully acknowledged. Please contact the corresponding author via email to request access to the data. Safeguarding of ethical standards will be ensured by submission of a study amendment to the Charité and Dresden ethics committees. Data access for questions of scientific integrity may additionally be regulated via the funder.

Code availability

Experimental code is freely available on request to the corresponding author. Analysis code will be provided with data access.

References

  1. Schultz, W., Dayan, P. & Montague, P. R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

    Article  CAS  PubMed  Google Scholar 

  2. Huys, Q. J. M., Tobler, P. N., Hasler, G. & Flagel, S. B. The role of learning-related dopamine signals in addiction vulnerability. Prog. Brain Res. 211, 31–77 (2014).

    Article  CAS  PubMed  Google Scholar 

  3. Lesaint, F., Sigaud, O., Flagel, S. B., Robinson, T. E. & Khamassi, M. Modelling individual differences in the form of Pavlovian conditioned approach responses: a dual learning systems approach with factored representations. PLoS Comput. Biol. 10, e1003466 (2014).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  4. Gläscher, J., Daw, N., Dayan, P. & O’Doherty, J. P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585–595 (2010).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  5. Daw, N. D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711 (2005).

    Article  CAS  PubMed  Google Scholar 

  6. Dickinson, A. & Balleine, B. in Stevens’ Handbook of Experimental Psychology 3rd edn 497–534 (2002).

  7. Doya, K. What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Netw. 12, 961–974 (1999).

    Article  CAS  PubMed  Google Scholar 

  8. Friedel, E. et al. Devaluation and sequential decisions: linking goal-directed and model-based behavior. Front. Hum. Neurosci. 8, 587 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  9. Ernst, M. & Paulus, M. P. Neurobiology of decision making: a selective review from a neurocognitive and clinical perspective. Biol. Psychiatry 58, 597–604 (2005).

    Article  PubMed  Google Scholar 

  10. Flagel, S. B. et al. A selective role for dopamine in stimulus–reward learning. Nature 469, 53–57 (2011).

    Article  CAS  PubMed  Google Scholar 

  11. Day, J. J., Roitman, M. F., Wightman, R. M. & Carelli, R. M. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat. Neurosci. 10, 1020–1028 (2007).

    Article  CAS  PubMed  Google Scholar 

  12. Berridge, K. C. & Robinson, T. E. What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Res. Rev. 28, 309–369 (1998).

    Article  CAS  PubMed  Google Scholar 

  13. Berridge, K. C. & Robinson, T. E. Parsing reward. Trends Neurosci. 26, 507–513 (2003).

    Article  CAS  PubMed  Google Scholar 

  14. Hickey, C. & Peelen, M. V. Neural mechanisms of incentive salience in naturalistic human vision. Neuron 85, 512–518 (2015).

    Article  CAS  PubMed  Google Scholar 

  15. Robinson, T. E. & Flagel, S. B. Dissociating the predictive and incentive motivational properties of reward-related cues through the study of individual differences. Biol. Psychiatry 65, 869–873 (2009).

    Article  PubMed  Google Scholar 

  16. McClure, S. M., Daw, N. D. & Montague, P. R. A computational substrate for incentive salience. Trends Neurosci. 26, 423–428 (2003).

    Article  CAS  PubMed  Google Scholar 

  17. Dayan, P., Niv, Y., Seymour, B. & Daw, N. D. The misbehavior of value and the discipline of the will. Neural Netw. 19, 1153–1160 (2006).

    Article  PubMed  Google Scholar 

  18. Dayan, P. & Berridge, K. C. Model-based and model-free Pavlovian reward learning: revaluation, revision, and revelation. Cogn. Affect. Behav. Neurosci. 14, 473–492 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  19. Garofalo, S. & di Pellegrino, G. Individual differences in the influence of task-irrelevant Pavlovian cues on human behavior. Front. Behav. Neurosci. 9, 163 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  20. Morrison, S. E., Bamkole, M. A. & Nicola, S. M. Sign-tracking, but not goal-tracking, is resistant to outcome devaluation. Front. Neurosci. 9, 468 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  21. Huys, Q. J. M. et al. Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding. PLoS Comput. Biol. 7, e1002028 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  22. Gottlieb, J. Attention, learning, and the value of information. Neuron 76, 281–295 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  23. Leclerc, R. & Reberg, D. Sign-tracking in aversive conditioning. Learn. Motiv. 11, 302–317 (1980).

    Article  Google Scholar 

  24. Yager, L. M., Pitchers, K. K., Flagel, S. B. & Robinson, T. E. Individual variation in the motivational and neurobiological effects of an opioid cue. Neuropsychopharmacology 40, 1269–1277 (2015).

    Article  CAS  PubMed  Google Scholar 

  25. Gottlieb, J., Oudeyer, P. Y., Lopes, M. & Baranes, A. Information-seeking, curiosity, and attention: computational and neural mechanisms. Trends Cogn. Sci. 17, 585–593 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  26. Renninger, L. W., Verghese, P. & Coughlan, J. Where to look next? Eye movements reduce local uncertainty. J. Vis. 7, 6 (2007).

    Article  PubMed  Google Scholar 

  27. Nassar, M. R. et al. Rational regulation of learning dynamics by pupil-linked arousal systems. Nat. Neurosci. 15, 1040–1046 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  28. Manohar, S. G. & Husain, M. Reduced pupillary reward sensitivity in Parkinson’s disease. NPJ Park. Dis. 1, 15026 (2015).

    Article  CAS  Google Scholar 

  29. Berridge, K. C. The debate over dopamine’s role in reward: the case for incentive salience. Psychopharmacology 191, 391–431 (2007).

    Article  CAS  PubMed  Google Scholar 

  30. Rutledge, R. B., Dean, M., Caplin, A. & Glimcher, P. W. Testing the reward prediction error hypothesis with an axiomatic model. J. Neurosci. 30, 13525–13536 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Seymour, B., Daw, N., Dayan, P., Singer, T. & Dolan, R. Differential encoding of losses and gains in the human striatum. J. Neurosci. 27, 4826–4831 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. Flagel, S. B. et al. A food predictive cue must be attributed with incentive salience for it to induce c-fos mRNA expression in cortico-striatal-thalamic brain regions. Neuroscience 196, 80–96 (2011).

    Article  CAS  PubMed  Google Scholar 

  33. Wilson, R. C. & Niv, Y. Is model fitting necessary for model-based fMRI? PLoS Comput. Biol. 11, e1004237 (2015).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  34. Sebold, M. et al. Don’t think, just feel the music: individuals with strong Pavlovian-to-instrumental transfer effects rely less on model-based reinforcement learning. J. Cogn. Neurosci. 28, 985–995 (2016).

    Article  PubMed  Google Scholar 

  35. Montague, P. R., Dayan, P. & Sejnowski, T. J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  36. Steinberg, E. E. et al. A causal link between prediction errors, dopamine neurons and learning. Nat. Neurosci. 16, 966–973 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. Dayan, P., Kakade, S. & Montague, P. R. Learning and selective attention. Nat. Neurosci. 3, 1218–1223 (2000).

    Article  CAS  PubMed  Google Scholar 

  38. Robinson, T. E. & Berridge, K. C. The neural basis of drug craving: an incentive-sensitization theory of addiction. Brain Res. Rev. 18, 247–291 (1993).

    Article  CAS  PubMed  Google Scholar 

  39. Saunders, B. T. & Robinson, T. E. Individual variation in resisting temptation: implications for addiction. Neurosci. Biobehav. Rev. 37, 1955–1975 (2013).

    Article  PubMed  Google Scholar 

  40. Garbusow, M. et al. Pavlovian-to-instrumental transfer effects in the nucleus accumbens relate to relapse in alcohol dependence. Addict. Biol. 21, 719–731 (2016).

    Article  CAS  PubMed  Google Scholar 

  41. Schad, D. J. et al. Neural correlates of instrumental responding in the context of alcohol-related cues index disorder severity and relapse risk. Eur. Arch. Psychiatry Clin. Neurosci. 269, 295–308 (2019).

    Article  PubMed  Google Scholar 

  42. Geurts, D. E., Huys, Q. J. M., den Ouden, H. & Cools, R. Aversive Pavlovian control of instrumental behavior in humans. J. Cogn. Neurosci. 25, 1428–1441 (2013).

    Article  PubMed  Google Scholar 

  43. Brainard, D. H. The psychophysics toolbox. Spat. Vis. 10, 433–436 (1997).

    Article  CAS  PubMed  Google Scholar 

  44. Pelli, D. G. The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat. Vis. 10, 437–442 (1997).

    Article  CAS  PubMed  Google Scholar 

  45. Garbusow, M. et al. Pavlovian-to-instrumental transfer in alcohol dependence: a pilot study. Neuropsychobiology 70, 111–121 (2014).

    Article  CAS  PubMed  Google Scholar 

  46. American Psychiatric Association. Diagnostic and statistical manual of mental disorders: DSM-IV (American Psychiatric Publishing, 1994).

  47. Wittchen, H.-U. & Pfister, H. DIA-X-Interviews: Manual für Screening-Verfahren und Interview; Interviewheft Längsschnittuntersuchung (DIA-X-Lifetime); Ergänzungsheft (DIA-X- Lifetime); Interviewheft Querschnittuntersuchung (DIA-X-12 Monate); Ergänzungsheft (DIA-X-12 Monate); PC-Programm zur Durchführung des Interviews (Längs- und Querschnittuntersuchung); Auswertungsprogramm (Swets and Zeitlinger, 1997).

  48. R Development Core Team. R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, 2016).

  49. Singmann, H., Bolker, B., Westfall, J. & Aust, F. afex: Analysis of Factorial Experiments R package version 0.18-0 https://cran.r-project.org/web/packages/afex/index.html (2017).

  50. Lenth, R. emmeans: Estimated Marginal Means, aka Least-Squares Means. R package version 1.1. https://cran.r-project.org/web/packages/emmeans/index.html (2018).

  51. Ruxton, G. D. The unequal variance t-test is an underused alternative to Student’s t-test and the Mann–Whitney U test. Behav. Ecol. 17, 688–690 (2006).

    Article  Google Scholar 

  52. Canty, A. & Ripley, B. D. boot: Bootstrap R (S-Plus) functions. R package version 1.3-18. https://cran.r-project.org/web/packages/boot/ (2017).

  53. Davison, A. C. & Hinkley, D. V. Bootstrap Methods and Their Applications (Cambridge Univ. Press, 1997).

  54. Morey, R. D. Confidence intervals from normalized data: a correction to Cousineau (2005). Tutor. Quant. Methods Psychol. 4, 81–84 (2008).

    Google Scholar 

  55. Kelley, K. MBESS: The MBESS R Package. R version 4.5.1. https://cran.r-project.org/web/packages/MBESS/index.html (2019).

  56. Hogarth, L., Dickinson, A. & Duka, T. in Attention and Associative Learning: From Brain to Behaviour (eds Mitchell, C. J. & Le Pelley, M. E.) 71–98 (Oxford Univ. Press, 2010).

  57. Peck, C. J., Jangraw, D. C., Suzuki, M., Efem, R. & Gottlieb, J. Reward modulates attention independently of action value in posterior parietal cortex. J. Neurosci. 29, 11182–11191 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  58. Hickey, C., Chelazzi, L. & Theeuwes, J. Reward changes salience in human vision via the anterior cingulate. J. Neurosci. 30, 11096–11103 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  59. Hickey, C. & van Zoest, W. Reward creates oculomotor salience. Curr. Biol. 22, R219–R220 (2012).

    Article  CAS  PubMed  Google Scholar 

  60. Itti, L. & Koch, C. Computational modelling of visual attention. Nat. Rev. Neurosci. 2, 194–203 (2001).

    Article  CAS  PubMed  Google Scholar 

  61. Gorgolewski, K. et al. Nipype: a flexible, lightweight and extensible neuroimaging data processing framework in python. Front. Neuroinform. 5, 13 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  62. Iglesias, S. et al. Hierarchical prediction errors in midbrain and basal forebrain during sensory learning. Neuron 80, 519–530 (2013).

    Article  CAS  PubMed  Google Scholar 

  63. Deserno, L. et al. Ventral striatal dopamine reflects behavioral and neural signatures of model-based control during sequential decision making. Proc. Natl Acad. Sci. USA 112, 1595–1600 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  64. White, D. M., Kraguljac, N. V., Reid, M. A. & Lahti, A. C. Contribution of substantia nigra glutamate to prediction error signals in schizophrenia: a combined magnetic resonance spectroscopy/functional imaging study. NPJ Schizophr. 1, 14001 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  65. Watanabe, N., Sakagami, M. & Haruno, M. Reward prediction error signal enhanced by striatum–amygdala interaction explains the acceleration of probabilistic reward learning by emotion. J. Neurosci. 33, 4487–4493 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  66. Gluth, S., Hotaling, J. M. & Rieskamp, J. The attraction effect modulates reward prediction errors and intertemporal choices. J. Neurosci. 37, 371–382 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  67. Garrison, J., Erdeniz, B. & Done, J. Prediction error in reinforcement learning: a meta-analysis of neuroimaging studies. Neurosci. Biobehav. Rev. 37, 1297–1310 (2013).

    Article  PubMed  Google Scholar 

  68. Logothetis, N. K., Pauls, J., Augath, M., Trinath, T. & Oeltermann, A. Neurophysiological investigation of the basis of the fMRI signal. Nature 412, 150–157 (2001).

    Article  CAS  PubMed  Google Scholar 

  69. Nebe, S. et al. No association of goal-directed and habitual control with alcohol consumption in young adults. Addict. Biol. 23, 379–393 (2018).

    Article  PubMed  Google Scholar 

  70. Neyens, V. et al. Representation of semantic similarity in the left intraparietal sulcus: functional magnetic resonance imaging evidence. Front. Hum. Neurosci. 11, 402 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  71. Eickhoff, S. B. et al. A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. NeuroImage 25, 1325–1335 (2005).

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

This work was supported by the German Research Foundation (FOR 1617: grants SCHA 1971/1-2, HE 2597/13-1, HE 2597/13-2, HE 2597/15-1, SCHL 1969/2-2, SCHL 1969/4-1, SM 80/7-1, SM 80/7-2, WI 709/10-1, WI 709/10-2, ZI 1119/3-1, ZI 1119/3-2, RA 1047/2-1 and RA 1047/2-2, and in part by CRC-TR 265). E.F. is a participant in the BIH Charité Clinician Scientist Program funded by the Charité—Universitätsmedizin Berlin and Berlin Institute of Health. Q.J.M.H. acknowledges support from the UCLH NIHR BRC. S.N. received funding from the University of Zurich Grants Office (FK-19-020). We thank N. B. Krömer for helpful feedback and advice on the analyses, S. Kuitunen-Paul for helpful feedback, and M. Rothkirch for help with setting up eye-tracking at the Berlin site. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Authors and Affiliations

Authors

Contributions

Q.J.M.H. conceived of the study idea. M.A.R., E.F., H.-U.W., U.S.Z., H.W., P.S., M.N.S., F.S., A.H. and Q.J.M.H. designed the study. D.J.S., M.G., M.S., S.N., E.O., E.F., U.S.Z., M.N.S., F.S., A.H. and Q.J.M.H. conducted the implementation, pilots and setup. M.G., M.S., S.N. and C.S. acquired the data, with supervision from N.R.-S., H.-U.W., U.S.Z., H.W., P.S., M.N.S., F.S., A.H. and Q.J.M.H. D.J.S. analysed the data, with supervision from M.A.R., P.D. and Q.J.M.H. and input from L.D., M.R., F.S. and A.H. D.J.S., M.A.R., P.D. and Q.J.M.H. wrote the manuscript. All authors read and revised the manuscript and provided critical intellectual contributions.

Corresponding author

Correspondence to Daniel J. Schad.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Primary Handling Editor: Marike Schiffer.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary methods, results, references and Supplementary Figs. 1–12.

Reporting Summary

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schad, D.J., Rapp, M.A., Garbusow, M. et al. Dissociating neural learning signals in human sign- and goal-trackers. Nat Hum Behav 4, 201–214 (2020). https://doi.org/10.1038/s41562-019-0765-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-019-0765-5

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing