Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Dissociating neural learning signals in human sign- and goal-trackers

Abstract

Individuals differ in how they learn from experience. In Pavlovian conditioning models, where cues predict reinforcer delivery at a different goal location, some animals—called sign-trackers—come to approach the cue, whereas others, called goal-trackers, approach the goal. In sign-trackers, model-free phasic dopaminergic reward-prediction errors underlie learning, which renders stimuli ‘wanted’. Goal-trackers do not rely on dopamine for learning and are thought to use model-based learning. We demonstrate this double dissociation in 129 male humans using eye-tracking, pupillometry and functional magnetic resonance imaging informed by computational models of sign- and goal-tracking. We show that sign-trackers exhibit a neural reward prediction error signal that is not detectable in goal-trackers. Model-free value only guides gaze and pupil dilation in sign-trackers. Goal-trackers instead exhibit a stronger model-based neural state prediction error signal. This model-based construct determines gaze and pupil dilation more in goal-trackers.

This is a preview of subscription content

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Assessment of sign- and goal-trackers via eye-tracking.
Fig. 2: Pupil dilation during Pavlovian conditioning in sign-trackers and goal-trackers.
Fig. 3: PIT in sign-trackers versus goal-trackers.
Fig. 4: NAc BOLD response in sign-trackers versus goal-trackers.
Fig. 5: Neural appetitive RPE signals in sign-trackers versus goal-trackers.
Fig. 6: Neural SPE learning signals in sign-trackers versus goal-trackers.

Data availability

Source data are available for Figs. 1–6 and Supplementary Figs. 2–12. Data sharing will be based on acceptance by the study team that: (1) a valid and timely scientific question, based on a written protocol, has been posed by those seeking to access the data; and (2) the role of the original study team will be fully acknowledged. Please contact the corresponding author via email to request access to the data. Safeguarding of ethical standards will be ensured by submission of a study amendment to the Charité and Dresden ethics committees. Data access for questions of scientific integrity may additionally be regulated via the funder.

Code availability

Experimental code is freely available on request to the corresponding author. Analysis code will be provided with data access.

References

  1. 1.

    Schultz, W., Dayan, P. & Montague, P. R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

    CAS  PubMed  Article  Google Scholar 

  2. 2.

    Huys, Q. J. M., Tobler, P. N., Hasler, G. & Flagel, S. B. The role of learning-related dopamine signals in addiction vulnerability. Prog. Brain Res. 211, 31–77 (2014).

    CAS  PubMed  Article  Google Scholar 

  3. 3.

    Lesaint, F., Sigaud, O., Flagel, S. B., Robinson, T. E. & Khamassi, M. Modelling individual differences in the form of Pavlovian conditioned approach responses: a dual learning systems approach with factored representations. PLoS Comput. Biol. 10, e1003466 (2014).

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  4. 4.

    Gläscher, J., Daw, N., Dayan, P. & O’Doherty, J. P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585–595 (2010).

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  5. 5.

    Daw, N. D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711 (2005).

    CAS  PubMed  Article  Google Scholar 

  6. 6.

    Dickinson, A. & Balleine, B. in Stevens’ Handbook of Experimental Psychology 3rd edn 497–534 (2002).

  7. 7.

    Doya, K. What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Netw. 12, 961–974 (1999).

    CAS  PubMed  Article  Google Scholar 

  8. 8.

    Friedel, E. et al. Devaluation and sequential decisions: linking goal-directed and model-based behavior. Front. Hum. Neurosci. 8, 587 (2014).

    PubMed  PubMed Central  Article  Google Scholar 

  9. 9.

    Ernst, M. & Paulus, M. P. Neurobiology of decision making: a selective review from a neurocognitive and clinical perspective. Biol. Psychiatry 58, 597–604 (2005).

    PubMed  Article  Google Scholar 

  10. 10.

    Flagel, S. B. et al. A selective role for dopamine in stimulus–reward learning. Nature 469, 53–57 (2011).

    CAS  PubMed  Article  Google Scholar 

  11. 11.

    Day, J. J., Roitman, M. F., Wightman, R. M. & Carelli, R. M. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat. Neurosci. 10, 1020–1028 (2007).

    CAS  PubMed  Article  Google Scholar 

  12. 12.

    Berridge, K. C. & Robinson, T. E. What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Res. Rev. 28, 309–369 (1998).

    CAS  PubMed  Article  Google Scholar 

  13. 13.

    Berridge, K. C. & Robinson, T. E. Parsing reward. Trends Neurosci. 26, 507–513 (2003).

    CAS  PubMed  Article  Google Scholar 

  14. 14.

    Hickey, C. & Peelen, M. V. Neural mechanisms of incentive salience in naturalistic human vision. Neuron 85, 512–518 (2015).

    CAS  PubMed  Article  Google Scholar 

  15. 15.

    Robinson, T. E. & Flagel, S. B. Dissociating the predictive and incentive motivational properties of reward-related cues through the study of individual differences. Biol. Psychiatry 65, 869–873 (2009).

    PubMed  Article  Google Scholar 

  16. 16.

    McClure, S. M., Daw, N. D. & Montague, P. R. A computational substrate for incentive salience. Trends Neurosci. 26, 423–428 (2003).

    CAS  PubMed  Article  Google Scholar 

  17. 17.

    Dayan, P., Niv, Y., Seymour, B. & Daw, N. D. The misbehavior of value and the discipline of the will. Neural Netw. 19, 1153–1160 (2006).

    PubMed  Article  Google Scholar 

  18. 18.

    Dayan, P. & Berridge, K. C. Model-based and model-free Pavlovian reward learning: revaluation, revision, and revelation. Cogn. Affect. Behav. Neurosci. 14, 473–492 (2014).

    PubMed  PubMed Central  Article  Google Scholar 

  19. 19.

    Garofalo, S. & di Pellegrino, G. Individual differences in the influence of task-irrelevant Pavlovian cues on human behavior. Front. Behav. Neurosci. 9, 163 (2015).

    PubMed  PubMed Central  Article  Google Scholar 

  20. 20.

    Morrison, S. E., Bamkole, M. A. & Nicola, S. M. Sign-tracking, but not goal-tracking, is resistant to outcome devaluation. Front. Neurosci. 9, 468 (2015).

    PubMed  PubMed Central  Article  Google Scholar 

  21. 21.

    Huys, Q. J. M. et al. Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding. PLoS Comput. Biol. 7, e1002028 (2011).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  22. 22.

    Gottlieb, J. Attention, learning, and the value of information. Neuron 76, 281–295 (2012).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  23. 23.

    Leclerc, R. & Reberg, D. Sign-tracking in aversive conditioning. Learn. Motiv. 11, 302–317 (1980).

    Article  Google Scholar 

  24. 24.

    Yager, L. M., Pitchers, K. K., Flagel, S. B. & Robinson, T. E. Individual variation in the motivational and neurobiological effects of an opioid cue. Neuropsychopharmacology 40, 1269–1277 (2015).

    CAS  PubMed  Article  Google Scholar 

  25. 25.

    Gottlieb, J., Oudeyer, P. Y., Lopes, M. & Baranes, A. Information-seeking, curiosity, and attention: computational and neural mechanisms. Trends Cogn. Sci. 17, 585–593 (2013).

    PubMed  PubMed Central  Article  Google Scholar 

  26. 26.

    Renninger, L. W., Verghese, P. & Coughlan, J. Where to look next? Eye movements reduce local uncertainty. J. Vis. 7, 6 (2007).

    PubMed  Article  Google Scholar 

  27. 27.

    Nassar, M. R. et al. Rational regulation of learning dynamics by pupil-linked arousal systems. Nat. Neurosci. 15, 1040–1046 (2012).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  28. 28.

    Manohar, S. G. & Husain, M. Reduced pupillary reward sensitivity in Parkinson’s disease. NPJ Park. Dis. 1, 15026 (2015).

    CAS  Article  Google Scholar 

  29. 29.

    Berridge, K. C. The debate over dopamine’s role in reward: the case for incentive salience. Psychopharmacology 191, 391–431 (2007).

    CAS  PubMed  Article  Google Scholar 

  30. 30.

    Rutledge, R. B., Dean, M., Caplin, A. & Glimcher, P. W. Testing the reward prediction error hypothesis with an axiomatic model. J. Neurosci. 30, 13525–13536 (2010).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  31. 31.

    Seymour, B., Daw, N., Dayan, P., Singer, T. & Dolan, R. Differential encoding of losses and gains in the human striatum. J. Neurosci. 27, 4826–4831 (2007).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  32. 32.

    Flagel, S. B. et al. A food predictive cue must be attributed with incentive salience for it to induce c-fos mRNA expression in cortico-striatal-thalamic brain regions. Neuroscience 196, 80–96 (2011).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  33. 33.

    Wilson, R. C. & Niv, Y. Is model fitting necessary for model-based fMRI? PLoS Comput. Biol. 11, e1004237 (2015).

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  34. 34.

    Sebold, M. et al. Don’t think, just feel the music: individuals with strong Pavlovian-to-instrumental transfer effects rely less on model-based reinforcement learning. J. Cogn. Neurosci. 28, 985–995 (2016).

    PubMed  Article  PubMed Central  Google Scholar 

  35. 35.

    Montague, P. R., Dayan, P. & Sejnowski, T. J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  36. 36.

    Steinberg, E. E. et al. A causal link between prediction errors, dopamine neurons and learning. Nat. Neurosci. 16, 966–973 (2013).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  37. 37.

    Dayan, P., Kakade, S. & Montague, P. R. Learning and selective attention. Nat. Neurosci. 3, 1218–1223 (2000).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  38. 38.

    Robinson, T. E. & Berridge, K. C. The neural basis of drug craving: an incentive-sensitization theory of addiction. Brain Res. Rev. 18, 247–291 (1993).

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  39. 39.

    Saunders, B. T. & Robinson, T. E. Individual variation in resisting temptation: implications for addiction. Neurosci. Biobehav. Rev. 37, 1955–1975 (2013).

    PubMed  Article  PubMed Central  Google Scholar 

  40. 40.

    Garbusow, M. et al. Pavlovian-to-instrumental transfer effects in the nucleus accumbens relate to relapse in alcohol dependence. Addict. Biol. 21, 719–731 (2016).

    CAS  PubMed  Article  Google Scholar 

  41. 41.

    Schad, D. J. et al. Neural correlates of instrumental responding in the context of alcohol-related cues index disorder severity and relapse risk. Eur. Arch. Psychiatry Clin. Neurosci. 269, 295–308 (2019).

    PubMed  Article  Google Scholar 

  42. 42.

    Geurts, D. E., Huys, Q. J. M., den Ouden, H. & Cools, R. Aversive Pavlovian control of instrumental behavior in humans. J. Cogn. Neurosci. 25, 1428–1441 (2013).

    PubMed  Article  Google Scholar 

  43. 43.

    Brainard, D. H. The psychophysics toolbox. Spat. Vis. 10, 433–436 (1997).

    CAS  PubMed  Article  Google Scholar 

  44. 44.

    Pelli, D. G. The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat. Vis. 10, 437–442 (1997).

    CAS  PubMed  Article  Google Scholar 

  45. 45.

    Garbusow, M. et al. Pavlovian-to-instrumental transfer in alcohol dependence: a pilot study. Neuropsychobiology 70, 111–121 (2014).

    CAS  PubMed  Article  Google Scholar 

  46. 46.

    American Psychiatric Association. Diagnostic and statistical manual of mental disorders: DSM-IV (American Psychiatric Publishing, 1994).

  47. 47.

    Wittchen, H.-U. & Pfister, H. DIA-X-Interviews: Manual für Screening-Verfahren und Interview; Interviewheft Längsschnittuntersuchung (DIA-X-Lifetime); Ergänzungsheft (DIA-X- Lifetime); Interviewheft Querschnittuntersuchung (DIA-X-12 Monate); Ergänzungsheft (DIA-X-12 Monate); PC-Programm zur Durchführung des Interviews (Längs- und Querschnittuntersuchung); Auswertungsprogramm (Swets and Zeitlinger, 1997).

  48. 48.

    R Development Core Team. R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, 2016).

  49. 49.

    Singmann, H., Bolker, B., Westfall, J. & Aust, F. afex: Analysis of Factorial Experiments R package version 0.18-0 https://cran.r-project.org/web/packages/afex/index.html (2017).

  50. 50.

    Lenth, R. emmeans: Estimated Marginal Means, aka Least-Squares Means. R package version 1.1. https://cran.r-project.org/web/packages/emmeans/index.html (2018).

  51. 51.

    Ruxton, G. D. The unequal variance t-test is an underused alternative to Student’s t-test and the Mann–Whitney U test. Behav. Ecol. 17, 688–690 (2006).

    Article  Google Scholar 

  52. 52.

    Canty, A. & Ripley, B. D. boot: Bootstrap R (S-Plus) functions. R package version 1.3-18. https://cran.r-project.org/web/packages/boot/ (2017).

  53. 53.

    Davison, A. C. & Hinkley, D. V. Bootstrap Methods and Their Applications (Cambridge Univ. Press, 1997).

  54. 54.

    Morey, R. D. Confidence intervals from normalized data: a correction to Cousineau (2005). Tutor. Quant. Methods Psychol. 4, 81–84 (2008).

    Google Scholar 

  55. 55.

    Kelley, K. MBESS: The MBESS R Package. R version 4.5.1. https://cran.r-project.org/web/packages/MBESS/index.html (2019).

  56. 56.

    Hogarth, L., Dickinson, A. & Duka, T. in Attention and Associative Learning: From Brain to Behaviour (eds Mitchell, C. J. & Le Pelley, M. E.) 71–98 (Oxford Univ. Press, 2010).

  57. 57.

    Peck, C. J., Jangraw, D. C., Suzuki, M., Efem, R. & Gottlieb, J. Reward modulates attention independently of action value in posterior parietal cortex. J. Neurosci. 29, 11182–11191 (2009).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  58. 58.

    Hickey, C., Chelazzi, L. & Theeuwes, J. Reward changes salience in human vision via the anterior cingulate. J. Neurosci. 30, 11096–11103 (2010).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  59. 59.

    Hickey, C. & van Zoest, W. Reward creates oculomotor salience. Curr. Biol. 22, R219–R220 (2012).

    CAS  PubMed  Article  Google Scholar 

  60. 60.

    Itti, L. & Koch, C. Computational modelling of visual attention. Nat. Rev. Neurosci. 2, 194–203 (2001).

    CAS  PubMed  Article  Google Scholar 

  61. 61.

    Gorgolewski, K. et al. Nipype: a flexible, lightweight and extensible neuroimaging data processing framework in python. Front. Neuroinform. 5, 13 (2011).

    PubMed  PubMed Central  Article  Google Scholar 

  62. 62.

    Iglesias, S. et al. Hierarchical prediction errors in midbrain and basal forebrain during sensory learning. Neuron 80, 519–530 (2013).

    CAS  PubMed  Article  Google Scholar 

  63. 63.

    Deserno, L. et al. Ventral striatal dopamine reflects behavioral and neural signatures of model-based control during sequential decision making. Proc. Natl Acad. Sci. USA 112, 1595–1600 (2015).

    CAS  PubMed  Article  Google Scholar 

  64. 64.

    White, D. M., Kraguljac, N. V., Reid, M. A. & Lahti, A. C. Contribution of substantia nigra glutamate to prediction error signals in schizophrenia: a combined magnetic resonance spectroscopy/functional imaging study. NPJ Schizophr. 1, 14001 (2015).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  65. 65.

    Watanabe, N., Sakagami, M. & Haruno, M. Reward prediction error signal enhanced by striatum–amygdala interaction explains the acceleration of probabilistic reward learning by emotion. J. Neurosci. 33, 4487–4493 (2013).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  66. 66.

    Gluth, S., Hotaling, J. M. & Rieskamp, J. The attraction effect modulates reward prediction errors and intertemporal choices. J. Neurosci. 37, 371–382 (2017).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  67. 67.

    Garrison, J., Erdeniz, B. & Done, J. Prediction error in reinforcement learning: a meta-analysis of neuroimaging studies. Neurosci. Biobehav. Rev. 37, 1297–1310 (2013).

    PubMed  Article  Google Scholar 

  68. 68.

    Logothetis, N. K., Pauls, J., Augath, M., Trinath, T. & Oeltermann, A. Neurophysiological investigation of the basis of the fMRI signal. Nature 412, 150–157 (2001).

    CAS  PubMed  Article  Google Scholar 

  69. 69.

    Nebe, S. et al. No association of goal-directed and habitual control with alcohol consumption in young adults. Addict. Biol. 23, 379–393 (2018).

    PubMed  Article  Google Scholar 

  70. 70.

    Neyens, V. et al. Representation of semantic similarity in the left intraparietal sulcus: functional magnetic resonance imaging evidence. Front. Hum. Neurosci. 11, 402 (2017).

    PubMed  PubMed Central  Article  Google Scholar 

  71. 71.

    Eickhoff, S. B. et al. A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. NeuroImage 25, 1325–1335 (2005).

    PubMed  Article  PubMed Central  Google Scholar 

Download references

Acknowledgements

This work was supported by the German Research Foundation (FOR 1617: grants SCHA 1971/1-2, HE 2597/13-1, HE 2597/13-2, HE 2597/15-1, SCHL 1969/2-2, SCHL 1969/4-1, SM 80/7-1, SM 80/7-2, WI 709/10-1, WI 709/10-2, ZI 1119/3-1, ZI 1119/3-2, RA 1047/2-1 and RA 1047/2-2, and in part by CRC-TR 265). E.F. is a participant in the BIH Charité Clinician Scientist Program funded by the Charité—Universitätsmedizin Berlin and Berlin Institute of Health. Q.J.M.H. acknowledges support from the UCLH NIHR BRC. S.N. received funding from the University of Zurich Grants Office (FK-19-020). We thank N. B. Krömer for helpful feedback and advice on the analyses, S. Kuitunen-Paul for helpful feedback, and M. Rothkirch for help with setting up eye-tracking at the Berlin site. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Affiliations

Authors

Contributions

Q.J.M.H. conceived of the study idea. M.A.R., E.F., H.-U.W., U.S.Z., H.W., P.S., M.N.S., F.S., A.H. and Q.J.M.H. designed the study. D.J.S., M.G., M.S., S.N., E.O., E.F., U.S.Z., M.N.S., F.S., A.H. and Q.J.M.H. conducted the implementation, pilots and setup. M.G., M.S., S.N. and C.S. acquired the data, with supervision from N.R.-S., H.-U.W., U.S.Z., H.W., P.S., M.N.S., F.S., A.H. and Q.J.M.H. D.J.S. analysed the data, with supervision from M.A.R., P.D. and Q.J.M.H. and input from L.D., M.R., F.S. and A.H. D.J.S., M.A.R., P.D. and Q.J.M.H. wrote the manuscript. All authors read and revised the manuscript and provided critical intellectual contributions.

Corresponding author

Correspondence to Daniel J. Schad.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Primary Handling Editor: Marike Schiffer.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary methods, results, references and Supplementary Figs. 1–12.

Reporting Summary

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Schad, D.J., Rapp, M.A., Garbusow, M. et al. Dissociating neural learning signals in human sign- and goal-trackers. Nat Hum Behav 4, 201–214 (2020). https://doi.org/10.1038/s41562-019-0765-5

Download citation

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing