Abstract
Reinforcement signals indicating success or failure are known to alter the probability of selecting between distinct actions. However, successful performance of many motor skills, such as speech articulation, also requires learning behavioral trajectories that vary continuously over time. Here, we investigated how temporally discrete reinforcement signals shape a continuous behavioral trajectory, the fundamental frequency of adult Bengalese finch song. We provided reinforcement contingent on fundamental frequency performance only at one point in the song. Learned changes to fundamental frequency were maximal at this point, but also extended both earlier and later in the fundamental frequency trajectory. A simple principle predicted the detailed structure of learning: birds learned to produce the average of the behavioral trajectories associated with successful outcomes. This learning rule accurately predicted the structure of learning at a millisecond timescale, demonstrating that the nervous system records fine-grained details of successful behavior and uses this information to guide learning.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Thorndike, E.L. Animal Intelligence (Macmillan, New York, 1911).
Skinner, B.F. Science and Human Behavior (Macmillan, New York, 1953).
Balleine, B.W. & Ostlund, S.B. Still at the choice-point: action selection and initiation in instrumental conditioning. Ann. NY Acad. Sci. 1104, 147–171 (2007).
Skinner, B.F. Superstition in the pigeon. J. Exp. Psychol. 38, 168–172 (1948).
Schoppik, D., Nagel, K.I. & Lisberger, S.G. Cortical mechanisms of smooth eye movements revealed by dynamic covariations of neural and behavioral responses. Neuron 58, 248–260 (2008).
Hira, R. et al. Transcranial optogenetic stimulation for functional mapping of the motor cortex. J. Neurosci. Methods 179, 258–263 (2009).
Fee, M.S., Kozhevnikov, A.A. & Hahnloser, R.H. Neural mechanisms of vocal sequence generation in the songbird. Ann. NY Acad. Sci. 1016, 153–170 (2004).
Tumer, E.C. & Brainard, M.S. Performance variability enables adaptive plasticity of 'crystallized' adult birdsong. Nature 450, 1240–1244 (2007).
Tchernichovski, O., Mitra, P.P., Lints, T. & Nottebohm, F. Dynamics of the vocal imitation process: how a zebra finch learns its song. Science 291, 2564–2569 (2001).
Franz, M. & Goller, F. Respiratory units of motor production and song imitation in the zebra finch. J. Neurobiol. 51, 129–141 (2002).
Méndez, J.M., Dall'asén, A.G., Cooper, B.G. & Goller, F. Acquisition of an acoustic template leads to refinement of song motor gestures. J. Neurophysiol. 104, 984–993 (2010).
Ashmore, R.C., Wild, J.M. & Schmidt, M.F. Brainstem and forebrain contributions to the generation of learned motor behaviors for song. J. Neurosci. 25, 8543–8554 (2005).
Glaze, C.M. & Troyer, T.W. Temporal structure in zebra finch song: implications for motor coding. J. Neurosci. 26, 991–1005 (2006).
Glaze, C.M. & Troyer, T.W. Behavioral measurements of a temporally precise motor code for birdsong. J. Neurosci. 27, 7631–7639 (2007).
Tchernichovski, O., Lints, T.J., Deregnaucourt, S., Cimenser, A. & Mitra, P.P. Studying the song development process: rationale and methods. Ann. NY Acad. Sci. 1016, 348–363 (2004).
Chi, Z. & Margoliash, D. Temporal precision and temporal drift in brain and behavior of zebra finch song. Neuron 32, 899–910 (2001).
Andalman, A.S. & Fee, M.S. A basal ganglia-forebrain circuit in the songbird biases motor output to avoid vocal errors. Proc. Natl. Acad. Sci. USA 106, 12518–12523 (2009).
Gardner, T.J. & Magnasco, M.O. Sparse time-frequency representations. Proc. Natl. Acad. Sci. USA 103, 6094–6099 (2006).
Todorov, E. Optimality principles in sensorimotor control. Nat. Neurosci. 7, 907–915 (2004).
Leonardo, A. & Konishi, M. Decrystallization of adult birdsong by perturbation of auditory feedback. Nature 399, 466–470 (1999).
Mauk, M.D. & Buonomano, D.V. The neural basis of temporal processing. Annu. Rev. Neurosci. 27, 307–340 (2004).
Cohen, J.D., McClure, S.M. & Yu, A.J. Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Phil. Trans. R. Soc. Lond. B 362, 933–942 (2007).
Staddon, J.E.R. Adaptive Behavior and Learning (Cambridge Univ. Press, Cambridge, UK, 1983).
Catania, A.C. Learning 2nd edn. (Prentice Hall, Englewood Cliffs, New Jersey, USA, 1984).
Churchland, M.M., Afshar, A. & Shenoy, K.V. A central source of movement variability. Neuron 52, 1085–1096 (2006).
Faisal, A.A., Selen, L.P. & Wolpert, D.M. Noise in the nervous system. Nat. Rev. Neurosci. 9, 292–303 (2008).
Todorov, E. & Jordan, M.I. Optimal feedback control as a theory of motor coordination. Nat. Neurosci. 5, 1226–1235 (2002).
Elemans, C.P., Spierts, I.L., Müller, U.K., Van Leeuwen, J.L. & Goller, F. Bird song: superfast muscles control dove's trill. Nature 431, 146 (2004).
Elemans, C.P., Mead, A.F., Rome, L.C. & Goller, F. Superfast vocal muscles control song production in songbirds. PLoS ONE 3, e2581 (2008).
Sober, S.J., Wohlgemuth, M.J. & Brainard, M.S. Central contributions to acoustic variation in birdsong. J. Neurosci. 28, 10370–10379 (2008).
van Beers, R.J. The sources of variability in saccadic eye movements. J. Neurosci. 27, 8757–8770 (2007).
van Beers, R.J., Haggard, P. & Wolpert, D.M. The role of execution noise in movement variability. J. Neurophysiol. 91, 1050–1063 (2004).
Fetz, E.E. & Finocchio, D.V. Operant conditioning of specific patterns of neural and muscular activity. Science 174, 431–435 (1971).
Rosenfeld, J.P. & Hetzler, B.E. Operant-controlled evoked responses: discrimination of conditioned and normally occurring components. Science 181, 767–770 (1973).
Rosenfeld, J.P. & Fox, S.S. Operant control of a brain potential evoked by a behavior. Physiol. Behav. 7, 489–493 (1971).
Bottjer, S.W., Miesner, E.A. & Arnold, A.P. Forebrain lesions disrupt development but not maintenance of song in passerine birds. Science 224, 901–903 (1984).
Scharff, C. & Nottebohm, F. A comparative study of the behavioral deficits following lesions of various parts of the zebra finch song system: implications for vocal learning. J. Neurosci. 11, 2896–2913 (1991).
Jarvis, E.D., Scharff, C., Grossman, M.R., Ramos, J.A. & Nottebohm, F. For whom the bird sings: context-dependent gene expression. Neuron 21, 775–788 (1998).
Hessler, N.A. & Doupe, A.J. Social context modulates singing-related neural activity in the songbird forebrain. Nat. Neurosci. 2, 209–211 (1999).
Doya, K. & Sejnowski, T.J. A computational model of avian song learning. in The New Cognitive Neurosciences (ed. Gazzaniga, M.S.) 469–482 (MIT Press, Cambridge, Massachusetts, USA, 2000).
Kao, M.H., Doupe, A.J. & Brainard, M.S. Contributions of an avian basal ganglia-forebrain circuit to real-time modulation of song. Nature 433, 638–643 (2005).
Olveczky, B.P., Andalman, A.S. & Fee, M.S. Vocal experimentation in the juvenile songbird requires a basal ganglia circuit. PLoS Biol. 3, e153 (2005).
Fiete, I.R., Fee, M.S. & Seung, H.S. Model of birdsong learning based on gradient estimation by dynamic perturbation of neural conductances. J. Neurophysiol. 98, 2038–2057 (2007).
Aronov, D., Andalman, A.S. & Fee, M.S. A specialized forebrain circuit for vocal babbling in the juvenile songbird. Science 320, 630–634 (2008).
Hampton, C.M., Sakata, J.T. & Brainard, M.S. An avian basal ganglia-forebrain circuit contributes differently to syllable versus sequence variability of adult Bengalese finch song. J. Neurophysiol. 101, 3235–3245 (2009).
Mauk, M.D. & Ruiz, B.P. Learning-dependent timing of Pavlovian eyelid responses: differential conditioning using multiple interstimulus intervals. Behav. Neurosci. 106, 666–681 (1992).
Medina, J.F., Carey, M.R. & Lisberger, S.G. The representation of time for motor learning. Neuron 45, 157–167 (2005).
Leonardo, A. & Fee, M.S. Ensemble coding of vocal control in birdsong. J. Neurosci. 25, 652–661 (2005).
House, D. Tonal Perception in Speech (Lund Univ. Press, Lund, Sweden, 1990).
Hermes, D.J. Stylization of pitch contours. in Methods in Empirical Prosody Research (ed. Sudhoff, S.) 29–62 (Walter de Gruyter, Berlin, 2006).
Acknowledgements
We thank P. Sabes, S. Sober, L. Frank, A. Doupe and S. Lisberger for discussion and comments on the manuscript and J. Wong and C. Brown for animal care. This work was supported by US National Institutes of Health R01 and P50 grants. E.C.T. was supported by a National Research Service Award postdoctoral fellowship from the US National Institute on Deafness and Other Communication Disorders and by the Sloan-Swartz Foundation. J.D.C. and T.L.W. were supported by US National Science Foundation graduate fellowships.
Author information
Authors and Affiliations
Contributions
All authors contributed to designing the experiments. E.C.T. developed the song acquisition and reinforcement delivery software. T.L.W., E.C.T. and J.D.C. performed the experiments and collected the data. J.D.C. developed the modeling and analysis techniques and prepared the manuscript, with assistance from T.L.W. and M.S.B.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Supplementary information
Supplementary Text and Figures
Supplementary Figure 1 (PDF 422 kb)
Rights and permissions
About this article
Cite this article
Charlesworth, J., Tumer, E., Warren, T. et al. Learning the microstructure of successful behavior. Nat Neurosci 14, 373–380 (2011). https://doi.org/10.1038/nn.2748
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn.2748
This article is cited by
-
Spontaneous variability predicts compensative motor response in vocal pitch control
Scientific Reports (2022)
-
Intrinsic motivation for singing in songbirds is enhanced by temporary singing suppression and regulated by dopamine
Scientific Reports (2021)
-
Sensory substitution reveals a manipulation bias
Nature Communications (2020)
-
Place preference and vocal learning rely on distinct reinforcers in songbirds
Scientific Reports (2018)
-
Learning by neural reassociation
Nature Neuroscience (2018)