Functional organization of human sensorimotor cortex for speech articulation

Journal name:
Nature
Volume:
495,
Pages:
327–332
Date published:
DOI:
doi:10.1038/nature11911
Received
Accepted
Published online

Abstract

Speaking is one of the most complex actions that we perform, but nearly all of us learn to do it effortlessly. Production of fluent speech requires the precise, coordinated movement of multiple articulators (for example, the lips, jaw, tongue and larynx) over rapid time scales. Here we used high-resolution, multi-electrode cortical recordings during the production of consonant-vowel syllables to determine the organization of speech sensorimotor cortex in humans. We found speech-articulator representations that are arranged somatotopically on ventral pre- and post-central gyri, and that partially overlap at individual electrodes. These representations were coordinated temporally as sequences during syllable production. Spatial patterns of cortical activity showed an emergent, population-level representation, which was organized by phonetic features. Over tens of milliseconds, the spatial patterns transitioned between distinct representations for different consonants and vowels. These results reveal the dynamic organization of speech sensorimotor cortex during the generation of multi-articulator movements that underlies our ability to speak.

At a glance

Figures

  1. vSMC physiology during syllable production.
    Figure 1: vSMC physiology during syllable production.

    a, Magnetic resonance imaging (MRI) reconstruction of a single subject brain with vSMC electrodes (dots), coloured according to distance from the Sylvian fissure (black and red are the most dorsal and ventral positions, respectively). b, Expanded view of vSMC anatomy. cs, central sulcus; PoCG, post-central gyrus; PrCG, pre-central gyrus; Sf, Sylvian fissure. Scale bars,1cm. c–e, Top, vocal tract schematics for three consonants (/b/, /d/, /g/), produced by occlusion at the lips, tongue tip and tongue body, respectively (red arrow). Middle, spectrograms of spoken consonant-vowel syllables. Bottom, average cortical activity from a subset of electrodes (electrode number on far right, same colouring as in a). Vertical dashed line, acoustic onset of consonant-vowel transition. f–h, Cortical activity at selected electrodes for different phonetic contrasts (mean±s.e.m.). Acoustic waveforms are displayed above. f, Fricatives (/θ/(‘th’ of ‘thin’), /s/, /∫/(‘sh’ of ‘shin’)) with different constriction locations. g, Front tongue consonants (/l/, /n/, /d/) with different constriction degree or shapes. h, Single consonant (/j/ (‘y’ of ‘yes’)) with different vowels (/a/, /i/, /u/). Purple arrows correspond to a tongue electrode with prolonged activity for /i/ and /u/ vowels. Black arrow corresponds to an active lip electrode for /u/.

  2. Spatial representation of articulators.
    Figure 2: Spatial representation of articulators.

    a, Localization of lips, jaw, tongue and larynx representations. Average magnitude of articulator weightings (colour scale) plotted as a function of anteroposterior (AP) distance from the central sulcus and dorsoventral (DV) distance from the Sylvian fissure (n = 3 subjects). b, Functional somatotopic organization of speech-articulator representations in vSMC. Lips (L, red); jaw (J, green); tongue (T, blue); larynx (X, black); mixed (yellow). Letters correspond to locations, based on direct measurement-derived regression weights; shaded rectangles correspond to regions classified by k-nearest neighbour.

  3. Temporal representation of articulators.
    Figure 3: Temporal representation of articulators.

    a, b, Timing of correlations between cortical activity and consonant (a) and vowel (b) articulator features. Colour maps display correlation coefficients (R) for a subset of electrodes. c, Acoustic landmarks. Onset (end of arrows, left), peak power (shown by a dot in each case) and offset (end of arrows, right) for consonant-vowel syllables (mean±s.e.m., n = 168 syllables, all subjects). Error bars are smaller than the symbols. d, Temporal sequence and range of correlations. Symbols are as in c. Data are mean (symbols)±s.e.m. (thick solid line) across electrodes from all subjects.

  4. Phonetic organization of spatial patterns.
    Figure 4: Phonetic organization of spatial patterns.

    a, b, Scatterplots of consonant-vowel syllables in the first three principal components for consonants (25ms before consonant-vowel transition) (a) and vowels (250ms after consonant-vowel transition) (b). A subset of consonant-vowels are labelled with international phonetic alphabet (IPA) symbols, all others have dots. Colouring denotes k-means cluster membership. c, d, Hierarchical clustering of cortical state-space at consonant (25 ms before consonant-vowel transition) and vowel time points (250 ms after consonant-vowel transition). Individual syllables and dendrogram branches are colour-coded and labelled by known linguistic categories using the same colour scheme as in Fig 4 a, b, with new subdivisions of the coronal tongue into front tongue and sibilant (in green and red, respectively). Lat, lateral; N. stop, nasal stop; O. stop, oral stop. e, f, Correlations between cortical state-space and phonetic features. Black vertical lines, medians; grey boxes, 25th and 75th percentiles. ***P<10−10, WSRT; n = 297 for both consonants and vowels. NS, not significant.

  5. Dynamics of phonetic representations.
    Figure 5: Dynamics of phonetic representations.

    a, b, Cortical state-space trajectories. a, Consonants transitioning to the vowel /u/. Each line corresponds to a single consonant-vowel trajectory. For each line, the left triangle indicates t = −500ms, the square indicates t = −25ms, the circle indicates t = 250ms, and the right triangle indicates t = 750ms. b, Trajectories of the labial consonants transitioning to /a/, /i/ and /u/. c, d, Across-subject averages of cluster separability (c) and correlation between cortical state-space structure and phonetic features (d) for consonants and vowels (mean±s.e.m.). e, Time-course of consonant-vowel syllable trajectories for one subject. Each colour corresponds to one of the consonant or vowel groups (colours are the same as in a and b above). The centre of each coloured tube is located at the centroid of the corresponding phonetic cluster. Tube diameter corresponds to cluster density and colour saturation represents the correlation between the structure of the cortical state-space and phonetic features.

References

  1. Levelt, W. J. M. Speaking: From Intention to Articulation (MIT Press, 1993)
  2. Ladefoged, P. & Johnson, K. A Course in Phonetics (Wadsworth Publishing, 2010)
  3. Browman, C. P. & Goldstein, L. Articulatory gestures as phonological units. Haskins Laboratories Status Report on Speech Research 99, 69101 (1989)
  4. Fowler, C. A., Rubin, P. E., Remez, R. E. & Turvey, M. T. in Language Production: Speech and Talk Vol. 1 (ed. Butterworth, B.) 373420 (Academic Press, 1980)
  5. Gracco, V. L. & Lofqvist, A. Speech motor coordination and control: evidence from lip, jaw, and laryngeal movements. J. Neurosci. 14, 65856597 (1994)
  6. Schöner, G. & Kelso, J. A. Dynamic pattern generation in behavioral and neural systems. Science 239, 15131520 (1988)
  7. Franklin, D. W. & Wolpert, D. M. Computational mechanisms of sensorimotor control. Neuron 72, 425442 (2011)
  8. Brown, S. et al. The somatotopy of speech: phonation and articulation in the human motor cortex. Brain Cogn. 70, 3141 (2009)
  9. Guenther, F. H., Ghosh, S. S. & Tourville, J. A. Neural modeling and imaging of the cortical interactions underlying syllable production. Brain Lang. 96, 280301 (2006)
  10. Schulz, G. M., Varga, M., Jeffires, K., Ludlow, C. L. & Braun, A. R. Functional neuroanatomy of human vocalization: an H215O PET study. Cereb. Cortex 15, 18351847 (2005)
  11. Jürgens, U. Neural pathways underlying vocal control. Neurosci. Biobehav. Rev. 26, 235258 (2002)
  12. Kuypers, H. G. Corticobular connexions to the pons and lower brain-stem in man: an anatomical study. Brain 81, 364388 (1958)
  13. Brodmann, K. Vergleichende Lokalisationslehre der Grosshirnrinde in ihren Prinzipien dargestellt auf Grund des Zellenbaues (Smith-Gordon, 1994)
  14. Penfield, W. & Boldrey, E. Somatic motor and sensory representation in the cerebral cortex of man studied by electrical stimulation. Brain 60, 389443 (1937)
  15. Foerster, O. The cerebral cortex in man. Lancet 221, 309312 (1931)
  16. Penfield, W. & Roberts, R. Speech and Brain: Mechanisms. (Princeton, 1959)
  17. Saltzman, E. & Munhall, K. A dynamical approach to gestural patterning in speech production. Ecol. Psychol. 1, 333382 (1989)
  18. Clements, G. N. & Hume, E. in The Handbook of Phonological Theory (ed. Goldsmith, J. A. ) 245306 (Basil Blackwell, 1995)
  19. Chomsky, N. & Halle, M. The Sound Pattern of English (MIT Press, 1991)
  20. Mesgarani, N. & Chang, E. F. Selective cortical representation of attended speaker in multi-talker speech perception. Nature 485, 233236 (2012)
  21. Crone, N. E., Miglioretti, D. L., Gordon, B. & Lesser, R. P. Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. II. Event-related synchronization in the gamma band. Brain 121, 23012315 (1998)
  22. Edwards, E. et al. Spatiotemporal imaging of cortical activation during verb generation and picture naming. Neuroimage 50, 291301 (2010)
  23. Ray, S. & Maunsell, J. H. Different origins of gamma rhythm and high-gamma activity in macaque visual cortex. PLoS Biol. 9, e1000610 (2011)
  24. Kent, R. D. in The Production of Speech (ed. MacNeilage, P. F.) (Springer-Verlag, 1983)
  25. McCarthy, G., Allison, T. & Spencer, D. D. Localization of the face area of human sensorimotor cortex by intracranial recording of somatosensory evoked potentials. J. Neurosurg. 79, 874884 (1993)
  26. Afshar, A. et al. Single-trial neural correlates of arm movement preparation. Neuron 71, 555564 (2011)
  27. Mazor, O. & Laurent, G. Transient dynamics versus fixed points in odor representations by locust antennal lobe projection neurons. Neuron 48, 661673 (2005)
  28. Sussillo, D. & Abbott, L. F. Generating coherent patterns of activity from chaotic neural networks. Neuron 63, 544557 (2009)
  29. Ahrens, M. B. et al. Brain-wide neuronal dynamics during motor adaptation in zebrafish. Nature 485, 471477 (2012)
  30. Churchland, M. M., Cunningham, J. P., Kaufman, M. T., Ryu, S. I. & Shenoy, K. V. Cortical preparatory activity: representation of movement or first cog in a dynamical machine? Neuron 68, 387400 (2010)
  31. McCarthy, J. J. Feature geometry and dependency: a review. Phonetica 45, 84108 (1988)
  32. Briggman, K. L. & Kristan, W. B. Multifunctional pattern-generating circuits. Annu. Rev. Neurosci. 31, 271294 (2008)
  33. Churchland, M. M. et al. Neural population dynamics during reaching. Nature 487, 5156 (2012)
  34. Brown, S., Ngan, E. & Liotti, M. A larynx area in the human motor cortex. Cereb. Cortex 18, 837845 (2008)
  35. Terumitsu, M., Fujii, Y., Suzuki, K., Kwee, I. L. & Nakada, T. Human primary motor cortex shows hemispheric specialization for speech. Neuroreport 17, 10911095 (2006)
  36. Hast, M. H., Fischer, J. M., Wetzel, A. B. & Thompson, V. E. Cortical motor representation of the laryngeal muscles in Macaca mulatta. Brain Res. 73, 229240 (1974)
  37. Jürgens, U. On the elicitability of vocalization from the cortical larynx area. Brain Res. 81, 564566 (1974)
  38. Pruszynski, J. A. et al. Primary motor cortex underlies multi-joint integration for fast feedback control. Nature 478, 387390 (2011)
  39. Hatsopoulos, N. G. & Suminski, A. J. Sensing with the motor cortex. Neuron 72, 477487 (2011)
  40. Tremblay, S., Shiller, D. M. & Ostry, D. J. Somatosensory basis of speech production. Nature 423, 866869 (2003)
  41. Matyas, F. et al. Motor control by sensory cortex. Science 330, 12401243 (2010)
  42. Rathelot, J. A. & Strick, P. L. Muscle representation in the macaque motor cortex: an anatomical perspective. Proc. Natl Acad. Sci. USA 103, 82578262 (2006)
  43. Gracco, V. L. & Abbs, J. H. Dynamic control of the perioral system during speech: kinematic analyses of autogenic and nonautogenic sensorimotor processes. J. Neurophysiol. 54, 418432 (1985)
  44. Sherrington, C. S. The Integrative Action of the Nervous System (Yale University Press, 1911)
  45. Jakobson, R., Fant, G. & Halle, M. Preliminaries to speech analysis: the distinctive features and their correlates (MIT Press, 1969)
  46. Keating, P. A. The Window Model of Coarticulation: Articulatory Evidence (Cambridge Univ. Press, 1990)
  47. Dell, G. S., Juliano, C. & Govindjee, A. Structure and content in language production:a theory of frame constraints in phonological speech errors. Cogn. Sci. 17, 149195 (1993)
  48. Mesgarani, N. & Chang, E. F. Selective cortical representation of attended speaker in multi-talker speech perception. Nature 485, 233236 (2012)
  49. Yang, X. et al. Auditory representations of acoustic signals. IEEE Transactions Inf. Theor. 38, 824839 (1992)
  50. International Phonetic Association Handbook of the International Phonetic Association (Cambridge Univ. Press, 1999)
  51. Rousseeuw, P. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, 5365 (1987)

Download references

Author information

Affiliations

  1. Department of Neurological Surgery and Department of Physiology, University of California, San Francisco, 505 Parnassus Avenue, San Francisco, California 94143, USA

    • Kristofer E. Bouchard,
    • Nima Mesgarani &
    • Edward F. Chang
  2. Center for Integrative Neuroscience, 675 Nelson Rising Lane, University of California, San Francisco, California 94158, USA

    • Kristofer E. Bouchard,
    • Nima Mesgarani &
    • Edward F. Chang
  3. Department of Linguistics, University of California, Berkeley, 1203 Dwinelle Hall, Berkeley, California 94720, USA

    • Keith Johnson
  4. UCSF Epilepsy Center, University of California, San Francisco, 400 Parnassus Avenue, San Francisco, California 94143, USA

    • Edward F. Chang

Contributions

E.F.C. conceived and collected the data for this project. K.E.B. designed and implemented the analysis with assistance from E.F.C. N.M. assisted with preliminary analysis. K.E.B. and E.F.C. wrote the manuscript. K.J. provided phonetic consultation on experimental design and interpretation of results. E.F.C. supervised the project.

Competing financial interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to:

Author details

Supplementary information

PDF files

  1. Supplementary Information (4.9 MB)

    This file contains Supplementary Figures 1-13, Supplementary Table 1 and Supplementary Methods.

Additional data