Article | Published:

Cortical tracking of hierarchical linguistic structures in connected speech

Nature Neuroscience 19, 158–164 (2016) | Download Citation

Subjects

Abstract

The most critical attribute of human language is its unbounded combinatorial nature: smaller elements can be combined into larger structures on the basis of a grammatical system, resulting in a hierarchy of linguistic units, such as words, phrases and sentences. Mentally parsing and representing such structures, however, poses challenges for speech comprehension. In speech, hierarchical linguistic structures do not have boundaries that are clearly defined by acoustic cues and must therefore be internally and incrementally constructed during comprehension. We found that, during listening to connected speech, cortical activity of different timescales concurrently tracked the time course of abstract linguistic structures at different hierarchical levels, such as words, phrases and sentences. Notably, the neural tracking of hierarchical linguistic structures was dissociated from the encoding of acoustic cues and from the predictability of incoming words. Our results indicate that a hierarchy of neural processing timescales underlies grammar-based internal construction of hierarchical linguistic structure.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

Rent or Buy

All prices are NET prices.

References

  1. 1.

    , , & Evolution, brain, and the nature of language. Trends Cogn. Sci. 17, 89–98 (2013).

  2. 2.

    Syntactic Structures (Mouton de Gruyter, 1957).

  3. 3.

    Linear order and constituency. Linguist. Inq. 34, 37–90 (2003).

  4. 4.

    & Basic linguistic composition recruits the left anterior temporal lobe and left angular gyrus during both listening and reading. Cereb. Cortex 23, 1859–1873 (2013).

  5. 5.

    & Cortical oscillations and speech processing: emerging computational principles and operations. Nat. Neurosci. 15, 511–517 (2012).

  6. 6.

    , & Segmenting nonsense: an event-related potential index of perceived onsets in continuous speech. Nat. Neurosci. 5, 700–703 (2002).

  7. 7.

    , & Syntactic unification operations are reflected in oscillatory dynamics during on-line sentence comprehension. J. Cogn. Neurosci. 22, 1333–1347 (2010).

  8. 8.

    , & Investigating the neural correlates of continuous speech computation with frequency-tagged neuroelectric responses. Neuroimage 44, 509–519 (2009).

  9. 9.

    , & Cortical representation of the constituent structure of sentences. Proc. Natl. Acad. Sci. USA 108, 2522–2527 (2011).

  10. 10.

    , , , & Neuronal oscillations and visual amplification of speech. Trends Cogn. Sci. 12, 106–113 (2008).

  11. 11.

    Neural syntax: cell assemblies, synapsembles and readers. Neuron 68, 362–385 (2010).

  12. 12.

    , , & A reservoir of time constants for memory traces in cortical neurons. Nat. Neurosci. 14, 366–372 (2011).

  13. 13.

    , , & Topographic mapping of a hierarchy of temporal receptive windows using a narrated story. J. Neurosci. 31, 2906–2915 (2011).

  14. 14.

    , & A hierarchy of time-scales and the brain. PLoS Comput. Biol. 4, e1000209 (2008).

  15. 15.

    & Phase patterns of neuronal responses reliably discriminate speech in human auditory cortex. Neuron 54, 1001–1010 (2007).

  16. 16.

    & Emergence of neural encoding of auditory objects while listening to competing speakers. Proc. Natl. Acad. Sci. USA 109, 11854–11859 (2012).

  17. 17.

    et al. Mechanisms underlying selective neuronal tracking of attended speech at a “cocktail party”. Neuron 77, 980–991 (2013).

  18. 18.

    , & Phase-locked responses to speech in human auditory cortex are enhanced during comprehension. Cereb. Cortex 23, 1378–1387 (2013).

  19. 19.

    et al. Reconstructing speech from human auditory cortex. PLoS Biol. 10, e1001251 (2012).

  20. 20.

    , & Brain potentials indicate immediate use of prosodic cues in natural speech processing. Nat. Neurosci. 2, 191–196 (1999).

  21. 21.

    , , & Signal-driven computations in speech processing. Science 298, 604–607 (2002).

  22. 22.

    , & Statistical learning by 8-month-old infants. Science 274, 1926–1928 (1996).

  23. 23.

    & Different origins of gamma rhythm and high-gamma activity in macaque visual cortex. PLoS Biol. 9, e1000610 (2011).

  24. 24.

    , , & Modeling and analysis of local field potentials for studying the function of cortical circuits. Nat. Rev. Neurosci. 14, 770–785 (2013).

  25. 25.

    & The neurobiology of language beyond single words. Annu. Rev. Neurosci. 37, 347–362 (2014).

  26. 26.

    & Neuroimaging of syntax and syntactic processing. Curr. Opin. Neurobiol. 16, 240–246 (2006).

  27. 27.

    & The cortical organization of speech processing. Nat. Rev. Neurosci. 8, 393–402 (2007).

  28. 28.

    , & Auditory language comprehension: an event-related fMRI study on the processing of syntactic and lexical information. Brain Lang. 74, 289–300 (2000).

  29. 29.

    et al. High gamma power is phase-locked to theta oscillations in human neocortex. Science 313, 1626–1628 (2006).

  30. 30.

    et al. An oscillatory hierarchy controlling neuronal excitability and stimulus processing in the auditory cortex. J. Neurophysiol. 94, 1904–1911 (2005).

  31. 31.

    , , & Communication between neocortex and hippocampus during sleep in rodents. Proc. Natl. Acad. Sci. USA 100, 2065–2069 (2003).

  32. 32.

    & Cortical oscillations and sensory predictions. Trends Cogn. Sci. 16, 390–398 (2012).

  33. 33.

    , & Speech perception at the interface of neurobiology and linguistics. Phil. Trans. R. Soc. Lond. B 363, 1071–1086 (2008).

  34. 34.

    & Brain oscillations during spoken sentence processing. J. Cogn. Neurosci. 24, 1149–1164 (2012).

  35. 35.

    et al. Speech rhythms and multiplexed oscillatory sensory coding in the human brain. PLoS Biol. 11, e1001752 (2013).

  36. 36.

    & Cortical entrainment to continuous speech: functional roles and interpretations. Front. Hum. Neurosci. 8, 311 (2014).

  37. 37.

    Foundations of Language: Brain, Meaning, Grammar, Evolution (Oxford University Press, 2002).

  38. 38.

    On Broca, brain, and binding: a new framework. Trends Cogn. Sci. 9, 416–423 (2005).

  39. 39.

    , & Prosody in the comprehension of spoken language: a literature review. Lang. Speech 40, 141–201 (1997).

  40. 40.

    , & Prosodic phrasing is central to language comprehension. Trends Cogn. Sci. 10, 244–249 (2006).

  41. 41.

    & Visual feature integration and the temporal correlation hypothesis. Annu. Rev. Neurosci. 18, 555–586 (1995).

  42. 42.

    Towards a neural basis of auditory sentence processing. Trends Cogn. Sci. 6, 78–84 (2002).

  43. 43.

    & Electrophysiology reveals semantic memory use in language comprehension. Trends Cogn. Sci. 4, 463–470 (2000).

  44. 44.

    , , , & Syntactically based sentence processing classes: evidence from event-related brain potentials. J. Cogn. Neurosci. 3, 151–165 (1991).

  45. 45.

    , & A cortical network for semantics: (de)constructing the N400. Nat. Rev. Neurosci. 9, 920–933 (2008).

  46. 46.

    et al. N400-like magnetoencephalography responses modulated by semantic context, word frequency and lexical class in sentences. Neuroimage 17, 1101–1116 (2002).

  47. 47.

    & Interactions between sentence context and word frequency in event-related brain potentials. Mem. Cognit. 18, 380–393 (1990).

  48. 48.

    , & A supramodal accumulation-to-bound signal that determines perceptual decisions in humans. Nat. Neurosci. 15, 1729–1735 (2012).

  49. 49.

    , & The architecture of cognitive control in the human prefrontal cortex. Science 302, 1181–1185 (2003).

  50. 50.

    , , & Tagging the neuronal entrainment to beat and meter. J. Neurosci. 31, 10234–10240 (2011).

  51. 51.

    The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9, 97–113 (1971).

  52. 52.

    & Denoising based on time-shift PCA. J. Neurosci. Methods 165, 297–305 (2007).

  53. 53.

    & Denoising based on spatial filtering. J. Neurosci. Methods 171, 331–339 (2008).

  54. 54.

    & Adaptive temporal encoding leads to a background-insensitive cortical representation of speech. J. Neurosci. 33, 5728–5735 (2013).

  55. 55.

    & Neural coding of continuous speech in auditory cortex during monaural and dichotic listening. J. Neurophysiol. 107, 78–89 (2012).

  56. 56.

    et al. Sensitivity to temporal modulation rate and spectral bandwidth in the human auditory system: MEG evidence. J. Neurophysiol. 107, 2033–2041 (2012).

  57. 57.

    & An Introduction to the Bootstrap (CRC press, 1993).

  58. 58.

    et al. Localization of dense intracranial electrode arrays using magnetic resonance imaging. Neuroimage 63, 157–165 (2012).

Download references

Acknowledgements

We thank J. Walker for MEG technical support, T. Thesen, W. Doyle and O. Devinsky for their instrumental help in collecting ECoG data, and G. Buzsaki, G. Cogan, S. Dehaene, A.-L. Giraud, G. Hickok, N. Hornstein, E. Lau, A. Marantz, N. Mesgarani, M. Peña, B. Pesaran, L. Pylkkänen, C. Schroeder, J. Simon and W. Singer for their comments on previous versions of the manuscript. This work supported by US National Institutes of Health grant 2R01DC05660 (D.P.) and Major Projects Program of the Shanghai Municipal Science and Technology Commission (STCSM) 15JC1400104 (X.T.) and National Natural Science Foundation of China 31500914 (X.T.).

Author information

Affiliations

  1. Department of Psychology, New York University, New York, New York, USA.

    • Nai Ding
    • , Hang Zhang
    • , Xing Tian
    •  & David Poeppel

  2. College of Biomedical Engineering and Instrument Sciences, Zhejiang University, Hangzhou, China.

    • Nai Ding

  3. Department of Neurology, New York University Langone Medical Center, New York, New York, USA.

    • Lucia Melloni

  4. Department of Neurophysiology, Max-Planck Institute for Brain Research, Frankfurt, Germany.

    • Lucia Melloni

  5. Department of Psychiatry, Columbia University, New York, New York, USA.

    • Lucia Melloni

  6. Department of Psychology and Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing, China.

    • Hang Zhang

  7. PKU-IDG/McGovern Institute for Brain Research, Peking University, Beijing, China.

    • Hang Zhang

  8. Peking-Tsinghua Center for Life Sciences, Beijing, China.

    • Hang Zhang

  9. New York University Shanghai, Shanghai, China.

    • Xing Tian

  10. NYU-ECNU Institute of Brain and Cognitive Science at NYU Shanghai, Shanghai, China.

    • Xing Tian

  11. Neuroscience Department, Max-Planck Institute for Empirical Aesthetics, Frankfurt, Germany.

    • David Poeppel

Authors

  1. Search for Nai Ding in:

  2. Search for Lucia Melloni in:

  3. Search for Hang Zhang in:

  4. Search for Xing Tian in:

  5. Search for David Poeppel in:

Contributions

N.D., L.M. and D.P. conceived and designed the experiments. N.D., H.Z. and X.T. performed the MEG experiments. L.M. performed the ECoG experiment. N.D., L.M. and D.P. wrote the paper. All of the authors discussed the results and edited the manuscript.

Competing interests

The authors declare no competing financial interests.

Corresponding authors

Correspondence to Nai Ding or David Poeppel.

Integrated supplementary information

Supplementary figures

  1. 1.

    Trial structure of Chinese (A-D) and English (EF) speech materials.

  2. 2.

    The spectrum of the temporal envelope for the Chinese (A) and English (B) 4-syllable sentence stimuli.

  3. 3.

    Comparisons between the responses to stimuli of different linguistic structures.

  4. 4.

    Dissociating neural encoding of sentential structures and transitional probability using Artifical Markovian Sentences (AMS).

  5. 5.

    Coverage of the ECoG electrodes.

Supplementary information

PDF files

  1. 1.

    Supplementary Text and Figures

    Supplementary Figures 1–5 and Supplementary Tables 1 and 2

  2. 2.

    Supplementary Methods Checklist

To obtain permission to re-use content from this article visit RightsLink.

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nn.4186