Abstract
Verbal working memory (vWM) involves storing and manipulating information in phonological sensory input. An influential theory of vWM proposes that manipulation is carried out by a central executive while storage is performed by two interacting systems: a phonological input buffer that captures sound-based information and an articulatory rehearsal system that controls speech motor output. Whether, when and how neural activity in the brain encodes these components remains unknown. Here we read out the contents of vWM from neural activity in human subjects as they manipulated stored speech sounds. As predicted, we identified storage systems that contained both phonological sensory and articulatory motor representations. Unexpectedly, however, we found that manipulation did not involve a single central executive but rather involved two systems with distinct contributions to successful manipulation. We propose, therefore, that multiple subsystems comprise the central executive needed to manipulate stored phonological input for articulatory motor output in vWM.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Rent or buy this article
Prices vary by article type
from$1.95
to$39.95
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Logie, R.H. Spatial and visual working memory: a mental workspace. Psychol. Learn. Motiv. 42, 37–78 (2003).
Baddeley, A. Working memory: looking back and looking forward. Nat. Rev. Neurosci. 4, 829–839 (2003).
Atkinson, R.C. & Shiffrin, R.M. in The Psychology of Learning and Motivation: Advances in Research and Theory (eds. Spence, K.W. & Spence, J.T.) 89–195 (Academic, New York, 1968).
Vallar, G. & Baddeley, A.D. Phonological short-term store, phonological processing and sentence comprehension: A neuropsychological case study. Cogn. Neuropsychol. 1, 121–141 (1984).
Martin, R.C. & Lesch, M. in Models of Short-Term Memory (ed. Gathercole, S.) 149–178 (Erlbaum, 1996).
Saffran, E.M. & Martin, N. in Neuropsychological Impairments of Short-term Memory (eds. Vallar, G. & Shallice, T.) 428–447 (Cambridge University Press, 1990).
Jacquemot, C., Dupoux, E., Decouche, O. & Bachoud-Levi, A.-C. Misperception in sentences but not in words: speech perception and the phonological buffer. Cogn. Neuropsychol. 23, 949–971 (2006).
Gupta, P. & MacWhinney, B. Vocabulary acquisition and verbal short-term memory: computational and neural bases. Brain Lang. 59, 267–333 (1997).
King, J. & Just, M.A. Individual differences in syntactic processing: the role of working memory. J. Mem. Lang. 30, 580–602 (1991).
Papagno, C., Valentine, T. & Baddeley, A.D. Phonological short-term memory and foreign-language vocabulary learning. J. Mem. Lang. 30, 331–347 (1991).
Hirschberg, J. & Manning, C.D. Advances in natural language processing. Science 349, 261–266 (2015).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).
Graves, A. & Schmidhuber, J. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 18, 602–610 (2005).
Sutskever, I., Vinyals, O. & Le, Q.V.V. Sequence to sequence learning with neural networks. in Advances in Neural Information Processing Systems 3104–3112 (2014).
Baddeley, A.D., Thomson, N. & Buchanan, M. Word length and the structure of short-term memory. J. Verbal Learn. 14, 575–589 (1975).
Baddeley, A. Working memory. Science 255, 556–559 (1992).
Jacquemot, C. & Scott, S.K. What is the relationship between phonological short-term memory and speech processing? Trends Cogn. Sci. 10, 480–486 (2006).
Buchsbaum, B.R. & D'Esposito, M. The search for the phonological store: from loop to convolution. J. Cogn. Neurosci. 20, 762–778 (2008).
Hickok, G., Buchsbaum, B., Humphries, C. & Muftuler, T. Auditory-motor interaction revealed by fMRI: speech, music, and working memory in area Spt. J. Cogn. Neurosci. 15, 673–682 (2003).
Hickok, G. & Poeppel, D. Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language. Cognition 92, 67–99 (2004).
Hickok, G. & Poeppel, D. Towards a functional neuroanatomy of speech perception. Trends Cogn. Sci. 4, 131–138 (2000).
Atkinson, R.C. & Shiffrin, R.M. Human memory: a proposed system and its control processes. Psychol. Learn. Motiv. 2, 89–195 (1968).
Shallice, T. in Principles of the Frontal Lobe Function (eds. Stuss, D.T. & Knight, R.T.) 261–284 (Oxford University Press, 2002).
Goldman-Rakic, P.S. The prefrontal landscape: implications of functional architecture for understanding human mentation and the central executive. Phil. Trans. R. Soc. Lond. B 351, 1445–1453 (1996).
Wallis, J.D., Anderson, K.C. & Miller, E.K. Single neurons in prefrontal cortex encode abstract rules. Nature 411, 953–956 (2001).
Vallar, G. & Baddeley, A.D. Fractionation of working memory: neuropsychological evidence for a phonological short-term store. J. Verbal Learn. Verbal Behav. 23, 151–161 (1984).
Paulesu, E., Frith, C.D. & Frackowiak, R.S.J. The neural correlates of the verbal component of working memory. Nature 362, 342–345 (1993).
Veltman, D.J., Rombouts, S.A.R. & Dolan, R.J. Maintenance versus manipulation in verbal working memory revisited: an fMRI study. Neuroimage 18, 247–256 (2003).
Becker, J.T., MacAndrew, D.K. & Fiez, J.A. A comment on the functional localization of the phonological storage subsystem of working memory. Brain Cogn. 41, 27–38 (1999).
Ladefoged, P. A Course in Phonetics (Wadsworth Publishing, 2005).
Zhang, M. & Barash, S. Neuronal switching of sensorimotor transformations for antisaccades. Nature 408, 971–975 (2000).
Sakai, K. & Miyashita, Y. Neural organization for the long-term memory of paired associates. Nature 354, 152–155 (1991).
Tomita, H., Ohbayashi, M., Nakahara, K., Hasegawa, I. & Miyashita, Y. Top-down signal from prefrontal cortex in executive control of memory retrieval. Nature 401, 699–703 (1999).
Nir, Y. et al. Coupling between neuronal firing rate, gamma LFP, and BOLD fMRI is related to interneuronal correlations. Curr. Biol. 17, 1275–1285 (2007).
Ray, S., Crone, N.E., Niebur, E., Franaszczuk, P.J. & Hsiao, S.S. Neural correlates of high-gamma oscillations (60-200 Hz) in macaque local field potentials and their potential implications in electrocorticography. J. Neurosci. 28, 11526–11536 (2008).
Cogan, G.B. et al. Sensory-motor transformations for speech occur bilaterally. Nature 507, 94–98 (2014).
D'Esposito, M. et al. The neural basis of the central executive system of working memory. Nature 378, 279–281 (1995).
Rowe, J.B., Toni, I., Josephs, O., Frackowiak, R.S. & Passingham, R.E. The prefrontal cortex: response selection or maintenance within working memory? Science 288, 1656–1660 (2000).
Voytek, B. et al. Oscillatory dynamics coordinating human frontal networks in support of goal maintenance. Nat. Neurosci. 18, 1318–1324 (2015).
Baddeley, A.D. The influence of acoustic and semantic similarity on long-term memory for word sequences. Q. J. Exp. Psychol. 18, 302–309 (1966).
Vallar, G. & Papagno, C. in The Handbook of Memory Disorders (eds. Baddeley, A.D., Kopelman, M.D. & Wilson, B.A.) 249–270 (John Wiley & Sons, Ltd, 2002).
Postle, B.R., Berger, J.S. & D'Esposito, M. Functional neuroanatomical double dissociation of mnemonic and executive control processes contributing to working memory performance. Proc. Natl. Acad. Sci. USA 96, 12959–12964 (1999).
Buchsbaum, B.R., Hickok, G. & Humphries, C. Role of left posterior superior temporal gyrus in phonological processing for speech perception and production. Cogn. Sci. 25, 663–678 (2001).
Jonides, J. et al. Verbal working memory load affects regional brain activation as measured by PET. J. Cogn. Neurosci. 9, 462–475 (1997).
Baddeley, A., Lewis, V. & Vallar, G. Exploring the articulatory loop. Q. J. Exp. Psychol. Human Exp. Psychol. 36, 233–252 (1984).
Awh, E., Vogel, E.K. & Oh, S.H. Interactions between attention and working memory. Neuroscience 139, 201–208 (2006).
Ikkai, A. & Curtis, C.E. Common neural mechanisms supporting spatial working memory, attention and motor intention. Neuropsychologia 49, 1428–1434 (2011).
Markowitz, D.A., Curtis, C.E. & Pesaran, B. Multiple component networks support working memory in prefrontal cortex. Proc. Natl. Acad. Sci. USA 112, 11084–11089 (2015).
Latimer, K.W., Yates, J.L., Meister, M.L.R., Huk, A.C. & Pillow, J.W. Neuronal modeling. Single-trial spike trains in parietal cortex reveal discrete steps during decision-making. Science 349, 184–187 (2015).
Harvey, C.D., Coen, P. & Tank, D.W. Choice-specific sequences in parietal cortex during a virtual-navigation decision task. Nature 484, 62–68 (2012).
Yang, A.I. et al. Localization of dense intracranial electrode arrays using magnetic resonance imaging. Neuroimage 63, 157–165 (2012).
Dale, A.M., Fischl, B. & Sereno, M.I. Cortical surface-based analysis. I. Segmentation and surface reconstruction. Neuroimage 9, 179–194 (1999).
Mitra, P.P. & Pesaran, B. Analysis of dynamic brain imaging data. Biophys. J. 76, 691–708 (1999).
Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. B 57, 289–300 (1995).
Acknowledgements
We thank A. Weiss, J. MacArthur and L. Frank for developing the data acquisition hardware and software, E. Londen, P. Purushothaman, C. Boomhaur, and P. Minhas for technical assistance, and C. Curtis for comments on the manuscript. This work was supported, in part, by R03-DC010475 from the NIDCD, an Award from the Simons Collaboration for the Global Brain (B.P.).
Author information
Authors and Affiliations
Contributions
G.B.C. designed the experiment, performed the research, analyzed the data and wrote the manuscript. A.I. performed the research and analyzed the data. L.M. performed the research and wrote the manuscript. T.T. performed the research. O.D. performed the research and wrote the manuscript. D.F. and W.D. performed the research. B.P. designed the experiment, performed the research, analyzed the data and wrote the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Integrated supplementary information
Supplementary Figure 1 Localization of responses
a. For each subject, we classified neural activity using an electrode’s response to a localizer task (see Experimental Procedures). Auditory (green), indicates an electrode responded to incoming sensory stimuli, Production (blue) indicates that an electrode responded to the motor output during the response, and Sensory-Motor (red) indicates that an electrode responded to both the incoming sensory stimuli and the motor production. Delay responses were also present in each electrode category (Delay+ - Purple outline) as well as Delay-Only Responses (Purple). b. Localization of electrodes across subjects. Color convention the same as in a.
Supplementary Figure 2 Example responses.
a. Example spectrograms of the neural responses for non-delay Electrodes. An Auditory electrode (top row) during the remembered sensory-motor dissociation task: A high gamma (70 Hz +) neural response is seen in the auditory epoch but not during the delay epoch. In an example Production electrode (second row) Significant neural activity is present in the production epoch but not during the delay period. In an example Sensory-Motor electrode (third row) significant neural activity is seen in the auditory and production epochs, but not during the delay. Note that the gray bar reflects the variable delay period (1.5 – 2 seconds post auditory onset). b. Average high gamma power traces (70 – 160 Hz) are shown for each electrode class as shown in a: Auditory, Production, and Sensory-Motor. c. Localization of each spectrogram from Fig 3a (left column) and Fig S2a (right column). Error values are SEM of power across electrodes (Auditory: N = 23 Electrodes, Production: N = 34, Sensory-Motor: N = 8).
Supplementary Figure 3 Example spectrograms for each class and subject.
We show for example spectrograms for each subject for each delay electrode type (where present).
Supplementary Figure 4 Representations of non-vWM responses.
Using the same models and procedure as illustrated in Fig 5, we demonstrate the representations of each non-delay electrode category. While the patterns are very similar to their Delay counterparts, crucially there is an absence of delay representations.
Supplementary Figure 5 A leave-one-out analysis.
We repeat our classifier/model analysis (see Fig 5) using iterations of our subject pool with a single subject removed and compare it to the entire subject pool. Removing some subjects weakened the effects, but the representation of vWM in each electrode class was qualitatively the same, indicating that our results are not due to a single subject effect. Delay+Auditory sites tended to encode in an Audtory representation. Delay+Production sites tended to encode in a Motor representation. Delay+Sensory-motor sites tended to encode in a Sensory-Motor representation. The Delay-only sites tended to encode in a Rule representation.
Supplementary information
Supplementary Text and Figures
Supplementary Figures 1–5 and Supplementary Tables 1–3 (PDF 1247 kb)
Rights and permissions
About this article
Cite this article
Cogan, G., Iyer, A., Melloni, L. et al. Manipulating stored phonological input during verbal working memory. Nat Neurosci 20, 279–286 (2017). https://doi.org/10.1038/nn.4459
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn.4459
This article is cited by
-
High-resolution neural recordings improve the accuracy of speech decoding
Nature Communications (2023)
-
Dataset of human medial temporal lobe neurons, scalp and intracranial EEG during a verbal working memory task
Scientific Data (2020)