Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Manipulating stored phonological input during verbal working memory

Abstract

Verbal working memory (vWM) involves storing and manipulating information in phonological sensory input. An influential theory of vWM proposes that manipulation is carried out by a central executive while storage is performed by two interacting systems: a phonological input buffer that captures sound-based information and an articulatory rehearsal system that controls speech motor output. Whether, when and how neural activity in the brain encodes these components remains unknown. Here we read out the contents of vWM from neural activity in human subjects as they manipulated stored speech sounds. As predicted, we identified storage systems that contained both phonological sensory and articulatory motor representations. Unexpectedly, however, we found that manipulation did not involve a single central executive but rather involved two systems with distinct contributions to successful manipulation. We propose, therefore, that multiple subsystems comprise the central executive needed to manipulate stored phonological input for articulatory motor output in vWM.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Match–mismatch task.
Figure 2: Example responses.
Figure 3: Subdivisions of delay activity.
Figure 4: Idealized and representative responses.
Figure 5: Representation of vWM.
Figure 6: Error analysis: during the delay epoch, all four response classes differentiated between trials with a correct utterance and trials with no utterance (chance is 0.5; dotted red lines).

Similar content being viewed by others

References

  1. Logie, R.H. Spatial and visual working memory: a mental workspace. Psychol. Learn. Motiv. 42, 37–78 (2003).

    Article  Google Scholar 

  2. Baddeley, A. Working memory: looking back and looking forward. Nat. Rev. Neurosci. 4, 829–839 (2003).

    Article  CAS  Google Scholar 

  3. Atkinson, R.C. & Shiffrin, R.M. in The Psychology of Learning and Motivation: Advances in Research and Theory (eds. Spence, K.W. & Spence, J.T.) 89–195 (Academic, New York, 1968).

  4. Vallar, G. & Baddeley, A.D. Phonological short-term store, phonological processing and sentence comprehension: A neuropsychological case study. Cogn. Neuropsychol. 1, 121–141 (1984).

    Article  Google Scholar 

  5. Martin, R.C. & Lesch, M. in Models of Short-Term Memory (ed. Gathercole, S.) 149–178 (Erlbaum, 1996).

  6. Saffran, E.M. & Martin, N. in Neuropsychological Impairments of Short-term Memory (eds. Vallar, G. & Shallice, T.) 428–447 (Cambridge University Press, 1990).

  7. Jacquemot, C., Dupoux, E., Decouche, O. & Bachoud-Levi, A.-C. Misperception in sentences but not in words: speech perception and the phonological buffer. Cogn. Neuropsychol. 23, 949–971 (2006).

    Article  Google Scholar 

  8. Gupta, P. & MacWhinney, B. Vocabulary acquisition and verbal short-term memory: computational and neural bases. Brain Lang. 59, 267–333 (1997).

    Article  CAS  Google Scholar 

  9. King, J. & Just, M.A. Individual differences in syntactic processing: the role of working memory. J. Mem. Lang. 30, 580–602 (1991).

    Article  Google Scholar 

  10. Papagno, C., Valentine, T. & Baddeley, A.D. Phonological short-term memory and foreign-language vocabulary learning. J. Mem. Lang. 30, 331–347 (1991).

    Article  Google Scholar 

  11. Hirschberg, J. & Manning, C.D. Advances in natural language processing. Science 349, 261–266 (2015).

    Article  CAS  Google Scholar 

  12. Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).

    Article  CAS  Google Scholar 

  13. Graves, A. & Schmidhuber, J. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 18, 602–610 (2005).

    Article  Google Scholar 

  14. Sutskever, I., Vinyals, O. & Le, Q.V.V. Sequence to sequence learning with neural networks. in Advances in Neural Information Processing Systems 3104–3112 (2014).

  15. Baddeley, A.D., Thomson, N. & Buchanan, M. Word length and the structure of short-term memory. J. Verbal Learn. 14, 575–589 (1975).

    Article  Google Scholar 

  16. Baddeley, A. Working memory. Science 255, 556–559 (1992).

    Article  CAS  Google Scholar 

  17. Jacquemot, C. & Scott, S.K. What is the relationship between phonological short-term memory and speech processing? Trends Cogn. Sci. 10, 480–486 (2006).

    Article  Google Scholar 

  18. Buchsbaum, B.R. & D'Esposito, M. The search for the phonological store: from loop to convolution. J. Cogn. Neurosci. 20, 762–778 (2008).

    Article  Google Scholar 

  19. Hickok, G., Buchsbaum, B., Humphries, C. & Muftuler, T. Auditory-motor interaction revealed by fMRI: speech, music, and working memory in area Spt. J. Cogn. Neurosci. 15, 673–682 (2003).

    Article  Google Scholar 

  20. Hickok, G. & Poeppel, D. Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language. Cognition 92, 67–99 (2004).

    Article  Google Scholar 

  21. Hickok, G. & Poeppel, D. Towards a functional neuroanatomy of speech perception. Trends Cogn. Sci. 4, 131–138 (2000).

    Article  CAS  Google Scholar 

  22. Atkinson, R.C. & Shiffrin, R.M. Human memory: a proposed system and its control processes. Psychol. Learn. Motiv. 2, 89–195 (1968).

    Article  Google Scholar 

  23. Shallice, T. in Principles of the Frontal Lobe Function (eds. Stuss, D.T. & Knight, R.T.) 261–284 (Oxford University Press, 2002).

  24. Goldman-Rakic, P.S. The prefrontal landscape: implications of functional architecture for understanding human mentation and the central executive. Phil. Trans. R. Soc. Lond. B 351, 1445–1453 (1996).

    Article  CAS  Google Scholar 

  25. Wallis, J.D., Anderson, K.C. & Miller, E.K. Single neurons in prefrontal cortex encode abstract rules. Nature 411, 953–956 (2001).

    Article  CAS  Google Scholar 

  26. Vallar, G. & Baddeley, A.D. Fractionation of working memory: neuropsychological evidence for a phonological short-term store. J. Verbal Learn. Verbal Behav. 23, 151–161 (1984).

    Article  Google Scholar 

  27. Paulesu, E., Frith, C.D. & Frackowiak, R.S.J. The neural correlates of the verbal component of working memory. Nature 362, 342–345 (1993).

    Article  CAS  Google Scholar 

  28. Veltman, D.J., Rombouts, S.A.R. & Dolan, R.J. Maintenance versus manipulation in verbal working memory revisited: an fMRI study. Neuroimage 18, 247–256 (2003).

    Article  Google Scholar 

  29. Becker, J.T., MacAndrew, D.K. & Fiez, J.A. A comment on the functional localization of the phonological storage subsystem of working memory. Brain Cogn. 41, 27–38 (1999).

    Article  CAS  Google Scholar 

  30. Ladefoged, P. A Course in Phonetics (Wadsworth Publishing, 2005).

  31. Zhang, M. & Barash, S. Neuronal switching of sensorimotor transformations for antisaccades. Nature 408, 971–975 (2000).

    Article  CAS  Google Scholar 

  32. Sakai, K. & Miyashita, Y. Neural organization for the long-term memory of paired associates. Nature 354, 152–155 (1991).

    Article  CAS  Google Scholar 

  33. Tomita, H., Ohbayashi, M., Nakahara, K., Hasegawa, I. & Miyashita, Y. Top-down signal from prefrontal cortex in executive control of memory retrieval. Nature 401, 699–703 (1999).

    Article  CAS  Google Scholar 

  34. Nir, Y. et al. Coupling between neuronal firing rate, gamma LFP, and BOLD fMRI is related to interneuronal correlations. Curr. Biol. 17, 1275–1285 (2007).

    Article  CAS  Google Scholar 

  35. Ray, S., Crone, N.E., Niebur, E., Franaszczuk, P.J. & Hsiao, S.S. Neural correlates of high-gamma oscillations (60-200 Hz) in macaque local field potentials and their potential implications in electrocorticography. J. Neurosci. 28, 11526–11536 (2008).

    Article  CAS  Google Scholar 

  36. Cogan, G.B. et al. Sensory-motor transformations for speech occur bilaterally. Nature 507, 94–98 (2014).

    Article  CAS  Google Scholar 

  37. D'Esposito, M. et al. The neural basis of the central executive system of working memory. Nature 378, 279–281 (1995).

    Article  CAS  Google Scholar 

  38. Rowe, J.B., Toni, I., Josephs, O., Frackowiak, R.S. & Passingham, R.E. The prefrontal cortex: response selection or maintenance within working memory? Science 288, 1656–1660 (2000).

    Article  CAS  Google Scholar 

  39. Voytek, B. et al. Oscillatory dynamics coordinating human frontal networks in support of goal maintenance. Nat. Neurosci. 18, 1318–1324 (2015).

    Article  CAS  Google Scholar 

  40. Baddeley, A.D. The influence of acoustic and semantic similarity on long-term memory for word sequences. Q. J. Exp. Psychol. 18, 302–309 (1966).

    Article  CAS  Google Scholar 

  41. Vallar, G. & Papagno, C. in The Handbook of Memory Disorders (eds. Baddeley, A.D., Kopelman, M.D. & Wilson, B.A.) 249–270 (John Wiley & Sons, Ltd, 2002).

  42. Postle, B.R., Berger, J.S. & D'Esposito, M. Functional neuroanatomical double dissociation of mnemonic and executive control processes contributing to working memory performance. Proc. Natl. Acad. Sci. USA 96, 12959–12964 (1999).

    Article  CAS  Google Scholar 

  43. Buchsbaum, B.R., Hickok, G. & Humphries, C. Role of left posterior superior temporal gyrus in phonological processing for speech perception and production. Cogn. Sci. 25, 663–678 (2001).

    Article  Google Scholar 

  44. Jonides, J. et al. Verbal working memory load affects regional brain activation as measured by PET. J. Cogn. Neurosci. 9, 462–475 (1997).

    Article  CAS  Google Scholar 

  45. Baddeley, A., Lewis, V. & Vallar, G. Exploring the articulatory loop. Q. J. Exp. Psychol. Human Exp. Psychol. 36, 233–252 (1984).

    Article  Google Scholar 

  46. Awh, E., Vogel, E.K. & Oh, S.H. Interactions between attention and working memory. Neuroscience 139, 201–208 (2006).

    Article  CAS  Google Scholar 

  47. Ikkai, A. & Curtis, C.E. Common neural mechanisms supporting spatial working memory, attention and motor intention. Neuropsychologia 49, 1428–1434 (2011).

    Article  Google Scholar 

  48. Markowitz, D.A., Curtis, C.E. & Pesaran, B. Multiple component networks support working memory in prefrontal cortex. Proc. Natl. Acad. Sci. USA 112, 11084–11089 (2015).

    Article  CAS  Google Scholar 

  49. Latimer, K.W., Yates, J.L., Meister, M.L.R., Huk, A.C. & Pillow, J.W. Neuronal modeling. Single-trial spike trains in parietal cortex reveal discrete steps during decision-making. Science 349, 184–187 (2015).

    Article  CAS  Google Scholar 

  50. Harvey, C.D., Coen, P. & Tank, D.W. Choice-specific sequences in parietal cortex during a virtual-navigation decision task. Nature 484, 62–68 (2012).

    Article  CAS  Google Scholar 

  51. Yang, A.I. et al. Localization of dense intracranial electrode arrays using magnetic resonance imaging. Neuroimage 63, 157–165 (2012).

    Article  Google Scholar 

  52. Dale, A.M., Fischl, B. & Sereno, M.I. Cortical surface-based analysis. I. Segmentation and surface reconstruction. Neuroimage 9, 179–194 (1999).

    Article  CAS  Google Scholar 

  53. Mitra, P.P. & Pesaran, B. Analysis of dynamic brain imaging data. Biophys. J. 76, 691–708 (1999).

    Article  CAS  Google Scholar 

  54. Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. B 57, 289–300 (1995).

    Google Scholar 

Download references

Acknowledgements

We thank A. Weiss, J. MacArthur and L. Frank for developing the data acquisition hardware and software, E. Londen, P. Purushothaman, C. Boomhaur, and P. Minhas for technical assistance, and C. Curtis for comments on the manuscript. This work was supported, in part, by R03-DC010475 from the NIDCD, an Award from the Simons Collaboration for the Global Brain (B.P.).

Author information

Authors and Affiliations

Authors

Contributions

G.B.C. designed the experiment, performed the research, analyzed the data and wrote the manuscript. A.I. performed the research and analyzed the data. L.M. performed the research and wrote the manuscript. T.T. performed the research. O.D. performed the research and wrote the manuscript. D.F. and W.D. performed the research. B.P. designed the experiment, performed the research, analyzed the data and wrote the manuscript.

Corresponding author

Correspondence to Bijan Pesaran.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Integrated supplementary information

Supplementary Figure 1 Localization of responses

a. For each subject, we classified neural activity using an electrode’s response to a localizer task (see Experimental Procedures). Auditory (green), indicates an electrode responded to incoming sensory stimuli, Production (blue) indicates that an electrode responded to the motor output during the response, and Sensory-Motor (red) indicates that an electrode responded to both the incoming sensory stimuli and the motor production. Delay responses were also present in each electrode category (Delay+ - Purple outline) as well as Delay-Only Responses (Purple). b. Localization of electrodes across subjects. Color convention the same as in a.

Supplementary Figure 2 Example responses.

a. Example spectrograms of the neural responses for non-delay Electrodes. An Auditory electrode (top row) during the remembered sensory-motor dissociation task: A high gamma (70 Hz +) neural response is seen in the auditory epoch but not during the delay epoch. In an example Production electrode (second row) Significant neural activity is present in the production epoch but not during the delay period. In an example Sensory-Motor electrode (third row) significant neural activity is seen in the auditory and production epochs, but not during the delay. Note that the gray bar reflects the variable delay period (1.5 – 2 seconds post auditory onset). b. Average high gamma power traces (70 – 160 Hz) are shown for each electrode class as shown in a: Auditory, Production, and Sensory-Motor. c. Localization of each spectrogram from Fig 3a (left column) and Fig S2a (right column). Error values are SEM of power across electrodes (Auditory: N = 23 Electrodes, Production: N = 34, Sensory-Motor: N = 8).

Supplementary Figure 3 Example spectrograms for each class and subject.

We show for example spectrograms for each subject for each delay electrode type (where present).

Supplementary Figure 4 Representations of non-vWM responses.

Using the same models and procedure as illustrated in Fig 5, we demonstrate the representations of each non-delay electrode category. While the patterns are very similar to their Delay counterparts, crucially there is an absence of delay representations.

Supplementary Figure 5 A leave-one-out analysis.

We repeat our classifier/model analysis (see Fig 5) using iterations of our subject pool with a single subject removed and compare it to the entire subject pool. Removing some subjects weakened the effects, but the representation of vWM in each electrode class was qualitatively the same, indicating that our results are not due to a single subject effect. Delay+Auditory sites tended to encode in an Audtory representation. Delay+Production sites tended to encode in a Motor representation. Delay+Sensory-motor sites tended to encode in a Sensory-Motor representation. The Delay-only sites tended to encode in a Rule representation.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–5 and Supplementary Tables 1–3 (PDF 1247 kb)

Supplementary Methods Checklist (PDF 430 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cogan, G., Iyer, A., Melloni, L. et al. Manipulating stored phonological input during verbal working memory. Nat Neurosci 20, 279–286 (2017). https://doi.org/10.1038/nn.4459

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn.4459

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing