Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Multiple spatial frames for immersive working memory

Nature Human Behaviour volume 6, pages 536–544 (2022)Cite this article

Subjects

Abstract

As we move around, relevant information that disappears from sight can still be held in working memory to serve upcoming behaviour. How we maintain and select visual information as we move through the environment remains poorly understood because most laboratory tasks of working memory rely on removing visual material while participants remain still. We used virtual reality to study visual working memory following self-movement in immersive environments. Directional biases in gaze revealed the recruitment of more than one spatial frame for maintaining and selecting memoranda following self-movement. The findings bring the important realization that multiple spatial frames support working memory in natural behaviour. The results also illustrate how virtual reality can be a critical experimental tool to characterize this core memory system.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Basic working memory task in VR and gaze signature of selection in working memory.
Fig. 2: Spatial frames for maintaining and selecting visual information following self-movement.
Fig. 3: Density plot for Experiment 3.
Fig. 4: Increasing memory demands increases reliance on both spatial frames.

Similar content being viewed by others

Data availability

All data are publicly available at https://osf.io/cj97y/.

Code availability

Code is available at https://osf.io/cj97y/.

References

  1. Baddeley, A. Working memory. Science 255, 556–559 (1992).

    Article  CAS  PubMed  Google Scholar 

  2. Ballard, D. H., Hayhoe, M. M., Pook, P. K. & Rao, R. P. N. Deictic codes for the embodiment of cognition. Behav. Brain Sci. 20, 723–767 (1997).

    Article  CAS  PubMed  Google Scholar 

  3. Tatler, B. W. & Land, M. F. Vision and the representation of the surroundings in spatial memory. Philos. Trans. R. Soc. Lond. B 366, 596–610 (2011).

    Article  Google Scholar 

  4. Hayhoe, M. & Ballard, D. Eye movements in natural behavior. Trends Cogn. Sci. 9, 188–194 (2005).

    Article  PubMed  Google Scholar 

  5. Draschkow, D., Kallmayer, M. & Nobre, A. C. When natural behavior engages working memory. Curr. Biol. 31, 869–874.e5 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Kristjánsson, Á. & Draschkow, D. Keeping it real: looking beyond capacity limits in visual cognition. Atten. Percept. Psychophys. 83, 1375–1390 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  7. Võ, M. L.-H., Boettcher, S. E. P. & Draschkow, D. Reading scenes: how scene grammar guides attention and aids perception in real-world environments. Curr. Opin. Psychol. 29, 205–210 (2019).

    Article  PubMed  Google Scholar 

  8. Fuster, J. M. & Alexander, G. E. Neuron activity related to short-term memory. Science 173, 652–654 (1971).

    Article  CAS  PubMed  Google Scholar 

  9. Funahashi, S., Chafee, M. V. & Goldman-Rakic, P. S. Prefrontal neuronal activity in rhesus monkeys performing a delayed anti-saccade task. Nature 365, 753–756 (1993).

    Article  CAS  PubMed  Google Scholar 

  10. Awh, E. & Jonides, J. Overlapping mechanisms of attention and spatial working memory. Trends Cogn. Sci. 5, 119–126 (2001).

    Article  CAS  PubMed  Google Scholar 

  11. Vogel, E. K. & Machizawa, M. G. Neural activity predicts individual differences in visual working memory capacity. Nature 428, 748–751 (2004).

    Article  CAS  PubMed  Google Scholar 

  12. Bays, P. M. & Husain, M. Dynamic shifts of limited working memory resources in human vision. Science 321, 851–854 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Griffin, I. C. & Nobre, A. C. Orienting attention to locations in internal representations. J. Cogn. Neurosci. 15, 1176–1194 (2003).

    Article  PubMed  Google Scholar 

  14. van Ede, F., Chekroud, S. R., Stokes, M. G. & Nobre, A. C. Concurrent visual and motor selection during visual working memory guided action. Nat. Neurosci. 22, 477–483 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. D’Esposito, M. & Postle, B. R. The cognitive neuroscience of working memory. Annu. Rev. Psychol. 66, 115–142 (2015).

    Article  PubMed  Google Scholar 

  16. Burgess, N. Spatial memory: how egocentric and allocentric combine. Trends Cogn. Sci. 10, 551–557 (2006).

    Article  PubMed  Google Scholar 

  17. Wolbers, T., Hegarty, M., Büchel, C. & Loomis, J. M. Spatial updating: how the brain keeps track of changing object locations during observer motion. Nat. Neurosci. 11, 1223–1230 (2008).

    Article  CAS  PubMed  Google Scholar 

  18. Aagten-Murphy, D. & Bays, P. M. Independent working memory resources for egocentric and allocentric spatial information. PLoS Comput. Biol. 5, e1006563 (2019).

    Article  CAS  Google Scholar 

  19. Klinghammer, M., Schütz, I., Blohm, G. & Fiehler, K. Allocentric information is used for memory-guided reaching in depth: a virtual reality study. Vision Res. 129, 13–24 (2016).

    Article  PubMed  Google Scholar 

  20. Boon, P. J., Theeuwes, J. & Belopolsky, A. V. Updating visual–spatial working memory during object movement. Vision Res. 94, 51–57 (2014).

    Article  PubMed  Google Scholar 

  21. Bellmund, J. L. S., Gärdenfors, P., Moser, E. I. & Doeller, C. F. Navigating cognition: spatial codes for human thinking. Science 362, eaat6766 (2018).

    Article  CAS  PubMed  Google Scholar 

  22. Meister, M. L. R. & Buffalo, E. A. Neurons in primate entorhinal cortex represent gaze position in multiple spatial reference frames. J. Neurosci. 38, 2430–2441 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  23. Klatzky, R. L. in Spatial Cognition Vol. 1404 (eds Freska, C. et al.) 1–17 (Springer, 1998).

  24. Postle, B. R. & D’Esposito, M. Spatial working memory activity of the caudate nucleus is sensitive to frame of reference. Cogn. Affect. Behav. Neurosci. 3, 133–144 (2003).

    Article  PubMed  Google Scholar 

  25. Shelton, A. L. & McNamara, T. P. Systems of spatial reference in human memory. Cogn. Psychol. 43, 274–310 (2001).

    Article  CAS  PubMed  Google Scholar 

  26. McNamara, T. P., Rump, B. & Werner, S. Egocentric and geocentric frames of reference in memory of large-scale space. Psychon. Bull. Rev. 10, 589–595 (2003).

    Article  PubMed  Google Scholar 

  27. McNamara, T. P. Mental representations of spatial relations. Cogn. Psychol. 18, 87–121 (1986).

    Article  CAS  PubMed  Google Scholar 

  28. Melcher, D. & Colby, C. L. Trans-saccadic perception. Trends Cogn. Sci. 12, 466–473 (2008).

    Article  PubMed  Google Scholar 

  29. Nakamura, K. & Colby, C. L. Updating of the visual representation in monkey striate and extrastriate cortex during saccades. Proc. Natl Acad. Sci. USA 99, 4026–4031 (2002).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Aagten-Murphy, D. & Bays, P. M. in Processes of Visuospatial Attention and Working Memory. Current Topics in Behavioral Neurosciences Vol. 41 (ed. Hodgson, T.) 155–183 (Springer, 2019).

  31. Van der Stigchel, S. & Hollingworth, A. Visuospatial working memory as a fundamental component of the eye movement system. Curr. Dir. Psychol. Sci. 27, 136–143 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  32. Colby, C. L. Action-oriented spatial reference frames in cortex. Neuron 20, 15–24 (1998).

    Article  CAS  PubMed  Google Scholar 

  33. van Ede, F., Chekroud, S. R. & Nobre, A. C. Human gaze tracks attentional focusing in memorized visual space. Nat. Human Behav. 3, 462–470 (2019).

    Article  Google Scholar 

  34. van Ede, F., Board, A. G. & Nobre, A. C. Goal-directed and stimulus-driven selection of internal representations. Proc. Natl Acad. Sci. USA 117, 24590–24598 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Maris, E. & Oostenveld, R. Nonparametric statistical testing of EEG- and MEG-data. J. Neurosci. Methods 164, 177–190 (2007).

    Article  PubMed  Google Scholar 

  36. Spivey, M. J. & Geng, J. J. Oculomotor mechanisms activated by imagery and memory: eye movements to absent objects. Psychol. Res. 65, 235–241 (2001).

    Article  CAS  PubMed  Google Scholar 

  37. Ferreira, F., Apel, J. & Henderson, J. M. Taking a new look at looking at nothing. Trends Cogn. Sci. 12, 405–410 (2008).

    Article  PubMed  Google Scholar 

  38. Johansson, R. & Johansson, M. Look here, eye movements play a functional role in memory retrieval. Psychol. Sci. 25, 236–242 (2014).

    Article  PubMed  Google Scholar 

  39. Engbert, R. & Kliegl, R. Microsaccades uncover the orientation of covert attention. Vision Res. 43, 1035–1045 (2003).

    Article  PubMed  Google Scholar 

  40. Hafed, Z. M. & Clark, J. J. Microsaccades as an overt measure of covert attention shifts. Vision Res. 42, 2533–2545 (2002).

    Article  PubMed  Google Scholar 

  41. Corneil, B. D. & Munoz, D. P. Overt responses during covert orienting. Neuron 82, 1230–1243 (2014).

    Article  CAS  PubMed  Google Scholar 

  42. Ballard, D., Hayhoe, M. M. & Pelz, J. B. Memory representations in natural tasks. J. Cogn. Neurosci. 7, 66–80 (1995).

    Article  CAS  PubMed  Google Scholar 

  43. Saredakis, D. et al. Factors associated with virtual reality sickness in head-mounted displays: a systematic review and meta-analysis. Front. Hum. Neurosci. 14, 96 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  44. Mou, W. & McNamara, T. P. Intrinsic frames of reference in spatial memory. J. Exp. Psychol. Learn. Mem. Cogn. 28, 162–170 (2002).

    Article  PubMed  Google Scholar 

  45. Mou, W., McNamara, T. P., Rump, B. & Xiao, C. Roles of egocentric and allocentric spatial representations in locomotion and reorientation. J. Exp. Psychol. Learn. Mem. Cogn. 32, 1274–1290 (2006).

    Article  PubMed  Google Scholar 

  46. Tatler, B. W. et al. Priorities for selection and representation in natural tasks. Philos. Trans. R. Soc. Lond. B 368, 20130066 (2013).

    Article  Google Scholar 

  47. Scarfe, P. & Glennerster, A. The science behind virtual reality displays. Annu. Rev. Vis. Sci. 5, 529–547 (2019).

    Article  PubMed  Google Scholar 

  48. Iglói, K., Doeller, C. F., Berthoz, A., Rondi-Reig, L. & Burgess, N. Lateralized human hippocampal activity predicts navigation based on sequence or place memory. Proc. Natl Acad. Sci. USA 107, 14466–14471 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  49. Meilinger, T., Knauff, M. & Bulthoff, H. H. Working memory in wayfinding – a dual task experiment in a virtual city. Cogn. Sci. 32, 755–770 (2008).

    Article  PubMed  Google Scholar 

  50. Draschkow, D. & Võ, M. L.-H. Scene grammar shapes the way we interact with objects, strengthens memories, and speeds search. Sci. Rep. 7, 16471 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  51. Li, C. L., Aivar, M. P., Kit, D. M., Tong, M. H. & Hayhoe, M. M. Memory and visual search in naturalistic 2D and 3D environments. J. Vis. 16, 9 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  52. Helbing, J., Draschkow, D. & Võ, M. L.-H. Search superiority: goal-directed attentional allocation creates more reliable incidental identity and location memory than explicit encoding in naturalistic virtual environments. Cognition 196, 104147 (2020).

    Article  PubMed  Google Scholar 

  53. Tarr, M. J. & Warren, W. H. Virtual reality in behavioral neuroscience and beyond. Nat. Neurosci. 5, 1089–1092 (2002).

    Article  CAS  PubMed  Google Scholar 

  54. Regan, C. An investigation into nausea and other side-effects of head-coupled immersive virtual reality. Virtual Real. 1, 17–31 (1995).

    Article  Google Scholar 

  55. Lo, W. T. & So, R. H. Y. Cybersickness in the presence of scene rotational movements along different axes. Appl. Ergon. 32, 1–14 (2001).

    Article  CAS  PubMed  Google Scholar 

  56. R Core Team. R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, 2017).

  57. Frossard, J. & Renaud, O. Permutation tests for regression, ANOVA, and comparison of signals: the permuco package. J. Stat. Softw. 99, 1–32 (2021).

    Article  Google Scholar 

  58. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D. & Iverson, G. Bayesian t tests for accepting and rejecting the null hypothesis. Psychon. Bull. Rev. 16, 225–237 (2009).

    Article  PubMed  Google Scholar 

  59. Morey, R. D. & Rouder, J. N. BayesFactor: Computation of Bayes Factors for Common Designs. R version 0.9.12-4.2 https://CRAN.R-project.org/package=BayesFactor (2018).

  60. Kass, R. E. & Raftery, A. E. Bayes factors. J. Am. Stat. Assoc. 90, 773–795 (1995).

    Article  Google Scholar 

  61. Schönbrodt, F. D. & Wagenmakers, E. J. Bayes factor design analysis: planning for compelling evidence. Psychon. Bull. Rev. 25, 128–142 (2018).

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

This research was funded by a Wellcome Trust Senior Investigator Award (104571/Z/14/Z) and a James S. McDonnell Foundation Understanding Human Cognition Collaborative Award (220020448) to A.C.N., a Marie Skłodowska-Curie Fellowship from the European Commission (ACCESS2WM) and an ERC Starting Grant from the European Research Council (MEMTICIPATION, 850636) to F.v.E., and by the National Institutes of Health Research Oxford Health Biomedical Research Centre. The Wellcome Centre for Integrative Neuroimaging is supported by core funding from the Wellcome Trust (203139/Z/16/Z). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. For the purpose of open access, the author has applied a CC BY public copyright license to any Author Accepted Manuscript version arising from this submission.

Author information

Authors and Affiliations

  1. Oxford Centre for Human Brain Activity, Wellcome Centre for Integrative Neuroimaging, Department of Psychiatry, University of Oxford, Oxford, UK

    Dejan Draschkow, Anna C. Nobre & Freek van Ede

  2. Department of Experimental Psychology, University of Oxford, Oxford, UK

    Dejan Draschkow & Anna C. Nobre

  3. Institute for Brain and Behavior Amsterdam, Department of Experimental and Applied Psychology, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands

    Freek van Ede

Authors
  1. Dejan Draschkow
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anna C. Nobre
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Freek van Ede
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

D.D., A.C.N. and F.v.E. conceived and designed the experiments. D.D. programmed the experiments and acquired the data. D.D. and F.v.E. analysed the data. D.D., A.C.N. and F.v.E. interpreted the data. D.D., A.C.N. and F.v.E. drafted and revised the manuscript.

Corresponding author

Correspondence to Dejan Draschkow.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Human Behaviour thanks Joy Geng, Andrey Nikolaev and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Table 1 and Figs. 1–4.

Reporting Summary.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Draschkow, D., Nobre, A.C. & van Ede, F. Multiple spatial frames for immersive working memory. Nat Hum Behav 6, 536–544 (2022). https://doi.org/10.1038/s41562-021-01245-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-021-01245-y

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing