Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

A retinotopic code structures the interaction between perception and memory systems

Abstract

Conventional views of brain organization suggest that regions at the top of the cortical hierarchy processes internally oriented information using an abstract amodal neural code. Despite this, recent reports have described the presence of retinotopic coding at the cortical apex, including the default mode network. What is the functional role of retinotopic coding atop the cortical hierarchy? Here we report that retinotopic coding structures interactions between internally oriented (mnemonic) and externally oriented (perceptual) brain areas. Using functional magnetic resonance imaging, we observed robust inverted (negative) retinotopic coding in category-selective memory areas at the cortical apex, which is functionally linked to the classic (positive) retinotopic coding in category-selective perceptual areas in high-level visual cortex. These functionally linked retinotopic populations in mnemonic and perceptual areas exhibit spatially specific opponent responses during both bottom-up perception and top-down recall, suggesting that these areas are interlocked in a mutually inhibitory dynamic. These results show that retinotopic coding structures interactions between perceptual and mnemonic neural systems, providing a scaffold for their dynamic interaction.

This is a preview of subscription content, access via your institution

Access options

Buy this article

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: pRF mapping reveals retinotopic coding throughout the posterior cortex, comprised of both positive- and negative-amplitude pRFs.
Fig. 2: Transition to mnemonic cortex is marked by the appearance of negative pRFs.
Fig. 3: Memory areas contain smaller pRFs compared to their paired perceptual areas.
Fig. 4: Shared visual-field representations between paired perception and memory areas.
Fig. 5: Positive pRFs in SPAs and negative pRFs in PMAs exhibit a spatially specific push–pull dynamic during memory recall.
Fig. 6: Positive pRFs in SPAs and negative pRFs in PMAs exhibit a push–pull dynamic during perception of familiar scene images.

Similar content being viewed by others

Data availability

Data are available on Open Science Framework (https://osf.io/sm2xf).

Code availability

Code used for data analysis is available on Open Science Framework (https://osf.io/sm2xf).

References

  1. Libby, A. & Buschman, T. J. Rotational dynamics reduce interference between sensory and memory representations. Nat. Neurosci. 24, 715–726 (2021).

    CAS  PubMed  PubMed Central  Google Scholar 

  2. Kiyonaga, A., Scimeca, J. M., Bliss, D. P. & Whitney, D. Serial dependence across perception, attention, and memory. Trends Cogn. Sci. 21, 493 (2017).

    PubMed  PubMed Central  Google Scholar 

  3. Summerfield, C. & de Lange, F. P. Expectation in perceptual decision making: neural and computational mechanisms. Nat. Rev. Neurosci. 15, 745–756 (2014).

    CAS  PubMed  Google Scholar 

  4. Rademaker, R. L., Chunharas, C. & Serences, J. T. Coexisting representations of sensory and mnemonic information in human visual cortex. Nat. Neurosci. 22, 1336–1344 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  5. Favila, S. E., Lee, H. & Kuhl, B. A. Transforming the concept of memory reactivation. Trends Neurosci. 43, 939–950 (2020).

    CAS  PubMed  PubMed Central  Google Scholar 

  6. Yassa, M. A. & Stark, C. E. L. Pattern separation in the hippocampus. Trends Neurosci. 34, 515–525 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  7. Holmes, G. Disturbances of vision by cerebral lesions. Br. J. Ophthalmol. 2, 353–384 (1918).

    CAS  PubMed  PubMed Central  Google Scholar 

  8. Wandell, B. A., Dumoulin, S. O. & Brewer, A. A. Visual field maps in human cortex. Neuron 56, 366–383 (2007).

    CAS  PubMed  Google Scholar 

  9. Guclu, U. & van Gerven, M. A. J. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35, 10005–10014 (2015).

    PubMed  PubMed Central  Google Scholar 

  10. Groen, I. I. A., Dekker, T. M., Knapen, T. & Silson, E. H. Visuospatial coding as ubiquitous scaffolding for human cognition. Trends Cogn. Sci. https://doi.org/10.1016/j.tics.2021.10.011 (2022).

  11. Popham, S. F. et al. Visual and linguistic semantic representations are aligned at the border of human visual cortex. Nat. Neurosci. 24, 1628–1636 (2021).

    CAS  PubMed  Google Scholar 

  12. Huntenburg, J. M., Bazin, P. L. & Margulies, D. S. Large-scale gradients in human cortical organization. Trends Cogn. Sci. 22, 21–31 (2018).

    PubMed  Google Scholar 

  13. Margulies, D. S. et al. Situating the default-mode network along a principal gradient of macroscale cortical organization. Proc. Natl Acad. Sci. USA 113, 12574–12579 (2016).

    ADS  CAS  PubMed  PubMed Central  Google Scholar 

  14. Bellmund, J. L. S., Gärdenfors, P., Moser, E. I. & Doeller, C. F. Navigating cognition: spatial codes for human thinking. Science (1979) 362, eaat6766 (2018).

    Google Scholar 

  15. Szinte, M. & Knapen, T. Visual organization of the default network. Cereb. Cortex 30, 3518–3527 (2020).

    PubMed  Google Scholar 

  16. Christiaan Klink, P., Chen, X., Vanduffel, W. & Roelfsema, P. R. Population receptive fields in non-human primates from whole-brain fMRI and large-scale neurophysiology in visual cortex. eLife 10, e67304 (2021).

    PubMed  PubMed Central  Google Scholar 

  17. Dumoulin, S. O. & Wandell, B. A. Population receptive field estimates in human visual cortex. Neuroimage 39, 647 (2008).

    PubMed  Google Scholar 

  18. Silson, E. H., Chan, A. W. Y., Reynolds, R. C., Kravitz, D. J. & Baker, C. I. A retinotopic basis for the division of high-level scene processing between lateral and ventral human occipitotemporal cortex. J. Neurosci. 35, 11921–11935 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  19. Kundu, P., Inati, S. J., Evans, J. W., Luh, W. M. & Bandettini, P. A. Differentiating BOLD and non-BOLD signals in fMRI time series using multi-echo EPI. Neuroimage 60, 1759–1770 (2012).

    PubMed  Google Scholar 

  20. Steel, A., Garcia, B. D., Silson, E. H. & Robertson, C. E. Evaluating the efficacy of multi-echo ICA denoising on model-based fMRI. Neuroimage 264, 119723 (2022).

    PubMed  Google Scholar 

  21. Wang, L., Mruczek, R. E. B., Arcaro, M. J. & Kastner, S. Probabilistic maps of visual topography in human cortex. Cereb. Cortex 25, 3911–3931 (2015).

    CAS  PubMed  Google Scholar 

  22. Thomas Yeo, B. T. et al. The organization of the human cerebral cortex estimated by intrinsic functional connectivity. J. Neurophysiol. 106, 1125–1165 (2011).

    PubMed Central  Google Scholar 

  23. Shmuel, A. et al. Sustained negative BOLD, blood flow and oxygen consumption response and its coupling to the positive response in the human brain. Neuron 36, 1195–1210 (2002).

    CAS  PubMed  Google Scholar 

  24. Steel, A., Billings, M. M., Silson, E. H. & Robertson, C. E. A network linking scene perception and spatial memory systems in posterior cerebral cortex. Nat. Commun. 12, 1–13 (2021).

    Google Scholar 

  25. Hasson, U., Levy, I., Behrmann, M., Hendler, T. & Malach, R. Eccentricity bias as an organizing principle for human high-order object areas. Neuron 34, 479–490 (2002).

    CAS  PubMed  Google Scholar 

  26. Dilks, D. D., Julian, J. B., Paunov, A. M. & Kanwisher, N. The occipital place area is causally and selectively involved in scene perception. J. Neurosci. 33, 1331–1336 (2013).

    PubMed  PubMed Central  Google Scholar 

  27. Epstein, R. & Kanwisher, N. A cortical representation the local visual environment. Nature 392, 598–601 (1998).

    ADS  CAS  PubMed  Google Scholar 

  28. Breedlove, J. L., St-Yves, G., Olman, C. A. & Naselaris, T. Generative feedback explains distinct brain activity codes for seen and mental images. Curr. Biol. https://doi.org/10.1016/j.cub.2020.04.014 (2020).

  29. Favila, S. E., Kuhl, B. A. & Winawer, J. Perception and memory have distinct spatial tuning properties in human visual cortex. Nat. Commun. 13, 5864 (2022).

    ADS  CAS  PubMed  PubMed Central  Google Scholar 

  30. Bastos, A. M. et al. Canonical microcircuits for predictive coding. Neuron 76, 695–711 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  31. Knapen, T. Topographic connectivity reveals task-dependent retinotopic processing throughout the human brain. Proc. Natl Acad. Sci. USA 118, e2017032118 (2021).

    CAS  PubMed  Google Scholar 

  32. Silson, E. H., Zeidman, P., Knapen, T. & Baker, C. I. Representation of contralateral visual space in the human hippocampus. J. Neurosci. 41, 2382–2392 (2021).

    CAS  PubMed  PubMed Central  Google Scholar 

  33. Zabelina, D. L. & Andrews-Hanna, J. R. Dynamic network interactions supporting internally-oriented cognition. Curr. Opin. Neurobiol. 40, 86–93 (2016).

    CAS  PubMed  Google Scholar 

  34. Shulman, G. L. et al. Common blood flow changes across visual tasks: II. Decreases in cerebral cortex. J. Cogn. Neurosci. 9, 648–663 (1997).

    CAS  PubMed  Google Scholar 

  35. Fox, M. D. et al. The human brain is intrinsically organized into dynamic, anticorrelated functional networks. Proc. Natl Acad. Sci. USA 102, 9673–9678 (2005).

    ADS  CAS  PubMed  PubMed Central  Google Scholar 

  36. Raichle, M. E. The brain’s default mode network. Annu Rev. Neurosci. 38, 433–447 (2015).

    CAS  PubMed  Google Scholar 

  37. Robertson, C. E. et al. Neural representations integrate the current field of view with the remembered 360° panorama in scene-selective cortex. Curr. Biol. 26, 2463–2468 (2016).

    CAS  PubMed  Google Scholar 

  38. Braga, R. M. & Buckner, R. L. Parallel interdigitated distributed networks within the individual estimated by intrinsic functional connectivity. Neuron 95, 457–471.e5 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  39. DiNicola, L. M., Braga, R. M. & Buckner, R. L. Parallel distributed networks dissociate episodic and social functions within the individual. J. Neurophysiol. 123, 1144–1179 (2020).

    PubMed  PubMed Central  Google Scholar 

  40. Silson, E. H., Steel, A. D. & Baker, C. I. Scene-selectivity and retinotopy in medial parietal cortex. Front Hum. Neurosci. 10, 412 (2016).

    PubMed  PubMed Central  Google Scholar 

  41. Silson, E. H., Steel, A., Kidder, A., Gilmore, A. W. & Baker, C. I. Distinct subdivisions of human medial parietal cortex support recollection of people and places. eLife 8, e47391 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  42. Deen, B. & Freiwald, W. A. Parallel systems for social and spatial reasoning within the cortical apex. Preprint at bioRxiv https://doi.org/10.1101/2021.09.23.461550 (2021).

  43. Ranganath, C. & Ritchey, M. Two cortical systems for memory-guided behaviour. Nat. Rev. Neurosci. 13, 713–726 (2012); https://doi.org/10.1038/nrn3338

  44. Pertzov, Y., Avidan, G. & Zohary, E. Multiple reference frames for saccadic planning in the human parietal cortex. J. Neurosci. 31, 1059 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  45. Gardner, J. L., Merriam, E. P., Movshon, J. A. & Heeger, D. J. Maps of visual space in human occipital cortex are retinotopic, not spatiotopic. J. Neurosci. 28, 3988 (2008).

    CAS  PubMed  PubMed Central  Google Scholar 

  46. Fetsch, C. R., Wang, S., Gu, Y., DeAngelis, G. C. & Angelaki, D. E. Spatial reference frames of visual, vestibular, and multimodal heading signals in the dorsal subdivision of the medial superior temporal area. J. Neurosci. 27, 700–712 (2007).

    CAS  PubMed  PubMed Central  Google Scholar 

  47. Golomb, J. D. & Kanwisher, N. Higher level visual cortex represents retinotopic, not spatiotopic, object location. Cereb. Cortex 22, 2794–2810 (2012).

    PubMed  Google Scholar 

  48. Silson, E. H., Groen, I. I. A., Kravitz, D. J. & Baker, C. I. Evaluating the correspondence between face-, scene-, and object-selectivity and retinotopic organization within lateral occipitotemporal cortex. J. Vis. 16, 14–14 (2016).

    PubMed  PubMed Central  Google Scholar 

  49. Swisher, J. D., Halko, M. A., Merabet, L. B., McMains, S. A. & Somers, D. C. Visual topography of human intraparietal sulcus. J. Neurosci. 27, 5326–5337 (2007).

    CAS  PubMed  PubMed Central  Google Scholar 

  50. Yeatman, J. D. et al. The vertical occipital fasciculus: a century of controversy resolved by in vivo measurements. Proc. Natl Acad. Sci. USA 111, E5214–E5223 (2014).

    CAS  PubMed  PubMed Central  Google Scholar 

  51. Peirce, J. W. PsychoPy—psychophysics software in Python. J. Neurosci. Methods. 162, 8–13 (2007).

    PubMed  PubMed Central  Google Scholar 

  52. Weiner, K. S. et al. Defining the most probable location of the parahippocampal place area using cortex-based alignment and cross-validation. Neuroimage 170, 373–384 (2018).

    PubMed  Google Scholar 

  53. Li, X., Morgan, P. S., Ashburner, J., Smith, J. & Rorden, C. The first step for neuroimaging data analysis: DICOM to NIfTI conversion. J. Neurosci. Methods. 264, 47–56 (2016).

    PubMed  Google Scholar 

  54. Fischl, B. FreeSurfer. NeuroImage 62, 774–781 (2012); https://doi.org/10.1016/j.neuroimage.2012.01.021

  55. Fischl, B. et al. Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron 33, 341–355 (2002).

    CAS  PubMed  Google Scholar 

  56. Dale, A. M., Fischl, B. & Sereno, M. I. Cortical surface-based analysis. I. Segmentation and surface reconstruction. Neuroimage 9, 179–194 (1999).

    CAS  PubMed  Google Scholar 

  57. Saad, Z. S. & Reynolds, R. C. SUMA. Neuroimage 62, 768–773 (2012).

    PubMed  Google Scholar 

  58. Cox, R. W. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Computers Biomed. Res. 29, 162–173 (1996).

    ADS  CAS  Google Scholar 

  59. Jo, H. J. et al. Effective preprocessing procedures virtually eliminate distance-dependent motion artifacts in resting state FMRI. J. Appl. Math. 2013, 935154 (2013).

    Google Scholar 

  60. DuPre, E. et al. TE-dependent analysis of multi-echo fMRI with *tedana*. J. Open Source Softw. 6, 3669 (2021).

    ADS  Google Scholar 

  61. DuPre, E. et al. ME-ICA/tedana: 0.0.6. Zenodo https://doi.org/10.5281/ZENODO.2558498 (2019).

  62. Evans, J. W., Kundu, P., Horovitz, S. G. & Bandettini, P. A. Separating slow BOLD from non-BOLD baseline drifts using multi-echo fMRI. Neuroimage 105, 189–197 (2015).

    PubMed  Google Scholar 

  63. Argall, B. D., Saad, Z. S. & Beauchamp, M. S. Simplified intersubject averaging on the cortical surface using SUMA. Hum. Brain Mapp. 27, 14–27 (2006).

    PubMed  Google Scholar 

  64. Julian, J. B., Fedorenko, E., Webster, J. & Kanwisher, N. An algorithmic method for functionally defining regions of interest in the ventral visual pathway. NeuroImage 60, 2357–2364, https://doi.org/10.1016/j.neuroimage.2012.02.055 (2012).

    Article  CAS  PubMed  Google Scholar 

  65. Gomez, J., Barnett, M. & Grill-Spector, K. Extensive childhood experience with Pokémon suggests eccentricity drives organization of visual cortex. Nat. Hum. Behav. 3, 611–624, https://doi.org/10.1038/s41562-019-0592-8 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  66. Gomez, J. et al. Development of population receptive fields in the lateral visual stream improves spatial coding amid stable structural-functional coupling. NeuroImage 188, 59–69 (2019).

    PubMed  Google Scholar 

  67. R Core Team. R: a language and environment for statistical computing (R Foundation for Statistical Computing, 2013); www.r-project.org/

  68. Lawrence, M. A. ez: easy analysis and visualization of factorial experiments. R package version 4.0.2 (2016).

Download references

Acknowledgements

We thank I. Groen for assistance with specific analysis code. This work was supported by the National Institute of Mental Health under award number R01MH130529 (C.E.R.). A.S. was supported by the Neukom Institute for Computational Science and E.H.S. by the Biotechnology and Biological Sciences Research Council award number BB/V003917/1.

Author information

Authors and Affiliations

Authors

Contributions

A.S., E.H.S. and C.E.R. conceived of and designed the experiment. E.H.S. and B.D.G. contributed stimulus code. A.S. and B.D.G. collected the data. A.S. processed the data. A.S. and E.H.S. analyzed the data. A.S., E.H.S. and C.E.R. wrote and edited the paper.

Corresponding authors

Correspondence to Adam Steel or Caroline E. Robertson.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Neuroscience thanks Christopher Baldassano, Alexander Huth and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Transition from positive to negative-amplitude population receptive fields (+pRF, -pRF) moving anteriorly from posterior cerebral cortex is evident in individual participants.

Figure depicts amplitude maps from all participants’ left hemispheres. Only vertices surviving the threshold applied in the main text (R2 > 0.08) are shown. Individual participant SPAs and PMAs used for analysis are drawn in white (PMAs) and black (SPAs).

Extended Data Fig. 2 Retinotopic coding in SPAs and PMAs.

To quantify the extent to which retinotopic coding is expressed within each ROI we first calculated the percentage of suprathreshold pRFs (R2 > 0.08) within our ROIs for each subject separately before testing each against a non-retinotopic prediction using t-tests (that is, t-test versus zero, with Bonferroni correction). Retinotopic coding was significantly present within each ROI (LH; OPA: t(12.51)=, pcorr = 3.35-9, D = 2.99; PPA: t(16) = 8.68, pcorr = 5.67-7, D = 2.17; LPMA: t(16) = 6.23, pcorr = 3.58-5, D = 1.59; VPMA: t(16) = 6.65, pcorr = 1.64-5, D = 1.66; RH; OPA: t(16) = 12.15, pcorr = 5.10-9, D = 3.03; PPA: t(16) = 11.97, pcorr = 6.32-9, D = 3.12; LPMA: t(16) = 8.75, pcorr = 5.05-7, D = 2.18; VPMA: t(16) = 6.39, pcorr = 2.68-5, D = 1.55). Bars represent the mean percentage of suprathreshold pRFs (R2 > 0.08) in each ROI/hemisphere for the lateral (left) and ventral (right) surfaces, respectively. Individual data points are overlaid. Each ROI exhibited a significant percentage of suprathreshold pRFs, ***ptwo-tailed < 0.001.

Extended Data Fig. 3 Comparison between the location of the SPAs, PMAs, and default mode network in one participant.

Comparison between the location of the SPAs, PMAs, and default mode network in one participant (example participant from Main text Fig. 2). This pattern was consistent in all individuals and at the group-level (Main text Fig. 1b). Default mode network defined using the Yeo et al., 2011 parcellation24.

Extended Data Fig. 4 Correlation in trial × trial activation during memory recall aggregated across participants.

Mean BOLD response amplitude relative to baseline during place recall trials for each ROI (OPA, LMPA) and pRF population (+/−).

Extended Data Fig. 5 Differential interaction between pRFs in SPAs with −/+ pRFs in memory areas is evident across all trials.

Each scatter plot and corresponding correlation values depict the unique correlation between pRFs in the SPAs with -pRFs (blue) and +pRFs (red) in the PMAs (for example, correlation between +pRFs in OPA with -pRFs in LPMA, controlling for +pRFs in LPMA) quantified using Pearson’s correlation. Each data point represents the z-scored activation on a given trial for all pRFS in the population (that is, all -LPMA pRFS) for a given subject on a trial.

Extended Data Fig. 6 Trial x trial interaction between −/+ pRFs in the place memory areas and scene perception areas exhibit push-pull interaction in independent data.

Recall trials were identical to the trials used in the localizer. Participants fixated on a dot projected in the center of the screen. They were then cued with the stimulus to be recalled for 1 second, followed by a 1 s dynamic mask, and 10 seconds of imagery. Trials were separated by a 4-8 s jittered interstimulus interval. Participants completed 32 imagery trials (16 for each landmark) separated into two imaging runs. One participant was excluded from the analysis for lack of familiarity with the landmarks; the remaining participants were familiar with the locations and had lived in the Hanover area for at least one year. Two participants did not have -pRFs in the ventral surface regions of interest. We tested for the relationship between +/− pRFs in the scene perception and place memory area using the same approach described in the Main text. We examined the unique correlation between the -/+ pRFs in the place memory areas and scene perception areas (that is, correlation between activation of -pRFs in memory areas with pRFs in scene perception areas, while controlling for activation of +pRFs in the memory areas). Using paired t-tests, we found evidence for the opponent interaction between -pRFs and +pRFs in this independent sample. We found that the relationship between the -/+ pRFs in the memory areas with the scene perception area pRFs was significantly different (Lateral – t(8) = 2.61, p = 0.018; Ventral – t(6) = 7.82, p < 0.0001). As we observed in our original analysis, the majority of participants showed a negative correlation in the trial x trial activation of the -pRFs in the place memory areas with pRFs in the scene perception areas (Ventral – 6/7 participants: t(6) = 2.79, p = 0.031; Lateral – 6/8 participants: t(8) = 1.79, p = 0.11). Likewise, most participants showed a positive relationship between activation of + pRFs in the memory areas and pRFs in the perception areas (Ventral – 7/7 participants; t(6) = 7.77, p = 0.0002; Lateral – 7/8 participants; t(8) = 3.30, p = 0.01). This result gives us confidence that our original analysis was not influenced by potential circularity. * = ptwo-tailed < 0.05, *** = ptwo-tailed < 0.005.

Extended Data Fig. 7 Activation during recall of personally familiar places.

Mean BOLD response amplitude relative to baseline when familiar scenes were presented in each lower quadrant (hemifield: ipsilateral and contralateral) for each ROI (OPA, LPMA) and pRF population (+/−).

Supplementary information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Steel, A., Silson, E.H., Garcia, B.D. et al. A retinotopic code structures the interaction between perception and memory systems. Nat Neurosci 27, 339–347 (2024). https://doi.org/10.1038/s41593-023-01512-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41593-023-01512-3

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing