Cognition depends on integrating sensory percepts with the memory of recent stimuli. However, the distributed nature of neural coding can lead to interference between sensory and memory representations. Here, we show that the brain mitigates such interference by rotating sensory representations into orthogonal memory representations over time. To study how sensory inputs and memories are represented, we recorded neurons from the auditory cortex of mice as they implicitly learned sequences of sounds. We found that the neural population represented sensory inputs and the memory of recent stimuli in two orthogonal dimensions. The transformation of sensory information into a memory was facilitated by a combination of ‘stable’ neurons, which maintained their selectivity over time, and ‘switching’ neurons, which inverted their selectivity over time. Together, these neural responses rotated the population representation, transforming sensory inputs into memory. Theoretical modeling showed that this rotational dynamic is an efficient mechanism for generating orthogonal representations, thereby protecting memories from sensory interference.
Subscribe to Journal
Get full journal access for 1 year
only $4.92 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
The data that support each main figure are included as source data. Original data are available upon reasonable request. Source data are provided with this paper.
The code supporting the implementation and analysis of the neural network model is available on our lab GitHub repository (www.github.com/buschman-lab). As the model was analyzed in the same way as the neural data, the same analysis code can be applied to neural data.
Summerfield, C. & de Lange, F. P. Expectation in perceptual decision making: neural and computational mechanisms. Nat. Rev. Neurosci. 15, 745–756 (2014).
Kiyonaga, A., Scimeca, J. M., Bliss, D. P. & Whitney, D. Serial dependence across perception, attention, and memory. Trends Cogn. Sci. 21, 493–497 (2017).
de Lange, F. P., Heilbron, M. & Kok, P. How do expectations shape perception? Trends Cogn. Sci. 22, 764–779 (2018).
Fiser, A. et al. Experience-dependent spatial expectations in mouse visual cortex. Nat. Neurosci. 19, 1658–1664 (2016).
Jaramillo, S. & Zador, A. M. The auditory cortex mediates the perceptual effects of acoustic temporal expectation. Nat. Neurosci. 14, 246–251 (2011).
Chun, M. M. & Jiang, Y. Contextual cueing: implicit learning and memory of visual context guides spatial attention. Cogn. Psychol. 36, 28–71 (1998).
Dehaene, S., Meyniel, F., Wacongne, C., Wang, L. & Pallier, C. The neural representation of sequences: from transition probabilities to algebraic patterns and linguistic trees. Neuron 88, 2–19 (2015).
Carandini, M. & Heeger, D. J. Normalization as a canonical neural computation. Nat. Rev. Neurosci. 13, 51–62 (2012).
Buschman, T. J., Siegel, M., Roy, J. E. & Miller, E. K. Neural substrates of cognitive capacity limitations. Proc. Natl Acad. Sci. USA 108, 11252–11255 (2011).
Sprague, T. C., Ester, E. F. & Serences, J. T. Reconstructions of information in visual spatial working memory degrade with memory load. Curr. Biol. 24, 2174–2180 (2014).
Bouchacourt, F. & Buschman, T. J. A flexible model of working memory. Neuron 103, 147–160.e8 (2019).
White, O. L., Lee, D. D. & Sompolinsky, H. Short-term memory in orthogonal neural networks. Phys. Rev. Lett. 92, 148102 (2004).
Botvinick, M. M. & Plaut, D. C. Short-term memory for serial order: a recurrent neural network model. Psychol. Rev. 113, 201–233 (2006).
Rigotti, M. et al. The importance of mixed selectivity in complex cognitive tasks. Nature 497, 585–590 (2013).
Sakai, K. & Miyashita, Y. Neural organization for the long-term memory of paired associates. Nature 354, 152–155 (1991).
Miyashita, Y. & Chang, H. S. Neuronal correlate of pictorial short-term memory in the primate temporal cortex. Nature 331, 68–70 (1988).
Gavornik, J. P. & Bear, M. F. Learned spatiotemporal sequence recognition and prediction in primary visual cortex. Nat. Neurosci. 17, 732–737 (2014).
Li, N. & DiCarlo, J. J. Unsupervised natural experience rapidly alters invariant object representation in visual cortex. Science 321, 1502–1507 (2008).
Maheu, M., Dehaene, S. & Meyniel, F. Brain signatures of a multiscale process of sequence learning in humans. eLife 8, e41541 (2019).
Kim, R., Seitz, A., Feenstra, H. & Shams, L. Testing assumptions of statistical learning: is it long-term and implicit? Neurosci. Lett. 461, 145–149 (2009).
Yakovlev, V., Fusi, S., Berman, E. & Zohary, E. Inter-trial neuronal activity in inferior temporal cortex: a putative vehicle to generate long-term visual associations. Nat. Neurosci. 1, 310–317 (1998).
Griniasty, M., Tsodyks, M. V. & Amit, D. J. Conversion of temporal correlations between stimuli to spatial correlations between attractors. Neural Comput. 5, 1–17 (1993).
Amit, D., Brunel, N. & Tsodyks, M. Correlations of cortical Hebbian reverberations: theory versus experiment. J. Neurosci. 14, 6435–6445 (1994).
den Ouden, H. E. M., Friston, K. J., Daw, N. D., McIntosh, A. R. & Stephan, K. E. A dual role for prediction error in associative learning. Cereb. Cortex 19, 1175–1185 (2009).
Eagleman, D. M. Motion integration and postdiction in visual awareness. Science 287, 2036–2038 (2000).
Aru, J., Tulver, K. & Bachmann, T. It’s all in your head: expectations create illusory perception in a dual-task setup. Conscious. Cogn. 65, 197–208 (2018).
Choi, H. & Scholl, B. J. Perceiving causality after the fact: postdiction in the temporal dynamics of causal perception. Perception 35, 385–399 (2006).
Fischer, J. & Whitney, D. Serial dependence in visual perception. Nat. Neurosci. 17, 738–743 (2014).
Elsayed, G. F., Lara, A. H., Kaufman, M. T., Churchland, M. M. & Cunningham, J. P. Reorganization between preparatory and movement population responses in motor cortex. Nat. Commun. 7, 13239 (2016).
Itskov, P. M., Vinnik, E. & Diamond, M. E. Hippocampal representation of touch-guided behavior in rats: persistent and independent traces of stimulus and reward location. PLoS ONE 6, e16462 (2011).
Levine, J. H. et al. Data-driven phenotypic dissection of AML reveals progenitor-like cells that correlate with prognosis. Cell 162, 184–197 (2015).
Rigotti, M., Ben Dayan Rubin, D. D., Wang, X.-J. & Fusi, S. Internal representation of task rules by recurrent dynamics: the importance of the diversity of neural responses. Front. Comput. Neurosci. 4, 24 (2010).
Olshausen, B. A. & Field, D. J. Sparse coding of sensory inputs. Curr. Opin. Neurobiol. 14, 481–487 (2004).
Rust, N. C. & DiCarlo, J. J. Balanced increases in selectivity and tolerance produce constant sparseness along the ventral visual stream. J. Neurosci. 32, 10170–10182 (2012).
Bassett, D. S. & Sporns, O. Network neuroscience. Nat. Neurosci. 20, 353–364 (2017).
Murray, J. D. et al. Stable population coding for working memory coexists with heterogeneous neural dynamics in prefrontal cortex. Proc. Natl Acad. Sci. USA 114, 394–399 (2017).
Stokes, M. G. et al. Dynamic coding for cognitive control in prefrontal cortex. Neuron 78, 364–375 (2013).
Freedman, D. J., Riesenhuber, M., Poggio, T. & Miller, E. K. Visual categorization and the primate prefrontal cortex: neurophysiology and behavior. J. Neurophysiol. 88, 929–941 (2002).
Fuster, J. M. & Alexander, G. E. Neuron activity related to short-term memory. Science 173, 652–654 (1971).
Warden, M. R. & Miller, E. K. The representation of multiple objects in prefrontal neuronal delay activity. Cereb. Cortex 17, i41–i50 (2007).
Spaak, E., Watanabe, K., Funahashi, S. & Stokes, M. G. Stable and dynamic coding for working memory in primate prefrontal cortex. J. Neurosci. 37, 6503–6516 (2017).
Miller, P. & Wang, X.-J. Inhibitory control by an integral feedback signal in prefrontal cortex: a model of discrimination between sequential stimuli. Proc. Natl Acad. Sci. USA 103, 201–206 (2006).
Postle, B. R. The cognitive neuroscience of visual short-term memory. Curr. Opin. Behav. Sci. 1, 40–46 (2015).
Chaudhuri, R. & Fiete, I. Computational principles of memory. Nat. Neurosci. 19, 394–403 (2016).
Meyers, E. M. Dynamic population coding and its relationship to working memory. J. Neurophysiol. 120, 2260–2268 (2018).
Riley, M. R. & Constantinidis, C. Role of prefrontal persistent activity in working memory. Front. Syst. Neurosci. 9, 181 (2016).
Perez, F. & Granger, B. E. IPython: a system for interactive scientific computing. Comput. Sci. Eng. 9, 21–29 (2007).
Millman, K. J. & Aivazis, M. Python for scientists and engineers. Comput. Sci. Eng. 13, 9–12 (2011).
Oliphant, T. E. Python for scientific computing. Comput. Sci. Eng. 9, 10–20 (2007).
Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).
Walt, S., van der Colbert, S. C. & Varoquaux, G. The NumPy Array: a structure for efficient numerical computation. Comput. Sci. Eng. 13, 22–30 (2011).
McKinney, W. Data structures for statistical computing in Python. Proc. 9th Python Sci. Conf. 445, 56–61 (2010).
Hunter, J. D. Matplotlib: a 2D graphics environment. Comput. Sci. Eng. 9, 90–95 (2007).
Manly, B. Randomization, Bootstrap and Monte Carlo Methods in Biology (Chapman & Hall/CRC, 1997).
Nicosia, V., Mangioni, G., Carchiolo, V. & Malgeri, M. Extending the definition of modularity to directed graphs with overlapping communities. J. Stat. Mech. Theory Exp. 2009, P03024 (2009).
Blondel, V. D., Guillaume, J.-L., Lambiotte, R. & Lefebvre, E. Fast unfolding of communities in large networks. J. Stat. Mech. Theory Exp. 2008, P10008 (2008).
Rousseeuw, P. J. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, 53–65 (1987).
McInnes, L., Healy, J. & Melville, J. UMAP: uniform manifold approximation and projection for dimension reduction. Preprint at arXiv https://arxiv.org/abs/1802.03426 (2018).
Beyer, K., Goldstein, J., Ramakrishnan, R. & Shaft, U. in Database Theory—ICDT’99 (eds Beeri, C. & Buneman, P.) 217–235 (Springer, 1999).
Wasmuht, D. F., Spaak, E., Buschman, T. J., Miller, E. K. & Stokes, M. G. Intrinsic neuronal dynamics predict distinct functional roles during working memory. Nat. Commun. 9, 3499 (2018).
The authors thank C. MacDowell, M. Panichello, C. Jahn, F. Bouchacourt, P. Hoyos and S. Henrickson for their detailed feedback during the writing of this manuscript. We also thank B. Briones for helping with histology and B. Morea for helping with surgery. We thank the Princeton Laboratory Animal Resources staff for their support. This work was supported by NIMH R01MH115042, ONR N000141410681 and NIH DP2EY025446 to T.J.B.
The authors declare no competing interests.
Peer review information Nature Neuroscience thanks Omri Barak and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
a, The neural population encoding of B/Y shown on (a) Day 1 and (b) Day 4. For each of the four conditions, the plot shows the mean ± s.e.m. of the population projection onto the B/Y sensory axis. Yellow outlines B/Y training period (185–285 ms). For panels a-e, n = 1064 withheld trials, z-scored and then combined across animals per day. Positive and negative projections indicate Y (green) and B (purple) encoding, respectively. Light and dark grey horizontal bars mark significant differences for AB vs XY and C vs. C*, respectively (two-sided t-tests, p ≤ 0.001, Bonferroni corrected). c, Data show mean ± s.e.m. of B/Y stimulus encoding strength on the B/Y sensory axis. Negatively labeled conditions (that is, B) were inverted, such that positive values on y-axis indicate B and Y trials are ‘correctly’ encoded as B and Y, respectively. Day 1 = 0.34 ± 0.021, Day 2 = 0.33 ± 0.22, Day 3 = 0.33 ± 0.022, Day 4 = 0.31 ± 0.021, all days p < 1/5000 two-sided bootstrap tests. Slope across days mean ± s.e.m. = −0.01 ± 0.01, p = 0.16, one-sided bootstrap test. d, Points show mean ± s.e.m. of A/X stimulus encoding strength on the B/Y sensory axis, during A/X stimulus presentation. For panels d-f, lines and shaded regions show mean and 95% CI of bootstrapped linear regressions. Positive values indicate correct A/X encoding: Day 1 = 0.13 ± 0.021, p < 1/5000, Day 2 = 0.062 ± 0.023, p = 0.0064, Day 3 = 0.096 ± 0.022, Day 4 = 0.031 ± 0.023, p = 0.17, all two-sided bootstrap tests. Slope across days mean ± s.e.m. = −0.028 ± 0.01, p = 0.0016, one-sided bootstrap test. e, Points show mean ± s.e.m. of C/C* stimulus encoding strength on the B/Y sensory axis. Positive values indicate correct encoding of C/C* association on the B/Y sensory axis (that is, C and C* should go in B and Y direction, respectively). Day 1 = −0.10 ± 0.023, p < 1/5000, Day 2 = −0.056 ± 0.024, p = 0.016, Day 3 = −0.006 ± 0.023, p = 0.81, Day 4 = −0.044 ± 0.023, p = 0.055, all two-sided bootstrap tests. Slope across days mean ± s.e.m. = 0.022 ± 0.01, p = 0.017, one-sided bootstrap test. Note, this trend does not appear in analysis of blocks of trials within a day (Supplementary Fig. 2f). f-g, Points show mean ± s.e.m. of angles between B/Y sensory axis and (f) A/X and (g) C/C* sensory axes (n = 5000 resamples of neurons). Significant differences from 90 degrees shown by grey boxes (p≤0.01, one-sided bootstrap tests). Significant change in angle to A/X sensory axis over time is shown by grey line (shaded region is 95% confidence interval of bootstrapped linear regression). The B/Y and A/X sensory axes were initially aligned, but became orthogonal over days: change over days, slope = 0.84 ± 0.24, p < 1/5000, one-sided bootstrap test. B/Y and C/C* sensory axes were always orthogonal. Change over days: slope = 0.29 ± 0.2, p = 0.077, one-sided bootstrap test. For all panels, p-values: * ≤ 0.05, ** ≤ 0.01, *** ≤ 0.001.
For each classifier, the accuracy (y-axis) was measured as the area under the curve (AUC; see methods). Accuracy was calculated using data from trials withheld from training (n = 152 trials per animal) and was calculated in a sliding window fashion (25 ms windows, stepped 10 ms). Lines show mean ± s.e.m. of accuracy timecourses (n = 7 animals). Day 1 and 4 shown in left and right panels, respectively. a, A/X sensory classifier performance over time, shown for decoding A/X (orange) and C/C* (blue) stimuli. Orange rectangle indicates A/X training period (10–110 ms). b, C/C* sensory classifier performance over time. Line colors follow panel a. Blue rectangle indicates C/C* training period (360–460 ms). Consistent with predictive coding shown by projections in Fig. 1g-h, on Day 4 the C/C* sensory classifier decoded A/X during the A/X stimulus and immediately before C/C*. c, A/X sensory classifier performance shown for expected trials (black line - ABCD vs. XYC*D) and unexpected trials (grey line - ABC*D vs. XYCD). Consistent with postdiction results shown in Fig. 3b, the A/X sensory classifier performs well on expected trials, but incorrectly classifies A/X during unexpected C/C* trials. d, A/X memory classifier performance over time, shown for A/X discrimination (dark blue) and C/C* (light blue). Dark blue rectangle indicates training period (360–460 ms). Consistent with projection results shown in Fig. 3c-d and Extended Data Figs. 5 and 6, A/X memory classifier can decode A/X near the end, but not beginning of the trial. A/X discrimination of C/C* is close to chance (AUC = 0.5), reflecting the fact that the two axes are independent. e, A/X memory classifier performance, divided by expectation. Colors follow panel c. The A/X memory classifier performs well at discriminating A/X on both expected and unexpected trials.
a-b, Lines show mean ± s.e.m. of neural activity (trials balanced across conditions) projected onto a C/C* chord encoding axis on (a) Day 1 (n = 4204) and (b) Day 4 (n = 4196). The C/C* chord encoding axis was trained using the firing rate response to the C/C* chord presented in isolation, outside of sequences (n = 300 trials). Line colors indicate trial types (ABCD – orange, ABC*D – pink, XYCD – green, XYC*D – blue) and line style indicates 3rd chord type (C – solid, C* – dashed). Positive and negative projections indicate C* and C encoding, respectively. Light and dark grey horizontal bars mark significant differences for AB vs. XY and C vs. C*, respectively (two-sided t-test, p ≤ 0.001, Bonferroni corrected). Results are consistent with projections onto the C/C* sensory axis (Fig. 1g,h). c, Points show mean ± s.e.m. of C/C* prediction strength during the A/X stimulus, which grew over days. Positive prediction (y-axis) indicates the C/C* chord sensory axis correctly encoded the association (that is, C during A and C* during X) during the A/X stimulus (black outline in panels a-b, 10–110 ms). Day 1 = 0.017 ± 0.023, p = 0.44, Day 2 = 0.025 ± 0.023, p = 0.27, Day 3 = 0.071 ± 0.023, p = 0.0008, Day 4 = 0.14 ± 0.022, p < 1/5000, two-sided bootstrap tests. Trials used in other projection analyses were also used here (n = 1064). For panels c-d, lines and shaded region show mean and 95% CI of bootstrapped linear regressions. Consistent with Fig. 1i, the prediction along the C/C* sensory chord axis increased across days; slope mean ± s.e.m. = 0.04 ± 0.01, p < 1/5000, one-sided bootstrap test. d, Violin plots show bootstrapped distributions of the angle between A/X sensory and C/C* chord sensory axes (n = 5000 resamples of neurons). The mean ± s.e.m. angle between axes by day (degrees): Day 1 = 95 ± 6.5, p = 0.19; Day 2 = 84 ± 5.9, p = 0.16, Day 3 = 78 ± 5.0, p = 0.011, Day 4 = 83 ± 4.4, p = 0.064, one-sided bootstrap tests against 90 degrees. Regression across days: slope = −4.3 ± 2.5, p = 0.039, one-sided bootstrap test. e, Angle between A/X memory and C/C* chord sensory axes. Angle (degrees) on Day 1 = 88 ± 7.0, p = 0.36; Day 2 = 103 ± 6.3, p = 0.019, Day 3 = 95 ± 4.5, p = 0.14, Day 4 = 83 ± 5.2, p = 0.10, one-sided bootstrap tests against 90 degrees. Regression across days: slope = −2.2 ± 2.8, p = 0.21, one-sided bootstrap test. For all panels, p-values: * ≤ 0.05, ** ≤ 0.01, *** ≤ 0.001.
a, Neural activity projected into A/X-C/C* state space for Day 1 (left) and Day 4 (right). Lines show mean projections of neural activity onto the A/X sensory axis (x-axis) and C/C* sensory axis (y-axis; n = 1064 trials). Activity is shown during the A/X stimulus presentation (−10–170 ms) for each of four trial types, indicated by legend. Marker saturation increases with time, as shown in sequence timecourse legend above graph. Inset shows principal components (PCs) of neural trajectories in grey; black arrow size matches percentage of explained variance per PC (see methods). On day 1, the neural trajectory moved predominately along the A/X encoding axis (x-axis). By day 4, the neural trajectories followed an angle, encoding both A/X and the expected C/C* information (y-axis). b, The angle of PC1 (relative to horizontal) during the A/X period increased across days. Radial lines show the circular mean ± s.e.m. of angle shown for Day 1 (light grey) and Day 4 (dark grey). Angle of PC1 per day (degrees): Day 1 = 18 ± 2.9, Day 2 = 14 ± 3.6, Day 3 = 11 ± 4.7, Day 4 = 31 ± 2.3 degrees (bootstrap, n = 5000 resamples of neurons). Change in angle across days, slope = 3.7 ± 2.4, p = 0.0028, one-sided bootstrap test. c, Neural activity during the C/C* stimulus period (340 to 520 ms) projected into A/X-C/C* state space, as in panel a. d, The angle of PC1 (relative to vertical) during the C/C* period decreased across days. Format follows panel b. Angle of PC1 per day (degrees): Day 1 = 79 ± 3.5, Day 2 = 74 ± 3.8, Day 3 = 77 ± 4.3, Day 4 = 58 ± 2.7 (bootstrap); change in angle across days, slope = −6.0 ± 1.4, p < 1/5000, one-sided bootstrap test. For all panels, p-values: * ≤ 0.05, ** ≤ 0.01, *** ≤ 0.001.
a,b, The neural population encoding of A/X memory shown on (a) Day 1 and (b) Day 4. For each of the four conditions, the lines show the mean ± s.e.m. of the population projection onto the A/X memory axis (blue outlines A/X memory training period; for all panels, n = 1064 withheld trials, combined across animals per day). Positive and negative projections indicate XY and AB encoding, respectively. Light and dark grey bars mark significant differences for AB vs. XY and C vs. C* respectively (two-sided t-test, p≤0.001, Bonferroni corrected). c-d, Neural activity projected into A/X memory - C/C* state space for (c) Day 1 and (d) Day 4, around the C/C* stimulus presentation (340–520 ms). The x-axis and y-axis are the projections of neural activity onto the A/X memory axis and the C/C* sensory axis, respectively. Marker saturation increases with time (shown across top). Inset shows PCs of neural trajectories in grey; black arrow size matches percentage of explained variance per PC (for distributions see Fig. 3h). e, Violin plots show distribution of the dimensionality of the full neural response during the C/C* stimulus presentation. For each day, PCA was performed on the firing rate responses across a pseudo population (neurons were concatenated across animals; see methods). Similar to Fig. 3h, the dimensionality was estimated using the explained variance ratio (EVR) of the first two PCs (see methods). Dimensionality of the neural responses tended to decrease over days, as shown by the increased in the EVR of first two PCs: Day 1 = 0.63 ± 0.062, Day 2 = 0.54 ± 0.056, Day 3 = 0.68 ± 0.064, Day 4 = 0.76 ± 0.091 (bootstrap, n = 5000 resamples of neurons). Change in EVR of first two PCs over days: slope mean ± s.e.m. = 0.053 ± 0.034, p = 0.065, one-sided bootstrap test.
a, Cross-temporal performance of A/X classifiers. A series of A/X classifiers were trained across the sequence (x-axis; 25 ms windows, stepping by 10 ms) and then each classifier was tested across the sequence (y-axis). Color indicates the average correct projection on withheld data for all combinations of training times and test times. White bars indicated onset and offset of A/X, B/Y, C/C* stimuli. Note, the low cross-temporal decoding performance between the A/X and C/C* time periods reflects the temporal dynamics of the representation of A/X during the sequence. b, Points show mean ± s.e.m. of correct projection along the A/X sensory axis (orange) and A/X memory axis (blue), during the first three stimuli in the sequence (A/X, B/Y, and C/C* columns). Positive values indicate correct encoding strength; negatively encoded conditions (that is, A) were inverted before averaging. Horizontal bars indicate significant differences between A/X encoding during the A/X stimulus and C/C* stimulus. The A/X sensory axis had stronger A/X encoding during A/X sensory compared to the C/C* stimulus (differences per day: Day 1 = 0.31, Day 2 = 0.26, Day 3 = 0.27, Day 4 = 0.32, all p≤1/5000, one-sided permutation tests). The A/X memory axis had stronger encoded A/X encoding during the C/C* stimulus compared to the A/X stimulus (differences per day: Day 1 = −0.19, Day 2 = −0.22, Day 3 = −0.11, Day 4 = −0.17, all p = 0.0002, one-sided permutation tests). (A/X Stimulus) Projections of neural activity during the A/X stimulus (10–110 ms) onto A/X sensory axis (mean ± s.e.m.): Day 1 = 0.37 ± 0.2, Day 2 = 0.27 ± 0.022, Day 3 = 0.28 ± 0.022, Day 4 = 0.33 ± 0.021, all p < 1/5000. Onto A/X memory axis: Day 1 = 0.053 ± 0.023, p = 0.022, Day 2 = 0.06 ± 0.024, p = 0.013, Day 3 = 0.12 ± 0.023, p < 1/5000, Day 4 = 0.053 ± 0.023, p = 0.019 (all two-sided bootstrap tests). During the A/X stimulus, A/X sensory encoding was stronger than A/X memory encoding on all days (Sen. – Mem. differences: Day 1 = 0.31, Day 2 = 0.21, Day 3 = 0.16, Day 4 = 0.27, all p≤1/5000, one-sided permutation tests). (B/Y Stimulus) Projections of neural activity during the B/Y stimulus (180–280 ms) onto the A/X sensory axis: Day 1 = 0.12 ± 0.022, p < 1/5000, Day 2 = 0.0038 ± 0.024, p = 0.87, Day 3 = 0.12 ± 0.023, p < 1/5000, Day 4 = 0.046 ± 0.024, p = 0.046. Onto A/X memory axis: Day 1 = 0.07 ± 0.023, p = 0.0044, Day 2 = 0.22 ± 0.23, p < 1/5000, Day 3 = 0.19 ± 0.23, p < 1/5000, Day 4 = 0.11 ± 0.024, p < 1/5000 (all two-sided bootstrap tests). During B/Y stimulus, A/X sensory encoding was slightly stronger than A/X memory on Day 1 (Sen. – Mem. diff. = 0.05, p = 0.064), but after experience, A/X memory encoding of A/X information was significantly stronger than A/X sensory encoding (Day 2 = −0.21, p = 0.0002, Day 3 = −0.07, p = 0.017, Day 4 diff. = −0.07, p = 0.02, all one-sided permutation tests). (C/C* Stimulus) Projections of neural activity during the C/C* stimulus (360–460 ms) onto A/X sensory axis: Day 1 = 0.053 ± 0.023, p = 0.021, Day 2 = 0.0034 ± 0.023, p = 0.87, Day 3 = 0.008 ± 0.023, p = 0.72, Day 4 = 0.0069 ± 0.023, p = 0.77. Onto A/X memory axis: Day 1 = 0.24 ± 0.023, Day 2 = 0.28 ± 0.023, Day 1 = 0.24 ± 0.024, Day 4 = 0.23 ± 0.024, all p < 1/5000 (all two-sided bootstrap tests). During the C/C* stimulus, the A/X memory encoding was stronger than A/X sensory encoding on all days (Sen – Mem. differences: Day 1 = −0.19, Day 2 = −0.28, Day 3 = −0.23, Day 4 = −0.22, p = 0.0002, all one-sided permutation tests). c, Violin plots show distribution of when A/X memory encoding strength crossed A/X sensory encoding strength. Horizontal line indicates mean. Mean ± s.e.m. of switch times (ms) relative to sequence onset: Day 1 = 248 ± 13, Day 2 = 182 ± 10, Day 3 = 178 ± 22, Day 4 = 194 ± 22 (n = 5000, bootstrap over trials). The switch time decreased over days (slope mean ± s.e.m. = −16 ± 8.2, p = 0.02, one-sided bootstrap test). The change in switch time decreased the most between days 1 and 3 and then stabilized by day 4 (Day 4-3 diff. = 16.11 ± 31, p = 0.31, one-sided bootstrap test). For all panels, p-values: * ≤ 0.05, ** ≤ 0.01, *** ≤ 0.001.
Trial-by-trial correlation of encoding strength along three relevant axes: A/X sensory, C/C* sensory, and A/X memory. Positive and negative values on each encoding axis indicate correct and incorrect projections, respectively. All lines show mean and 95% confidence interval of bootstrapped linear regressions; slope, correlation (r) and p-values (all one-sided bootstrap tests, uncorrected for multiple comparisons across panels) are listed in plots. a-b, Correlation between A/X sensory encoding strength (x-axis; 10–110 ms) and A/X memory encoding strength (y-axis; 360–460 ms) on (a) Day 1 and (b) Day 4. Consistent with a transformation of A/X information from sensory to memory, there is a significant correlation on Day 1 and 4. c-f, Relationship between A/X encoding strength (x-axis) and C/C* sensory encoding strength (y-axis). A/X encoding strength by the sensory and memory axes was estimated during the 50 ms prior to C/C* onset (300–350 ms). C/C* sensory encoding strength was estimated during C/C* (360–460 ms). Panels show correlations between C/C* representation and A/X sensory representation (c and d) or A/X memory representation (e and f). Correlations are shown for both expected stimuli (c and e; ABCD, XYC*D) and unexpected stimuli (d and f; ABC*D, XYCD). c, On day 4, A/X sensory encoding was positively correlated with C/C* encoding accuracy on expected trials (ABCD, XYC*D). d, On day 4, A/X sensory encoding was negatively correlated with C/C* encoding accuracy on unexpected trials (ABC*D, XYCD). e, On day 4 there was no significant correlation between A/X memory encoding accuracy and C/C* encoding on expected trials. f, On day 4, A/X memory encoding accuracy was positively correlated with C/C* encoding during unexpected trials.
Phenograph clustering (Fig. 5a) was applied to z-scored firing rate differences calculated for specific C/C*/Cmix stimuli. a, Z-scored differences were calculated for XYCD-ABCD (left), XYC*D-ABC*D (middle), and XYCmixD-ABCmixD (right) pairs of conditions. Lines show mean ± s.e.m. of A/X selectivity over time per original Phenograph cluster (n = 522). Note, Cmix trials involved presenting a novel stimulus that was a mix between the two chords making up C and C*; ABCmixD and XYCmixD sequences occurred on 12% of trials, randomly distributed throughout the day (see methods). b, Data points show the individual neurons’ original AB-XY z-scored differences (x-axis) were highly correlated with z-scored differences calculated on the ‘C’ trials (XYCD-ABCD), during sensory (dark blue, left; r = 0.91 ± 0.01) and memory (dark blue, right; r = 0.89 ± 0.02) time periods. Similarly, the correlation was high to ‘C*’ trials (XYCD-ABC*D) during sensory (blue, left; r = 0.92 ± 0.01) and memory (blue, right; r = 0.87 ± 0.02) time periods. Finally, this correlation was also seen on Cmix trials (XYCmixD-ABCmixD) during both sensory (green, left; r = 0.8 ± 0.02) and memory (green, right; r = 0.73 ± 0.02) time periods. All correlations were significant (p < 1/5000, one-sided bootstrapped linear regressions, n = 5000 resamples across neurons).
a, Sensitivity index (d-prime) calculated between all pairs of the four Phenograph clusters (see methods). Red line shows observed d-prime; histograms show d-prime after permutation (1000 shuffles). All clusters were more separated than expected by chance (all p≤0.001, one-sided permutation tests). b, Plot shows how systematically varying the number of neighbors in the Phenograph algorithm (K; color-axis) changed the goodness of clustering, as measured by the silhouette score (x-axis) and modularity (y-axis, see methods). White text shows the resulting number of identified clusters. A K value between 35 and 45 results in 4 clusters and high silhouette scores and modularity. Increasing the K value beyond this recommended range leads to unstable clustering with highly variable silhouette scores and low modularity. c, Density of UMAP projection of A/X temporal selectivity. Dot colors indicate Phenograph clustering (left) and K-means clustering (right, number of clusters = 4) labels. Area of circle indicates number of data points in region (max size = 8). d, K-means silhouette score as a function of cluster number. K-means was performed on UMAP projections for timecourse of A/X selectivity, random selectivity, and C/C* selectivity. e-g, A/X temporal selectivity profile clustered by K-means applied to UMAP, as shown in panel c. Lines show mean ± s.e.m. of each cluster’s selectivity timecourse, after each K-means run, when the number of clusters set to (e) k = 2, (f) k = 3, and (g) k = 4.
a, Intrinsic variability was higher in stable neurons. Lines show mean ± s.e.m. of fano factor of stable (orange, n = 355) and switching (blue, n = 167) neurons over the sequence (neurons combined days). b, Violin plots show distribution of fano factor during pre-stimulus period (−400–0 ms), stimulus presentation (A/X, B/Y, C/C* and D/D* combined, 100 ms each) and inter-chord interval (ICI; 75 ms each). Fano factor was higher in stable neurons compared to switching neurons before the stimulus (stable, mean ± s.e.m. = 1.06 ± 0.01, switching = 1.04 ± 0.01; diff. = −0.02, p = 0.02), during stimulus presentation (stable = 1.03 ± 0.01, switching = 1.01 ± 0.01; diff. = −0.02, p = 0.016), and during the ICI (stable = 1.03 ± 0.01, switching = 1.01 ± 0.01; diff. = −0.02, p = 0.01, all one-sided permutation tests). Difference between the pre-stimulus and stimulus periods were significant for both neuron types (stable = 0.03, p = 0.001; switching = 0.03, p = 0.03; one-sided permutation tests). c, Line show mean ± s.e.m. of intrinsic autocorrelation of functional neuron types, calculated during the pre-stimulus period. The autocorrelation at lag zero was removed for clarity. d, Histograms show distribution of time constants from autocorrelations. The time constant (tau; x-axis) provides a measure of each neuron types’s intrinsic timescale; it was estimated by fitting an exponential function to the autocorrelation shown in panel c. No difference was observed between neuron types: switching mean ± s.e.m. = 94 ± 51 ms, stable = 86 ± 31 ms (bootstrapped exponential fit; n = 1000 resamples with replacement). e, Switching neurons carried slightly more of the A/X-C/C* association than stable neurons. Lines show mean ± s.e.m. of stable (orange) and switching (blue) neurons’ A/X and C/C* temporal selectivity profiles. Neurons without significant C/C* selectivity were removed (stable n = 123; switching, n = 24, data combined across days). Selectivity of neurons is plotted with respect to their initial A/X preference (that is, initial selectivity is always positive). Dashed lines show C/C* selectivity of the same neurons. Responses to associated stimuli (AC/XC*) are positive, while responses to unassociated stimuli (AC*/XC) are negative. f, Violin plots show distribution of average predictive selectivity during C/C* stimulus presentation (350–450 ms). Each dot is a neuron; all days included. The mean ± s.e.m. of prediction in stable neurons = −0.13 ± 0.31, p = 0.67; switching neurons = 1.16 ± 0.53, p = 0.022, two-sided bootstrap tests. g, Data points show estimated locations of neurons in each functional cluster along recording array. Switching (blue), stable (orange), and none (grey) neurons are plotted according to their estimated electrode location (x-axis – AP, y-axis – depth (DV) based on implant coordinates; 6 probes had 4 shanks separated by 200 µm). Small, random jitter in anterior-posterior (AP) direction was added for clarity of presentation and does not reflect actual differences. For all panels, p-values: * ≤ 0.05, ** ≤ 0.01, *** ≤ 0.001.
About this article
Cite this article
Libby, A., Buschman, T.J. Rotational dynamics reduce interference between sensory and memory representations. Nat Neurosci (2021). https://doi.org/10.1038/s41593-021-00821-9