Abstract
Although we are continuously bombarded with visual input, only a fraction of incoming visual events is perceived, remembered or acted on. The neural underpinnings of various forms of visual priority coding, including perceptual expertise, goal-directed attention, visual salience, image memorability and preferential looking, have been studied. Here, we synthesize information from these different examples to review recent developments in our understanding of visual priority coding and its neural correlates, with a focus on the role of behaviour to evaluate candidate correlates. We propose that the brain combines different types of priority into a unified priority signal while also retaining the ability to differentiate between them, and that this happens by leveraging partially overlapping low-dimensional neural subspaces for each type of priority that are shared with the downstream neural populations involved in decision-making. Finally, we describe the gulfs in understanding that have resulted from different research approaches, and we point towards future directions that will lead to fundamental insights about neural coding and how prioritization influences visually guided behaviours.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$189.00 per year
only $15.75 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Srinivasan, M. V. Honey bees as a model for vision, perception, and cognition. Annu. Rev. Entomol. 55, 267–284 (2010).
Stoeger, A. S. & Manger, P. Vocal learning in elephants: neural bases and adaptive context. Curr. Opin. Neurobiol. 28, 101–107 (2014).
Ungerleider, L. G., Mishkin, M. in Analysis of Visual Behavior 549–586 (MIT Press, 1982).
Pasupathy, A., Popovkina, D. V. & Kim, T. Visual functions of primate area V4. Annu. Rev. Vis. Sci. 6, 363–385 (2020).
DiCarlo, J. J., Zoccolan, D. & Rust, N. C. How does the brain solve visual object recognition? Neuron 73, 415–34 (2012).
Gilbert, C. D., Sigman, M. & Crist, R. E. The neural basis of perceptual learning. Neuron 31, 681–697 (2001).
Fahle, M. Perceptual learning: specificity versus generalization. Curr. Opin. Neurobiol. 15, 154–160 (2005).
Seitz, A. R. & Dinse, H. R. A common framework for perceptual learning. Curr. Opin. Neurobiol. 17, 148–153 (2007).
Sasaki, Y., Nanez, J. E. & Watanabe, T. Advances in visual perceptual learning and plasticity. Nat. Rev. Neurosci. 11, 53–60 (2010).
Lu, Z.-L., Hua, T., Huang, C.-B., Zhou, Y. & Dosher, B. A. Visual perceptual learning. Neurobiol. Learn. Mem. 95, 145–151 (2011).
Sagi, D. Perceptual learning in vision research. Vis. Res. 51, 1552–1566 (2011).
Watanabe, T. & Sasaki, Y. Perceptual learning: toward a comprehensive theory. Annu. Rev. Psychol. 66, 197–221 (2015).
Tamaki, M. et al. Reward does not facilitate visual perceptual learning until sleep occurs. Proc. Natl Acad. Sci. USA 117, 959–968 (2020).
Tamaki, M., Wang, Z., Watanabe, T. & Sasaki, Y. Trained-feature-specific offline learning by sleep in an orientation detection task. J. Vis. 19, 12 (2019).
Szpiro, S. F. A. & Carrasco, M. Exogenous attention enables perceptual learning. Psychol. Sci. 26, 1854–1862 (2015).
Kim, R., Seitz, A., Feenstra, H. & Shams, L. Testing assumptions of statistical learning: is it long-term and implicit? Neurosci. Lett. 461, 145–149 (2009).
Fiorentini, A. & Berardi, N. Perceptual learning specific for orientation and spatial frequency. Nature 287, 43–44 (1980).
Pourtois, G., Rauss, K. S., Vuilleumier, P. & Schwartz, S. Effects of perceptual learning on primary visual cortex activity in humans. Vis. Res. 48, 55–62 (2008).
Mollon, J. D. & Danilova, M. V. Three remarks on perceptual learning. Spat. Vis. 10, 51–58 (1996).
Little, D. F., Zhang, Y.-X. & Wright, B. A. Disruption of perceptual learning by a brief practice break. Curr. Biol. 27, 3699–3705.e3 (2017).
Kang, D.-W. et al. Structural and functional connectivity changes beyond visual cortex in a later phase of visual perceptual learning. Sci. Rep. 8, 5186 (2018).
Kastner, S. & Ungerleider, L. G. Mechanisms of visual attention in the human cortex. Annu. Rev. Neurosci. 23, 315–341 (2000).
Anton-Erxleben, K. & Carrasco, M. Attentional enhancement of spatial resolution: linking behavioural and neurophysiological evidence. Nat. Rev. Neurosci. 14, 188–200 (2013).
Krauzlis, R. J., Lovejoy, L. P. & Zénon, A. Superior colliculus and visual spatial attention. Annu. Rev. Neurosci. 36, 165–182 (2013).
Amso, D. & Scerif, G. The attentive brain: insights from developmental cognitive neuroscience. Nat. Rev. Neurosci. 16, 606–619 (2015).
Clark, K., Squire, R. F., Merrikhi, Y. & Noudoost, B. Visual attention: linking prefrontal sources to neuronal and behavioral correlates. Prog. Neurobiol. 132, 59–80 (2015).
Treisman, A. M. & Gelade, G. A feature-integration theory of attention. Cogn. Psychol. 12, 97–136 (1980).
Treisman, A. & Sato, S. Conjunction search revisited. J. Exp. Psychol. Hum. Percept. Perform. 16, 459–478 (1990).
Wolfe, J. M. Visual search in continuous, naturalistic stimuli. Vis. Res. 34, 1187–1195 (1994).
Posner, M. I. Orienting of attention. Q. J. Exp. Psychol. 32, 3–25 (1980).
Carrasco, M. Visual attention: the past 25 years. Vis. Res. 51, 1484–1525 (2011).
Bacon, W. F. & Egeth, H. E. Overriding stimulus-driven attentional capture. Percept. Psychophys. 55, 485–496 (1994).
Wolfe, J. M., Cave, K. R. & Franzel, S. L. Guided search: an alternative to the feature integration model for visual search. J. Exp. Psychol. Hum. Percept. Perform. 15, 419–433 (1989).
Nakayama, K. & Martini, P. Situating visual search. Vis. Res. 51, 1526–1537 (2011).
Eckstein, M. P. Visual search: a retrospective. J. Vis. 11, 14 (2011).
Judd, T., Ehinger, K., Durand, F. & Torralba, A. Learning to predict where humans look. IEEE 12th International Conference on Computer Vision 2106–2113 (IEEE, 2009).
Isola, P., Xiao, J., Parikh, D., Torralba, A. & Oliva, A. What makes a photograph memorable? IEEE Trans. Pattern Anal. Mach. Intell. 36, 1469–1482 (2014). This paper describes image memorability-associated behavioural variation across a large and arbitrarily selected set of photographs.
Bainbridge, W. A., Isola, P. & Oliva, A. The intrinsic memorability of face photographs. J. Exp. Psychol. Gen. 142, 1323–1334 (2013).
Goetschalckx, L. & Wagemans, J. MemCat: a new category-based image set quantified on memorability. PeerJ 7, e8169 (2019).
Khosla, A., Raju, A. S., Torralba, A. & Oliva, A. Understanding and predicting image memorability at a large scale. International Conference on Computer Vision 2390–2398 (IEEE, 2015).
Rust, N. C. & Mehrpour, V. Understanding image memorability. Trends Cogn. Sci. 24, 557–568 (2020).
Bainbridge, W. A. The memorability of people: intrinsic memorability across transformations of a person’s face. J. Exp. Psychol. Learn. Mem. Cogn. 43, 706–716 (2017).
Bainbridge, W. A. in Psychology of Learning and Motivation (eds Federmeier, K. D. & Beck, D. M.) Ch. 1 vol. 70 1–27 (Academic, 2019).
Bylinskii, Z., Isola, P., Bainbridge, C., Torralba, A. & Oliva, A. Intrinsic and extrinsic effects on image memorability. Vis. Res. 116, 165–178 (2015).
Jutras, M. J. & Buffalo, E. A. Recognition memory signals in the macaque hippocampus. Proc. Natl Acad. Sci. USA 107, 401–406 (2010).
Hall, B. A., Melfi, V., Burns, A., McGill, D. M. & Doyle, R. E. Curious creatures: a multi-taxa investigation of responses to novelty in a zoo environment. PeerJ 6, e4454 (2018).
Teller, D. Y. The forced-choice preferential looking procedure: a psychophysical technique for use with human infants. Infant. Behav. Dev. 2, 135–153 (1979).
Meyer, T. & Rust, N. C. Single-exposure visual memory judgments are reflected in inferotemporal cortex. eLife 7, e32259 (2018).
Rich, J. B. in Encyclopedia of Clinical Neuropsychology (eds Kreutzer, J. S., DeLuca, J. & Caplan, B.) 2124–2128 (Springer, 2011).
Desimone, R. & Duncan, J. Neural mechanisms of selective visual attention. Annu. Rev. Neurosci. 18, 193–222 (1995).
Desimone, R. Neural mechanisms for visual memory and their role in attention. Proc. Natl Acad. Sci. USA 93, 13494–13499 (1996).
Reynolds, J. H. & Chelazzi, L. Attentional modulation of visual processing. Annu. Rev. Neurosci. 27, 611–647 (2004).
Maunsell, J. H. R. Neuronal mechanisms of visual attention. Annu. Rev. Vis. Sci. 1, 373–391 (2015).
Cohen, M. R. & Kohn, A. Measuring and interpreting neuronal correlations. Nat. Neurosci. 14, 811–819 (2011).
Roth, N. & Rust, N. C. Rethinking assumptions about how trial and nuisance variability impact neural task performance in a fast-processing regime. J. Neurophysiol. 121, 115–130 (2019).
Law, C.-T. & Gold, J. I. Neural correlates of perceptual learning in a sensory-motor, but not a sensory, cortical area. Nat. Neurosci. 11, 505–513 (2008).
Fries, P. Rhythms for cognition: communication through coherence. Neuron 88, 220–235 (2015).
Womelsdorf, T. & Fries, P. The role of neuronal synchronization in selective attention. Curr. Opin. Neurobiol. 17, 154–160 (2007).
Kohn, A. et al. Principles of corticocortical communication: proposed schemes and design considerations. Trends Neurosci. 43, 725–737 (2020).
Vyas, S., Golub, M. D., Sussillo, D. & Shenoy, K. V. Computation through neural population dynamics. Annu. Rev. Neurosci. 43, 249–275 (2020).
Srinath, R., Ruff, D. A. & Cohen, M. R. Attention improves information flow between neuronal populations without changing the communication subspace. Curr. Biol. 31, 5299–5313 (2021). This study demonstrated that attention improves the ability to predict activity in decision neurons from the activity of neurons in the visual cortex without changing the dimensionality of the communication subspace.
Kaufman, M. T., Churchland, M. M., Ryu, S. I. & Shenoy, K. V. Cortical activity in the null space: permitting preparation without movement. Nat. Neurosci. 17, 440–448 (2014). This study demonstrated that only a subspace of activity in a given population affects behaviour, while the remaining subset (null space) does not, laying a foundation for the subspace framework.
Saxena, S. & Cunningham, J. P. Towards the neural population doctrine. Curr. Opin. Neurobiol. 55, 103–111 (2019).
Gawne, T. J. & Richmond, B. J. How independent are the messages carried by adjacent inferior temporal cortical neurons? J. Neurosci. 13, 2758–2771 (1993).
Chen, Y.-P., Lin, C.-P., Hsu, Y.-C. & Hung, C. P. Network anisotropy trumps noise for efficient object coding in macaque inferior temporal cortex. J. Neurosci. 35, 9889–9899 (2015).
Hung, C. P., Cui, D., Chen, Y.-P., Lin, C.-P. & Levine, M. R. Correlated activity supports efficient cortical processing. Front. Comput. Neurosci. 8, 171 (2014).
Ziemba, C. M., Freeman, J., Movshon, J. A. & Simoncelli, E. P. Selectivity and tolerance for visual texture in macaque V2. Proc. Natl Acad. Sci. USA 113, E3140–3149 (2016).
Dapello, J. et al. Simulating a primary visual cortex at the front of CNNs improves robustness to image perturbations. Advances in Neural Information Processing Systems 33 https://proceedings.neurips.cc/paper/2020/hash/98b17f068d5d9b7668e19fb8ae470841-Abstract.html (2020).
Parker, A. J. & Newsome, W. T. Sense and the single neuron: probing the physiology of perception. Annu. Rev. Neurosci. 21, 227–277 (1998).
Schoups, A., Vogels, R., Qian, N. & Orban, G. Practising orientation identification improves orientation coding in V1 neurons. Nature 412, 549–553 (2001).
Ghose, G. M., Yang, T. & Maunsell, J. H. R. Physiological correlates of perceptual learning in monkey V1 and V2. J. Neurophysiol. 87, 1867–1888 (2002).
Raiguel, S., Vogels, R., Mysore, S. G. & Orban, G. A. Learning to see the difference specifically alters the most informative V4 neurons. J. Neurosci. 26, 6589–6602 (2006).
Uka, T., Sasaki, R. & Kumano, H. Change in choice-related response modulation in area MT during learning of a depth-discrimination task is consistent with task learning. J. Neurosci. 32, 13689–13700 (2012).
Ni, A. M., Ruff, D. A., Alberts, J. J., Symmonds, J. & Cohen, M. R. Learning and attention reveal a general relationship between population activity and behavior. Science 359, 463–465 (2018). This paper demonstrated that perceptual learning and attention were reflected in V4 primarily by an overlapping neural correlate: changes in correlated trial variability, which is strongly linked to performance.
Yotsumoto, Y. et al. Location-specific cortical activation changes during sleep after training for perceptual learning. Curr. Biol. 19, 1278–1282 (2009).
Poort, J. et al. Learning enhances sensory and multiple non-sensory representations in primary visual cortex. Neuron 86, 1478–1490 (2015).
Karmarkar, U. R. & Dan, Y. Experience-dependent plasticity in adult visual cortex. Neuron 52, 577–585 (2006).
Yan, Y., Zhaoping, L. & Li, W. Bottom-up saliency and top-down learning in the primary visual cortex of monkeys. Proc. Natl Acad. Sci. USA 115, 10499–10504 (2018).
Jia, K. et al. Recurrent processing drives perceptual plasticity. Curr. Biol. 30, 4177–4187.e4 (2020).
Yang, T. & Maunsell, J. H. R. The effect of perceptual learning on neuronal responses in monkey visual area V4. J. Neurosci. 24, 1617–1626 (2004).
Gu, Y. et al. Perceptual learning reduces interneuronal correlations in macaque visual cortex. Neuron 71, 750–761 (2011).
Huang, C. et al. Circuit models of low-dimensional shared variability in cortical networks. Neuron 101, 337–348.e4 (2019). This study presents a circuit model in which realistic variability that is low-dimensional and modulated by attention arises from a biophysically realistic network.
Semedo, J. D., Zandvakili, A., Machens, C. K., Yu, B. M. & Kohn, A. Cortical areas interact through a communication subspace. Neuron 102, 249–259.e4 (2019). This study demonstrated that populations of neurons in visual areas V1 and V2, which are highly interconnected, share activity via a subspace of activity that is lower-dimensional than the activity in either area.
Treue, S. & Martínez Trujillo, J. C. Feature-based attention influences motion processing gain in macaque visual cortex. Nature 399, 575–579 (1999).
McAlonan, K., Cavanaugh, J. & Wurtz, R. H. Guarding the gateway to cortex with attention in visual thalamus. Nature 456, 391–394 (2008).
Moran, J. & Desimone, R. Selective attention gates visual processing in the extrastriate cortex. Science 229, 782–784 (1985).
Buffalo, E. A., Fries, P., Landman, R., Liang, H. & Desimone, R. A backward progression of attentional effects in the ventral stream. Proc. Natl Acad. Sci. USA 107, 361–365 (2010).
Saenz, M., Buracas, G. T. & Boynton, G. M. Global effects of feature-based attention in human visual cortex. Nat. Neurosci. 5, 631–632 (2002).
Cohen, M. R. & Maunsell, J. H. R. When attention wanders: how uncontrolled fluctuations in attention affect performance. J. Neurosci. 31, 15802–15806 (2011).
Cohen, M. R. & Maunsell, J. H. R. Attention improves performance primarily by reducing interneuronal correlations. Nat. Neurosci. 12, 1594–1600 (2009).
Gregoriou, G. G., Rossi, A. F., Ungerleider, L. G. & Desimone, R. Lesions of prefrontal cortex reduce attentional modulation of neuronal responses and synchrony in V4. Nat. Neurosci. 17, 1003–1011 (2014).
Herrero, J. L., Gieselmann, M. A., Sanayei, M. & Thiele, A. Attention-induced variance and noise correlation reduction in macaque V1 is mediated by NMDA receptors. Neuron 78, 729–739 (2013).
Luo, T. Z. & Maunsell, J. H. R. Neuronal modulations in visual cortex are associated with only one of multiple components of attention. Neuron 86, 1182–1188 (2015).
Mayo, J. P. & Maunsell, J. H. R. Graded neuronal modulations related to visual spatial attention. J. Neurosci. 36, 5353–5361 (2016).
Mitchell, J. F., Sundberg, K. A. & Reynolds, J. H. Spatial attention decorrelates intrinsic activity fluctuations in macaque area V4. Neuron 63, 879–888 (2009).
Nandy, A. S., Nassi, J. J. & Reynolds, J. H. Laminar organization of attentional modulation in macaque visual area V4. Neuron 93, 235–246 (2017).
Ruff, D. A. & Cohen, M. R. Attention can either increase or decrease spike count correlations in visual cortex. Nat. Neurosci. 17, 1591–1597 (2014).
Ruff, D. A. & Cohen, M. R. Stimulus dependence of correlated variability across cortical areas. J. Neurosci. 36, 7546–7556 (2016).
Ruff, D. A. & Cohen, M. R. Simultaneous multi-area recordings suggest that attention improves performance by reshaping stimulus representations. Nat. Neurosci. 22, 1669–1676 (2019).
Verhoef, B.-E. & Maunsell, J. H. R. Attention-related changes in correlated neuronal activity arise from normalization mechanisms. Nat. Neurosci. 20, 969–977 (2017).
Yan, Y. et al. Perceptual training continuously refines neuronal population codes in primary visual cortex. Nat. Neurosci. 17, 1380–1387 (2014).
Zénon, A. & Krauzlis, R. J. Attention deficits without cortical neuronal deficits. Nature 489, 434–437 (2012).
Ruff, D. A. & Cohen, M. R. Global cognitive factors modulate correlated response variability between V4 neurons. J. Neurosci. 34, 16408–16416 (2014).
Oemisch, M., Westendorff, S., Everling, S. & Womelsdorf, T. Interareal spike-train correlations of anterior cingulate and dorsal prefrontal cortex during attention shifts. J. Neurosci. 35, 13076–13089 (2015).
Pooresmaeili, A. & Roelfsema, P. R. A growth-cone model for the spread of object-based attention during contour grouping. Curr. Biol. 24, 2869–2877 (2014).
Ruff, D. A., Alberts, J. J. & Cohen, M. R. Relating normalization to neuronal populations across cortical areas. J. Neurophysiol. 116, 1375–1386 (2016).
Bichot, N. P., Rossi, A. F. & Desimone, R. Parallel and serial neural mechanisms for visual search in macaque area V4. Science 308, 529–534 (2005).
Bosman, C. A. et al. Attentional stimulus selection through selective synchronization between monkey visual areas. Neuron 75, 875–888 (2012).
Buschman, T. J. & Miller, E. K. Top-down versus bottom-up control of attention in the prefrontal and posterior parietal cortices. Science 315, 1860–1862 (2007).
Fries, P., Reynolds, J. H., Rorie, A. E. & Desimone, R. Modulation of oscillatory neuronal synchronization by selective visual attention. Science 291, 1560–1563 (2001).
Gregoriou, G. G., Gotts, S. J., Zhou, H. & Desimone, R. Long-range neural coupling through synchronization with attention. Prog. Brain Res. 176, 35–45 (2009).
Lakatos, P., Karmos, G., Mehta, A. D., Ulbert, I. & Schroeder, C. E. Entrainment of neuronal oscillations as a mechanism of attentional selection. Science 320, 110–113 (2008).
Miller, E. K. & Buschman, T. J. Cortical circuits for the control of attention. Curr. Opin. Neurobiol. 23, 216–222 (2013).
Saalmann, Y. B., Pigarev, I. N. & Vidyasagar, T. R. Neural mechanisms of visual attention: how top-down feedback highlights relevant locations. Science 316, 1612–1615 (2007).
Saproo, S. & Serences, J. T. Attention improves transfer of motion information between V1 and MT. J. Neurosci. 34, 3586–3596 (2014).
Womelsdorf, T. & Fries, P. Neuronal coherence during selective attentional processing and sensory-motor integration. J. Physiol. Paris 100, 182–193 (2006).
Briggs, F., Mangun, G. R. & Usrey, W. M. Attention enhances synaptic efficacy and the signal-to-noise ratio in neural circuits. Nature 499, 476–480 (2013).
Dagnino, B., Gariel-Mathis, M.-A. & Roelfsema, P. R. Microstimulation of area V4 has little effect on spatial attention and on perception of phosphenes evoked in area V1. J. Neurophysiol. 113, 730–739 (2015).
Klink, P. C., Jeurissen, D., Theeuwes, J., Denys, D. & Roelfsema, P. R. Working memory accuracy for multiple targets is driven by reward expectation and stimulus contrast with different time-courses. Sci. Rep. 7, 9082 (2017).
Moore, T. & Armstrong, K. M. Selective gating of visual signals by microstimulation of frontal cortex. Nature 421, 370–373 (2003).
Ruff, D. A. & Cohen, M. R. A normalization model suggests that attention changes the weighting of inputs between visual areas. Proc. Natl Acad. Sci. USA 114, E4085–E4094 (2017).
Salinas, E. & Sejnowski, T. J. Correlated neuronal activity and the flow of neural information. Nat. Rev. Neurosci. 2, 539–550 (2001).
Azouz, R. & Gray, C. M. Adaptive coincidence detection and dynamic gain control in visual cortical neurons in vivo. Neuron 37, 513–523 (2003).
Buffalo, E. A., Fries, P., Landman, R., Buschman, T. J. & Desimone, R. Laminar differences in gamma and alpha coherence in the ventral stream. Proc. Natl Acad. Sci. USA 108, 11262–11267 (2011).
Womelsdorf, T., Fries, P., Mitra, P. P. & Desimone, R. Gamma-band synchronization in visual cortex predicts speed of change detection. Nature 439, 733–736 (2006).
Bosman, C. A., Womelsdorf, T., Desimone, R. & Fries, P. A microsaccadic rhythm modulates gamma-band synchronization and behavior. J. Neurosci. 29, 9471–9480 (2009).
Knierim, J. J. & van Essen, D. C. Neuronal responses to static texture patterns in area V1 of the alert macaque monkey. J. Neurophysiol. 67, 961–980 (1992).
Burrows, B. E. & Moore, T. Influence and limitations of popout in the selection of salient visual stimuli by area V4 neurons. J. Neurosci. 29, 15169–15177 (2009).
Hegdé, J. & Felleman, D. J. How selective are V1 cells for pop-out stimuli? J. Neurosci. 23, 9968–9980 (2003).
Chelazzi, L., Duncan, J., Miller, E. K. & Desimone, R. Responses of neurons in inferior temporal cortex during memory-guided visual search. J. Neurophysiol. 80, 2918–2940 (1998).
Thompson, K. G. & Bichot, N. P. A visual salience map in the primate frontal eye field. Prog. Brain Res. 147, 251–262 (2005).
Ogawa, T. & Komatsu, H. Neuronal dynamics of bottom-up and top-down processes in area V4 of macaque monkeys performing a visual search. Exp. Brain Res. 173, 1–13 (2006).
Bichot, N. P., Heard, M. T., DeGennaro, E. M. & Desimone, R. A source for feature-based attention in the prefrontal cortex. Neuron 88, 832–844 (2015).
Pagan, M., Urban, L. S., Wohl, M. P. & Rust, N. C. Signals in inferotemporal cortex and perirhinal cortex suggest an untangling of visual target information. Nat. Neurosci. 16, 1132–1139 (2013).
Roth, N. & Rust, N. C. Inferotemporal cortex multiplexes behaviorally-relevant target match signals and visual representations in a manner that minimizes their interference. PLoS One 13, e0200528 (2018).
Roth, N. & Rust, N. C. The integration of visual and target signals in V4 and IT during visual object search. J. Neurophysiol. 122, 2522–2540 (2019).
Chen, X. et al. Parietal cortex regulates visual salience and salience-driven behavior. Neuron 106, 177–187.e4 (2020).
Koch, C. & Ullman, S. Shifts in selective visual attention: towards the underlying neural circuitry. Hum. Neurobiol. 4, 219–227 (1985).
Borji, A., Sihite, D. N. & Itti, L. What stands out in a scene? A study of human explicit saliency judgment. Vis. Res. 91, 62–77 (2013).
Itti, L., Koch, C. & Niebur, E. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20, 1254–1259 (1998).
Wang, J., Borji, A., Jay Kuo, C.-C. & Itti, L. Learning a combined model of visual saliency for fixation prediction. IEEE Trans. Image Process. 25, 1566–1579 (2016).
Jaegle, A. et al. Population response magnitude variation in inferotemporal cortex predicts image memorability. eLife 8, e47596 (2019). This paper describes the neural correlate of one behavioural priority signature, image memorability, and demonstrates that it emerges naturally from deep neural networks trained to identify objects.
Fahy, F. L., Riches, I. P. & Brown, M. W. Neuronal activity related to visual recognition memory: long-term memory and the encoding of recency and familiarity information in the primate anterior and medial inferior temporal and rhinal cortex. Exp. Brain Res. 96, 457–472 (1993).
Li, L., Miller, E. K. & Desimone, R. The representation of stimulus familiarity in anterior inferior temporal cortex. J. Neurophysiol. 69, 1918–1929 (1993).
Xiang, J. Z. & Brown, M. W. Differential neuronal encoding of novelty, familiarity and recency in regions of the anterior temporal lobe. Neuropharmacology 37, 657–676 (1998).
Womelsdorf, T. Translating expectation into visual selection through a beta-synchronous fronto-parietal neural subnetwork. Neuron 109, 8–10 (2021).
Ossandón, T. et al. Efficient ‘pop-out’ visual search elicits sustained broadband γ activity in the dorsal attention network. J. Neurosci. 32, 3414–3421 (2012).
Jutras, M. J., Fries, P. & Buffalo, E. A. Oscillatory activity in the monkey hippocampus during visual exploration and memory formation. Proc. Natl Acad. Sci. USA 110, 13144–13149 (2013).
Jutras, M. J., Fries, P. & Buffalo, E. A. Gamma-band synchronization in the macaque hippocampus and memory formation. J. Neurosci. 29, 12521–12531 (2009).
Sederberg, P. B. et al. Gamma oscillations distinguish true from false memories. Psychol. Sci. 18, 927–932 (2007).
Sederberg, P. B. et al. Hippocampal and neocortical gamma oscillations predict memory formation in humans. Cereb. Cortex 17, 1190–1196 (2007).
Fell, J. et al. Human memory formation is accompanied by rhinal-hippocampal coupling and decoupling. Nat. Neurosci. 4, 1259–1264 (2001).
Naya, Y. & Suzuki, W. A. Integrating what and when across the primate medial temporal lobe. Science 333, 773–776 (2011).
Kiani, R., Esteky, H., Mirpour, K. & Tanaka, K. Object category structure in response patterns of neuronal population in monkey inferior temporal cortex. J. Neurophysiol. 97, 4296–4309 (2007).
Yu, B. M. et al. Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity. J. Neurophysiol. 102, 614–635 (2009).
Cunningham, J. P. & Yu, B. M. Dimensionality reduction for large-scale neural recordings. Nat. Neurosci. 17, 1500–1509 (2014).
Sadtler, P. T. et al. Neural constraints on learning. Nature 512, 423–426 (2014).
Kiani, R. et al. Natural grouping of neural responses reveals spatially segregated clusters in prearcuate cortex. Neuron 85, 1359–1373 (2015).
Cowley, B. R., Smith, M. A., Kohn, A. & Yu, B. M. Stimulus-driven population activity patterns in macaque primary visual cortex. PLoS Comput. Biol. 12, e1005185 (2016).
Elsayed, G. F., Lara, A. H., Kaufman, M. T., Churchland, M. M. & Cunningham, J. P. Reorganization between preparatory and movement population responses in motor cortex. Nat. Commun. 7, 13239 (2016).
Golub, M. D., Chase, S. M., Batista, A. P. & Yu, B. M. Brain-computer interfaces for dissecting cognitive processes underlying sensorimotor control. Curr. Opin. Neurobiol. 37, 53–58 (2016).
Morcos, A. S. & Harvey, C. D. History-dependent variability in population dynamics during evidence accumulation in cortex. Nat. Neurosci. 19, 1672–1681 (2016).
Elsayed, G. F. & Cunningham, J. P. Structure in neural population recordings: an expected byproduct of simpler phenomena? Nat. Neurosci. 20, 1310–1318 (2017).
Jazayeri, M. & Afraz, A. Navigating the neural space in search of the neural code. Neuron 93, 1003–1014 (2017).
Miri, A. et al. Behaviorally selective engagement of short-latency effector pathways by motor cortex. Neuron 95, 683–696.e11 (2017).
Pitkow, X. & Angelaki, D. E. Inference in the brain: statistics flowing in redundant population codes. Neuron 94, 943–953 (2017).
Pandarinath, C. et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nat. Methods 15, 805–815 (2018).
Ruff, D. A., Ni, A. M. & Cohen, M. R. Cognition as a window into neuronal population space. Annu. Rev. Neurosci. 41, 77–97 (2018).
Stringer, C., Pachitariu, M., Steinmetz, N., Carandini, M. & Harris, K. D. High-dimensional geometry of population responses in visual cortex. Nature 571, 361–365 (2019).
Rajan, K. & Bialek, W. Maximally informative “stimulus energies” in the analysis of neural responses to natural signals. PLoS One 8, e71959 (2013).
Ecker, A. S. et al. State dependence of noise correlations in macaque primary visual cortex. Neuron 82, 235–48 (2014).
Ruff, D. A., Xue, C., Kramer, L. E., Baqai, F. & Cohen, M. R. Low rank mechanisms underlying flexible visual representations. Proc. Natl Acad. Sci. USA 117, 29321–29329 (2020).
Mehrpour, V., Meyer, T., Simoncelli, E. P. & Rust, N. C. Pinpointing the neural signatures of single-exposure recognition memory. Proc. Natl Acad. Sci. USA 118, e2021660118 (2021). In the context of a visual memory task, this study demonstrated that the population subspaces encoding novelty and a task-irrelevant variable, contrast, were partially overlapping and could be disambiguated.
McAdams, C. J. & Maunsell, J. H. Effects of attention on orientation-tuning functions of single neurons in macaque cortical area V4. J. Neurosci. 19, 431–441 (1999).
Reynolds, J. H. & Heeger, D. J. The normalization model of attention. Neuron 61, 168–185 (2009).
Herrmann, K., Heeger, D. J. & Carrasco, M. Feature-based attention enhances performance by increasing response gain. Vis. Res. 74, 10–20 (2012).
Eldar, E., Cohen, J. D. & Niv, Y. The effects of neural gain on attention and learning. Nat. Neurosci. 16, 1146–1153 (2013).
McMahon, D. B. & Olson, C. R. Repetition suppression in monkey inferotemporal cortex: relation to behavioral priming. J. Neurophysiol. 97, 3532–43 (2007).
Grill-Spector, K., Henson, R. & Martin, A. Repetition and the brain: neural models of stimulus-specific effects. Trends Cogn. Sci. 10, 14–23 (2006).
Woloszyn, L. & Sheinberg, D. L. Effects of long-term visual experience on responses of distinct classes of single units in inferior temporal cortex. Neuron 74, 193–205 (2012).
Arandia-Romero, I., Tanabe, S., Drugowitsch, J., Kohn, A. & Moreno-Bote, R. Multiplicative and additive modulation of neuronal tuning with population activity affects encoded information. Neuron 89, 1305–1316 (2016).
Heeger, D. J. Normalization of cell responses in cat striate cortex. Vis. Neurosci. 9, 181–197 (1992).
Rigotti, M. et al. The importance of mixed selectivity in complex cognitive tasks. Nature 497, 585–590 (2013).
Fusi, S., Miller, E. K. & Rigotti, M. Why neurons mix: high dimensionality for higher cognition. Curr. Opin. Neurobiol. 37, 66–74 (2016).
Panichello, M. F. & Buschman, T. J. Shared mechanisms underlie the control of working memory and attention. Nature 592, 601–605 (2021). This study directly compared the neural correlates of working memory and attention, which is a critical step toward determining whether different forms of prioritization are mediated by similar mechanisms.
Moreno-Bote, R. et al. Information-limiting correlations. Nat. Neurosci. 17, 1410–1417 (2014).
Kohn, A., Coen-Cagli, R., Kanitscheider, I. & Pouget, A. Correlations and neuronal population information. Annu. Rev. Neurosci. 39, 237–256 (2016).
Deneve, S., Latham, P. E. & Pouget, A. Efficient computation and cue integration with noisy population codes. Nat. Neurosci. 4, 826–831 (2001).
Lee, J. & Maunsell, J. H. R. A normalization model of attentional modulation of single unit responses. PLoS One 4, e4651 (2009).
Boynton, G. M. A framework for describing the effects of attention on visual responses. Vis. Res. 49, 1129–1143 (2009).
Carandini, M. & Heeger, D. J. Normalization as a canonical neural computation. Nat. Rev. Neurosci. 13, 51–62 (2011).
Zhaoping, L. A new framework for understanding vision from the perspective of the primary visual cortex. Curr. Opin. Neurobiol. 58, 1–10 (2019).
Bogacz, R. & Brown, M. W. Comparison of computational models of familiarity discrimination in the perirhinal cortex. Hippocampus 13, 494–524 (2003).
Itti, L. & Koch, C. Computational modelling of visual attention. Nat. Rev. Neurosci. 2, 194–203 (2001).
Shadlen, M. N. & Newsome, W. T. The variable discharge of cortical neurons: implications for connectivity, computation, and information coding. J. Neurosci. 18, 3870–3896 (1998).
van Vreeswijk, C. & Sompolinsky, H. Chaos in neuronal networks with balanced excitatory and inhibitory activity. Science 274, 1724–1726 (1996).
Amit, D. J. & Brunel, N. Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex. Cereb. Cortex 7, 237–252 (1997).
Doiron, B., Litwin-Kumar, A., Rosenbaum, R., Ocker, G. K. & Josić, K. The mechanics of state-dependent neural correlations. Nat. Neurosci. 19, 383–393 (2016).
Deco, G. et al. Resting-state functional connectivity emerges from structurally and dynamically shaped slow linear fluctuations. J. Neurosci. 33, 11239–11252 (2013).
Wimmer, K. et al. Sensory integration dynamics in a hierarchical network explains choice probabilities in cortical area MT. Nat. Commun. 6, 6177 (2015).
Kanashiro, T., Ocker, G. K., Cohen, M. R. & Doiron, B. Attentional modulation of neuronal variability in circuit models of cortex. eLife 6, e23978 (2017).
Hennequin, G., Ahmadian, Y., Rubin, D. B., Lengyel, M. & Miller, K. D. The dynamical regime of sensory cortex: stable dynamics around a single stimulus-tuned attractor account for patterns of noise variability. Neuron 98, 846–860.e5 (2018).
Sadeh, S. & Clopath, C. Theory of neuronal perturbome in cortical networks. Proc. Natl Acad. Sci. USA 117, 26966–26976 (2020).
Sweeney, Y. & Clopath, C. Population coupling predicts the plasticity of stimulus responses in cortical circuits. eLife 9, e56053 (2020).
Ni, A. M., Ray, S. & Maunsell, J. H. R. Tuned normalization explains the size of attention modulations. Neuron 73, 803–813 (2012).
Khaligh-Razavi, S.-M. & Kriegeskorte, N. Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Comput. Biol. 10, e1003915 (2014).
Yamins, D. L. K. et al. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proc. Natl Acad. Sci. USA 111, 8619–8624 (2014).
Yamins, D. L. K. & DiCarlo, J. J. Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci. 19, 356–365 (2016).
Tamietto, M. & de Gelder, B. Neural bases of the non-conscious perception of emotional signals. Nat. Rev. Neurosci. 11, 697–709 (2010).
Pessoa, L., Kastner, S. & Ungerleider, L. G. Neuroimaging studies of attention: from modulation of sensory processing to top-down control. J. Neurosci. 23, 3990–3998 (2003).
Markovic, J., Anderson, A. K. & Todd, R. M. Tuning to the significant: neural and genetic processes underlying affective enhancement of visual perception and memory. Behav. Brain Res. 259, 229–241 (2014).
Wenliang, L. K. & Seitz, A. R. Deep neural networks for modeling visual perceptual learning. J. Neurosci. 38, 6028–6044 (2018). One of the only studies to investigate the consequences of adding trial variability that is modulated by priority coding to an otherwise deterministic deep network.
Zenke, F. et al. Visualizing a joint future of neuroscience and neuromorphic engineering. Neuron 109, 571–575 (2021).
Sigman, M. & Gilbert, C. D. Learning to find a shape. Nat. Neurosci. 3, 264–269 (2000).
Sarafyazd, M. & Jazayeri, M. Hierarchical reasoning by neural circuits in the frontal cortex. Science 364, eaav8911 (2019).
Cohen, M. R. & Maunsell, J. H. R. Using neuronal populations to study the mechanisms underlying spatial and feature attention. Neuron 70, 1192–1204 (2011).
Purcell, B. A. & Kiani, R. Neural mechanisms of post-error adjustments of decision policy in parietal cortex. Neuron 89, 658–671 (2016).
Ebitz, R. B., Tu, J. C. & Hayden, B. Y. Rules warp feature encoding in decision-making circuits. PLoS Biol. 18, e3000951 (2020).
Dworkin, J. D. et al. The extent and drivers of gender imbalance in neuroscience reference lists. Nat. Neurosci. 23, 918–926 (2020).
Maliniak, D., Powers, R. & Walter, B. F. The gender citation gap in international relations. Int. Organ. 67, 889–922 (2013).
Caplar, N., Tacchella, S. & Birrer, S. Quantitative evaluation of gender bias in astronomical publications from citation counts. Nat. Astron. 1, 0141 (2017).
Zurn, P., Bassett, D. S. & Rust, N. C. The citation diversity statement: a practice of transparency, a way of life. Trends Cogn. Sci. 24, 669–672 (2020).
Bertolero, M. Balanced Citer https://github.com/mb3152/balanced_citer (2021).
Ambekar, A., Ward, C., Mohammed, J., Male, S. & Skiena, S. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 49–58 (ACM, 2009).
Sood, G. & Laohaprapanon, S. Predicting race and ethnicity from the sequence of characters in a name. arXiv https://arxiv.org/abs/1805.02109 (2018).
Bertolero, M. A. et al. Racial and ethnic imbalance in neuroscience reference lists and intersections with gender. bioRxiv https://doi.org/10.1101/2020.10.12.336230 (2020).
Acknowledgements
This work was supported by the Simons Foundation (Simons Collaboration on the Global Brain award 543033 to N.C.R. and 542961SPI to M.R.C.), the US National Eye Institute of the National Institutes of Health (awards R01EY020851 and R01EY032878 to N.C.R. and awards R01EY022930 and R01NS121913 to M.R.C.) and the US National Science Foundation (award 2043255 to N.C.R.).
Author information
Authors and Affiliations
Contributions
The authors contributed equally to all aspects of the article.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Reviews Neuroscience thanks the anonymous reviewers for their contribution to the peer review of this work.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Glossary
- Receptive fields
-
The restricted region of visual space within which changes in the visual stimulus lead to changes in a neuron’s firing rate response.
- Ventral stream
-
Also called the ‘form processing’ or ‘what are you looking at?’ pathway owing to its association with object identification. Includes primate visual brain areas of visual cortex V1, V2, and V4 and the inferotemporal cortex.
- Search template
-
Defines the combined set of features that are sought in a visual search task.
- Population responses
-
Snapshots of the spiking activity of a collection of individual neurons in response to a single trial in one experimental condition.
- Trial variability
-
Variability in the responses of an individual neuron across repeated instances of the same experimental conditions and visual stimulus.
- Noise correlations
-
The degree to which trial variability is correlated between different units in response to repeated presentations of the same visual stimuli and other experimental conditions.
- Nuisance variability
-
The spiking variability induced by parameters not relevant to a task such as object position or size in a task that requires extraction of, for example, object identity.
- Synchrony
-
Simultaneous activation and/or inactivation of different neurons on fast timescales (less than 10 ms). Often measured as the coherence between activity in two areas in a particular temporal frequency band.
- Weights
-
Used to determine the output of a linear decoder, computed as a weighted sum of the population response on a single trial (for example, output = weight 1 × neuron 1 response + weight 2 × neuron 2 response…).
- Coherence
-
A measure of the similarity of oscillatory activity between two brain regions.
- Population vector direction
-
The position of a population response vector in an N-dimensional space (where N equals the number of neurons) after normalizing for population vector length (or magnitude).
- Multiplicatively
-
Modulations that impact a neuron’s response by multiplying it by a factor.
- Decoder
-
A single (typically linear) axis in a high-dimensional space, most often created to extract a particular type of information (such as ‘is this image an A or a B?’) from a neuronal population.
- Adaptation
-
Changes in the response of an individual (behavioural) or neuron with repeated or prolonged exposure to a stimulus.
- Units
-
Individual neurons or groups of a few neurons whose spiking activity is recorded typically via extracellular techniques. Measures of unit activity may or may not reflect the responses of a single neuron.
- Linear subspaces
-
Given a population of N neurons that define a population dimensionality with an upper bound of N, a linear subspace is a subset of the full space with dimensions M < N.
- Covariance matrix
-
Describes the covariation between different neurons across visual stimuli and repeated trials.
- Divisive normalization
-
A model that describes the responses of an individual neuron or population as a combination of the image within its ‘classic’ receptive field, adjusted by the combined response of other neurons.
Rights and permissions
About this article
Cite this article
Rust, N.C., Cohen, M.R. Priority coding in the visual system. Nat Rev Neurosci 23, 376–388 (2022). https://doi.org/10.1038/s41583-022-00582-9
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41583-022-00582-9
