Abstract
Ensemble perception is a process by which we summarize complex scenes. Despite the importance of ensemble perception to everyday cognition, there are few computational models that provide a formal account of this process. Here we develop and test a model in which ensemble representations reflect the global sum of activation signals across all individual items. We leverage this set of minimal assumptions to formally connect a model of memory for individual items to ensembles. We compare our ensemble model against a set of alternative models in five experiments. Our approach uses performance on a visual memory task for individual items to generate zero-free-parameter predictions of interindividual and intraindividual differences in performance on an ensemble continuous-report task. Our top-down modelling approach formally unifies models of memory for individual items and ensembles and opens a venue for building and comparing models of distinct memory processes and representations.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The data are publicly available at the following OSF link: https://osf.io/mt29p/.
Code availability
The code is publicly available on OSF (https://osf.io/mt29p/).
References
Baddeley, A. Working memory. Science 255, 556–559 (1992).
Miller, G. A. The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychol. Rev. 63, 81–97 (1956).
Pashler, H. Processing stages in overlapping tasks: evidence for a central bottleneck. J. Exp. Psychol. Hum. Percept. Perform. 10, 358–377 (1984).
Simon, H. A. Invariants of human behavior. Annu. Rev. Psychol. 41, 1–20 (1990).
Kahneman, D. A psychological perspective on economics. Am. Econ. Rev. 93, 162–168 (2003).
Ariely, D. Seeing sets: representation by statistical properties. Psychol. Sci. 12, 157–162 (2001).
Brady, T. F. & Oliva, A. Statistical learning using real-world scenes. Psychol. Sci. 19, 678–685 (2008).
Goldstein, M. H. et al. General cognitive principles for learning structure in time and space. Trends Cogn. Sci. 14, 249–258 (2010).
Whitney, D. & Yamanashi Leib, A. Ensemble perception. Annu. Rev. Psychol. 69, 105–129 (2018).
Alvarez, G. A. & Oliva, A. The representation of simple ensemble visual features outside the focus of attention. Psychol. Sci. 19, 392–398 (2008).
Brady, T. F., Shafer-Skelton, A. & Alvarez, G. A. Global ensemble texture representations are critical to rapid scene perception. J. Exp. Psychol. Hum. Percept. Perform. 43, 1160–1176 (2017).
Utochkin, I. Ensemble summary statistics as a basis for visual categorization. J. Vis. 15, 8 (2015).
Balas, B., Nakano, L. & Rosenholtz, R. A summary-statistic representation in peripheral vision explains visual crowding. J. Vis. 9, 13 (2009).
Block, N. Perceptual consciousness overflows cognitive access. Trends Cogn. Sci. 15, 567–575 (2011).
Cohen, M. A., Dennett, D. C. & Kanwisher, N. What is the bandwidth of perceptual experience? Trends Cogn. Sci. 20, 324–335 (2016).
Grahek, I., Schaller, M. & Tackett, J. L. Anatomy of a psychological theory: integrating construct-validation and computational-modeling methods to advance theorizing. Perspect. Psychol. Sci. 16, 803–815 (2021).
Guest, O. & Martin, A. E. How computational modeling can force theory building in psychological science. Perspect. Psychol. Sci. 16, 789–802 (2021).
Navarro, D. J. If mathematical psychology did not exist we might need to invent it: a comment on theory building in psychology. Perspect. Psychol. Sci. 16, 707–716 (2021).
Oberauer, K. & Lewandowsky, S. Addressing the theory crisis in psychology. Psychon. Bull. Rev. 26, 1596–1618 (2019).
Busemeyer, J. R. & Wang, Y. M. Model comparisons and model selections based on generalization criterion methodology. J. Math. Psychol. 44, 171–189 (2000).
Lee, M. D. How cognitive modeling can benefit from hierarchical Bayesian models. J. Math. Psychol. 55, 1–7 (2011).
Yarkoni, T. The generalizability crisis. Behav. Brain Sci. 45, e1 (2021).
Rust, N. C. in The Cognitive Neurosciences 5th edn (eds Gazzaniga, M. S. & Mangun, G. R.) 337–348 (MIT Press, 2014).
Alvarez, G. A. Representing multiple objects as an ensemble enhances visual cognition. Trends Cogn. Sci. 15, 122–131 (2011).
Ward, E. J., Bear, A. & Scholl, B. J. Can you perceive ensembles without perceiving individuals? The role of statistical perception in determining whether awareness overflows access. Cognition 152, 78–86 (2016).
Oriet, C., Giesinger, C. & Stewart, K. M. Can change detection succeed when change localization fails? J. Exp. Psychol. Hum. Percept. Perform. 46, 1127–1147 (2020).
Haberman, J. & Whitney, D. Efficient summary statistical representation when change localization fails. Psychon. Bull. Rev. 18, 855–859 (2011).
Marchant, A. P., Simons, D. J. & de Fockert, J. W. Ensemble representations: effects of set size and item heterogeneity on average size perception. Acta Psychol. 142, 245–250 (2013).
Šetić, M., Švegar, D. & Domijan, D. Modelling the statistical processing of visual information. Neurocomputing 70, 1808–1812 (2007).
Baek, J. & Chong, S. C. Ensemble perception and focused attention: two different modes of visual processing to cope with limited capacity. Psychon. Bull. Rev. 27, 602–606 (2020).
Solomon, J. A. Five dichotomies in the psychophysics of ensemble perception. Atten. Percept. Psychophys. 83, 904–910 (2021).
Chetverikov, A., Campana, G. & Kristjánsson, R. Building ensemble representations: how the shape of preceding distractor distributions affects visual search. Cognition 153, 196–210 (2016).
Hansmann-Roth, S., Thorsteinsdóttir, S., Geng, J. & Kristjánsson, R. Temporal integration of feature probability distributions in visual working memory. J. Vis. 21, 1969 (2021).
van Rooij, I. & Baggio, G. Theory before the test: how to build high-verisimilitude explanatory theories in psychological science. Perspect. Psychol. Sci. 16, 682–697 (2021).
Schurgin, M. W., Wixted, J. T. & Brady, T. F. Psychophysical scaling reveals a unified theory of visual memory strength. Nat. Hum. Behav. 4, 1156–1172 (2020).
Thurstone, L. L. A law of comparative judgment. Psychol. Rev. 34, 273–286 (1927).
Swets, J. A. Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurement of performance. Psychol. Bull. 99, 181–198 (1986).
Luce, R. D. & Galanter, E. Psychophysical scaling. Handb. Math. Psychol. 1, 245–307 (1963).
Shepard, R. N. Toward a universal law of generalization for psychological science. Science 237, 1317–1323 (1987).
Stevens, S. S. A scale for the measurement of a psychological magnitude: loudness. Psychol. Rev. 43, 405–416 (1936).
Wickens, T. D. Elementary Signal Detection Theory (Oxford Univ. Press, 2001).
Wixted, J. T. The forgotten history of signal detection theory. J. Exp. Psychol. Learn. Mem. Cogn. 46, 201–233 (2020).
Brady, T. F., Schacter, D. L. & Alvarez, G. The adaptive nature of false memories is revealed by gist-based distortion of true memories. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/zeg95 (2018).
Chater, N., Tenenbaum, J. B. & Yuille, A. Probabilistic models of cognition: where next. Trends Cogn. Sci. 10, 292–293 (2006).
Hemmer, P. & Steyvers, M. A Bayesian account of reconstructive memory. Top. Cogn. Sci. 1, 189–202 (2009).
McCarley, J. S. & Benjamin, A. S. in The Oxford Handbook of Cognitive Engineering (eds Lee, J. D. & Kirlik, A.) 465–475 (Oxford Univ. Press, 2013).
Hintzman, D. L. ‘Schema abstraction’ in a multiple-trace memory model. Psychol. Rev. 93, 411–428 (1986).
Howard, M. W. & Kahana, M. J. A distributed representation of temporal context. J. Math. Psychol. 46, 269–299 (2002).
Murdock, B. B. A theory for the storage and retrieval of item and associative information. Psychol. Rev. 89, 609–626 (1982).
Reder, L. M. et al. A mechanistic account of the mirror effect for word frequency: a computational model of remember–know judgments in a continuous recognition paradigm. J. Exp. Psychol. Learn. Mem. Cogn. 26, 294–320 (2000).
Shiffrin, R. M. & Steyvers, M. A model for recognition memory: REM—retrieving effectively from memory. Psychon. Bull. Rev. 4, 145–166 (1997).
Kriegeskorte, N. & Wei, X. X. Neural tuning and representational geometry. Nat. Rev. Neurosci. 22, 703–718 (2021).
Xiong, H. D. & Wei, X. X. Optimal encoding of prior information in noisy working memory systems. In Conference on Computational Cognitive Neuroscience (CCN, 2022).
Nosofsky, R. M. Attention and learning processes in the identification and categorization of integral stimuli. J. Exp. Psychol. Learn. Mem. Cogn. 13, 87–108 (1987).
Tenenbaum, J. B. Bayesian modeling of human concept learning. Adv. Neural Inf. Process. Syst. 11, 59–68 (1999).
Shamir, M. Emerging principles of population coding: in search for the neural code. Curr. Opin. Neurobiol. 25, 140–148 (2014).
Averbeck, B. B., Latham, P. E. & Pouget, A. Neural correlations, population coding and computation. Nat. Rev. Neurosci. 7, 358–366 (2006).
Bartolo, R., Saunders, R. C., Mitz, A. R. & Averbeck, B. B. Information-limiting correlations in large neural populations. J. Neurosci. 40, 1668–1678 (2020).
Kohn, A., Coen-Cagli, R., Kanitscheider, I. & Pouget, A. Correlations and neuronal population information. Annu. Rev. Neurosci. 39, 237–256 (2016).
Williams, J. R., Robinson, M. M., Schurgin, M. W., Wixted, J. T. & Brady, T. F. You can’t ‘count’ how many items people remember in working memory: the importance of signal detection-based measures for understanding change detection performance. J. Exp. Psychol. Hum. Percept. Perform. 48, 1390–1409 (2022).
Robinson, M. M., Benjamin, A. S. & Irwin, D. E. Is there a K in capacity? Assessing the structure of visual short-term memory. Cogn. Psychol. 121, 101305 (2020).
Tong, K., Dubé, C. & Sekuler, R. What makes a prototype a prototype? Averaging visual features in a sequence. Atten. Percept. Psychophys. 81, 1962–1978 (2019).
VanderWeele, T. J. & Mathur, M. B. Some desirable properties of the Bonferroni correction: is the Bonferroni correction really so bad? Am. J. Epidemiol. 188, 617–618 (2019).
Rahnev, D., Block, N., Denison, R. N. & Jehee, J. Is perception probabilistic? Clarifying the definitions. Preprint at PsyArXiv https://psyarxiv.com/f8v5r/ (2021).
Eckstein, M. P. Probabilistic computations for attention, eye movements, and search. Annu. Rev. Vis. Sci. 3, 319–342 (2017).
Ma, W. J. Organizing probabilistic models of perception. Trends Cogn. Sci. 16, 511–518 (2012).
Zeng, T., Tompary, A., Schapiro, A. C. & Thompson-Schill, S. L. Tracking the relation between gist and item memory over the course of long-term memory consolidation. eLife https://doi.org/10.7554/elife.65588 (2021).
Rosenbaum, D. & Bowman, H. Extraction of gist without encoding of individual items in RSVP of numerical sequences. Preprint at OSF https://osf.io/n2rcj (2021).
Hommel, B. et al. No one knows what attention is. Atten. Percept. Psychophys. 81, 2288–2303 (2019).
Greene, N. R. & Naveh-Benjamin, M. The effects of divided attention at encoding on specific and gist-based associative episodic memory. Mem. Cogn. 50, 59–76 (2021).
Chen, Z., Zhuang, R., Wang, X., Ren, Y. & Abrams, R. A. Ensemble perception without attention depends upon attentional control settings. Atten. Percept. Psychophys. 83, 1240–1250 (2021).
Zepp, J., Dubé, C. & Melcher, D. A direct comparison of central tendency recall and temporal integration in the successive field iconic memory task. Atten. Percept. Psychophys. 83, 1337–1356 (2021).
Gershman, S. J. in The Oxford Handbook of Human Memory (eds Kahana, M. & Wagner, A.) (Oxford Univ. Press, 2021).
Li, A. Y., Liang, J. C., Lee, A. C. & Barense, M. D. The validated circular shape space: quantifying the visual similarity of shape. J. Exp. Psychol. Gen. 149, 949–966 (2020).
Zhang, W. & Luck, S. J. Discrete fixed-resolution representations in visual working memory. Nature 453, 233–235 (2008).
Smith, J. D. & Minda, J. P. Prototypes in the mist: the early epochs of category learning. J. Exp. Psychol. Learn. Mem. Cogn. 24, 1411–1436 (1998).
Nadarajah, S., Afuecheta, E. & Chan, S. On the distribution of maximum of multivariate normal random vectors. Commun. Stat. Theory Methods 48, 2425–2445 (2019).
Acknowledgements
We acknowledge funding from the National Institutes of Health (National Research Service Award Fellowship No. 1F32MH127823-01 to M.M.R.) and the National Science Foundation (grant nos. BCS-1653457 and BCS-2146988 to T.F.B.). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
Author information
Authors and Affiliations
Contributions
M.M.R. and T.F.B. conceived and designed the experiments, developed the material and analytic tools and models, and wrote the paper. M.M.R. implemented the main experiments and modelling analyses.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Human Behaviour thanks Bernhard Spitzer, Eddie Ester and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 The Perceptual Summation model predicts ensemble memory for color with a range manipulation.
Graphical representation of TCC models’ fit and prediction of data in Experiment 2. In this experiment participants had to remember colors of simultaneously presented circles, and the range of colors was manipulated in the ensemble task. The top row of panel A shows the fits of the TCC model for individual items to aggregate data from the visual working memory task for six items. The bottom row of panel A shows results from the predictive analysis in which d’ estimates from the visual working memory task were substituted into the TCC Perceptual Summation (blue), Post-perceptual (red) and Automatic Averaging (green) models to predict the ensemble data. The bottom panel (B) shows model predictions for a few example participants. Schurgin et al.35).
Extended Data Fig. 2 Comparison in predictive accuracy between Perceptual Summation model and competing models of ensemble memory for shape with the set size manipulation.
The top panel shows violin plots for the difference in predicted negative log likelihood scores between each of the six alternative competing models (PNLLAlt) and the main Perceptual Summation model (PNLLPerSum) for Experiment 3 (n = 50 participants). Lower values of PNLL indicate higher predictive accuracy, therefore, PNLL difference scores higher (or lower) than zero indicate support for the Perceptual Summation (or a competing) model. In both experiments, the vast majority of participants are better predicted by the Perceptual Summation model than any of the alternatives. The bottom panel shows a table with a summary of descriptive and inferential statistics from all comparisons in Experiment 3, including the mean and standard error of the mean across participants. PNLL values were compared with a paired two-tailed t-test, corrected for multiple comparisons and all p-values were statistically significant (p < 0.001).
Extended Data Fig. 3 The Perceptual Summation model predicts ensemble memory for shape with a set size manipulation.
Graphical representation of the TCC models’ fit and prediction of data in Experiment 3. In this experiment participants had to remember different shapes, and the number of shapes was manipulated in the working memory and ensemble task. The top row of panel A shows the fits of the TCC model for individual items to aggregate data from the visual working memory task for six items and the second row of panel A shows results from the predictive analysis in which d’ estimates from the visual working memory task were substituted into the TCC Perceptual Summation (blue), Post-perceptual (red) and Automatic Averaging (green) models to predict the ensemble data. Panel B shows data and model predictions for a few example participants.
Extended Data Fig. 4 Comparison in predictive accuracy between Perceptual Summation model and competing models of ensemble memory for shape with the range manipulation.
The top panel shows violin plots with the difference in predicted negative log likelihood scores between each of the six alternative competing models (PNLLAlt) and the main Perceptual Summation model (PNLLPerSum) for Experiment 4 (n = 50 participants). Lower values of PNLL indicate higher predictive accuracy, therefore, PNLL difference scores higher (or lower) than zero indicate support for the Perceptual Summation (or a competing) model. In both experiments, the vast majority of participants are better predicted by the Perceptual Summation model than any of the alternatives. The bottom panel shows a table with a summary of descriptive and inferential statistics from all comparisons in Experiment 4, including the mean and standard error of the mean across participants. PNLL values were compared with a paired two-tailed t-test, corrected for multiple comparisons and all p-values were statistically significant (p < 0.001).
Extended Data Fig. 5 The Perceptual Summation model predicts ensemble memory for shape with a range manipulation.
Graphical representation of TCC model’s fit and prediction of data in Experiment 4. In this experiment participants had to remember simultaneously presented shapes, and the range of shapes was manipulated in the ensemble task. The top row of panel A shows the fits of the TCC model for individual items to aggregate data from the visual working memory task for six items, and the second row of panel A shows results from the predictive analysis in which d’ estimates from the visual working memory task were substituted into the TCC Perceptual Summation (blue), Post-perceptual (red) and Automatic Averaging (green) models to predict the ensemble data. Panel B shows data and model predictions for a few example participants.
Extended Data Fig. 6 Comparison in predictive accuracy between Sequential Perceptual Summation model and competing models of ensemble memory for sequentially presented stimuli.
The top panel shows violin plots of the difference in predicted negative log likelihood scores between each of the eight alternative competing models (PNLLAlt) and the main Sequential Perceptual Summation model (PNLLPerSum) (n = 50 participants). Lower values of PNLL indicate higher predictive accuracy, therefore, PNLL difference scores higher (or lower) than zero indicate support for the Sequential Perceptual Summation (or a competing) model. The vast majority of participants are better predicted by the Sequential Perceptual Summation model than any of the alternatives. Note that the baseline here is the Sequential Perceptual Summation model that relies on fitting a decay rate. The independent d’ Perceptual Summation model, the last model above, is the same model but without this parametric assumption about how d’ changes across the items in the working memory task. This independent model is instead one in which we used separate d’ estimates to quantify familiarity of items as a function of serial position, rather than a single d’ and rate parameter. This model is marked with an * because it is also a version of the Sequential Perceptual Summation model and so shows comparable predictive accuracy to the main Sequential Perceptual Summation model we use, as expected. The bottom panel shows a table with a summary of descriptive and inferential statistics from all comparisons in Experiment 5, including the mean and standard error of the mean across participants. PNLL values were compared with a paired two-tailed t-test, corrected for multiple comparisons and for all comparisons between competing models p-values were statistically significant (p < 0.001).
Extended Data Fig. 7 The Perceptual Summation model predicts ensemble memory for sequentially presented stimuli.
Summary of results from Experiment 5, in which participants had to remember colors of sequentially presented real-world objects. The top row of panel A shows the fits of the Sequential TCC model to individual data and the second row of panel A shows the TCC Sequential Perceptual Summation (blue), Post-perceptual (red) and Automatic Averaging (green) models’ predictions of the ensemble data in two conditions. In the clockwise (counterclockwise) condition the most recently shown items were from the clockwise (counterclockwise) direction from the mean color, producing a clockwise (counterclockwise) bias. Panel B shows data and model predictions for a few example participants.
Supplementary information
Supplementary Information
Supplementary Figs. 1–3, Table 1, Discussion and references.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Robinson, M.M., Brady, T.F. A quantitative model of ensemble perception as summed activation in feature space. Nat Hum Behav 7, 1638–1651 (2023). https://doi.org/10.1038/s41562-023-01602-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41562-023-01602-z