In crowds, where scrutinizing individual facial expressions is inefficient, humans can make snap judgments about the prevailing mood by reading ‘crowd emotion’. We investigated how the brain accomplishes this feat in a set of behavioural and functional magnetic resonance imaging studies. Participants were asked to either avoid or approach one of two crowds of faces presented in the left and right visual hemifields. Perception of crowd emotion was improved when crowd stimuli contained goal-congruent cues and was highly lateralized to the right hemisphere. The dorsal visual stream was preferentially activated in crowd emotion processing, with activity in the intraparietal sulcus and superior frontal gyrus predicting perceptual accuracy for crowd emotion perception, whereas activity in the fusiform cortex in the ventral stream predicted better perception of individual facial expressions. Our findings thus reveal significant behavioural differences and differential involvement of the hemispheres and the major visual streams in reading crowd versus individual face expressions.
This is a preview of subscription content, access via your institution
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Rent or buy this article
Get just this article for as long as you need it
Prices may be subject to local taxes which are calculated during checkout
Alvarez, G. A. Representing multiple objects as an ensemble enhances visual cognition. Trends Cogn. Sci. 15, 122–131 (2011).
Fischer, J. & Whitney, D. Object-level visual information gets through the bottleneck of crowding. J. Neurophysiol. 106, 1389–1398 (2011).
Haberman, J., Harp, T. & Whitney, D. Averaging facial expression over time. J. Vis. 9, 1–13 (2009).
Haberman, J. & Whitney, D. Efficient summary statistical representation when change localization fails. Psychon. Bull. Rev. 18, 855–859 (2011).
Dakin, S. C. & Watt, R. J. The computation of orientation statistics from visual texture. Vision Res. 37, 3181–3192 (1997).
Parkes, L., Lund, J., Angelucci, A., Solomon, J. A. & Morgan, M. Compulsory averaging of crowded orientation signals in human vision. Nat. Neurosci. 4, 739–744 (2001).
Ariely, D. Seeing sets: representation by statistical properties. Psychol. Sci. 12, 157–162 (2001).
Chong, S. C. & Treisman, A. Representation of statistical properties. Vision Res. 43, 393–404 (2003).
Watamaniuk, S. N. J. & Sekuler, R. Temporal and spatial integration in dynamic random-dot stimuli. Vision Res. 32, 2341–2347 (1992).
Halberda, J., Sires, S. F. & Feigenson, L. Multiple spatially overlapping sets can be enumerated in parallel. Psychol. Sci. 17, 572–576 (2006).
Alvarez, G. A. & Oliva, A. The representation of simple ensemble visual features outside the focus of attention. Psychol. Sci. 19, 392–398 (2008).
Choo, H. & Franconeri, S. L. Objects with reduced visibility still contribute to size averaging. Atten. Percept. Psychophys. 72, 86–99 (2010).
Corbett, J. E. & Oriet, C. The whole is indeed more than the sum of its parts: Perceptual averaging in the absence of individual item representation. Acta Psychol. 138, 289–301 (2011).
Haberman, J. & Whitney, D. The visual system discounts emotional deviants when extracting average expression. Atten. Percept. Psychophys. 72, 1825–1838 (2010).
Im, H. Y. & Halberda, J. The effects of sampling and internal noise on the representation of ensemble average size. Atten. Percept. Psychophys. 75, 278–286 (2013).
Haberman, J. & Whitney, D. Rapid extraction of mean emotion and gender from sets of faces. Curr. Biol. 17, R751–R753 (2007).
Hubert-Wallander, B. & Boynton, G. M. Not all summary statistics are made equal: evidence from extracting summaries across time. J. Vis. 15, 1–12 (2015).
Ji, L., Chen, W. & Fu, X. Different roles of foveal and extrafoveal vision in ensemble representation for facial expressions. EPCE 8532, 164–173 (2014).
Yang, J.-W., Yoon, K. L., Chong, S. C. & Oh, K. J. Accurate but pathological: social anxiety and ensemble coding of emotion. Cog. Ther. Res. 37, 572–578 (2013).
De Fockert, J. W. & Wolfenstein, C. Rapid extraction of mean identity from sets of faces. Q. J. Exp. Psychol. 62, 1716–1722 (2009).
Leib, A. Y., Puri, A. M., Fischer, J., Bentin, S., Whitney, D. & Robertson, L. Crowd perception in prosopagnosia. Neuropsychologia 50, 1698–1707 (2012).
Leib, A. Y., Fischer, J., Liu, Y., Qiu, S., Robertson, L. & Whitney, D. Ensemble crowd perception: a viewpoint-invariant mechanism to represent average crowd identity. J. Vis. 14, 1–13 (2014).
Neumann, M. F., Schweinberger, S. R. & Burton, A. M. Viewers extract mean and individual identity from sets of famous faces. Cognition 128, 56–63 (2013).
Brunyé, T. T., Howe, J. L. & Mahoney, C. R. Seeing the crowd for the bomber: spontaneous threat perception from static and randomly moving crowd simulations. J. Exp. Psychol. Appl. 20, 303–322 (2014).
Sweeny, T. D., Haroz, S. & Whitney, D. Perceiving group behavior: sensitive ensemble coding mechanisms for biological motion of human crowds. J. Exp. Psychol. Hum. Percept. Perform. 39, 329–337 (2013).
Florey, J., Clifford, C. W., Dakin, S. & Mareschal, I. Spatial limitations in averaging social cues. Sci. Rep. 6, 32210 (2016).
Sweeny, T. D. & Whitney, D. Perceiving crowd attention: ensemble perception of a crowd’s gaze. Psychol. Sci. 25, 1903–1913 (2014).
Adams, R. B. Jr., Ambady, N., Macrae, C. N. & Kleck, R. E. Emotional expressions forecast approach-avoidance behavior. Motiv. Emot. 30, 179–188 (2006).
Horstmann, G. What do facial expressions convey: feeling states, behavioral intentions, or action requests? Emotion 3, 150–166 (2003).
Marsh, A. A., Ambady, N. & Kleck, R. E. The effects of fear and anger facial expressions on approach- and avoidance-related behaviors. Emotion 5, 119–124 (2005).
Davidson, R. J. Anterior cerebral asymmetry and the nature of emotion. Brain Cogn. 20, 125–151 (1992).
Elliot, A. J. Approach and avoidance motivation and achievement goals. Educ. Psychol. 34, 169–189 (1999).
Adams, R. B. Jr., Hess, U. & Kleck, R. E. The intersection of gender-related facial appearance and facial displays of emotion. Emot. Rev. 7, 5–13 (2015).
Bargh, J. A., Chen, M. & Burrows, L. Automaticity of social behavior: direct effects of trait construct and stereotype activation on action. J. Pers. Soc. Psychol. 71, 230–244 (1996).
Davidson, R. J. & Irwin, W. The functional neuroanatomy of emotion and affective style. Trends Cogn. Sci. 3, 11–21 (1999).
De Renzi, E. Prosopagnosia in two patients with CT scan evidence of damage confined to the right hemisphere. Neuropsychologia 24, 385–389 (1986).
Fabes, R. A. & Martin, C. L. Gender and age stereotypes of emotionality. Pers. Soc. Psychol. Bull. 17, 532–540 (1991).
Kanwisher, N., McDermott, J. & Chun, M. M. The fusiform face area: a module in human extrastriate cortex specialized for face perception. J. Neurosci. 17, 4302–4311 (1997).
Wada, Y. & Yamamoto, T. Selective impairment of facial recognition due to a haematoma restricted to the right fusiform and lateral occipital region. J. Neurol. Neurosurg. Psychiatry 71, 254–257 (2001).
Merigan, W. H. & Maunsell, J. H. How parallel are the primate visual pathways? Annu. Rev. Neurosci. 16, 369–402 (1993).
Sawatari, A. & Callaway, E. M. Convergence of magno- and parvocellular pathways in layer 4B of macaque primary visual cortex. Nature 380, 442–446 (1996).
Freud, E., Plaut, D. C. & Behrmann, M. ‘What’ is happening in the dorsal visual pathway. Trends Cogn. Sci. 20, 773–784 (2016).
Goodale, M. A. & Milner, A. D. Separate visual pathways for perception and action. Trends Neurosci. 15, 20–25 (1992).
Kveraga, K., Boshyan, J. & Bar, M. The magnocellular trigger of top-down facilitation in object recognition. J. Neurosci. 27, 13232–13240 (2007).
Kveraga, K., Ghuman, A. S. & Bar, M. Top-down predictions in the cognitive brain. Brain Cogn. 65, 145–168 (2007).
Livingstone, M. S. & Hubel, D. E. Segregation of form, color, movement, and depth: anatomy, physiology, and perception. Science 240, 740–749 (1988).
Milner, A. D. & Goodale, M. A. The Visual Brain in Action (Oxford Univ. Press, Oxford, 1995).
Milner, A. D. & Goodale, M. A. Two visual systems re-viewed. Neuropsychologia 46, 774–785 (2008).
Schiller, P. H. & Logothetis, N. K. The color- opponent and broad-band channels of the primate visual system. Trends Neurosci. 13, 392–398 (1990).
Thomas, C., Kveraga, K., Huberle, E., Karnath, H.-O. & Bar, M. Enabling global processing in simultanagnosia by psychophysical biasing of visual pathways. Brain 135, 1578–1585 (2012).
Vuilleumier, P., Armony, J. L., Driver, J. & Dolan, R. J. Distinct spatial frequency sensitivities for processing faces and emotional expressions. Nat. Neurosci. 6, 624–631 (2003).
Winston, J. S., Vuilleumier, P. & Dolan, R. J. Effects of low-spatial frequency components of fearful faces on fusiform cortex activity. Curr. Biol. 13, 1824–1829 (2003).
Courtney, S. M., Ungerleider, L. G., Keil, K. & Haxby, J. V. Object and spatial visual working memory activate separate neural systems in human cortex. Cereb. Cortex 6, 39–49 (1996).
Courtney, S. M., Petit, L., Maisog, J. M., Ungerleider, L. G. & Haxby, J. V. An area specialized for spatial working memory in human frontal cortex. Science 279, 1347–1351 (1998).
Sala, J. B. & Courtney, S. M. Binding of what and where during working memory maintenance. Cortex 43, 5–21 (2007).
Takahashi, E., Ohki, K. & Kim, D.-S. Dissociation and convergence of the dorsal and ventral visual streams in the human prefrontal cortex. Neuroimage 65, 488–498 (2013).
Wilson, F. A., Scalaidhe, S. P. & Goldman-Rakic, P. S. Dissociation of object and spatial processing domains in primate prefrontal cortex. Science 260, 1955–1958 (1993).
Denys, K. et al. The processing of visual shape in the cerebral cortex of human and nonhuman primates: a functional magnetic resonance imaging study. J. Neurosci. 24, 2551–2565 (2004).
Grill-Spector, K. & Malach, R. The human visual cortex. Annu. Rev. Neurosci. 27, 649–677 (2004).
Haxby, J. V. et al. Dissociation of object and spatial visual processing pathways in human extrastriate cortex. Proc. Natl Acad. Sci. USA 88, 1621–1625 (1991).
Purves, D. et al. Neuroscience 3rd edn (Sinauer Associates, Sunderland, MA, 2004).
Taylor, J. C., Wiggett, A. J. & Downing, P. E. Functional MRI analysis of body and body part representations in extrastriate and fusiform body parts. J. Neurophysiol. 98, 1626–1633 (2007).
LeDoux, J. E. The Emotional Brain (Simon and Schuster, New York, NY, 1996).
Morris, J. S. et al. A differential neural response in the human amygdala to fearful and happy facial expressions. Nature 383, 812–815 (1996).
Damasio, A. R. et al. Subcortical and cortical brain activity during the feeling of self-generated emotions. Nat. Neurosci. 3, 1049–1056 (2000).
Vuilleumier, P., Armony, J. L., Driver, J. & Dolan, R. J. Effects of attention and emotion on face processing in the human brain: an event-related fMRI study. Neuron 30, 829–841 (2001).
Zald, D. H. The human amygdala and the emotional evaluation of sensory stimuli. Brain Res. Brain Res. Rev. 41, 88–123 (2003).
Whalen, P. J. et al. Human amygdala responsivity to masked fearful eye whites. Science 306, 2061 (2004).
Ekman, P. & Friesen, W. V. Pictures of Facial Affect (Consulting Psychologists Press, Palo Alto, CA, 1976).
Botvinick, M. M., Cohen, J. D. & Carter, C. S. Conflict monitoring and anterior cingulate cortex: an update. Trends Cogn. Sci. 8, 539–545 (2004).
Jonides, J. & Nee, D. E. Brain mechanisms of proactive interference in working memory. Neuroscience 139, 181–193 (2006).
Monsell, S. Task switching. Trends Cogn. Sci. 7, 134–140 (2003).
Rogers, R. D. & Monsell, S. The costs of a predictable switch between simple cognitive tasks. J. Exp. Psychol. Gen. 124, 207–231 (1995).
Theios, J. in Attention and Performance V (eds Rabbitt, P. M. A. & Dornic, S.) 418–440 (Academic Press, New York, NY, 1975).
Christie, J., Ginsberg, J. P., Steedman, J., Fridriksson, J., Bonilha, L. & Rorden, C. Global versus local processing: seeing the left side of the forest and the right side of the trees. Front. Hum. Neurosci. 6, 28 (2012).
Delis, D. C., Robertson, L. C. & Efron, R. Hemispheric specialization of memory for visual hierarchical stimuli. Neuropsychologia 24, 205–214 (1986).
Robertson, L. C., Lamb, M. R. & Knight, R. T. Effects of lesions of temporal-parietal junction on perceptual and attentional processing in humans. J. Neurosci. 8, 3757–3769 (1988).
Robertson, L. C. & Ivry, R. Hemispheric asymmetry: attention to visual and auditory primitives. Curr. Dir. Psychol. Sci. 9, 59–63 (2000).
Yovel, G., Levy, J. & Yovel, I. Hemispheric asymmetries for global and local visual perception: effects of stimulus and task factors. J. Exp. Psychol. Hum. Percept. Perform. 27, 1369–1385 (2001).
Becker, D. V., Kenrick, D. T., Neuberg, S. L., Blackwell, K. C. & Smith, D. M. The confounded nature of angry men and happy women. J. Pers. Soc. Psychol. 92, 179–190 (2007).
Borod, J. C. et al. Right hemisphere emotional perception: evidence across multiple channels. Neuropsychology 12, 446–458 (1998).
Silberman, E. K. & Weingartner, H. Hemispheric lateralization of functions related to emotion. Brain Cogn. 5, 322–353 (1986).
Tzourio-Mazoyer, N. et al. Automatic anatomical labelling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage 15, 273–289 (2002).
Denison, R. N., Vu, A. T., Yacoub, E., Feinberg, D. A. & Silver, M. A. Functional mapping of the magnocellular and parvocellular subdivisions of human LGN. Neuroimage 102, 358–369 (2014).
Adams, R. B. Jr. et al. Compound facial threat cue perception: contributions of visual pathways, aging, and anxiety. J. Vis. 16, 1375 (2016).
Im, H. Y., Adams, R. B., Jr., Boshyan, J., Ward, N., Cushing, C. & Kveraga, K. Anxiety modulates perception of facial fear in a pathway-specific, lateralized manner. Preprint at http://www.biorxiv.org/content/early/2017/05/24/141838 (2017).
Kveraga, K. in Scene Vision: Making Sense of What We See (eds Kveraga, K. & Bar, M.) 291–307 (MIT Press, Cambridge, MA, 2014).
Erk, S., Kleczar, A. & Walter, H. Valence-specific regulation effects in a working memory task with emotional context. Neuroimage 37, 623–632 (2007).
Heller, W. & Nitscke, J. B. Regional brain activity in emotion: a framework for understanding cognition in depression. Cogn. Emot. 11, 637–661 (1997).
Yang, Q., Wang, X., Yin, S., Zhao, X., Tan, J. & Chen, A. Improved emotional conflict control triggered by the processing priority of negative emotion. Sci. Rep. 6, 24302 (2016).
Adams, R. B. Jr. & Kveraga, K. Social vision: functional forecasting and the integration of compound social cues. Rev. Philos. Psychol. 6, 591–610 (2015).
Rogers, L. J., Vallortigara, G. & Andrew, R. J. Divided Brains: The Biology and Behavior of Brain Asymmetries (Cambridge Univ. Press, Cambridge, 2013).
Scott, W. A. Cognitive complexity and cognitive flexibility. Sociometry 25, 405–414 (1962).
Craig, A. D. Forebrain emotional asymmetry: a neuroanatomical basis? Trends Cogn. Sci. 9, 566–571 (2005).
Davidson, R. J. in Brain Asymmetry, Cerebral Asymmetry, Emotion, and Affective Style (eds Davidson, R. J. & Hugdahl, K.) 361–387 (MIT Press, Cambridge, MA, 1995).
Ivry, R. B. & Robertson, L. C. The Two Sides of Perception (MIT Press, Cambridge, MA, 1998).
Carroll, N. C. & Young, A. W. Priming of emotion recognition. Q. J. Exp. Psychol. 58, 1173–1197 (2005).
Norman, D. A. & Shallice, T. in Consciousness and Self-Regulation: Advances in Research and Theory IV (eds Davidson, R., Schwartz, R. & Shapiro, D.) 376–390 (Plenum Press, New York, NY, 1986).
Posner, M. I., Rueda, M. R. & Kanske, P. in Handbook of Psychophysiology (eds Cacioppo, J. T., Tassinary, J. G. & Berntson, G. G.) 410–414 (Cambridge Univ. Press, Cambridge, 2007).
Cant, J. S., Sun, S. Z. & Xu, Y. Distinct cognitive mechanisms involved in the processing of single objects and object ensembles. J. Vis. 15, 1–21 (2015).
Chong, S. C., Joo, S. J., Emmmanouil, T. & Treisman, A. Statistical processing: not so implausible after all. Percept. Psychophys. 70, 1327–1334 (2008).
Chong, S. C. & Evans, K. K. Distributed vs. focused attention (count vs. estimate). Wiley Interdiscip. Rev. Cogn. Sci. 2, 634–638 (2011).
Chong, S. C. & Treisman, A. Attentional spread in the statistical processing of visual displays. Percept. Psychophys. 67, 1–13 (2005).
Haberman, J., Brady, T. F. & Alvarez, G. A. Individual differences in ensemble perception reveal multiple, independent levels of ensemble representation. J. Exp. Psychol. Gen. 144, 432–446 (2015).
Myczek, K. & Simons, D. J. Better than average: alternatives to statistical summary representations for rapid judgments of average size. Percept. Psychophys. 70, 772–788 (2008).
Cant, J. S. & Xu, Y. The impact of density and ratio on object-ensemble representation in human anterior-medial ventral visual cortex. Cereb. Cortex 25, 4226–4239 (2015).
Cant, J. S. & Xu, Y. The contribution of object shape and surface properties to object-ensemble representation in anterior-medial ventral visual cortex. J. Cogn. Neurosci. 29, 398–412 (2017).
Huis In ‘t Veld, E. M. J. & de Gelder, B. From individual fear to mass panic. The neurological basis of crowd perception. Hum. Brain Mapp. 36, 2338–1351 (2015).
Utochkin, I. S. Ensemble summary statistics as a basis for rapid visual categorization. J. Vis. 15, 1–14 (2015).
Brady, T. F. & Alvarez, G. A. No evidence for a fixed object limit in working memory: ensemble representations inflate estimates of working memory capacity for complex objects. J. Exp. Psychol. Learn. Mem. Cogn. 41, 921–929 (2015).
Cohen, M. A., Dennett, D. C. & Kanwisher, N. What is the bandwidth of perceptual experience? Trends Cogn. Sci. 19, 324–335 (2016).
Feigenson, L. in Space, Time, and Number in the Brain: Searching for the Foundations of Mathematical Thought (eds Dehaene, S. & Brannon, E.) 13–22 (Elsevier, London, 2011).
Im, H. Y. & Chong, S. C. Mean size as a unit of visual working memory. Perception 43, 663–676 (2014).
Im, H. Y., Park, W. J. & Chong, S. C. Ensemble statistics as units of selection. J. Cogn. Psychol. 27, 114–127 (2015).
Cowan, N. The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behav. Brain Sci. 24, 87–185 (2001).
Luck, S. & Vogel, E. K. The capacity of visual working memory for features and conjunctions. Nature 390, 279–281 (1997).
Pylyshyn, Z. W. & Storm, R. W. Tracking multiple independent targets: evidence for a parallel tracking mechanism. Spat. Vis. 3, 179–197 (1988).
Treisman, A. M. & Gelade, G. A feature-integration theory of attention. Cogn. Psychol. 12, 97–136 (1980).
Haberman, J. & Whitney, D. in From Perception to Consciousness: Searching with Anne Treisman (eds Wolfe, J. & Robertson, L.) Ch. 16 (Oxford Univ. Press, Oxford, 2012).
Brady, T. F. & Alvarez, G. A. Hierarchical encoding in visual working memory: ensemble statistics bias memory for individual items. Psychol. Sci. 22, 384–392 (2011).
Cohen, J. Statistical Power Analysis for the Behavioral Sciences (Lawrence Erlbaum Associates, Hillsdale, NJ, 1988).
Brainard, D. H. The Psychophysics Toolbox. Spat. Vis. 10, 433–436 (1997).
Pelli, D. G. The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat. Vis. 10, 437–442 (1997).
Maule, J. & Franklin, A. Effects of ensemble complexity and perceptual similarity on rapid averaging of hue. J. Vis. 15, 1–18 (2015).
Utochkin, I. S. & Tiurina, N. A. Parallel averaging of size is possible but range-limited: a reply to Marchant, Simons, and De Fockert. Acta Psychologica 146, 7–18 (2014).
Fitts, P. M. & Seeger, C. M. S-R compatibility: spatial characteristics of stimulus and response codes. J. Exp. Psychol. 46, 199–201 (1953).
Deichmann, R., Gottfried, J. A., Hutton, C. & Turner, R. Optimized EPI for fMRI studies of the orbitofrontal cortex. Neuroimage 19, 430–441 (2003).
Wall, M. B., Walker, R. & Smith, A. T. Functional imaging of the human superior colliculus: an optimised approach. Neuroimage 47, 1620–1627 (2009).
Kringelbach, M. L. & Rolls, E. T. The functional neuroanatomy of the human orbitofrontal cortex: evidence from neuroimaging and neuropsychology. Prog. Neurobiol. 72, 341–372 (2004).
Mazaika P. K., Hoeft, F., Glover G. H. & Reiss A. L. Methods and software for fMRI analysis for clinical subjects. In Poster Session Presented at the Meeting of Human Brain Mapping 2009 (2009).
Fischl, B. et al. Automatically parcellating the human cerebral cortex. Cereb. Cortex 14, 11–22 (2004).
Spiridon, M., Fischl, B. & Kanwisher, N. Location and spatial profile of category-specific regions in human extrastriate cortex. Hum. Brain Mapp. 27, 77–89 (2006).
This work was supported by the National Institutes of Health (R01MH101194) to K.K. and R.B.A. Jr. Data collection was conducted at the Pennsylvania State University. Informed written consent was obtained for all the studies according to the procedures of the Institutional Review Board at the Pennsylvania State University. The participants received a course credit for their participation. No funders had any role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
The authors declare no competing interests.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Supplementary Results; Supplementary Figures 1–10; Supplementary Tables 1–4
Rights and permissions
About this article
Cite this article
Im, H., Albohn, D.N., Steiner, T.G. et al. Differential hemispheric and visual stream contributions to ensemble coding of crowd emotion. Nat Hum Behav 1, 828–842 (2017). https://doi.org/10.1038/s41562-017-0225-z
This article is cited by
Mechanisms for individual, group-based and crowd-based attention to social information
Nature Reviews Psychology (2022)
Dissociable causal roles of the frontal and parietal cortices in the effect of object location on object identity detection: a TMS study
Experimental Brain Research (2022)
Spatial and feature-based attention to expressive faces
Experimental Brain Research (2019)
Cross-cultural and hemispheric laterality effects on the ensemble coding of emotion in facial crowds
Culture and Brain (2017)