Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Amplification in the evaluation of multiple emotional expressions over time

Nature Human Behaviour volume 6, pages 1408–1416 (2022)Cite this article

Subjects

Abstract

Social interactions are dynamic and unfold over time. To make sense of social interactions, people must aggregate sequential information into summary, global evaluations. But how do people do this? Here, to address this question, we conducted nine studies (N = 1,583) using a diverse set of stimuli. Our focus was a central aspect of social interaction—namely, the evaluation of others’ emotional responses. The results suggest that when aggregating sequences of images and videos expressing varying degrees of emotion, perceivers overestimate the sequence’s average emotional intensity. This tendency for overestimation is driven by stronger memory of more emotional expressions. A computational model supports this account and shows that amplification cannot be explained only by nonlinear perception of individual exemplars. Our results demonstrate an amplification effect in the perception of sequential emotional information, which may have implications for the many types of social interactions that involve repeated emotion estimation.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Get just this article for as long as you need it

$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Structure and results of Studies 1–4.
Fig. 2: Results from the similarity analysis in Study 8 (N = 100).
Fig. 3: The difference between post-ratings and continuous ratings for the three types of videos in Study 9 (N = 565): neutral, negative and positive.

Data availability

The data for Studies 1–8 are available at https://osf.io/krgcv/. The data for Study 9 are available at https://github.com/StanfordSocialNeuroscienceLab/SEND.

Code availability

The code for the analysis of Studies 1–8 is available at https://osf.io/krgcv/. The code for the tasks can be found at https://github.com/GoldenbergLab/task-sequential-faces-emotion-estimationMe.

References

  1. Haberman, J., Harp, T. & Whitney, D. Averaging facial expressions over time. J. Vis. 9, 1 (2009).

    Article  Google Scholar 

  2. Alvarez, G. A. Representing multiple objects as an ensemble enhances visual cognition. Trends Cogn. Sci. 15, 122–131 (2011).

    Article  Google Scholar 

  3. Whitney, D. & Yamanashi Leib, A. Ensemble perception. Annu. Rev. Psychol. 69, 105–129 (2018).

    Article  Google Scholar 

  4. Oriet, C. & Brand, J. Size averaging of irrelevant stimuli cannot be prevented. Vis. Res. 79, 8–16 (2013).

    Article  Google Scholar 

  5. Oriet, C. & Hozempa, K. Incidental statistical summary representation over time. J. Vis. 16, 3 (2016).

    Article  Google Scholar 

  6. Schirmer, A. & Adolphs, R. Emotion perception from face, voice, and touch: comparisons and convergence. Trends Cogn. Sci. 21, 216–228 (2017).

    Article  Google Scholar 

  7. Eimer, M. & Holmes, A. Event-related brain potential correlates of emotional face processing. Neuropsychologia 45, 15–31 (2007).

    Article  Google Scholar 

  8. Goldenberg, A., Weisz, E., Sweeny, T., Cikara, M. & Gross, J. J. The crowd emotion amplification effect. Psychol. Sci. 32, 437–450 (2021).

    Article  Google Scholar 

  9. Jackson, M. C., Wu, C. Y., Linden, D. E. J. & Raymond, J. E. Enhanced visual short-term memory for angry faces. J. Exp. Psychol. Hum. Percept. Perform. 35, 363–374 (2009).

    Article  Google Scholar 

  10. Sessa, P., Luria, R., Gotler, A., Jolicœur, P. & Dell’acqua, R. Interhemispheric ERP asymmetries over inferior parietal cortex reveal differential visual working memory maintenance for fearful versus neutral facial identities. Psychophysiology 48, 187–197 (2011).

    Article  Google Scholar 

  11. Jackson, M. C., Wolf, C., Johnston, S. J., Raymond, J. E. & Linden, D. E. J. Neural correlates of enhanced visual short-term memory for angry faces: an fMRI study. PLoS ONE 3, e3536 (2008).

    Article  Google Scholar 

  12. Lee, H. J. & Cho, Y. S. Memory facilitation for emotional faces: visual working memory trade-offs resulting from attentional preference for emotional facial expressions. Mem. Cogn. 47, 1231–1243 (2019).

    Article  Google Scholar 

  13. Kaufmann, J. M. & Schweinberger, S. R. Expression influences the recognition of familiar faces. Perception 33, 399–408 (2004).

    Article  Google Scholar 

  14. Gallegos, D. R. & Tranel, D. Positive facial affect facilitates the identification of famous faces. Brain Lang. 93, 338–348 (2005).

    Article  Google Scholar 

  15. Jackson, M. C., Linden, D. E. J. & Raymond, J. E. Angry expressions strengthen the encoding and maintenance of face identity representations in visual working memory. Cogn. Emot. 28, 278–297 (2014).

    Article  Google Scholar 

  16. D’Argembeau, A. & Van der Linden, M. Facial expressions of emotion influence memory for facial identity in an automatic way. Emotion 7, 507–515 (2007).

    Article  Google Scholar 

  17. Tong, K., Dubé, C. & Sekuler, R. What makes a prototype a prototype? Averaging visual features in a sequence. Atten. Percept. Psychophys. 81, 1962–1978 (2019).

    Article  Google Scholar 

  18. Wilken, P. & Ma, W. J. A detection theory account of change detection. J. Vis. 4, 1120–1135 (2004).

    Article  Google Scholar 

  19. Fiedler, K. Beware of samples! A cognitive–ecological sampling approach to judgment biases. Psychol. Rev. 107, 659–676 (2000).

    Article  CAS  Google Scholar 

  20. Hubert-Wallander, B. & Boynton, G. M. Not all summary statistics are made equal: evidence from extracting summaries across time. J. Vis. 15, 5 (2015).

    Article  Google Scholar 

  21. Potter, M. Meaning in visual search. Science 187, 965–966 (1975).

    Article  CAS  Google Scholar 

  22. Albrecht, A. R. & Scholl, B. J. Perceptually averaging in a continuous visual world. Psychol. Sci. 21, 560–567 (2010).

    Article  Google Scholar 

  23. Goldenberg, A., Sweeny, T. D., Shpigel, E. & Gross, J. J. Is this my group or not? The role of ensemble coding of emotional expressions in group categorization. J. Exp. Psychol. Gen. https://doi.org/10.1037/xge0000651 (2019).

  24. Krumhuber, E. G., Kappas, A. & Manstead, A. S. R. Effects of dynamic aspects of facial expressions: a review. Emot. Rev. 5, 41–46 (2013).

    Article  Google Scholar 

  25. Krumhuber, E. G. & Skora, L. in Handbook of Human Motion (eds Müller, B. & Wolf, S. I.) 1–15 (Springer International, 2016); https://doi.org/10.1007/978-3-319-30808-1

  26. Yoshikawa, S. & Sato, W. Dynamic facial expressions of emotion induce representational momentum. Cogn. Affect. Behav. Neurosci. 8, 25–31 (2008).

    Article  Google Scholar 

  27. Kahneman, D., Fredrickson, B. L., Schreiber, C. A. & Redelmeier, D. A. When more pain is preferred to less: adding a better end. Psychol. Sci. 4, 401–405 (1993).

    Article  Google Scholar 

  28. Redelmeier, D. A. & Kahneman, D. Patients’ memories of painful medical treatments: real-time and retrospective evaluations of two minimally invasive procedures. Pain 66, 3–8 (1996).

    Article  Google Scholar 

  29. Fredrickson, B. L. Extracting meaning from past affective experiences: the importance of peaks, ends, and specific emotions. Cogn. Emot. 14, 577–606 (2000).

    Article  Google Scholar 

  30. Rozin, P. & Rozin, A. Advancing understanding of the aesthetics of temporal sequences by combining some principles and practices in music and cuisine with psychology. Perspect. Psychol. Sci. 13, 598–617 (2018).

    Article  Google Scholar 

  31. Cojuharenco, I. & Ryvkin, D. Peak–end rule versus average utility: how utility aggregation affects evaluations of experiences. J. Math. Psychol. 52, 326–335 (2008).

    Article  Google Scholar 

  32. Ong, D. C. et al. Modeling emotion in complex stories: the Stanford Emotional Narratives Dataset. IEEE Trans. Affect. Comput. https://doi.org/10.1109/taffc.2019.2955949 (2019).

  33. Koller, M. robustlmm: an R package for robust estimation of linear mixed-effects models. J. Stat. Softw. 75, 1–24 (2016).

    Article  Google Scholar 

  34. Robinson, M. & Brady, T. A quantitative model of ensemble perception as summed patterns of activation in feature space. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/k3d26 (2022).

  35. Eid, M. & Diener, E. Norms for experiencing emotions in different cultures: inter- and intranational differences. J. Pers. Soc. Psychol. 81, 869–885 (2001).

    Article  CAS  Google Scholar 

  36. Kardosh, R., Sklar, A. Y., Goldstein, A., Pertzov, Y. & Hassin, R. R. Minority salience and the overestimation of individuals from minority groups in perception and memory. Proc. Natl Acad. Sci. USA 119, e2116884119 (2022).

    Article  CAS  Google Scholar 

  37. Zacks, J. M. & Swallow, K. M. Event segmentation. Curr. Dir. Psychol. Sci. 16, 80–84 (2007).

    Article  Google Scholar 

  38. Dubé, C., Zhou, F., Kahana, M. J. & Sekuler, R. Similarity-based distortion of visual short-term memory is due to perceptual averaging. Vis. Res. 96, 8–16 (2014).

    Article  Google Scholar 

  39. Elias, E., Dyer, M. & Sweeny, T. D. Ensemble perception of dynamic emotional groups. Psychol. Sci. 28, 193–203 (2017).

    Article  Google Scholar 

  40. Tottenham, N. et al. The NimStim set of facial expressions: judgments from untrained research participants. Psychiatry Res. 168, 242–249 (2009).

    Article  Google Scholar 

  41. Langner, O. et al. Presentation and validation of the Radboud Faces Database. Cogn. Emot. 24, 1377–1388 (2010).

    Article  Google Scholar 

  42. Schurgin, M. W., Wixted, J. T. & Brady, T. F. Psychophysical scaling reveals a unified theory of visual memory strength. Nat. Hum. Behav. 4, 1156–1172 (2020).

    Article  Google Scholar 

  43. Wickens, T. D. Elementary Signal Detection Theory (Oxford Univ. Press, 2001).

Download references

Acknowledgements

J.S. is supported in part by the German Academic Scholarship Foundation (Promotionsförderung der Studienstiftung des deutschen Volkes) and in part by NIH grant no. 1R01MH112560-01. D.C.O. is supported in part by a Singapore Ministry of Education Academic Research Fund Tier 1 grant. T.F.B. is supported in part by NSF CAREER grant no. BCS-1653457. Finally, M.M.R. is supported by the National Research Service Award fellowship from the National Institute of Health no. 1F32MH127823-01. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Author notes

  1. These authors contributed equally: Amit Goldenberg, Jonas Schöne.

Authors and Affiliations

  1. Harvard Business School, Harvard University, Boston, MA, USA

    Amit Goldenberg, Zi Huang & David Levari

  2. Department of Experimental Psychology, University of Oxford, Oxford, UK

    Jonas Schöne

  3. Department of Psychology, University of Denver, Denver, CO, USA

    Timothy D. Sweeny

  4. Department of Information Systems and Analytics, National University of Singapore, Singapore, Singapore

    Desmond C. Ong

  5. Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore, Singapore

    Desmond C. Ong

  6. University of California, San Diego, CA, USA

    Timothy F. Brady & Maria M. Robinson

  7. Department of Psychology, Stanford University, Stanford, CA, USA

    Jamil Zaki & James J. Gross

Authors
  1. Amit Goldenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jonas Schöne
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zi Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Timothy D. Sweeny
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Desmond C. Ong
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Timothy F. Brady
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Maria M. Robinson
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. David Levari
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jamil Zaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. James J. Gross
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

A.G., J.S. and Z.H. conceived and designed the experiments. A.G. and J.S. performed the experiments (Studies 1–8), analysed the data for these experiments and wrote the paper. D.C.O. and J.Z. designed and ran Study 9, and D.C.O. provided an initial analysis of the data. D.C.O. and J.Z. reviewed the paper. T.F.B. and M.M.R. analysed the data of Study 8, conducted the computational model of that study and reviewed the paper. D.L., T.D.S. and J.J.G. were involved in writing and reviewing the manuscript.

Corresponding author

Correspondence to Amit Goldenberg.

Ethics declarations

Competing interests

The authors declare no competing interests. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Peer review

Peer review information

Nature Human Behaviour thanks Raoul Bell and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Goldenberg, A., Schöne, J., Huang, Z. et al. Amplification in the evaluation of multiple emotional expressions over time. Nat Hum Behav 6, 1408–1416 (2022). https://doi.org/10.1038/s41562-022-01390-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-022-01390-y

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing