Article | Published:

The primacy of categories in the recognition of 12 emotions in speech prosody across two cultures

Abstract

Central to emotion science is the degree to which categories, such as Awe, or broader affective features, such as Valence, underlie the recognition of emotional expression. To explore the processes by which people recognize emotion from prosody, US and Indian participants were asked to judge the emotion categories or affective features communicated by 2,519 speech samples produced by 100 actors from 5 cultures. With large-scale statistical inference methods, we find that prosody can communicate at least 12 distinct kinds of emotion that are preserved across the 2 cultures. Analyses of the semantic and acoustic structure of the recognition of emotions reveal that emotion categories drive the recognition of emotions more so than affective features, including Valence. In contrast to discrete emotion theories, however, emotion categories are bridged by gradients representing blends of emotions. Our findings, visualized within an interactive map, reveal a complex, high-dimensional space of emotional states recognized cross-culturally in speech prosody.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Code availability

Custom MATLAB analysis code can be requested from https://goo.gl/forms/3q0y2Vvi1KinMft13.

Data availability

The 2,519 speech samples used in the present study and their ratings can be requested from https://goo.gl/forms/3q0y2Vvi1KinMft13. Publications incorporating the speech samples should reference the previous study33.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    Keltner, D. & Haidt, J. Social functions of emotions at four levels of analysis. Cogn. Emot. 13, 505–521 (1999).

  2. 2.

    Nesse, R. M. Evolutionary explanations of emotions. Hum. Nat. 1, 261–289 (1990).

  3. 3.

    Campos, B., Shiota, M. N., Keltner, D., Gonzaga, G. C. & Goetz, J. L. What is shared, what is different? Core relational themes and expressive displays of eight positive emotions. Cogn. Emot. 27, 37–52 (2013).

  4. 4.

    Oveis, C., Spectre, A., Smith, P. K., Liu, M. Y. & Keltner, D. Laughter conveys status. J. Exp. Soc. Psychol. 65, 109–115 (2016).

  5. 5.

    Gonzaga, G. C., Keltner, D., Londahl, E. A. & Smith, M. D. Love and the commitment problem in romantic relations and friendship. J. Pers. Soc. Psychol. 81, 247–262 (2001).

  6. 6.

    ten Brinke, L. & Adams, G. S. Saving face? When emotion displays during public apologies mitigate damage to organizational performance. Organ. Behav. Hum. Decis. Process. 130, 1–12 (2015).

  7. 7.

    Cowen, A. S. & Keltner, D. Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proc. Natl Acad. Sci. USA 114, E7900–E7909 (2017).

  8. 8.

    Schirmer, A. & Adolphs, R. Emotion perception from face, voice, and touch: comparisons and convergence. Trends Cogn. Sci. 21, 216–228 (2017).

  9. 9.

    Singer, T. & Lamm, C. The social neuroscience of empathy. Ann. NY Acad. Sci. 1156, 81–96 (2009).

  10. 10.

    Frühholz, S., Ceravolo, L. & Grandjean, D. Specific brain networks during explicit and implicit decoding of emotional prosody. Cereb. Cortex 22, 1107–1117 (2012).

  11. 11.

    Bach, D. R. et al. The effect of appraisal level on processing of emotional prosody in meaningless speech. Neuroimage 42, 919–927 (2008).

  12. 12.

    Cordaro, D. T. et al. Universals and cultural variations in 22 emotional expressions across five cultures. Emotion 18, 75–93 (2018).

  13. 13.

    Elfenbein, H. A. & Ambady, N. On the universality and cultural specificity of emotion recognition: a meta-analysis. Psychol. Bull. 128, 203–235 (2002).

  14. 14.

    Keltner, D. & Cordaro, D. T. in Emotion Researcher (Scarantino, A. ed.) Available at http://emotionresearcher.com/understanding-multimodal-emotional-expressions-recent-advances-in-basic-emotion-theory/ (2015).

  15. 15.

    Norenzayan, A. & Heine, S. J. Psychological universals: what are they and how can we know? Psychol. Bull. 131, 763–784 (2005).

  16. 16.

    Sauter, D. A., Eisner, F., Ekman, P. & Scott, S. K. Cross-cultural recognition of basic emotions through nonverbal emotional vocalizations. Proc. Natl Acad. Sci. USA 107, 2408–2412 (2010).

  17. 17.

    Filippi, P. et al. Humans recognize emotional arousal in vocalizations across all classes of terrestrial vertebrates: evidence for acoustic universals. Proc. R. Soc. B 284, 20170990 (2017).

  18. 18.

    Parr, L. A., Waller, B. M. & Vick, S. J. New developments in understanding emotional facial signals in chimpanzees. Curr. Dir. Psychol. Sci. 16, 117–122 (2007).

  19. 19.

    Snowdon, C. T. in Handbook of Affective Sciences (eds Davidson, R. J. & Scherer, K. R.) 457-480 (Oxford Univ. Press, Oxford, 2002).

  20. 20.

    Filippi, P. Emotional and interactional prosody across animal communication systems: a comparative approach to the emergence of language. Front. Psychol. 7, 1393 (2016).

  21. 21.

    Adolphs, R. Neural systems for recognizing emotion. Curr. Opin. Neurobiol. 12, 169–177 (2002).

  22. 22.

    Russell, J. A. Is there universal recognition of emotion from facial expressions? A review of the cross-cultural studies. Psychol. Bull. 115, 102–141 (1994).

  23. 23.

    Cordaro, D. T., Keltner, D., Tshering, S., Wangchuk, D. & Flynn, L. M. The voice conveys emotion in ten globalized cultures and one remote village in Bhutan. Emotion 16, 117–128 (2016).

  24. 24.

    Gendron, M., Roberson, D., van der Vyver, J. M. & Barrett, L. F. Cultural relativity in perceiving emotion from vocalizations. Psychol. Sci. 25, 911–920 (2014).

  25. 25.

    Hertenstein, M. J. & Campos, J. J. The retention effects of an adult’s emotional displays on infant behavior. Child Dev. 75, 595–613 (2004).

  26. 26.

    Juslin, P. N. & Laukka, P. Communication of emotions in vocal expression and music performance: different channels, same code? Psychol. Bull. 129, 770–814 (2003).

  27. 27.

    Keltner, D. et al. in Handbook of Emotions 4th edn (eds Lewis M., Haviland-Jones, J. M. & Barrett, L. F.) 467–482 (Guilford, New York, 2016).

  28. 28.

    Wu, Y., Muentener, P. & Schulz, L. E. One- to four-year-olds connect diverse positive emotional vocalizations to their probable causes. Proc. Natl Acad. Sci. USA 114, 11896–11901 (2017).

  29. 29.

    Titze, I. R. & Martin, D. W. Principles of voice production. J. Acoust. Soc. Am. 104, 1148 (1998).

  30. 30.

    Scherer, K. R. & Bänziger, T. Emotional expression in prosody: a review and an agenda for future research. In Proc. Speech Prosody 2004 359–366 (2004).

  31. 31.

    Mitchell, R. L. C. & Ross, E. D. Attitudinal prosody: what we know and directions for future study. Neurosci. Biobehav. Rev. 37, 471–479 (2013).

  32. 32.

    Hancil, S. The Role of Prosody in Affective Speech (Peter Lang, New York, 2009).

  33. 33.

    Laukka, P. et al. The expression and recognition of emotions in the voice across five nations: a lens model analysis based on acoustic features. J. Pers. Soc. Psychol. 111, 686–705 (2016).

  34. 34.

    Nordström, H., Laukka, P., Thingujam, N. S., Schubert, E. & Elfenbein, H. A. Emotion appraisal dimensions inferred from vocal expressions are consistent across cultures: a comparison between Australia and India. R. Soc. Open Sci. 4, 170912 (2017).

  35. 35.

    Paulmann, S. & Uskul, A. K. Cross-cultural emotional prosody recognition: evidence from Chinese and British listeners. Cogn. Emot. 28, 230–244 (2014).

  36. 36.

    Scherer, K. R., Banse, R. & Wallbott, H. G. Emotion inferences from vocal expression correlate across languages and cultures. J. Cross Cult. Psychol. 32, 76–92 (2001).

  37. 37.

    Cowen, A. S. & Keltner, D. Clarifying the conceptualization, dimensionality, and structure of emotion: response to Barrett and colleagues. Trends Cogn. Sci. 22, 274–276 (2018).

  38. 38.

    Laukka, P. et al. Cross-cultural decoding of positive and negative non-linguistic emotion vocalizations. Front. Psychol. 4, 353 (2013).

  39. 39.

    Parr, L. A., Cohen, M. & de Waal, F. Influence of social context on the use of blended and graded facial displays in chimpanzees. Int. J. Primatol. 26, 73–103 (2005).

  40. 40.

    Ekman, P. in The Nature of Emotion (eds Ekman, P. & Davidson, R. J.) 15–19 (Oxford Univ. Press, Oxford, 1992).

  41. 41.

    Harris, R. J., Young, A. W. & Andrews, T. J. Morphing between expressions dissociates continuous from categorical representations of facial expression in the human brain. Proc. Natl Acad. Sci. USA 190, 21164–21169 (2012).

  42. 42.

    Russell, J. A. Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychol. Bull. 115, 102–141 (1994).

  43. 43.

    Russell, J. A. Core affect and the psychological construction of emotion. Psychol. Rev. 110, 145–172 (2003).

  44. 44.

    Smith, C. A. & Ellsworth, P. C. Patterns of cognitive appraisal in emotion. J. Pers. Soc. Psychol. 48, 813–838 (1985).

  45. 45.

    Frijda, N. H., Kuipers, P. & ter Schure, E. Relations among emotion, appraisal, and emotional action readiness. J. Pers. Soc. Psychol. 57, 212–228 (1989).

  46. 46.

    Scherer, K. R. The dynamic architecture of emotion: evidence for the component process model. Cogn. Emot. 23, 1307–1351 (2009).

  47. 47.

    Watson, D. & Tellegen, A. Toward a consensual structure of mood. Psychol. Bull. 98, 219–235 (1985).

  48. 48.

    Posner, J., Russell, J. A. & Peterson, B. S. The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17, 715–734 (2005).

  49. 49.

    Russell, J. A circumplex model of affect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980).

  50. 50.

    Ang, J., Dhillon, R., Krupski, A., Shriberg, E. & Stolcke, A. Prosody-based automatic detection of annoyance and frustration in human-computer dialog. In Proc. 7th International Conference on Spoken Language Processing 2037–2040 (2002).

  51. 51.

    Laukka, P., Neiberg, D., Forsell, M., Karlsson, I. & Elenius, K. Expression of affect in spontaneous speech: acoustic correlates and automatic detection of irritation and resignation. Comput. Speech Lang. 25, 84–104 (2011).

  52. 52.

    Provine, R. R. & Fischer, K. R. Laughing, smiling, and talking: relation to sleeping and social context in humans. Ethology 83, 295–305 (1989).

  53. 53.

    Vidrascu, L. & Devillers, L. Real-life emotion representation and detection in call centers data. In Proc. 3784th Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 739–746 (Springer, 2005).

  54. 54.

    Sauter, D. A. & Fischer, A. H. Can perceivers recognise emotions from spontaneous expressions?. Cogn. Emot. 32, 504–515 (2018).

  55. 55.

    Anikin, A. & Lima, C. F. Perceptual and acoustic differences between authentic and acted nonverbal emotional vocalizations. Q. J. Exp. Psychol. 71, 622–641 (2018).

  56. 56.

    Scherer, K. R. Vocal markers of emotion: comparing induction and acting elicitation. Comput. Speech Lang. 27, 40–58 (2013).

  57. 57.

    Juslin, P. N., Laukka, P. & Bänziger, T. The mirror to our soul? Comparisons of spontaneous and posed vocal expression of emotion. J. Nonverbal Behav. 42, 1–40 (2018).

  58. 58.

    Gupta, V., Hanges, P. J. & Dorfman, P. Cultural clusters: methodology and findings. J. World Bus. 37, 11–15 (2002).

  59. 59.

    Jaju, A., Kwak, H. & Zinkhan, G. M. Learning styles of undergraduate business students: cross-cultural comparison between the US, India, and Korea. Mark. Educ. Rev. 12, 49–60 (2002).

  60. 60.

    Barrett, L. F. Are emotions natural kinds? Perspect. Psychol. Sci. 1, 28–58 (2006).

  61. 61.

    Ekman, P. What scientists who study emotion agree about. Perspect. Psychol. Sci. 11, 31–34 (2016).

  62. 62.

    Ekman, P. & Cordaro, D. What is meant by calling emotions basic. Emot. Rev. 3, 364–370 (2011).

  63. 63.

    Keltner, D. & Lerner, J. S. in Handbook of Social Psychology 5th edn (eds Fiske, S. T. et al., Wiley Online Library, Hoboken NJ, 2010).

  64. 64.

    Lazarus, R. S. Progress on a cognitive–motivational–relational theory of emotion. Am. Psychol. 46, 819–834 (1991).

  65. 65.

    Roseman, I. J. Appraisal determinants of discrete emotions. Cogn. Emot. 5, 161–200 (1991).

  66. 66.

    Etcoff, N. L. & Magee, J. J. Categorical perception of facial expressions. Cognition 44, 227–240 (1992).

  67. 67.

    Harmon-Jones, C., Bastian, B. & Harmon-Jones, E. The discrete emotions questionnaire: a new tool for measuring state self-reported emotions. PLoS ONE 11, e0159915 (2016).

  68. 68.

    Izard, C. E. Four systems for emotion activation: cognitive and noncognitive processes. Psychol. Rev. 100, 68–90 (1993).

  69. 69.

    Johnson-Laird, P. N. & Oatley, K. The language of emotions: an analysis of a semantic field. Cogn. Emot. 3, 81–123 (1989).

  70. 70.

    Shiota, M. N. et al. Beyond happiness: building a science of discrete positive emotions. Am. Psychol. 72, 617–643 (2017).

  71. 71.

    Samson, A. C., Kreibig, S. D., Soderstrom, B., Wade, A. A. & Gross, J. J. Eliciting positive, negative and mixed emotional states: a film library for affective scientists. Cogn. Emot. 30, 827–856 (2016).

  72. 72.

    Gendron, M., Roberson, D., van der Vyver, J. M. & Barrett, L. F. Perceptions of emotion from facial expressions are not culturally universal: evidence from a remote culture. Emotion 14, 251–262 (2014).

  73. 73.

    Laukka, P., Neiberg, D. & Elfenbein, H. A. Evidence for cultural dialects in vocal emotion expression: acoustic classification within and across five nations. Emotion 14, 445–449 (2014).

  74. 74.

    Mehrabian, A. & Russell, J. An Approach to Environmental Psychology (MIT Press, Cambridge MA, 1974).

  75. 75.

    Osgood, C. E. Dimensionality of the semantic space for communication via facial expressions. Scand. J. Psychol. 7, 1–30 (1966).

  76. 76.

    Sauter, D. A. & Scott, S. K. More than one kind of happiness: can we recognize vocal expressions of different positive states? Motiv. Emot. 31, 192–199 (2007).

  77. 77.

    Simon-Thomas, E. R., Keltner, D. J., Sauter, D., Sinicropi-Yao, L. & Abramson, A. The voice conveys specific emotions: evidence from vocal burst displays. Emotion 9, 838–846 (2009).

  78. 78.

    Benjamini, Y. & Yu, B. The shuffle estimator for explainable variance in FMRI experiments. Ann. Appl. Stat. 7, 2007–2033 (2013).

  79. 79.

    Barrett, L. F. Valence is a basic building block of emotional life. J. Res. Pers. 40, 35–55 (2006).

  80. 80.

    Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. B 57, 289–300 (1995).

  81. 81.

    Barrett, L. F., Lindquist, K. A. & Gendron, M. Language as context for the perception of emotion. Trends Cogn. Sci. 11, 327–332 (2007).

  82. 82.

    Abdi, H. & Williams, L. J. Partial least squares methods: partial least squares correlation and partial least square regression. Comput. Toxicol. 930, 549–579 (2013).

  83. 83.

    Hardoon, D. R., Szedmak, S. & Shawe-Taylor, J. Canonical correlation analysis: an overview with application to learning methods. Neural Comput. 16, 2639–2664 (2004).

  84. 84.

    Wilcoxon, F. Individual comparisons by ranking methods. Biom. Bull. 1, 80–83 (1945).

  85. 85.

    Van Der Maaten, L. & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).

  86. 86.

    Scherer, K. R. Vocal affect expression: a review and a model for future research. Psychol. Bull. 99, 143–165 (1986).

  87. 87.

    Ringeval, F. et al. AV+EC 2015: The first affect recognition challenge bridging across audio, video, and physiological data. In Proc. 5th International Workshop on Audio/Visual Emotion Challenge 3–8 (ACM, 2015).

  88. 88.

    Haidt, J. & Keltner, D. Culture and facial expression: open-ended methods find more expressions and a gradient of recognition. Cogn. Emot. 13, 225–266 (1999).

  89. 89.

    Kragel, P. A. & LaBar, K. S. Multivariate neural biomarkers of emotional states are categorically distinct. Soc. Cogn. Affect. Neurosci. 10, 1437–1448 (2015).

  90. 90.

    Kreibig, S. D. Autonomic nervous system activity in emotion: a review. Biol. Psychol. 84, 394–421 (2010).

  91. 91.

    Lench, H. C., Flores, S. A. & Bench, S. W. Discrete emotions predict changes in cognition, judgment, experience, behavior, and physiology: a meta-analysis of experimental emotion elicitations. Psychol. Bull. 137, 834–855 (2011).

  92. 92.

    Vytal, K. & Hamann, S. Neuroimaging support for discrete neural correlates of basic emotions: a voxel-based meta-analysis. J. Cogn. Neurosci. 22, 2864–2885 (2010).

  93. 93.

    Wager, T. D. et al. in Handbook of Emotions 3rd edn (eds Lewis, M. et al.) 249–271 (Guilford, New York, 2008).

  94. 94.

    Scherer, K. & Bänziger, T. in Blueprint for Affective Computing: A Sourcebook (eds Scherer, K. R., Banziger, T., & Roesch, E.) 166–176 (Oxford Univ. Press, Oxford, 2010).

  95. 95.

    G’Sell, M. G., Wager, S., Chouldechova, A. & Tibshirani, R. Sequential selection procedures and false discovery rate control. J. R. Stat. Soc. B. 78, 423–444 (2016).

Download references

Acknowledgements

We thank R. Rosipal for devising a correlational version of PPCA and F. Theunissen for providing input regarding acoustic analyses. Research reported in this publication was supported by the US National Institute of Mental Health under award number T32-MH020006-16A1 and by the Thomas and Ruth Ann Hornaday Chair in Psychology at the University of California, Berkeley. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

P.L. and H.A.E. contributed all speech samples; A.S.C. and D.K. designed the research with input from P.L. and H.A.E.; A.S.C. performed research, contributed analytic tools and analysed data; and A.S.C. and D.K. wrote the paper with input from P.L., H.A.E. and R.L.

Competing interests

The authors declare no competing interests.

Correspondence to Alan S. Cowen.

Supplementary information

Supplementary Text and Figures

Supplementary Notes 1–11, Supplementary Discussions 1–7, Supplementary Figures 1–8, Supplementary Tables 1 and 2, and Supplementary References

Reporting Summary

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark
Fig. 1: Correlations in the meaning of emotional prosody across cultures.
Fig. 2: The preserved recognition of emotion categories accounts for the preservation of affective feature judgments across cultures.
Fig. 3: Verifying that PPCA accurately estimates the number of shared dimensions.
Fig. 4: 12 distinct varieties of emotional prosody are preserved across cultures via category recognition.
Fig. 5: Correlations between coefficients of components extracted from US and Indian category judgments using different methods.
Fig. 6: Visualizing the 12-dimensional structure of emotion conveyed by prosody.
Fig. 7: The 12 distinct categories can be blended together in a number of ways.
Fig. 8: Low-level acoustic correlates of emotion recognition and their preservation across cultures.