Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

The primacy of categories in the recognition of 12 emotions in speech prosody across two cultures

Abstract

Central to emotion science is the degree to which categories, such as Awe, or broader affective features, such as Valence, underlie the recognition of emotional expression. To explore the processes by which people recognize emotion from prosody, US and Indian participants were asked to judge the emotion categories or affective features communicated by 2,519 speech samples produced by 100 actors from 5 cultures. With large-scale statistical inference methods, we find that prosody can communicate at least 12 distinct kinds of emotion that are preserved across the 2 cultures. Analyses of the semantic and acoustic structure of the recognition of emotions reveal that emotion categories drive the recognition of emotions more so than affective features, including Valence. In contrast to discrete emotion theories, however, emotion categories are bridged by gradients representing blends of emotions. Our findings, visualized within an interactive map, reveal a complex, high-dimensional space of emotional states recognized cross-culturally in speech prosody.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Correlations in the meaning of emotional prosody across cultures.
Fig. 2: The preserved recognition of emotion categories accounts for the preservation of affective feature judgments across cultures.
Fig. 3: Verifying that PPCA accurately estimates the number of shared dimensions.
Fig. 4: 12 distinct varieties of emotional prosody are preserved across cultures via category recognition.
Fig. 5: Correlations between coefficients of components extracted from US and Indian category judgments using different methods.
Fig. 6: Visualizing the 12-dimensional structure of emotion conveyed by prosody.
Fig. 7: The 12 distinct categories can be blended together in a number of ways.
Fig. 8: Low-level acoustic correlates of emotion recognition and their preservation across cultures.

Code availability

Custom MATLAB analysis code can be requested from https://goo.gl/forms/3q0y2Vvi1KinMft13.

Data availability

The 2,519 speech samples used in the present study and their ratings can be requested from https://goo.gl/forms/3q0y2Vvi1KinMft13. Publications incorporating the speech samples should reference the previous study33.

References

  1. 1.

    Keltner, D. & Haidt, J. Social functions of emotions at four levels of analysis. Cogn. Emot. 13, 505–521 (1999).

    Article  Google Scholar 

  2. 2.

    Nesse, R. M. Evolutionary explanations of emotions. Hum. Nat. 1, 261–289 (1990).

    CAS  Article  Google Scholar 

  3. 3.

    Campos, B., Shiota, M. N., Keltner, D., Gonzaga, G. C. & Goetz, J. L. What is shared, what is different? Core relational themes and expressive displays of eight positive emotions. Cogn. Emot. 27, 37–52 (2013).

    Article  Google Scholar 

  4. 4.

    Oveis, C., Spectre, A., Smith, P. K., Liu, M. Y. & Keltner, D. Laughter conveys status. J. Exp. Soc. Psychol. 65, 109–115 (2016).

    Article  Google Scholar 

  5. 5.

    Gonzaga, G. C., Keltner, D., Londahl, E. A. & Smith, M. D. Love and the commitment problem in romantic relations and friendship. J. Pers. Soc. Psychol. 81, 247–262 (2001).

    CAS  Article  Google Scholar 

  6. 6.

    ten Brinke, L. & Adams, G. S. Saving face? When emotion displays during public apologies mitigate damage to organizational performance. Organ. Behav. Hum. Decis. Process. 130, 1–12 (2015).

    Article  Google Scholar 

  7. 7.

    Cowen, A. S. & Keltner, D. Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proc. Natl Acad. Sci. USA 114, E7900–E7909 (2017).

  8. 8.

    Schirmer, A. & Adolphs, R. Emotion perception from face, voice, and touch: comparisons and convergence. Trends Cogn. Sci. 21, 216–228 (2017).

    Article  Google Scholar 

  9. 9.

    Singer, T. & Lamm, C. The social neuroscience of empathy. Ann. NY Acad. Sci. 1156, 81–96 (2009).

  10. 10.

    Frühholz, S., Ceravolo, L. & Grandjean, D. Specific brain networks during explicit and implicit decoding of emotional prosody. Cereb. Cortex 22, 1107–1117 (2012).

    Article  Google Scholar 

  11. 11.

    Bach, D. R. et al. The effect of appraisal level on processing of emotional prosody in meaningless speech. Neuroimage 42, 919–927 (2008).

    Article  Google Scholar 

  12. 12.

    Cordaro, D. T. et al. Universals and cultural variations in 22 emotional expressions across five cultures. Emotion 18, 75–93 (2018).

    Article  Google Scholar 

  13. 13.

    Elfenbein, H. A. & Ambady, N. On the universality and cultural specificity of emotion recognition: a meta-analysis. Psychol. Bull. 128, 203–235 (2002).

  14. 14.

    Keltner, D. & Cordaro, D. T. in Emotion Researcher (Scarantino, A. ed.) Available at http://emotionresearcher.com/understanding-multimodal-emotional-expressions-recent-advances-in-basic-emotion-theory/ (2015).

  15. 15.

    Norenzayan, A. & Heine, S. J. Psychological universals: what are they and how can we know? Psychol. Bull. 131, 763–784 (2005).

  16. 16.

    Sauter, D. A., Eisner, F., Ekman, P. & Scott, S. K. Cross-cultural recognition of basic emotions through nonverbal emotional vocalizations. Proc. Natl Acad. Sci. USA 107, 2408–2412 (2010).

  17. 17.

    Filippi, P. et al. Humans recognize emotional arousal in vocalizations across all classes of terrestrial vertebrates: evidence for acoustic universals. Proc. R. Soc. B 284, 20170990 (2017).

    Article  Google Scholar 

  18. 18.

    Parr, L. A., Waller, B. M. & Vick, S. J. New developments in understanding emotional facial signals in chimpanzees. Curr. Dir. Psychol. Sci. 16, 117–122 (2007).

  19. 19.

    Snowdon, C. T. in Handbook of Affective Sciences (eds Davidson, R. J. & Scherer, K. R.) 457-480 (Oxford Univ. Press, Oxford, 2002).

  20. 20.

    Filippi, P. Emotional and interactional prosody across animal communication systems: a comparative approach to the emergence of language. Front. Psychol. 7, 1393 (2016).

  21. 21.

    Adolphs, R. Neural systems for recognizing emotion. Curr. Opin. Neurobiol. 12, 169–177 (2002).

    CAS  Article  Google Scholar 

  22. 22.

    Russell, J. A. Is there universal recognition of emotion from facial expressions? A review of the cross-cultural studies. Psychol. Bull. 115, 102–141 (1994).

    CAS  Article  Google Scholar 

  23. 23.

    Cordaro, D. T., Keltner, D., Tshering, S., Wangchuk, D. & Flynn, L. M. The voice conveys emotion in ten globalized cultures and one remote village in Bhutan. Emotion 16, 117–128 (2016).

  24. 24.

    Gendron, M., Roberson, D., van der Vyver, J. M. & Barrett, L. F. Cultural relativity in perceiving emotion from vocalizations. Psychol. Sci. 25, 911–920 (2014).

    Article  Google Scholar 

  25. 25.

    Hertenstein, M. J. & Campos, J. J. The retention effects of an adult’s emotional displays on infant behavior. Child Dev. 75, 595–613 (2004).

    Article  Google Scholar 

  26. 26.

    Juslin, P. N. & Laukka, P. Communication of emotions in vocal expression and music performance: different channels, same code? Psychol. Bull. 129, 770–814 (2003).

    Article  Google Scholar 

  27. 27.

    Keltner, D. et al. in Handbook of Emotions 4th edn (eds Lewis M., Haviland-Jones, J. M. & Barrett, L. F.) 467–482 (Guilford, New York, 2016).

  28. 28.

    Wu, Y., Muentener, P. & Schulz, L. E. One- to four-year-olds connect diverse positive emotional vocalizations to their probable causes. Proc. Natl Acad. Sci. USA 114, 11896–11901 (2017).

  29. 29.

    Titze, I. R. & Martin, D. W. Principles of voice production. J. Acoust. Soc. Am. 104, 1148 (1998).

    Article  Google Scholar 

  30. 30.

    Scherer, K. R. & Bänziger, T. Emotional expression in prosody: a review and an agenda for future research. In Proc. Speech Prosody 2004 359–366 (2004).

  31. 31.

    Mitchell, R. L. C. & Ross, E. D. Attitudinal prosody: what we know and directions for future study. Neurosci. Biobehav. Rev. 37, 471–479 (2013).

    Article  Google Scholar 

  32. 32.

    Hancil, S. The Role of Prosody in Affective Speech (Peter Lang, New York, 2009).

  33. 33.

    Laukka, P. et al. The expression and recognition of emotions in the voice across five nations: a lens model analysis based on acoustic features. J. Pers. Soc. Psychol. 111, 686–705 (2016).

    Article  Google Scholar 

  34. 34.

    Nordström, H., Laukka, P., Thingujam, N. S., Schubert, E. & Elfenbein, H. A. Emotion appraisal dimensions inferred from vocal expressions are consistent across cultures: a comparison between Australia and India. R. Soc. Open Sci. 4, 170912 (2017).

  35. 35.

    Paulmann, S. & Uskul, A. K. Cross-cultural emotional prosody recognition: evidence from Chinese and British listeners. Cogn. Emot. 28, 230–244 (2014).

  36. 36.

    Scherer, K. R., Banse, R. & Wallbott, H. G. Emotion inferences from vocal expression correlate across languages and cultures. J. Cross Cult. Psychol. 32, 76–92 (2001).

  37. 37.

    Cowen, A. S. & Keltner, D. Clarifying the conceptualization, dimensionality, and structure of emotion: response to Barrett and colleagues. Trends Cogn. Sci. 22, 274–276 (2018).

    Article  Google Scholar 

  38. 38.

    Laukka, P. et al. Cross-cultural decoding of positive and negative non-linguistic emotion vocalizations. Front. Psychol. 4, 353 (2013).

  39. 39.

    Parr, L. A., Cohen, M. & de Waal, F. Influence of social context on the use of blended and graded facial displays in chimpanzees. Int. J. Primatol. 26, 73–103 (2005).

  40. 40.

    Ekman, P. in The Nature of Emotion (eds Ekman, P. & Davidson, R. J.) 15–19 (Oxford Univ. Press, Oxford, 1992).

  41. 41.

    Harris, R. J., Young, A. W. & Andrews, T. J. Morphing between expressions dissociates continuous from categorical representations of facial expression in the human brain. Proc. Natl Acad. Sci. USA 190, 21164–21169 (2012).

  42. 42.

    Russell, J. A. Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychol. Bull. 115, 102–141 (1994).

    CAS  Article  Google Scholar 

  43. 43.

    Russell, J. A. Core affect and the psychological construction of emotion. Psychol. Rev. 110, 145–172 (2003).

    Article  Google Scholar 

  44. 44.

    Smith, C. A. & Ellsworth, P. C. Patterns of cognitive appraisal in emotion. J. Pers. Soc. Psychol. 48, 813–838 (1985).

    Article  Google Scholar 

  45. 45.

    Frijda, N. H., Kuipers, P. & ter Schure, E. Relations among emotion, appraisal, and emotional action readiness. J. Pers. Soc. Psychol. 57, 212–228 (1989).

  46. 46.

    Scherer, K. R. The dynamic architecture of emotion: evidence for the component process model. Cogn. Emot. 23, 1307–1351 (2009).

    Article  Google Scholar 

  47. 47.

    Watson, D. & Tellegen, A. Toward a consensual structure of mood. Psychol. Bull. 98, 219–235 (1985).

  48. 48.

    Posner, J., Russell, J. A. & Peterson, B. S. The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17, 715–734 (2005).

    Article  Google Scholar 

  49. 49.

    Russell, J. A circumplex model of affect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980).

    Article  Google Scholar 

  50. 50.

    Ang, J., Dhillon, R., Krupski, A., Shriberg, E. & Stolcke, A. Prosody-based automatic detection of annoyance and frustration in human-computer dialog. In Proc. 7th International Conference on Spoken Language Processing 2037–2040 (2002).

  51. 51.

    Laukka, P., Neiberg, D., Forsell, M., Karlsson, I. & Elenius, K. Expression of affect in spontaneous speech: acoustic correlates and automatic detection of irritation and resignation. Comput. Speech Lang. 25, 84–104 (2011).

    Article  Google Scholar 

  52. 52.

    Provine, R. R. & Fischer, K. R. Laughing, smiling, and talking: relation to sleeping and social context in humans. Ethology 83, 295–305 (1989).

    Article  Google Scholar 

  53. 53.

    Vidrascu, L. & Devillers, L. Real-life emotion representation and detection in call centers data. In Proc. 3784th Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 739–746 (Springer, 2005).

  54. 54.

    Sauter, D. A. & Fischer, A. H. Can perceivers recognise emotions from spontaneous expressions?. Cogn. Emot. 32, 504–515 (2018).

    Article  Google Scholar 

  55. 55.

    Anikin, A. & Lima, C. F. Perceptual and acoustic differences between authentic and acted nonverbal emotional vocalizations. Q. J. Exp. Psychol. 71, 622–641 (2018).

  56. 56.

    Scherer, K. R. Vocal markers of emotion: comparing induction and acting elicitation. Comput. Speech Lang. 27, 40–58 (2013).

    Article  Google Scholar 

  57. 57.

    Juslin, P. N., Laukka, P. & Bänziger, T. The mirror to our soul? Comparisons of spontaneous and posed vocal expression of emotion. J. Nonverbal Behav. 42, 1–40 (2018).

  58. 58.

    Gupta, V., Hanges, P. J. & Dorfman, P. Cultural clusters: methodology and findings. J. World Bus. 37, 11–15 (2002).

  59. 59.

    Jaju, A., Kwak, H. & Zinkhan, G. M. Learning styles of undergraduate business students: cross-cultural comparison between the US, India, and Korea. Mark. Educ. Rev. 12, 49–60 (2002).

  60. 60.

    Barrett, L. F. Are emotions natural kinds? Perspect. Psychol. Sci. 1, 28–58 (2006).

    Article  Google Scholar 

  61. 61.

    Ekman, P. What scientists who study emotion agree about. Perspect. Psychol. Sci. 11, 31–34 (2016).

    Article  Google Scholar 

  62. 62.

    Ekman, P. & Cordaro, D. What is meant by calling emotions basic. Emot. Rev. 3, 364–370 (2011).

    Article  Google Scholar 

  63. 63.

    Keltner, D. & Lerner, J. S. in Handbook of Social Psychology 5th edn (eds Fiske, S. T. et al., Wiley Online Library, Hoboken NJ, 2010).

  64. 64.

    Lazarus, R. S. Progress on a cognitive–motivational–relational theory of emotion. Am. Psychol. 46, 819–834 (1991).

    CAS  Article  Google Scholar 

  65. 65.

    Roseman, I. J. Appraisal determinants of discrete emotions. Cogn. Emot. 5, 161–200 (1991).

    Article  Google Scholar 

  66. 66.

    Etcoff, N. L. & Magee, J. J. Categorical perception of facial expressions. Cognition 44, 227–240 (1992).

    CAS  Article  Google Scholar 

  67. 67.

    Harmon-Jones, C., Bastian, B. & Harmon-Jones, E. The discrete emotions questionnaire: a new tool for measuring state self-reported emotions. PLoS ONE 11, e0159915 (2016).

    Article  Google Scholar 

  68. 68.

    Izard, C. E. Four systems for emotion activation: cognitive and noncognitive processes. Psychol. Rev. 100, 68–90 (1993).

    CAS  Article  Google Scholar 

  69. 69.

    Johnson-Laird, P. N. & Oatley, K. The language of emotions: an analysis of a semantic field. Cogn. Emot. 3, 81–123 (1989).

  70. 70.

    Shiota, M. N. et al. Beyond happiness: building a science of discrete positive emotions. Am. Psychol. 72, 617–643 (2017).

    Article  Google Scholar 

  71. 71.

    Samson, A. C., Kreibig, S. D., Soderstrom, B., Wade, A. A. & Gross, J. J. Eliciting positive, negative and mixed emotional states: a film library for affective scientists. Cogn. Emot. 30, 827–856 (2016).

  72. 72.

    Gendron, M., Roberson, D., van der Vyver, J. M. & Barrett, L. F. Perceptions of emotion from facial expressions are not culturally universal: evidence from a remote culture. Emotion 14, 251–262 (2014).

    Article  Google Scholar 

  73. 73.

    Laukka, P., Neiberg, D. & Elfenbein, H. A. Evidence for cultural dialects in vocal emotion expression: acoustic classification within and across five nations. Emotion 14, 445–449 (2014).

  74. 74.

    Mehrabian, A. & Russell, J. An Approach to Environmental Psychology (MIT Press, Cambridge MA, 1974).

  75. 75.

    Osgood, C. E. Dimensionality of the semantic space for communication via facial expressions. Scand. J. Psychol. 7, 1–30 (1966).

    CAS  Article  Google Scholar 

  76. 76.

    Sauter, D. A. & Scott, S. K. More than one kind of happiness: can we recognize vocal expressions of different positive states? Motiv. Emot. 31, 192–199 (2007).

  77. 77.

    Simon-Thomas, E. R., Keltner, D. J., Sauter, D., Sinicropi-Yao, L. & Abramson, A. The voice conveys specific emotions: evidence from vocal burst displays. Emotion 9, 838–846 (2009).

  78. 78.

    Benjamini, Y. & Yu, B. The shuffle estimator for explainable variance in FMRI experiments. Ann. Appl. Stat. 7, 2007–2033 (2013).

    Article  Google Scholar 

  79. 79.

    Barrett, L. F. Valence is a basic building block of emotional life. J. Res. Pers. 40, 35–55 (2006).

    Article  Google Scholar 

  80. 80.

    Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. B 57, 289–300 (1995).

    Google Scholar 

  81. 81.

    Barrett, L. F., Lindquist, K. A. & Gendron, M. Language as context for the perception of emotion. Trends Cogn. Sci. 11, 327–332 (2007).

    Article  Google Scholar 

  82. 82.

    Abdi, H. & Williams, L. J. Partial least squares methods: partial least squares correlation and partial least square regression. Comput. Toxicol. 930, 549–579 (2013).

  83. 83.

    Hardoon, D. R., Szedmak, S. & Shawe-Taylor, J. Canonical correlation analysis: an overview with application to learning methods. Neural Comput. 16, 2639–2664 (2004).

    Article  Google Scholar 

  84. 84.

    Wilcoxon, F. Individual comparisons by ranking methods. Biom. Bull. 1, 80–83 (1945).

    Article  Google Scholar 

  85. 85.

    Van Der Maaten, L. & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).

    Google Scholar 

  86. 86.

    Scherer, K. R. Vocal affect expression: a review and a model for future research. Psychol. Bull. 99, 143–165 (1986).

    CAS  Article  Google Scholar 

  87. 87.

    Ringeval, F. et al. AV+EC 2015: The first affect recognition challenge bridging across audio, video, and physiological data. In Proc. 5th International Workshop on Audio/Visual Emotion Challenge 3–8 (ACM, 2015).

  88. 88.

    Haidt, J. & Keltner, D. Culture and facial expression: open-ended methods find more expressions and a gradient of recognition. Cogn. Emot. 13, 225–266 (1999).

  89. 89.

    Kragel, P. A. & LaBar, K. S. Multivariate neural biomarkers of emotional states are categorically distinct. Soc. Cogn. Affect. Neurosci. 10, 1437–1448 (2015).

    Article  Google Scholar 

  90. 90.

    Kreibig, S. D. Autonomic nervous system activity in emotion: a review. Biol. Psychol. 84, 394–421 (2010).

    Article  Google Scholar 

  91. 91.

    Lench, H. C., Flores, S. A. & Bench, S. W. Discrete emotions predict changes in cognition, judgment, experience, behavior, and physiology: a meta-analysis of experimental emotion elicitations. Psychol. Bull. 137, 834–855 (2011).

    Article  Google Scholar 

  92. 92.

    Vytal, K. & Hamann, S. Neuroimaging support for discrete neural correlates of basic emotions: a voxel-based meta-analysis. J. Cogn. Neurosci. 22, 2864–2885 (2010).

    Article  Google Scholar 

  93. 93.

    Wager, T. D. et al. in Handbook of Emotions 3rd edn (eds Lewis, M. et al.) 249–271 (Guilford, New York, 2008).

  94. 94.

    Scherer, K. & Bänziger, T. in Blueprint for Affective Computing: A Sourcebook (eds Scherer, K. R., Banziger, T., & Roesch, E.) 166–176 (Oxford Univ. Press, Oxford, 2010).

  95. 95.

    G’Sell, M. G., Wager, S., Chouldechova, A. & Tibshirani, R. Sequential selection procedures and false discovery rate control. J. R. Stat. Soc. B. 78, 423–444 (2016).

Download references

Acknowledgements

We thank R. Rosipal for devising a correlational version of PPCA and F. Theunissen for providing input regarding acoustic analyses. Research reported in this publication was supported by the US National Institute of Mental Health under award number T32-MH020006-16A1 and by the Thomas and Ruth Ann Hornaday Chair in Psychology at the University of California, Berkeley. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Affiliations

Authors

Contributions

P.L. and H.A.E. contributed all speech samples; A.S.C. and D.K. designed the research with input from P.L. and H.A.E.; A.S.C. performed research, contributed analytic tools and analysed data; and A.S.C. and D.K. wrote the paper with input from P.L., H.A.E. and R.L.

Corresponding author

Correspondence to Alan S. Cowen.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Text and Figures

Supplementary Notes 1–11, Supplementary Discussions 1–7, Supplementary Figures 1–8, Supplementary Tables 1 and 2, and Supplementary References

Reporting Summary

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Cowen, A.S., Laukka, P., Elfenbein, H.A. et al. The primacy of categories in the recognition of 12 emotions in speech prosody across two cultures. Nat Hum Behav 3, 369–382 (2019). https://doi.org/10.1038/s41562-019-0533-6

Download citation

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing