Letter | Published:

Conceptual knowledge predicts the representational structure of facial emotion perception

Nature Human Behaviourvolume 2pages581591 (2018) | Download Citation

Abstract

Recent theoretical accounts argue that conceptual knowledge dynamically interacts with processing of facial cues, fundamentally influencing visual perception of social and emotion categories. Evidence is accumulating for the idea that a perceiver’s conceptual knowledge about emotion is involved in emotion perception, even when stereotypic facial expressions are presented in isolation1,2,3,4. However, existing methods have not allowed a comprehensive assessment of the relationship between conceptual knowledge and emotion perception across individuals and emotion categories. Here we use a representational similarity analysis approach to show that conceptual knowledge predicts the representational structure of facial emotion perception. We conducted three studies using computer mouse-tracking5 and reverse-correlation6 paradigms. Overall, we found that when individuals believed two emotions to be conceptually more similar, faces from those categories were perceived with a corresponding similarity, even when controlling for any physical similarity in the stimuli themselves. When emotions were rated conceptually more similar, computer-mouse trajectories during emotion perception exhibited a greater simultaneous attraction to both category responses (despite only one emotion being depicted; studies 1 and 2), and reverse-correlated face prototypes exhibited a greater visual resemblance (study 3). Together, our findings suggest that differences in conceptual knowledge are reflected in the perceptual processing of facial emotion.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    Gendron, M., Lindquist, K. A., Barsalou, L. & Barrett, L. F. Emotion words shape emotion percepts. Emotion 12, 314–325 (2012).

  2. 2.

    Lindquist, K. A., Barrett, L. F., Bliss-Moreau, E. & Russell, J. A. Language and the perception of emotion. Emotion 6, 125–138 (2006).

  3. 3.

    Lindquist, K. A., Gendron, M., Barrett, L. F. & Dickerson, B. C. Emotion perception, but not affect perception, is impaired with semantic memory loss. Emotion 14, 375–387 (2014).

  4. 4.

    Nook, E. C., Lindquist, K. A. & Zaki, J. A new look at emotion perception: concepts speed and shape facial emotion recognition. Emotion 15, 569–578 (2015).

  5. 5.

    Freeman, J. B. & Ambady, N. MouseTracker: software for studying real-time mental processing using a computer mouse-tracking method. Behav. Res. Methods 42, 226–241 (2010).

  6. 6.

    Dotsch, R., Wigboldus, D. H. J., Langner, O. & van Knippenberg, A. Ethnic out-group faces are biased in the prejudiced mind. Psychol. Sci. 19, 978–980 (2008).

  7. 7.

    Ekman, P. Universals and cultural differences in facial expressions of emotion. In Nebraska Symposium on Emotion and Motivation, 1971 (ed. Cole, J.) 207–283 (Univ. Nebraska Press, Lincoln, NE, 1972).

  8. 8.

    Ekman, P. & Cordaro, D. What is meant by calling emotions basic. Emot. Rev. 3, 364–370 (2011).

  9. 9.

    Adolphs, R. Neural systems for recognizing emotion. Curr. Opin. Neurobiol. 12, 169–177 (2002).

  10. 10.

    Adolphs, R. Cognitive neuroscience of human social behaviour. Nat. Rev. Neurosci. 4, 165–178 (2003).

  11. 11.

    Darwin, C. The Expression of the Emotions in Man and Animals (Oxford Univ. Press, New York, NY, 1872).

  12. 12.

    Ekman, P. Facial expressions of emotion: new findings, new questions. Psychol. Sci. 3, 34–38 (1992).

  13. 13.

    Ekman, P. Facial expression and emotion. Am. Psychol. 48, 384–392 (1993).

  14. 14.

    Smith, M. L., Cottrell, G. W., Gosselin, F. & Schyns, P. G. Transmitting and decoding facial expressions. Psychol. Sci. 16, 184–189 (2005).

  15. 15.

    Tracy, J. L. & Robins, R. W. The automaticity of emotion recognition. Emotion 8, 81–95 (2008).

  16. 16.

    Ambady, N. & Weisbuch, M. in Oxford Handbook of Face Perception (eds Calder, A. J., Rhodes, G., Haxby, J. V. & Johnson, M. H.) 479–488 (Oxford Univ. Press, Oxford, 2011).

  17. 17.

    Hassin, R. R., Aviezer, H. & Bentin, S. Inherently ambiguous: facial expressions of emotions, in context. Emot. Rev. 5, 60–65 (2013).

  18. 18.

    Aviezer, H., Hassin, R., Bentin, S. & Trope, Y. in First Impressions (eds Ambady, N. & Skowronski, J.) 255–286 (Guilford Press, New York, NY, 2008).

  19. 19.

    Aviezer, H. et al. Not on the face alone: perception of contextualized face expressions in Huntington’s disease. Brain 132, 1633–1644 (2009).

  20. 20.

    Aviezer, H., Dudarev, V., Bentin, S. & Hassin, R. R. The automaticity of emotional face-context integration. Emotion 11, 1406–1414 (2011).

  21. 21.

    Meeren, H. K. M., van Heijnsbergen, C. C. R. J. & de Gelder, B. Rapid perceptual integration of facial expression and emotional body language. Proc. Natl Acad. Sci. USA 102, 16518–16523 (2005).

  22. 22.

    Van den Stock, J., Righart, R. & de Gelder, B. Body expressions influence recognition of emotions in the face and voice. Emotion 7, 487–494 (2007).

  23. 23.

    Barrett, L. F. The theory of constructed emotion: an active inference account of interoception and categorization. Soc. Cogn. Affect. Neurosci 12, 1–23 (2017).

  24. 24.

    Freeman, J. B. & Ambady, N. A dynamic interactive theory of person construal. Psychol. Rev. 118, 247–279 (2011).

  25. 25.

    Lindquist, K. A. Emotions emerge from more basic psychological ingredients: a modern psychological constructionist approach. Emot. Rev. 5, 356–368 (2013).

  26. 26.

    Carroll, N. C. & Young, A. W. Priming of emotion recognition. Q. J. Exp. Psychol. A 58, 1173–1197 (2005).

  27. 27.

    Doyle, C. M. & Lindquist, K. A. When a word is worth a thousand pictures: language shapes perceptual memory for emotion. J. Exp. Psychol. Gen 147, 62–73 (2018).

  28. 28.

    Fugate, J. M. B., Gendron, M., Nakashima, S. & Barrett, L.F. Emotion words: adding face value. Emotion (in the press).

  29. 29.

    Kriegeskorte, N., Mur, M. & Bandettini, P. Representational similarity analysis—connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008).

  30. 30.

    Stolier, R. M. & Freeman, J. B. Neural pattern similarity reveals the inherent intersection of social categories. Nat. Neurosci. 19, 795–797 (2016).

  31. 31.

    Skerry, A. E. & Saxe, R. Neural representations of emotion are organized around abstract event features. Curr. Biol. 25, 1945–1954 (2015).

  32. 32.

    Freeman, J. B. Doing psychological science by hand. Curr. Dir. Psychol. Sci. (in the press).

  33. 33.

    Stolier, R. M. & Freeman, J. B. A neural mechanism of social categorization. J. Neurosci. 37, 5711–5721 (2017).

  34. 34.

    Liang, K.-Y. & Zeger, S. L. Longitudinal data analysis using generalized linear models. Biometrika 73, 13–22 (1986).

  35. 35.

    Nili, H. et al. A toolbox for representational similarity analysis. PLoS Comput. Biol. 10, e1003553 (2014).

  36. 36.

    Khaligh-Razavi, S.-M., Henriksson, L., Kay, K. & Kriegeskorte, N. Fixed versus mixed RSA: explaining visual representations by fixed and mixed feature sets from shallow and deep computational models. J. Math. Psychol. 76B, 184–197 (2017).

  37. 37.

    Barsalou, L. W. Grounded cognition. Annu. Rev. Psychol. 59, 617–645 (2008).

  38. 38.

    Chanes, L., Wormwood, J. B., Betz, N. & Barrett, L. F. Facial expression predictions as drivers of social perception. J. Pers. Soc. Psychol. 114, 380–396 (2018).

  39. 39.

    Lieberman, M. D. in Social Neuroscience: Toward Understanding the Underpinnings of the Social Mind (eds Todorov, A., Fiske, S. T. & Prentice, D.) 188–209 (Oxford Univ. Press, Oxford, 2011).

  40. 40.

    Riley, R. D., Lambert, P. C. & Abo-Zaid, G. Meta-analysis of individual subject data: rationale, conduct, and reporting. Br. Med. J. 340, c221 (2010).

  41. 41.

    Freeman, J. B. & Johnson, K. L. More than meets the eye: split-second social perception. Trends Cogn. Sci. 20, 362–374 (2016).

  42. 42.

    O’Callaghan, C., Kveraga, K., Shine, J. M., Adams, R. B. & Bar, M. Predictions penetrate perception: converging insights from brain, behavior and disorder. Conscious. Cogn. 47, 63–74 (2017).

  43. 43.

    Summerfield, C. & Egner, T. Expectation (and attention) in visual cognition. Trends Cogn. Sci. 13, 403–409 (2009).

  44. 44.

    Skerry, A. E. & Saxe, R. A common neural code for perceived and inferred emotion. J. Neurosci. 34, 15997–16008 (2014).

  45. 45.

    Kuhn, L. K., Wydell, T., Lavan, N., McGettigan, C. & Garrido, L. Similar representations of emotions across faces and voices. Emotion 17, 912–937 (2017).

  46. 46.

    Elfenbein, H. A. Nonverbal dialects and accents in facial expressions of emotion. Emot. Rev. 5, 90–96 (2013).

  47. 47.

    Elfenbein, H. A., Beaupré, M., Lévesque, M. & Hess, U. Toward a dialect theory: cultural differences in the expression and recognition of posed facial expressions. Emotion 7, 131–146 (2007).

  48. 48.

    Jack, R. E., Caldara, R. & Schyns, P. G. Internal representations reveal cultural diversity in expectations of facial expressions of emotion. J. Exp. Psychol. Gen. 141, 19–25 (2012).

  49. 49.

    Jack, R. E., Garrod, O. G. B., Yu, H., Caldara, R. & Schyns, P. G. Facial expressions of emotion are not culturally universal. Proc. Natl Acad. Sci. USA 109, 7241–7244 (2012).

  50. 50.

    Nook, E. C., Sasse, S. F., Lambert, H. K., McLaughlin, K. A. & Somerville, L. H. Increasing verbal knowledge mediates development of multidimensional emotion representations. Nat. Hum. Behav. 1, 881–889 (2017).

  51. 51.

    Barrett, L. F., Gross, J., Christensen, T. C. & Benvenuto, M. Knowing what you’re feeling and knowing what to do about it: mapping the relation between emotion differentiation and emotion regulation. Cogn. Emot. 15, 713–724 (2001).

  52. 52.

    Kang, S. M. & Shaver, P. R. Individual differences in emotional complexity: their psychological implications. J. Pers. 72, 687–726 (2004).

  53. 53.

    Lindquist, K. A. & Barrett, L. F. in Handbook of Emotions 3rd edn (eds Lewis, M., Haviland-Jones, J. M. & Barrett, L. F.) 513–530 (Guilford, New York, NY, 2008).

  54. 54.

    Russell, J. A., Bachorowski, J. A. & Fernandez-Dols, J. M. Facial and vocal expressions of emotion. Annu. Rev. Psychol. 54, 329–349 (2003).

  55. 55.

    Barrett, L. F., Mesquita, B. & Gendron, M. Context in emotion perception. Curr. Dir. Psychol. Sci. 20, 286–290 (2011).

  56. 56.

    Tottenham, N. et al. The NimStim set of facial expressions: judgments from untrained research subjects. Psychiatry Res. 168, 242–249 (2009).

  57. 57.

    Hugenberg, K. & Bodenhausen, G. V. Ambiguity in social categorization: the role of prejudice and facial affect in race categorization. Psychol. Sci. 15, 342–345 (2004).

  58. 58.

    Bagby, R. M., Parker, J. D. A. & Taylor, G. J. The twenty-item Toronto Alexithymia scale—I. Item selection and cross-validation of the factor structure. J. Psychosom. Res. 38, 23–32 (1994).

  59. 59.

    Watson, D., Clark, L. A. & Tellegen, A. Development and validation of brief measures of positive and negative affect: the PANAS scales. J. Pers. Soc. Psychol. 54, 1063–70 (1988).

  60. 60.

    Freeman, J. B., Dale, R. & Farmer, T. A. Hand in motion reveals mind in motion. Front. Psychol. 2, 59 (2011).

  61. 61.

    Ekman, P. & Friesen, W. V. Facial Action Coding System: A Technique For The Measurement Of Facial Movement (Consulting Psychologists Press, Palo Alto, CA, 1978).

  62. 62.

    Ekman, P., Irwin, W. & Rosenberg, E. EMFACS: Coders Instructions (EMFACS-8) (Univ. California San Francisco Press, San Francisco, CA, 1994).

  63. 63.

    Ekman, P., Friesen, W. & Hager, J. Facial Action Coding System: Investigator’s Guide 2nd edn (Research Nexus eBook, Salt Lake City, UT, 2002).

  64. 64.

    Carlin, J. D. & Kriegeskorte, N. Adjudicating between face-coding models with individual-face fMRI responses. PLoS Comput. Biol. 13, e1005604 (2017).

  65. 65.

    Barsalou, L.W. & Hale, C.R. in Categories and Concepts: Theoretical Views and Inductive Data analysis (eds Van Mechelen, I., Hampton, J., Michalski, R. & Theuns, P.) 97–144 (Academic Press, San Diego, CA, 1993).

  66. 66.

    Brooks, J. A., Stolier, R. M. & Freeman, J. B. Stereotypes bias visual prototypes for sex and emotion categories. Soc. Cogn. (in the press).

  67. 67.

    Lundqvist, D. & Litton, J. E. The Averaged Karolinska Directed Emotional Faces—AKDEF (CD ROM) (Karolinska Institute, Stockholm, 1998).

  68. 68.

    Dotsch, R. & Todorov, A. Reverse correlating social face perception. Soc. Psychol. Personal. Sci. 3, 562–571 (2012).

Download references

Acknowledgements

We thank L.I. Reed for assistance with FACS coding. This work was supported in part by research grant NIH-R01-MH112640 (J.B.F.). The funders of this research had no role in the conceptualization, design, data collection, analysis, decision to publish or preparation of the manuscript.

Author information

Affiliations

  1. Department of Psychology, New York University, New York, NY, USA

    • Jeffrey A. Brooks
    •  & Jonathan B. Freeman
  2. Center for Neural Science, New York University, New York, NY, USA

    • Jonathan B. Freeman

Authors

  1. Search for Jeffrey A. Brooks in:

  2. Search for Jonathan B. Freeman in:

Contributions

Both authors collaborated on the study concept, design, and interpretation of the data. J.A.B. collected and analysed the data. J.A.B. drafted the manuscript and J.B.F. provided critical revisions. Both authors approved the final version of the manuscript for submission.

Competing interests

The authors declare no competing interests.

Corresponding authors

Correspondence to Jeffrey A. Brooks or Jonathan B. Freeman.

Supplementary information

  1. Supplementary Information

    Supplementary Figures 1–4, Supplementary Tables 1–5

  2. Reporting Summary

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/s41562-018-0376-6