Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Letter
  • Published:

Conceptual knowledge predicts the representational structure of facial emotion perception

Nature Human Behaviour volume 2, pages 581–591 (2018)Cite this article

Subjects

Abstract

Recent theoretical accounts argue that conceptual knowledge dynamically interacts with processing of facial cues, fundamentally influencing visual perception of social and emotion categories. Evidence is accumulating for the idea that a perceiver’s conceptual knowledge about emotion is involved in emotion perception, even when stereotypic facial expressions are presented in isolation1,2,3,4. However, existing methods have not allowed a comprehensive assessment of the relationship between conceptual knowledge and emotion perception across individuals and emotion categories. Here we use a representational similarity analysis approach to show that conceptual knowledge predicts the representational structure of facial emotion perception. We conducted three studies using computer mouse-tracking5 and reverse-correlation6 paradigms. Overall, we found that when individuals believed two emotions to be conceptually more similar, faces from those categories were perceived with a corresponding similarity, even when controlling for any physical similarity in the stimuli themselves. When emotions were rated conceptually more similar, computer-mouse trajectories during emotion perception exhibited a greater simultaneous attraction to both category responses (despite only one emotion being depicted; studies 1 and 2), and reverse-correlated face prototypes exhibited a greater visual resemblance (study 3). Together, our findings suggest that differences in conceptual knowledge are reflected in the perceptual processing of facial emotion.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Mouse-tracking effects.
Fig. 2: Average conceptual and perceptual DMs for studies 1–3 and schematic of the analytic approach.
Fig. 3: Multilevel regression results for studies 1–3.
Fig. 4: Reverse-correlation effects.

Similar content being viewed by others

References

  1. Gendron, M., Lindquist, K. A., Barsalou, L. & Barrett, L. F. Emotion words shape emotion percepts. Emotion 12, 314–325 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  2. Lindquist, K. A., Barrett, L. F., Bliss-Moreau, E. & Russell, J. A. Language and the perception of emotion. Emotion 6, 125–138 (2006).

    Article  PubMed  Google Scholar 

  3. Lindquist, K. A., Gendron, M., Barrett, L. F. & Dickerson, B. C. Emotion perception, but not affect perception, is impaired with semantic memory loss. Emotion 14, 375–387 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  4. Nook, E. C., Lindquist, K. A. & Zaki, J. A new look at emotion perception: concepts speed and shape facial emotion recognition. Emotion 15, 569–578 (2015).

    Article  PubMed  Google Scholar 

  5. Freeman, J. B. & Ambady, N. MouseTracker: software for studying real-time mental processing using a computer mouse-tracking method. Behav. Res. Methods 42, 226–241 (2010).

    Article  PubMed  Google Scholar 

  6. Dotsch, R., Wigboldus, D. H. J., Langner, O. & van Knippenberg, A. Ethnic out-group faces are biased in the prejudiced mind. Psychol. Sci. 19, 978–980 (2008).

    Article  PubMed  Google Scholar 

  7. Ekman, P. Universals and cultural differences in facial expressions of emotion. In Nebraska Symposium on Emotion and Motivation, 1971 (ed. Cole, J.) 207–283 (Univ. Nebraska Press, Lincoln, NE, 1972).

  8. Ekman, P. & Cordaro, D. What is meant by calling emotions basic. Emot. Rev. 3, 364–370 (2011).

    Article  Google Scholar 

  9. Adolphs, R. Neural systems for recognizing emotion. Curr. Opin. Neurobiol. 12, 169–177 (2002).

    Article  PubMed  CAS  Google Scholar 

  10. Adolphs, R. Cognitive neuroscience of human social behaviour. Nat. Rev. Neurosci. 4, 165–178 (2003).

    Article  PubMed  CAS  Google Scholar 

  11. Darwin, C. The Expression of the Emotions in Man and Animals (Oxford Univ. Press, New York, NY, 1872).

    Book  Google Scholar 

  12. Ekman, P. Facial expressions of emotion: new findings, new questions. Psychol. Sci. 3, 34–38 (1992).

    Article  Google Scholar 

  13. Ekman, P. Facial expression and emotion. Am. Psychol. 48, 384–392 (1993).

    Article  PubMed  CAS  Google Scholar 

  14. Smith, M. L., Cottrell, G. W., Gosselin, F. & Schyns, P. G. Transmitting and decoding facial expressions. Psychol. Sci. 16, 184–189 (2005).

    Article  PubMed  Google Scholar 

  15. Tracy, J. L. & Robins, R. W. The automaticity of emotion recognition. Emotion 8, 81–95 (2008).

    Article  PubMed  Google Scholar 

  16. Ambady, N. & Weisbuch, M. in Oxford Handbook of Face Perception (eds Calder, A. J., Rhodes, G., Haxby, J. V. & Johnson, M. H.) 479–488 (Oxford Univ. Press, Oxford, 2011).

  17. Hassin, R. R., Aviezer, H. & Bentin, S. Inherently ambiguous: facial expressions of emotions, in context. Emot. Rev. 5, 60–65 (2013).

    Article  Google Scholar 

  18. Aviezer, H., Hassin, R., Bentin, S. & Trope, Y. in First Impressions (eds Ambady, N. & Skowronski, J.) 255–286 (Guilford Press, New York, NY, 2008).

  19. Aviezer, H. et al. Not on the face alone: perception of contextualized face expressions in Huntington’s disease. Brain 132, 1633–1644 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  20. Aviezer, H., Dudarev, V., Bentin, S. & Hassin, R. R. The automaticity of emotional face-context integration. Emotion 11, 1406–1414 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  21. Meeren, H. K. M., van Heijnsbergen, C. C. R. J. & de Gelder, B. Rapid perceptual integration of facial expression and emotional body language. Proc. Natl Acad. Sci. USA 102, 16518–16523 (2005).

    Article  PubMed  CAS  Google Scholar 

  22. Van den Stock, J., Righart, R. & de Gelder, B. Body expressions influence recognition of emotions in the face and voice. Emotion 7, 487–494 (2007).

    Article  PubMed  Google Scholar 

  23. Barrett, L. F. The theory of constructed emotion: an active inference account of interoception and categorization. Soc. Cogn. Affect. Neurosci 12, 1–23 (2017).

    Article  PubMed  Google Scholar 

  24. Freeman, J. B. & Ambady, N. A dynamic interactive theory of person construal. Psychol. Rev. 118, 247–279 (2011).

    Article  PubMed  Google Scholar 

  25. Lindquist, K. A. Emotions emerge from more basic psychological ingredients: a modern psychological constructionist approach. Emot. Rev. 5, 356–368 (2013).

    Article  Google Scholar 

  26. Carroll, N. C. & Young, A. W. Priming of emotion recognition. Q. J. Exp. Psychol. A 58, 1173–1197 (2005).

    Article  PubMed  Google Scholar 

  27. Doyle, C. M. & Lindquist, K. A. When a word is worth a thousand pictures: language shapes perceptual memory for emotion. J. Exp. Psychol. Gen 147, 62–73 (2018).

    Article  PubMed  Google Scholar 

  28. Fugate, J. M. B., Gendron, M., Nakashima, S. & Barrett, L.F. Emotion words: adding face value. Emotion (in the press).

  29. Kriegeskorte, N., Mur, M. & Bandettini, P. Representational similarity analysis—connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008).

    Article  PubMed  PubMed Central  Google Scholar 

  30. Stolier, R. M. & Freeman, J. B. Neural pattern similarity reveals the inherent intersection of social categories. Nat. Neurosci. 19, 795–797 (2016).

    Article  PubMed  CAS  Google Scholar 

  31. Skerry, A. E. & Saxe, R. Neural representations of emotion are organized around abstract event features. Curr. Biol. 25, 1945–1954 (2015).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  32. Freeman, J. B. Doing psychological science by hand. Curr. Dir. Psychol. Sci. (in the press).

  33. Stolier, R. M. & Freeman, J. B. A neural mechanism of social categorization. J. Neurosci. 37, 5711–5721 (2017).

    Article  PubMed  CAS  Google Scholar 

  34. Liang, K.-Y. & Zeger, S. L. Longitudinal data analysis using generalized linear models. Biometrika 73, 13–22 (1986).

    Article  Google Scholar 

  35. Nili, H. et al. A toolbox for representational similarity analysis. PLoS Comput. Biol. 10, e1003553 (2014).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  36. Khaligh-Razavi, S.-M., Henriksson, L., Kay, K. & Kriegeskorte, N. Fixed versus mixed RSA: explaining visual representations by fixed and mixed feature sets from shallow and deep computational models. J. Math. Psychol. 76B, 184–197 (2017).

    Article  Google Scholar 

  37. Barsalou, L. W. Grounded cognition. Annu. Rev. Psychol. 59, 617–645 (2008).

    Article  PubMed  Google Scholar 

  38. Chanes, L., Wormwood, J. B., Betz, N. & Barrett, L. F. Facial expression predictions as drivers of social perception. J. Pers. Soc. Psychol. 114, 380–396 (2018).

    Article  PubMed  Google Scholar 

  39. Lieberman, M. D. in Social Neuroscience: Toward Understanding the Underpinnings of the Social Mind (eds Todorov, A., Fiske, S. T. & Prentice, D.) 188–209 (Oxford Univ. Press, Oxford, 2011).

  40. Riley, R. D., Lambert, P. C. & Abo-Zaid, G. Meta-analysis of individual subject data: rationale, conduct, and reporting. Br. Med. J. 340, c221 (2010).

    Article  Google Scholar 

  41. Freeman, J. B. & Johnson, K. L. More than meets the eye: split-second social perception. Trends Cogn. Sci. 20, 362–374 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  42. O’Callaghan, C., Kveraga, K., Shine, J. M., Adams, R. B. & Bar, M. Predictions penetrate perception: converging insights from brain, behavior and disorder. Conscious. Cogn. 47, 63–74 (2017).

    Article  PubMed  Google Scholar 

  43. Summerfield, C. & Egner, T. Expectation (and attention) in visual cognition. Trends Cogn. Sci. 13, 403–409 (2009).

    Article  PubMed  Google Scholar 

  44. Skerry, A. E. & Saxe, R. A common neural code for perceived and inferred emotion. J. Neurosci. 34, 15997–16008 (2014).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  45. Kuhn, L. K., Wydell, T., Lavan, N., McGettigan, C. & Garrido, L. Similar representations of emotions across faces and voices. Emotion 17, 912–937 (2017).

    Article  PubMed  Google Scholar 

  46. Elfenbein, H. A. Nonverbal dialects and accents in facial expressions of emotion. Emot. Rev. 5, 90–96 (2013).

    Article  Google Scholar 

  47. Elfenbein, H. A., Beaupré, M., Lévesque, M. & Hess, U. Toward a dialect theory: cultural differences in the expression and recognition of posed facial expressions. Emotion 7, 131–146 (2007).

    Article  PubMed  Google Scholar 

  48. Jack, R. E., Caldara, R. & Schyns, P. G. Internal representations reveal cultural diversity in expectations of facial expressions of emotion. J. Exp. Psychol. Gen. 141, 19–25 (2012).

    Article  PubMed  Google Scholar 

  49. Jack, R. E., Garrod, O. G. B., Yu, H., Caldara, R. & Schyns, P. G. Facial expressions of emotion are not culturally universal. Proc. Natl Acad. Sci. USA 109, 7241–7244 (2012).

    Article  PubMed  Google Scholar 

  50. Nook, E. C., Sasse, S. F., Lambert, H. K., McLaughlin, K. A. & Somerville, L. H. Increasing verbal knowledge mediates development of multidimensional emotion representations. Nat. Hum. Behav. 1, 881–889 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  51. Barrett, L. F., Gross, J., Christensen, T. C. & Benvenuto, M. Knowing what you’re feeling and knowing what to do about it: mapping the relation between emotion differentiation and emotion regulation. Cogn. Emot. 15, 713–724 (2001).

    Article  Google Scholar 

  52. Kang, S. M. & Shaver, P. R. Individual differences in emotional complexity: their psychological implications. J. Pers. 72, 687–726 (2004).

    Article  PubMed  Google Scholar 

  53. Lindquist, K. A. & Barrett, L. F. in Handbook of Emotions 3rd edn (eds Lewis, M., Haviland-Jones, J. M. & Barrett, L. F.) 513–530 (Guilford, New York, NY, 2008).

  54. Russell, J. A., Bachorowski, J. A. & Fernandez-Dols, J. M. Facial and vocal expressions of emotion. Annu. Rev. Psychol. 54, 329–349 (2003).

    Article  PubMed  Google Scholar 

  55. Barrett, L. F., Mesquita, B. & Gendron, M. Context in emotion perception. Curr. Dir. Psychol. Sci. 20, 286–290 (2011).

    Article  Google Scholar 

  56. Tottenham, N. et al. The NimStim set of facial expressions: judgments from untrained research subjects. Psychiatry Res. 168, 242–249 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  57. Hugenberg, K. & Bodenhausen, G. V. Ambiguity in social categorization: the role of prejudice and facial affect in race categorization. Psychol. Sci. 15, 342–345 (2004).

    Article  PubMed  Google Scholar 

  58. Bagby, R. M., Parker, J. D. A. & Taylor, G. J. The twenty-item Toronto Alexithymia scale—I. Item selection and cross-validation of the factor structure. J. Psychosom. Res. 38, 23–32 (1994).

    Article  PubMed  CAS  Google Scholar 

  59. Watson, D., Clark, L. A. & Tellegen, A. Development and validation of brief measures of positive and negative affect: the PANAS scales. J. Pers. Soc. Psychol. 54, 1063–70 (1988).

    Article  PubMed  CAS  Google Scholar 

  60. Freeman, J. B., Dale, R. & Farmer, T. A. Hand in motion reveals mind in motion. Front. Psychol. 2, 59 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  61. Ekman, P. & Friesen, W. V. Facial Action Coding System: A Technique For The Measurement Of Facial Movement (Consulting Psychologists Press, Palo Alto, CA, 1978).

    Google Scholar 

  62. Ekman, P., Irwin, W. & Rosenberg, E. EMFACS: Coders Instructions (EMFACS-8) (Univ. California San Francisco Press, San Francisco, CA, 1994).

    Google Scholar 

  63. Ekman, P., Friesen, W. & Hager, J. Facial Action Coding System: Investigator’s Guide 2nd edn (Research Nexus eBook, Salt Lake City, UT, 2002).

  64. Carlin, J. D. & Kriegeskorte, N. Adjudicating between face-coding models with individual-face fMRI responses. PLoS Comput. Biol. 13, e1005604 (2017).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  65. Barsalou, L.W. & Hale, C.R. in Categories and Concepts: Theoretical Views and Inductive Data analysis (eds Van Mechelen, I., Hampton, J., Michalski, R. & Theuns, P.) 97–144 (Academic Press, San Diego, CA, 1993).

  66. Brooks, J. A., Stolier, R. M. & Freeman, J. B. Stereotypes bias visual prototypes for sex and emotion categories. Soc. Cogn. (in the press).

  67. Lundqvist, D. & Litton, J. E. The Averaged Karolinska Directed Emotional Faces—AKDEF (CD ROM) (Karolinska Institute, Stockholm, 1998).

    Google Scholar 

  68. Dotsch, R. & Todorov, A. Reverse correlating social face perception. Soc. Psychol. Personal. Sci. 3, 562–571 (2012).

    Article  Google Scholar 

Download references

Acknowledgements

We thank L.I. Reed for assistance with FACS coding. This work was supported in part by research grant NIH-R01-MH112640 (J.B.F.). The funders of this research had no role in the conceptualization, design, data collection, analysis, decision to publish or preparation of the manuscript.

Author information

Authors and Affiliations

  1. Department of Psychology, New York University, New York, NY, USA

    Jeffrey A. Brooks & Jonathan B. Freeman

  2. Center for Neural Science, New York University, New York, NY, USA

    Jonathan B. Freeman

Authors
  1. Jeffrey A. Brooks
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jonathan B. Freeman
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Both authors collaborated on the study concept, design, and interpretation of the data. J.A.B. collected and analysed the data. J.A.B. drafted the manuscript and J.B.F. provided critical revisions. Both authors approved the final version of the manuscript for submission.

Corresponding authors

Correspondence to Jeffrey A. Brooks or Jonathan B. Freeman.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figures 1–4, Supplementary Tables 1–5

Reporting Summary

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Brooks, J.A., Freeman, J.B. Conceptual knowledge predicts the representational structure of facial emotion perception. Nat Hum Behav 2, 581–591 (2018). https://doi.org/10.1038/s41562-018-0376-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-018-0376-6

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing