Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Letter
  • Published:

Modelling face memory reveals task-generalizable representations

Nature Human Behaviour volume 3, pages 817–826 (2019)Cite this article

Subjects

Abstract

Current cognitive theories are cast in terms of information-processing mechanisms that use mental representations1,2,3,4. For example, people use their mental representations to identify familiar faces under various conditions of pose, illumination and ageing, or to draw resemblance between family members. Yet, the actual information contents of these representations are rarely characterized, which hinders knowledge of the mechanisms that use them. Here, we modelled the three-dimensional representational contents of 4 faces that were familiar to 14 participants as work colleagues. The representational contents were created by reverse-correlating identity information generated on each trial with judgements of the face’s similarity to the individual participant’s memory of this face. In a second study, testing new participants, we demonstrated the validity of the modelled contents using everyday face tasks that generalize identity judgements to new viewpoints, age and sex. Our work highlights that such models of mental representations are critical to understanding generalization behaviour and its underlying information-processing mechanisms.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Reverse-correlating mental representations of familiar faces.
Fig. 2: Contents of mental representations of familiar faces.
Fig. 3: NNMF multivariate and compact representations.
Fig. 4: Generalization of performance across tasks.

Similar content being viewed by others

Data availability

Data are available in Mendeley Data with the identifier https://doi.org/10.17632/nyt677xwfm.150.

Code availability

Analysis scripts are available in Mendeley Data with the identifier https://doi.org/10.17632/nyt677xwfm.150.

References

  1. Bar, M. The proactive brain: memory for predictions. Phil. Trans. R. Soc. B 364, 1235–1243 (2009).

    Article  Google Scholar 

  2. Bar, M. et al. Top-down facilitation of visual recognition. Proc. Natl Acad. Sci. USA 103, 449–454 (2006).

    Article  CAS  Google Scholar 

  3. Ullman, S., Assif, L., Fetaya, E. & Harari, D. Atoms of recognition in human and computer vision. Proc. Natl Acad. Sci. USA 113, 2744–2749 (2016).

    Article  CAS  Google Scholar 

  4. Harel, A., Kravitz, D. J. & Baker, C. I. Task context impacts visual object processing differentially across the cortex. Proc. Natl Acad. Sci. USA 111, E962–E971 (2014).

    Article  CAS  Google Scholar 

  5. O’Toole, A. J. in The Oxford Handbook of Face Perception (eds Rhodes, G. et al.) 15–30 (Oxford Univ. Press, 2011).

  6. Tsao, D. Y. & Livingstone, M. S. Mechanisms of face perception. Annu. Rev. Neurosci. 31, 411–437 (2008).

    Article  CAS  Google Scholar 

  7. Rosch, E. & Mervis, C. B. Family resemblances—studies in internal structure of categories. Cogn. Psychol. 7, 573–605 (1975).

    Article  Google Scholar 

  8. Ahumada, A. & Lovell, J. Stimulus features in signal detection. J. Acoust. Soc. Am. 49, 1751 (1971).

    Article  Google Scholar 

  9. Yu, H., Garrod, O. G. B. & Schyns, P. G. Perception-driven facial expression synthesis. Comput. Graph. 36, 152–162 (2012).

    Article  Google Scholar 

  10. Lee, D. D. & Seung, H. S. Learning the parts of objects by non-negative matrix factorization. Nature 401, 788–791 (1999).

    Article  CAS  Google Scholar 

  11. Lee, H. & Kuhl, B. A. Reconstructing perceived and retrieved faces from activity patterns in lateral parietal cortex. J. Neurosci. 36, 6069–6082 (2016).

    Article  CAS  Google Scholar 

  12. Nestor, A., Plaut, D. C. & Behrmann, M. Feature-based face representations and image reconstruction from behavioral and neural data. Proc. Natl Acad. Sci. USA 113, 416–421 (2016).

    Article  CAS  Google Scholar 

  13. Chang, C. H., Nemrodov, D., Lee, A. C. H. & Nestor, A. Memory and perception-based facial image reconstruction. Sci. Rep. 7, 6499 (2017).

    Article  Google Scholar 

  14. Turk, M. & Pentland, A. Eigenfaces for recognition. J. Cogn. Neurosci. 3, 71–86 (1991).

    Article  CAS  Google Scholar 

  15. Cootes, T. F., Edwards, G. J. & Taylor, C. J. Active appearance models. IEEE Trans. Pattern Anal. 23, 681–685 (2001).

    Article  Google Scholar 

  16. Blanz, V. & Vetter, T. A morphable model for the synthesis of 3D faces. In Proc. 26th Annual Conference on Computer Graphics and Interactive Techniques 187–194 (ACM Press/Addison–Wesley, 1999).

  17. Rhodes, G. & Jeffery, L. Adaptive norm-based coding of facial identity. Vis. Res 46, 2977–2987 (2006).

    Article  Google Scholar 

  18. O’Toole, A. J., Castillo, C. D., Parde, C. J., Hill, M. Q. & Chellappa, R. Face space representations in deep convolutional neural networks. Trends Cogn. Sci. 22, 794–809 (2018).

    Article  Google Scholar 

  19. Young, A. W. & Burton, A. M. Are we face experts? Trends Cogn. Sci. 22, 100–110 (2018).

    Article  Google Scholar 

  20. White, D., Phillips, P. J., Hahn, C. A., Hill, M. & O’Toole, A. J. Perceptual expertise in forensic facial image comparison. Proc. Biol. Sci. 282, 20151292 (2015).

    Article  Google Scholar 

  21. Eger, E., Schweinberger, S. R., Dolan, R. J. & Henson, R. N. Familiarity enhances invariance of face representations in human ventral visual cortex: fMRI evidence. Neuroimage 26, 1128–1139 (2005).

    Article  CAS  Google Scholar 

  22. Jenkins, R., White, D., Van Montfort, X. & Burton, A. M. Variability in photos of the same face. Cognition 121, 313–323 (2011).

    Article  Google Scholar 

  23. Gosselin, F. & Schyns, P. G. RAP: a new framework for visual categorization. Trends Cogn. Sci. 6, 70–77 (2002).

    Article  Google Scholar 

  24. Schyns, P. G. Diagnostic recognition: task constraints, object information, and their interactions. Cognition 67, 147–179 (1998).

    Article  CAS  Google Scholar 

  25. Palmeri, T. J., Wong, A. C. N. & Gauthier, I. Computational approaches to the development of perceptual expertise. Trends Cogn. Sci. 8, 378–386 (2004).

    Article  Google Scholar 

  26. Burton, A. M., Schweinberger, S. R., Jenkins, R. & Kaufmann, J. M. Arguments against a configural processing account of familiar face recognition. Perspect. Psychol. Sci. 10, 482–496 (2015).

    Article  Google Scholar 

  27. Erens, R. G., Kappers, A. M. & Koenderink, J. J. Perception of local shape from shading. Percept. Psychophys. 54, 145–156 (1993).

    Article  CAS  Google Scholar 

  28. Phong, B. T. Illumination for computer generated pictures. Commun. ACM 18, 311–317 (1975).

    Article  Google Scholar 

  29. Liu, Z. L. Viewpoint dependency in object representation and recognition. Spat. Vis. 9, 491–521 (1996).

    Article  CAS  Google Scholar 

  30. Schyns, P. G., Goldstone, R. L. & Thibaut, J. P. The development of features in object concepts. Behav. Brain Sci. 21, 1–17 (1998); discussion 17–54.

    Article  CAS  Google Scholar 

  31. Mangini, M. C. & Biederman, I. Making the ineffable explicit: estimating the information employed for face classifications. Cogn. Sci. 28, 209–226 (2004).

    Article  Google Scholar 

  32. Baxter, M. G. Involvement of medial temporal lobe structures in memory and perception. Neuron 61, 667–677 (2009).

    Article  CAS  Google Scholar 

  33. Xu, T. et al. Deeper interpretability of deep networks. Preprint at https://arxiv.org/abs/1811.07807 (2018).

  34. Leopold, D. A., O’Toole, A. J., Vetter, T. & Blanz, V. Prototype-referenced shape encoding revealed by high-level aftereffects. Nat. Neurosci. 4, 89–94 (2001).

    Article  CAS  Google Scholar 

  35. Leopold, D. A., Bondar, I. V. & Giese, M. A. Norm-based face encoding by single neurons in the monkey inferotemporal cortex. Nature 442, 572–575 (2006).

    Article  CAS  Google Scholar 

  36. Chang, L. & Tsao, D. Y. The code for facial identity in the primate brain. Cell 169, 1013–1028 (2017).

    Article  CAS  Google Scholar 

  37. Zhan, J., Ince, R. A. A., van Rijsbergen, N. & Schyns, P. G. Dynamic construction of reduced representations in the brain for perceptual decision behavior. Curr. Biol. 29, 319–326 e314 (2019).

    Article  CAS  Google Scholar 

  38. Kay, K. N., Naselaris, T., Prenger, R. J. & Gallant, J. L. Identifying natural images from human brain activity. Nature 452, 352–357 (2008).

    Article  CAS  Google Scholar 

  39. Smith, F. W. & Muckli, L. Nonstimulated early visual areas carry information about surrounding context. Proc. Natl Acad. Sci. USA 107, 20099–20103 (2010).

    Article  CAS  Google Scholar 

  40. Peirce, J. W. Understanding mid-level representations in visual processing. J. Vis. 15, 5 (2015).

    Article  Google Scholar 

  41. Kubilius, J., Wagemans, J. & Op de Beeck, H. P. A conceptual framework of computations in mid-level vision. Front. Comput. Neurosci. 8, 158 (2014).

    Article  Google Scholar 

  42. Friston, K. J. & Kiebel, S. Predictive coding under the free-energy principle. Phil. Trans. R. Soc. B 364, 1211–1221 (2009).

    Article  Google Scholar 

  43. Clark, A. Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behav. Brain Sci. 36, 181–204 (2013).

    Article  Google Scholar 

  44. Gosselin, F. & Schyns, P. G. Superstitious perceptions reveal properties of internal representations. Psychol. Sci. 14, 505–509 (2003).

    Article  Google Scholar 

  45. Smith, M. L., Gosselin, F. & Schyns, P. G. Measuring internal representations from behavioral and brain data. Curr. Biol. 22, 191–196 (2012).

    Article  CAS  Google Scholar 

  46. Nestor, A., Plaut, D. C. & Behrmann, M. Unraveling the distributed neural code of facial identity through spatiotemporal pattern analysis. Proc. Natl Acad. Sci. USA 108, 9998–10003 (2011).

    Article  CAS  Google Scholar 

  47. Gobbini, M. I. et al. Prioritized detection of personally familiar faces. PLoS ONE 8, e66620 (2013).

    Article  CAS  Google Scholar 

  48. van Belle, G., Ramon, M., Lefevre, P. & Rossion, B. Fixation patterns during recognition of personally familiar and unfamiliar faces. Front. Psychol. 1, 20 (2010).

    PubMed  PubMed Central  Google Scholar 

  49. Ramon, M., Vizioli, L., Liu-Shuang, J. & Rossion, B. Neural microgenesis of personally familiar face recognition. Proc. Natl Acad. Sci. USA 112, E4835–E4844 (2015).

    Article  CAS  Google Scholar 

  50. Zhan, J., Garrod, O. G., van Rijsbergen, N. & Schyns, P. Modelling face memory reveals task-generalizable representations. Mendeley Data https://doi.org/10.17632/nyt677xwfm.1 (2019).

Download references

Acknowledgements

P.G.S. received support from the Wellcome Trust (Senior Investigator Award, UK; 107802) and the Multidisciplinary University Research Initiative/Engineering and Physical Sciences Research Council (USA, UK; 172046-01). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Authors and Affiliations

  1. Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, UK

    Jiayu Zhan, Oliver G. B. Garrod, Nicola van Rijsbergen & Philippe G. Schyns

  2. School of Psychology, University of Glasgow, Glasgow, UK

    Philippe G. Schyns

Authors
  1. Jiayu Zhan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Oliver G. B. Garrod
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicola van Rijsbergen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Philippe G. Schyns
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

J.Z., N.v.R. and P.G.S. designed the research; O.G.B.G. and P.G.S. developed the GMF; J.Z. performed the research; J.Z. and N.v.R. analysed the data; and J.Z., N.v.R. and P.G.S. wrote the paper.

Corresponding author

Correspondence to Philippe G. Schyns.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information: Primary Handling Editor: Marike Schiffer

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary Information

Supplementary Methods, Supplementary Figs. 1–9 and Supplementary Tables 1–7.

Reporting Summary

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhan, J., Garrod, O.G.B., van Rijsbergen, N. et al. Modelling face memory reveals task-generalizable representations. Nat Hum Behav 3, 817–826 (2019). https://doi.org/10.1038/s41562-019-0625-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-019-0625-3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing