Modelling face memory reveals task-generalizable representations

Subjects

Abstract

Current cognitive theories are cast in terms of information-processing mechanisms that use mental representations1,2,3,4. For example, people use their mental representations to identify familiar faces under various conditions of pose, illumination and ageing, or to draw resemblance between family members. Yet, the actual information contents of these representations are rarely characterized, which hinders knowledge of the mechanisms that use them. Here, we modelled the three-dimensional representational contents of 4 faces that were familiar to 14 participants as work colleagues. The representational contents were created by reverse-correlating identity information generated on each trial with judgements of the face’s similarity to the individual participant’s memory of this face. In a second study, testing new participants, we demonstrated the validity of the modelled contents using everyday face tasks that generalize identity judgements to new viewpoints, age and sex. Our work highlights that such models of mental representations are critical to understanding generalization behaviour and its underlying information-processing mechanisms.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Reverse-correlating mental representations of familiar faces.
Fig. 2: Contents of mental representations of familiar faces.
Fig. 3: NNMF multivariate and compact representations.
Fig. 4: Generalization of performance across tasks.

Data availability

Data are available in Mendeley Data with the identifier https://doi.org/10.17632/nyt677xwfm.150.

Code availability

Analysis scripts are available in Mendeley Data with the identifier https://doi.org/10.17632/nyt677xwfm.150.

References

  1. 1.

    Bar, M. The proactive brain: memory for predictions. Phil. Trans. R. Soc. B 364, 1235–1243 (2009).

    Article  Google Scholar 

  2. 2.

    Bar, M. et al. Top-down facilitation of visual recognition. Proc. Natl Acad. Sci. USA 103, 449–454 (2006).

    CAS  Article  Google Scholar 

  3. 3.

    Ullman, S., Assif, L., Fetaya, E. & Harari, D. Atoms of recognition in human and computer vision. Proc. Natl Acad. Sci. USA 113, 2744–2749 (2016).

    CAS  Article  Google Scholar 

  4. 4.

    Harel, A., Kravitz, D. J. & Baker, C. I. Task context impacts visual object processing differentially across the cortex. Proc. Natl Acad. Sci. USA 111, E962–E971 (2014).

    CAS  Article  Google Scholar 

  5. 5.

    O’Toole, A. J. in The Oxford Handbook of Face Perception (eds Rhodes, G. et al.) 15–30 (Oxford Univ. Press, 2011).

  6. 6.

    Tsao, D. Y. & Livingstone, M. S. Mechanisms of face perception. Annu. Rev. Neurosci. 31, 411–437 (2008).

    CAS  Article  Google Scholar 

  7. 7.

    Rosch, E. & Mervis, C. B. Family resemblances—studies in internal structure of categories. Cogn. Psychol. 7, 573–605 (1975).

    Article  Google Scholar 

  8. 8.

    Ahumada, A. & Lovell, J. Stimulus features in signal detection. J. Acoust. Soc. Am. 49, 1751 (1971).

    Article  Google Scholar 

  9. 9.

    Yu, H., Garrod, O. G. B. & Schyns, P. G. Perception-driven facial expression synthesis. Comput. Graph. 36, 152–162 (2012).

    Article  Google Scholar 

  10. 10.

    Lee, D. D. & Seung, H. S. Learning the parts of objects by non-negative matrix factorization. Nature 401, 788–791 (1999).

    CAS  Article  Google Scholar 

  11. 11.

    Lee, H. & Kuhl, B. A. Reconstructing perceived and retrieved faces from activity patterns in lateral parietal cortex. J. Neurosci. 36, 6069–6082 (2016).

    CAS  Article  Google Scholar 

  12. 12.

    Nestor, A., Plaut, D. C. & Behrmann, M. Feature-based face representations and image reconstruction from behavioral and neural data. Proc. Natl Acad. Sci. USA 113, 416–421 (2016).

    CAS  Article  Google Scholar 

  13. 13.

    Chang, C. H., Nemrodov, D., Lee, A. C. H. & Nestor, A. Memory and perception-based facial image reconstruction. Sci. Rep. 7, 6499 (2017).

    Article  Google Scholar 

  14. 14.

    Turk, M. & Pentland, A. Eigenfaces for recognition. J. Cogn. Neurosci. 3, 71–86 (1991).

    CAS  Article  Google Scholar 

  15. 15.

    Cootes, T. F., Edwards, G. J. & Taylor, C. J. Active appearance models. IEEE Trans. Pattern Anal. 23, 681–685 (2001).

    Article  Google Scholar 

  16. 16.

    Blanz, V. & Vetter, T. A morphable model for the synthesis of 3D faces. In Proc. 26th Annual Conference on Computer Graphics and Interactive Techniques 187–194 (ACM Press/Addison–Wesley, 1999).

  17. 17.

    Rhodes, G. & Jeffery, L. Adaptive norm-based coding of facial identity. Vis. Res 46, 2977–2987 (2006).

    Article  Google Scholar 

  18. 18.

    O’Toole, A. J., Castillo, C. D., Parde, C. J., Hill, M. Q. & Chellappa, R. Face space representations in deep convolutional neural networks. Trends Cogn. Sci. 22, 794–809 (2018).

    Article  Google Scholar 

  19. 19.

    Young, A. W. & Burton, A. M. Are we face experts? Trends Cogn. Sci. 22, 100–110 (2018).

    Article  Google Scholar 

  20. 20.

    White, D., Phillips, P. J., Hahn, C. A., Hill, M. & O’Toole, A. J. Perceptual expertise in forensic facial image comparison. Proc. Biol. Sci. 282, 20151292 (2015).

    Article  Google Scholar 

  21. 21.

    Eger, E., Schweinberger, S. R., Dolan, R. J. & Henson, R. N. Familiarity enhances invariance of face representations in human ventral visual cortex: fMRI evidence. Neuroimage 26, 1128–1139 (2005).

    CAS  Article  Google Scholar 

  22. 22.

    Jenkins, R., White, D., Van Montfort, X. & Burton, A. M. Variability in photos of the same face. Cognition 121, 313–323 (2011).

    Article  Google Scholar 

  23. 23.

    Gosselin, F. & Schyns, P. G. RAP: a new framework for visual categorization. Trends Cogn. Sci. 6, 70–77 (2002).

    Article  Google Scholar 

  24. 24.

    Schyns, P. G. Diagnostic recognition: task constraints, object information, and their interactions. Cognition 67, 147–179 (1998).

    CAS  Article  Google Scholar 

  25. 25.

    Palmeri, T. J., Wong, A. C. N. & Gauthier, I. Computational approaches to the development of perceptual expertise. Trends Cogn. Sci. 8, 378–386 (2004).

    Article  Google Scholar 

  26. 26.

    Burton, A. M., Schweinberger, S. R., Jenkins, R. & Kaufmann, J. M. Arguments against a configural processing account of familiar face recognition. Perspect. Psychol. Sci. 10, 482–496 (2015).

    Article  Google Scholar 

  27. 27.

    Erens, R. G., Kappers, A. M. & Koenderink, J. J. Perception of local shape from shading. Percept. Psychophys. 54, 145–156 (1993).

    CAS  Article  Google Scholar 

  28. 28.

    Phong, B. T. Illumination for computer generated pictures. Commun. ACM 18, 311–317 (1975).

    Article  Google Scholar 

  29. 29.

    Liu, Z. L. Viewpoint dependency in object representation and recognition. Spat. Vis. 9, 491–521 (1996).

    CAS  Article  Google Scholar 

  30. 30.

    Schyns, P. G., Goldstone, R. L. & Thibaut, J. P. The development of features in object concepts. Behav. Brain Sci. 21, 1–17 (1998); discussion 17–54.

    CAS  Article  Google Scholar 

  31. 31.

    Mangini, M. C. & Biederman, I. Making the ineffable explicit: estimating the information employed for face classifications. Cogn. Sci. 28, 209–226 (2004).

    Article  Google Scholar 

  32. 32.

    Baxter, M. G. Involvement of medial temporal lobe structures in memory and perception. Neuron 61, 667–677 (2009).

    CAS  Article  Google Scholar 

  33. 33.

    Xu, T. et al. Deeper interpretability of deep networks. Preprint at https://arxiv.org/abs/1811.07807 (2018).

  34. 34.

    Leopold, D. A., O’Toole, A. J., Vetter, T. & Blanz, V. Prototype-referenced shape encoding revealed by high-level aftereffects. Nat. Neurosci. 4, 89–94 (2001).

    CAS  Article  Google Scholar 

  35. 35.

    Leopold, D. A., Bondar, I. V. & Giese, M. A. Norm-based face encoding by single neurons in the monkey inferotemporal cortex. Nature 442, 572–575 (2006).

    CAS  Article  Google Scholar 

  36. 36.

    Chang, L. & Tsao, D. Y. The code for facial identity in the primate brain. Cell 169, 1013–1028 (2017).

    CAS  Article  Google Scholar 

  37. 37.

    Zhan, J., Ince, R. A. A., van Rijsbergen, N. & Schyns, P. G. Dynamic construction of reduced representations in the brain for perceptual decision behavior. Curr. Biol. 29, 319–326 e314 (2019).

    CAS  Article  Google Scholar 

  38. 38.

    Kay, K. N., Naselaris, T., Prenger, R. J. & Gallant, J. L. Identifying natural images from human brain activity. Nature 452, 352–357 (2008).

    CAS  Article  Google Scholar 

  39. 39.

    Smith, F. W. & Muckli, L. Nonstimulated early visual areas carry information about surrounding context. Proc. Natl Acad. Sci. USA 107, 20099–20103 (2010).

    CAS  Article  Google Scholar 

  40. 40.

    Peirce, J. W. Understanding mid-level representations in visual processing. J. Vis. 15, 5 (2015).

    Article  Google Scholar 

  41. 41.

    Kubilius, J., Wagemans, J. & Op de Beeck, H. P. A conceptual framework of computations in mid-level vision. Front. Comput. Neurosci. 8, 158 (2014).

    Article  Google Scholar 

  42. 42.

    Friston, K. J. & Kiebel, S. Predictive coding under the free-energy principle. Phil. Trans. R. Soc. B 364, 1211–1221 (2009).

    Article  Google Scholar 

  43. 43.

    Clark, A. Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behav. Brain Sci. 36, 181–204 (2013).

    Article  Google Scholar 

  44. 44.

    Gosselin, F. & Schyns, P. G. Superstitious perceptions reveal properties of internal representations. Psychol. Sci. 14, 505–509 (2003).

    Article  Google Scholar 

  45. 45.

    Smith, M. L., Gosselin, F. & Schyns, P. G. Measuring internal representations from behavioral and brain data. Curr. Biol. 22, 191–196 (2012).

    CAS  Article  Google Scholar 

  46. 46.

    Nestor, A., Plaut, D. C. & Behrmann, M. Unraveling the distributed neural code of facial identity through spatiotemporal pattern analysis. Proc. Natl Acad. Sci. USA 108, 9998–10003 (2011).

    CAS  Article  Google Scholar 

  47. 47.

    Gobbini, M. I. et al. Prioritized detection of personally familiar faces. PLoS ONE 8, e66620 (2013).

    CAS  Article  Google Scholar 

  48. 48.

    van Belle, G., Ramon, M., Lefevre, P. & Rossion, B. Fixation patterns during recognition of personally familiar and unfamiliar faces. Front. Psychol. 1, 20 (2010).

    PubMed  PubMed Central  Google Scholar 

  49. 49.

    Ramon, M., Vizioli, L., Liu-Shuang, J. & Rossion, B. Neural microgenesis of personally familiar face recognition. Proc. Natl Acad. Sci. USA 112, E4835–E4844 (2015).

    CAS  Article  Google Scholar 

  50. 50.

    Zhan, J., Garrod, O. G., van Rijsbergen, N. & Schyns, P. Modelling face memory reveals task-generalizable representations. Mendeley Data https://doi.org/10.17632/nyt677xwfm.1 (2019).

Download references

Acknowledgements

P.G.S. received support from the Wellcome Trust (Senior Investigator Award, UK; 107802) and the Multidisciplinary University Research Initiative/Engineering and Physical Sciences Research Council (USA, UK; 172046-01). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Affiliations

Authors

Contributions

J.Z., N.v.R. and P.G.S. designed the research; O.G.B.G. and P.G.S. developed the GMF; J.Z. performed the research; J.Z. and N.v.R. analysed the data; and J.Z., N.v.R. and P.G.S. wrote the paper.

Corresponding author

Correspondence to Philippe G. Schyns.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information: Primary Handling Editor: Marike Schiffer

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary Information

Supplementary Methods, Supplementary Figs. 1–9 and Supplementary Tables 1–7.

Reporting Summary

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhan, J., Garrod, O.G.B., van Rijsbergen, N. et al. Modelling face memory reveals task-generalizable representations. Nat Hum Behav 3, 817–826 (2019). https://doi.org/10.1038/s41562-019-0625-3

Download citation

Further reading

Search

Sign up for the Nature Briefing newsletter for a daily update on COVID-19 science.
Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing