Modelling face memory reveals task-generalizable representations

Article metrics



Current cognitive theories are cast in terms of information-processing mechanisms that use mental representations1,2,3,4. For example, people use their mental representations to identify familiar faces under various conditions of pose, illumination and ageing, or to draw resemblance between family members. Yet, the actual information contents of these representations are rarely characterized, which hinders knowledge of the mechanisms that use them. Here, we modelled the three-dimensional representational contents of 4 faces that were familiar to 14 participants as work colleagues. The representational contents were created by reverse-correlating identity information generated on each trial with judgements of the face’s similarity to the individual participant’s memory of this face. In a second study, testing new participants, we demonstrated the validity of the modelled contents using everyday face tasks that generalize identity judgements to new viewpoints, age and sex. Our work highlights that such models of mental representations are critical to understanding generalization behaviour and its underlying information-processing mechanisms.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.


All prices are NET prices.

Fig. 1: Reverse-correlating mental representations of familiar faces.
Fig. 2: Contents of mental representations of familiar faces.
Fig. 3: NNMF multivariate and compact representations.
Fig. 4: Generalization of performance across tasks.

Data availability

Data are available in Mendeley Data with the identifier

Code availability

Analysis scripts are available in Mendeley Data with the identifier


  1. 1.

    Bar, M. The proactive brain: memory for predictions. Phil. Trans. R. Soc. B 364, 1235–1243 (2009).

  2. 2.

    Bar, M. et al. Top-down facilitation of visual recognition. Proc. Natl Acad. Sci. USA 103, 449–454 (2006).

  3. 3.

    Ullman, S., Assif, L., Fetaya, E. & Harari, D. Atoms of recognition in human and computer vision. Proc. Natl Acad. Sci. USA 113, 2744–2749 (2016).

  4. 4.

    Harel, A., Kravitz, D. J. & Baker, C. I. Task context impacts visual object processing differentially across the cortex. Proc. Natl Acad. Sci. USA 111, E962–E971 (2014).

  5. 5.

    O’Toole, A. J. in The Oxford Handbook of Face Perception (eds Rhodes, G. et al.) 15–30 (Oxford Univ. Press, 2011).

  6. 6.

    Tsao, D. Y. & Livingstone, M. S. Mechanisms of face perception. Annu. Rev. Neurosci. 31, 411–437 (2008).

  7. 7.

    Rosch, E. & Mervis, C. B. Family resemblances—studies in internal structure of categories. Cogn. Psychol. 7, 573–605 (1975).

  8. 8.

    Ahumada, A. & Lovell, J. Stimulus features in signal detection. J. Acoust. Soc. Am. 49, 1751 (1971).

  9. 9.

    Yu, H., Garrod, O. G. B. & Schyns, P. G. Perception-driven facial expression synthesis. Comput. Graph. 36, 152–162 (2012).

  10. 10.

    Lee, D. D. & Seung, H. S. Learning the parts of objects by non-negative matrix factorization. Nature 401, 788–791 (1999).

  11. 11.

    Lee, H. & Kuhl, B. A. Reconstructing perceived and retrieved faces from activity patterns in lateral parietal cortex. J. Neurosci. 36, 6069–6082 (2016).

  12. 12.

    Nestor, A., Plaut, D. C. & Behrmann, M. Feature-based face representations and image reconstruction from behavioral and neural data. Proc. Natl Acad. Sci. USA 113, 416–421 (2016).

  13. 13.

    Chang, C. H., Nemrodov, D., Lee, A. C. H. & Nestor, A. Memory and perception-based facial image reconstruction. Sci. Rep. 7, 6499 (2017).

  14. 14.

    Turk, M. & Pentland, A. Eigenfaces for recognition. J. Cogn. Neurosci. 3, 71–86 (1991).

  15. 15.

    Cootes, T. F., Edwards, G. J. & Taylor, C. J. Active appearance models. IEEE Trans. Pattern Anal. 23, 681–685 (2001).

  16. 16.

    Blanz, V. & Vetter, T. A morphable model for the synthesis of 3D faces. In Proc. 26th Annual Conference on Computer Graphics and Interactive Techniques 187–194 (ACM Press/Addison–Wesley, 1999).

  17. 17.

    Rhodes, G. & Jeffery, L. Adaptive norm-based coding of facial identity. Vis. Res 46, 2977–2987 (2006).

  18. 18.

    O’Toole, A. J., Castillo, C. D., Parde, C. J., Hill, M. Q. & Chellappa, R. Face space representations in deep convolutional neural networks. Trends Cogn. Sci. 22, 794–809 (2018).

  19. 19.

    Young, A. W. & Burton, A. M. Are we face experts? Trends Cogn. Sci. 22, 100–110 (2018).

  20. 20.

    White, D., Phillips, P. J., Hahn, C. A., Hill, M. & O’Toole, A. J. Perceptual expertise in forensic facial image comparison. Proc. Biol. Sci. 282, 20151292 (2015).

  21. 21.

    Eger, E., Schweinberger, S. R., Dolan, R. J. & Henson, R. N. Familiarity enhances invariance of face representations in human ventral visual cortex: fMRI evidence. Neuroimage 26, 1128–1139 (2005).

  22. 22.

    Jenkins, R., White, D., Van Montfort, X. & Burton, A. M. Variability in photos of the same face. Cognition 121, 313–323 (2011).

  23. 23.

    Gosselin, F. & Schyns, P. G. RAP: a new framework for visual categorization. Trends Cogn. Sci. 6, 70–77 (2002).

  24. 24.

    Schyns, P. G. Diagnostic recognition: task constraints, object information, and their interactions. Cognition 67, 147–179 (1998).

  25. 25.

    Palmeri, T. J., Wong, A. C. N. & Gauthier, I. Computational approaches to the development of perceptual expertise. Trends Cogn. Sci. 8, 378–386 (2004).

  26. 26.

    Burton, A. M., Schweinberger, S. R., Jenkins, R. & Kaufmann, J. M. Arguments against a configural processing account of familiar face recognition. Perspect. Psychol. Sci. 10, 482–496 (2015).

  27. 27.

    Erens, R. G., Kappers, A. M. & Koenderink, J. J. Perception of local shape from shading. Percept. Psychophys. 54, 145–156 (1993).

  28. 28.

    Phong, B. T. Illumination for computer generated pictures. Commun. ACM 18, 311–317 (1975).

  29. 29.

    Liu, Z. L. Viewpoint dependency in object representation and recognition. Spat. Vis. 9, 491–521 (1996).

  30. 30.

    Schyns, P. G., Goldstone, R. L. & Thibaut, J. P. The development of features in object concepts. Behav. Brain Sci. 21, 1–17 (1998); discussion 17–54.

  31. 31.

    Mangini, M. C. & Biederman, I. Making the ineffable explicit: estimating the information employed for face classifications. Cogn. Sci. 28, 209–226 (2004).

  32. 32.

    Baxter, M. G. Involvement of medial temporal lobe structures in memory and perception. Neuron 61, 667–677 (2009).

  33. 33.

    Xu, T. et al. Deeper interpretability of deep networks. Preprint at (2018).

  34. 34.

    Leopold, D. A., O’Toole, A. J., Vetter, T. & Blanz, V. Prototype-referenced shape encoding revealed by high-level aftereffects. Nat. Neurosci. 4, 89–94 (2001).

  35. 35.

    Leopold, D. A., Bondar, I. V. & Giese, M. A. Norm-based face encoding by single neurons in the monkey inferotemporal cortex. Nature 442, 572–575 (2006).

  36. 36.

    Chang, L. & Tsao, D. Y. The code for facial identity in the primate brain. Cell 169, 1013–1028 (2017).

  37. 37.

    Zhan, J., Ince, R. A. A., van Rijsbergen, N. & Schyns, P. G. Dynamic construction of reduced representations in the brain for perceptual decision behavior. Curr. Biol. 29, 319–326 e314 (2019).

  38. 38.

    Kay, K. N., Naselaris, T., Prenger, R. J. & Gallant, J. L. Identifying natural images from human brain activity. Nature 452, 352–357 (2008).

  39. 39.

    Smith, F. W. & Muckli, L. Nonstimulated early visual areas carry information about surrounding context. Proc. Natl Acad. Sci. USA 107, 20099–20103 (2010).

  40. 40.

    Peirce, J. W. Understanding mid-level representations in visual processing. J. Vis. 15, 5 (2015).

  41. 41.

    Kubilius, J., Wagemans, J. & Op de Beeck, H. P. A conceptual framework of computations in mid-level vision. Front. Comput. Neurosci. 8, 158 (2014).

  42. 42.

    Friston, K. J. & Kiebel, S. Predictive coding under the free-energy principle. Phil. Trans. R. Soc. B 364, 1211–1221 (2009).

  43. 43.

    Clark, A. Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behav. Brain Sci. 36, 181–204 (2013).

  44. 44.

    Gosselin, F. & Schyns, P. G. Superstitious perceptions reveal properties of internal representations. Psychol. Sci. 14, 505–509 (2003).

  45. 45.

    Smith, M. L., Gosselin, F. & Schyns, P. G. Measuring internal representations from behavioral and brain data. Curr. Biol. 22, 191–196 (2012).

  46. 46.

    Nestor, A., Plaut, D. C. & Behrmann, M. Unraveling the distributed neural code of facial identity through spatiotemporal pattern analysis. Proc. Natl Acad. Sci. USA 108, 9998–10003 (2011).

  47. 47.

    Gobbini, M. I. et al. Prioritized detection of personally familiar faces. PLoS ONE 8, e66620 (2013).

  48. 48.

    van Belle, G., Ramon, M., Lefevre, P. & Rossion, B. Fixation patterns during recognition of personally familiar and unfamiliar faces. Front. Psychol. 1, 20 (2010).

  49. 49.

    Ramon, M., Vizioli, L., Liu-Shuang, J. & Rossion, B. Neural microgenesis of personally familiar face recognition. Proc. Natl Acad. Sci. USA 112, E4835–E4844 (2015).

  50. 50.

    Zhan, J., Garrod, O. G., van Rijsbergen, N. & Schyns, P. Modelling face memory reveals task-generalizable representations. Mendeley Data (2019).

Download references


P.G.S. received support from the Wellcome Trust (Senior Investigator Award, UK; 107802) and the Multidisciplinary University Research Initiative/Engineering and Physical Sciences Research Council (USA, UK; 172046-01). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

J.Z., N.v.R. and P.G.S. designed the research; O.G.B.G. and P.G.S. developed the GMF; J.Z. performed the research; J.Z. and N.v.R. analysed the data; and J.Z., N.v.R. and P.G.S. wrote the paper.

Correspondence to Philippe G. Schyns.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information: Primary Handling Editor: Marike Schiffer

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary Information

Supplementary Methods, Supplementary Figs. 1–9 and Supplementary Tables 1–7.

Reporting Summary

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Further reading