Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Revealing the multidimensional mental representations of natural objects underlying human similarity judgements

Nature Human Behaviour volume 4, pages 1173–1185 (2020)Cite this article

Subjects

Abstract

Objects can be characterized according to a vast number of possible criteria (such as animacy, shape, colour and function), but some dimensions are more useful than others for making sense of the objects around us. To identify these core dimensions of object representations, we developed a data-driven computational model of similarity judgements for real-world images of 1,854 objects. The model captured most explainable variance in similarity judgements and produced 49 highly reproducible and meaningful object dimensions that reflect various conceptual and perceptual properties of those objects. These dimensions predicted external categorization behaviour and reflected typicality judgements of those categories. Furthermore, humans can accurately rate objects along these dimensions, highlighting their interpretability and opening up a way to generate similarity estimates from object dimensions alone. Collectively, these results demonstrate that human similarity judgements can be captured by a fairly low-dimensional, interpretable embedding that generalizes to external behaviour.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Task and modelling procedure for the large-scale identification of mental object representations.
Fig. 2: Predictiveness of the computational model for single-trial behavioural judgements and similarity.
Fig. 3: Example object dimensions illustrating their interpretability.
Fig. 4: Illustration of example objects with their respective dimensions, using rose plots.
Fig. 5: Two-dimensional visualization of the similarity embedding, combining dimensionality reduction (multidimensional-scaling-initialized t-distributed stochastic neighbourhood embedding; dual perplexity, 5 and 30; 1,000 iterations) with rose plots for each object (see Fig. 4).
Fig. 6: How many dimensions are required to capture behavioural judgements and object similarity?
Fig. 7: The relationships between seemingly categorical dimensions and typicality ratings of those categories.
Fig. 8: Task and results of direct ratings of dimensions.

Similar content being viewed by others

Data availability

The learned embedding, triplet odd-one-out behavioural data for testing model performance, typicality scores, participant-generated dimension labels and dimension ratings are available at https://osf.io/z2784. The behavioural data used for training the model are available from the corresponding author upon request.

Code availability

To reproduce the relevant analyses and figures, the relevant MATLAB scripts and functions are available at https://osf.io/z2784. The computational modelling code to create an embedding is available from the corresponding author upon request.

References

  1. Biederman, I. Recognition-by-components: a theory of human image understanding. Psychol. Rev. 94, 115–147 (1987).

    Article  Google Scholar 

  2. Edelman, S. Representation is representation of similarities. Behav. Brain Sci. 21, 449–467 (1998).

    Article  CAS  Google Scholar 

  3. Nosofsky, R. M. Attention, similarity, and the identification–categorization relationship. J. Exp. Psychol. Gen. 115, 39–57 (1986).

    Article  CAS  Google Scholar 

  4. Goldstone, R. L. The role of similarity in categorization: providing a groundwork. Cognition 52, 125–157 (1994).

    Article  CAS  Google Scholar 

  5. Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M. & Boyes-Braem, P. Basic objects in natural categories. Cognit. Psychol. 8, 382–439 (1976).

    Article  Google Scholar 

  6. Hahn, U. & Chater, N. in Knowledge, Concepts and Categories (eds Lamberts, K. & Shanks, D.) 43–92 (Psychology Press, 1997).

  7. Rips, L. J., Smith, E. E. & Medin, D. L. in The Oxford Handbook of Thinking and Reasoning (eds Holyoak, K. J. & Morrison, R. G.) 177–209 (Oxford Univ. Press, 2012).

  8. Rogers, T. T. & McClelland, J. L. Semantic Cognition: A Parallel Distributed Processing Approach (MIT Press, 2004).

  9. Goldstone, R. L. & Son, J. Y. in The Oxford Handbook of Thinking and Reasoning (eds Holyoak, K. J. & Morrison, R. G.) 155–176 (Oxford Univ. Press, 2012).

  10. Kriegeskorte, N. & Kievit, R. A. Representational geometry: integrating cognition, computation, and the brain. Trends Cogn. Sci. 17, 401–412 (2013).

    Article  Google Scholar 

  11. Caramazza, A. & Shelton, J. R. Domain-specific knowledge systems in the brain: the animate–inanimate distinction. J. Cogn. Neurosci. 10, 1–34 (1998).

    Article  CAS  Google Scholar 

  12. Chao, L. L., Haxby, J. V. & Martin, A. Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects. Nat. Neurosci. 2, 913–919 (1999).

    Article  CAS  Google Scholar 

  13. Konkle, T. & Oliva, A. Canonical visual size for real-world objects. J. Exp. Psychol. Hum. Percept. Perform. 37, 23–37 (2011).

    Article  Google Scholar 

  14. Murphy, G. The Big Book of Concepts (MIT Press, 2004).

  15. McRae, K., Cree, G. S., Seidenberg, M. S. & McNorgan, C. Semantic feature production norms for a large set of living and nonliving things. Behav. Res. Methods 37, 547–559 (2005).

    Article  Google Scholar 

  16. Devereux, B. J., Tyler, L. K., Geertzen, J. & Randall, B. The Centre for Speech, Language and the Brain (CSLB) concept property norms. Behav. Res. Methods 46, 1119–1127 (2014).

    Article  Google Scholar 

  17. Hebart, M. N. et al. THINGS: a database of 1,854 object concepts and more than 26,000 naturalistic object images. PLoS ONE 14, e0223792 (2019).

    Article  CAS  Google Scholar 

  18. Tversky, A. Features of similarity. Psychol. Rev. 84, 327–352 (1977).

    Article  Google Scholar 

  19. Barsalou, L. W. Context-independent and context-dependent information in concepts. Mem. Cognit. 10, 82–93 (1982).

    Article  CAS  Google Scholar 

  20. Maddox, W. T. & Ashby, F. G. Comparing decision bound and exemplar models of categorization. Percept. Psychophys. 53, 49–70 (1993).

    Article  CAS  Google Scholar 

  21. Hoyer, P. O. Modeling receptive fields with non-negative sparse coding. Neurocomputing 52, 547–552 (2003).

    Article  Google Scholar 

  22. Murphy, B., Talukdar, P. & Mitchell, T. Learning effective and interpretable semantic models using non-negative sparse embedding. In Proc. of COLING 2012 1933–1950 (2012).

  23. Shepard, R. N. Stimulus and response generalization: a stochastic model relating generalization to distance in psychological space. Psychometrika 22, 325–345 (1957).

    Article  Google Scholar 

  24. Kobak, D. & Berens, P. The art of using t-SNE for single-cell transcriptomics. Nat. Commun. 10, 5416 (2019).

    Article  Google Scholar 

  25. Shelton, J. R., Fouch, E. & Caramazza, A. The selective sparing of body part knowledge: a case study. Neurocase 4, 339–351 (1998).

    Article  Google Scholar 

  26. Pedersen, T., Patwardhan, S. & Michelizzi, J. WordNet::Similarity—measuring the relatedness of concepts. In HLT-NAACL 2004: Demonstration Papers (eds Dumais, S. et al.) 38–41 (ACL Press, 2004).

  27. Warrington, E. K. & Shallice, T. Category specific semantic impairments. Brain 107, 829–853 (1984).

    Article  Google Scholar 

  28. Rips, L. J. in Similarity and Analogical Reasoning (eds Vosniadou, S. & Ortony, A.) 21–59 (Cambridge Univ. Press, 1989).

  29. Smith, E. E. & Sloman, S. A. Similarity- versus rule-based categorization. Mem. Cognit. 22, 377–386 (1994).

    Article  CAS  Google Scholar 

  30. Pilehvar, M. T. & Collier, N. De-conflated semantic representations. In 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP) 1680–1690 (2016).

  31. Nosofsky, R. M., Sanders, C. A., Meagher, B. J. & Douglas, B. J. Toward the development of a feature-space representation for a complex natural category domain. Behav. Res. Methods 50, 530–556 (2018).

    Article  Google Scholar 

  32. Nosofsky, R. M., Sanders, C. A., Meagher, B. J. & Douglas, B. J. Search for the missing dimensions: building a feature-space representation for a natural-science category domain. Comput. Brain Behav. 3, 13–33 (2020).

    Article  Google Scholar 

  33. Keil, F. C. Constraints on knowledge and cognitive development. Psychol. Rev. 88, 187–227 (1981).

    Article  Google Scholar 

  34. Shepard, R. N. The analysis of proximities: multidimensional scaling with an unknown distance function. Psychometrika 27, 125–140 (1962).

    Article  Google Scholar 

  35. Torgerson, W. S. Multidimensional scaling: I. Theory and method. Psychometrika 17, 401–419 (1952).

    Article  Google Scholar 

  36. Thurstone, L. L. Multiple factor analysis. Psychol. Rev. 38, 406–427 (1931).

    Article  Google Scholar 

  37. Tranel, D., Logan, C. G., Frank, R. J. & Damasio, A. R. Explaining category-related effects in the retrieval of conceptual and lexical knowledge for concrete entities: operationalization and analysis of factors. Neuropsychologia 35, 1329–1339 (1997).

    Article  CAS  Google Scholar 

  38. Shepard, R. N. & Arabie, P. Additive clustering: representation of similarities as combinations of discrete overlapping properties. Psychol. Rev. 86, 87–123 (1979).

    Article  Google Scholar 

  39. Navarro, D. J. & Lee, M. D. Common and distinctive features in stimulus similarity: a modified version of the contrast model. Psychon. Bull. Rev. 11, 961–974 (2004).

    Article  Google Scholar 

  40. Carlson, T. A., Ritchie, J. B., Kriegeskorte, N., Durvasula, S. & Ma, J. Reaction time for object categorization is predicted by representational distance. J. Cogn. Neurosci. 26, 132–142 (2013).

    Article  Google Scholar 

  41. Yee, E. & Thompson-Schill, S. L. Putting concepts into context. Psychon. Bull. Rev. 23, 1015–1027 (2016).

    Article  Google Scholar 

  42. Charest, I., Kievit, R. A., Schmitz, T. W., Deca, D. & Kriegeskorte, N. Unique semantic space in the brain of each beholder predicts perceived similarity. Proc. Natl Acad. Sci. USA 111, 14565–14570 (2014).

    Article  CAS  Google Scholar 

  43. De Haas, B., Iakovidis, A. L., Schwarzkopf, D. S. & Gegenfurtner, K. R. Individual differences in visual salience vary along semantic dimensions. Proc. Natl Acad. Sci. USA 116, 11687–11692 (2019).

    Google Scholar 

  44. Peterson, J. C., Abbott, J. T. & Griffiths, T. L. Evaluating (and improving) the correspondence between deep neural networks and human representations. Cogn. Sci. 42, 2648–2669 (2018).

    Article  Google Scholar 

  45. Rajalingham, R. et al. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks. J. Neurosci. 38, 7255–7269 (2018).

    Article  CAS  Google Scholar 

  46. Jozwik, K. M., Kriegeskorte, N., Storrs, K. R. & Mur, M. Deep convolutional neural networks outperform feature-based but not categorical models in explaining object similarity judgments. Front. Psychol. 8, 1726 (2017).

    Article  Google Scholar 

  47. Iordan, M. C., Giallanza, T., Ellis, C. T., Beckage, N. & Cohen, J. D. Context matters: recovering human semantic structure from machine learning analysis of large-scale text corpora. Preprint at arXiv https://arxiv.org/abs/1910.06954 (2019).

  48. Bauer, A. J. & Just, M. A. in The Oxford Handbook of Neurolinguistics (eds de Zubicaray, G. I. & Schiller, N. O.) 518–547 (Oxford Univ. Press, 2019).

  49. Binder, J. R. et al. Toward a brain-based componential semantic representation. Cogn. Neuropsychol. 33, 130–174 (2016).

    Article  Google Scholar 

  50. Huth, A. G., Nishimoto, S., Vu, A. T. & Gallant, J. L. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron 76, 1210–1224 (2012).

    Article  CAS  Google Scholar 

  51. Cichy, R. M., Kriegeskorte, N., Jozwik, K. M., van den Bosch, J. J. & Charest, I. The spatiotemporal neural dynamics underlying perceived similarity for real-world objects. NeuroImage 194, 12–24 (2019).

    Article  Google Scholar 

  52. Bankson, B. B., Hebart, M. N., Groen, I. I. A. & Baker, C. I. The temporal evolution of conceptual object representations revealed through models of behavior, semantics and deep neural networks. NeuroImage 178, 172–182 (2018).

    Article  CAS  Google Scholar 

  53. Zheng, C. Y., Pereira, F., Baker, C. I. & Hebart, M. N. Revealing interpretable object representations from human behavior. Preprint at arXiv https://arxiv.org/abs/1901.02915 (2019).

  54. Abadi, M. et al. TensorFlow: a system for large-scale machine learning. In 12th Symposium on Operating Systems Design and Implementation 265–283 (2016).

  55. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at arXiv https://arxiv.org/abs/1412.6980 (2015).

Download references

Acknowledgements

We thank A. Corriveau for help with the data collection in the laboratory experiment, L. Stoinksi and J. Perkuhn for help with finding public-domain images for this publication, and I. Charest, B. Love, A. Martin and P. McClure for helpful discussions and/or comments on the manuscript. This research was supported by the Intramural Research Program of the National Institutes of Health (grant nos ZIA-MH-002909 and ZIC-MH002968), under National Institute of Mental Health Clinical Study Protocol 93-M-1070 (NCT00001360), and by a research group grant awarded to M.N.H. by the Max Planck Society. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Authors and Affiliations

  1. Laboratory of Brain and Cognition, National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA

    Martin N. Hebart & Chris I. Baker

  2. Vision and Computational Cognition Group, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany

    Martin N. Hebart

  3. Machine Learning Core, National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA

    Charles Y. Zheng & Francisco Pereira

Authors
  1. Martin N. Hebart
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Charles Y. Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Francisco Pereira
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chris I. Baker
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

M.N.H. and C.I.B. conceived and designed the study. M.N.H. collected the data. C.Y.Z., M.N.H. and F.P. designed the computational model. M.N.H., C.Y.Z. and F.P. analysed the data. M.N.H., C.I.B., F.P. and C.Y.Z. wrote the manuscript and provided critical revisions.

Corresponding author

Correspondence to Martin N. Hebart.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Primary Handling Editor: Marike Schiffer.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Reproducibility of dimensions.

Reproducibility of dimensions in the chosen 49-dimensional embedding across 20 random initializations (see Extended Data Fig. 2 for a list of all dimension labels). Shaded areas reflect 95% confidence intervals.

Extended Data Fig. 2 Labels and word clouds for all 49 model dimensions.

Labels for all 49 dimensions, with respective word clouds reflecting the naming frequency across 20 participants. The dimensions appear to reflect both perceptual and conceptual properties of objects. A visual comparison between labels and word clouds indicates a generally good agreement between participant naming and the labels we provided for the dimensions.

Extended Data Fig. 3 Category-typicality correlations.

Detailed results of inferential statistical analyses correlating category-related dimensions with typicality of their category. One-sided p-values were generated using randomization tests and were controlled for false discovery rate (FDR) across multiple tests. 90% confidence intervals were used to complement one-sided tests.

Extended Data Fig. 4 Model performance and dimensionality as a function of training data size.

Model performance and dimensionality varied as a function of the amount of data used for training the model. Models were trained in steps of 100,000 trials. Six models with random initialization and random subsets of data were trained per step and all models applied to the same test data as in the main text, making it a total of 78 trained models. For each step, computation of up to two models did not complete on the compute server for technical reasons, making the total between 4 and 6 models per step. Results for each individual model and the average for each step are shown in the Figure. a. Model performance was already high for 100,000 trials as training data but increased with more data, saturating around the final model performance. b. Dimensionality increased steadily with the amount of training data.

Supplementary information

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hebart, M.N., Zheng, C.Y., Pereira, F. et al. Revealing the multidimensional mental representations of natural objects underlying human similarity judgements. Nat Hum Behav 4, 1173–1185 (2020). https://doi.org/10.1038/s41562-020-00951-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-020-00951-3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing