Abstract
Objects can be characterized according to a vast number of possible criteria (such as animacy, shape, colour and function), but some dimensions are more useful than others for making sense of the objects around us. To identify these core dimensions of object representations, we developed a data-driven computational model of similarity judgements for real-world images of 1,854 objects. The model captured most explainable variance in similarity judgements and produced 49 highly reproducible and meaningful object dimensions that reflect various conceptual and perceptual properties of those objects. These dimensions predicted external categorization behaviour and reflected typicality judgements of those categories. Furthermore, humans can accurately rate objects along these dimensions, highlighting their interpretability and opening up a way to generate similarity estimates from object dimensions alone. Collectively, these results demonstrate that human similarity judgements can be captured by a fairly low-dimensional, interpretable embedding that generalizes to external behaviour.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The learned embedding, triplet odd-one-out behavioural data for testing model performance, typicality scores, participant-generated dimension labels and dimension ratings are available at https://osf.io/z2784. The behavioural data used for training the model are available from the corresponding author upon request.
Code availability
To reproduce the relevant analyses and figures, the relevant MATLAB scripts and functions are available at https://osf.io/z2784. The computational modelling code to create an embedding is available from the corresponding author upon request.
References
Biederman, I. Recognition-by-components: a theory of human image understanding. Psychol. Rev. 94, 115–147 (1987).
Edelman, S. Representation is representation of similarities. Behav. Brain Sci. 21, 449–467 (1998).
Nosofsky, R. M. Attention, similarity, and the identification–categorization relationship. J. Exp. Psychol. Gen. 115, 39–57 (1986).
Goldstone, R. L. The role of similarity in categorization: providing a groundwork. Cognition 52, 125–157 (1994).
Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M. & Boyes-Braem, P. Basic objects in natural categories. Cognit. Psychol. 8, 382–439 (1976).
Hahn, U. & Chater, N. in Knowledge, Concepts and Categories (eds Lamberts, K. & Shanks, D.) 43–92 (Psychology Press, 1997).
Rips, L. J., Smith, E. E. & Medin, D. L. in The Oxford Handbook of Thinking and Reasoning (eds Holyoak, K. J. & Morrison, R. G.) 177–209 (Oxford Univ. Press, 2012).
Rogers, T. T. & McClelland, J. L. Semantic Cognition: A Parallel Distributed Processing Approach (MIT Press, 2004).
Goldstone, R. L. & Son, J. Y. in The Oxford Handbook of Thinking and Reasoning (eds Holyoak, K. J. & Morrison, R. G.) 155–176 (Oxford Univ. Press, 2012).
Kriegeskorte, N. & Kievit, R. A. Representational geometry: integrating cognition, computation, and the brain. Trends Cogn. Sci. 17, 401–412 (2013).
Caramazza, A. & Shelton, J. R. Domain-specific knowledge systems in the brain: the animate–inanimate distinction. J. Cogn. Neurosci. 10, 1–34 (1998).
Chao, L. L., Haxby, J. V. & Martin, A. Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects. Nat. Neurosci. 2, 913–919 (1999).
Konkle, T. & Oliva, A. Canonical visual size for real-world objects. J. Exp. Psychol. Hum. Percept. Perform. 37, 23–37 (2011).
Murphy, G. The Big Book of Concepts (MIT Press, 2004).
McRae, K., Cree, G. S., Seidenberg, M. S. & McNorgan, C. Semantic feature production norms for a large set of living and nonliving things. Behav. Res. Methods 37, 547–559 (2005).
Devereux, B. J., Tyler, L. K., Geertzen, J. & Randall, B. The Centre for Speech, Language and the Brain (CSLB) concept property norms. Behav. Res. Methods 46, 1119–1127 (2014).
Hebart, M. N. et al. THINGS: a database of 1,854 object concepts and more than 26,000 naturalistic object images. PLoS ONE 14, e0223792 (2019).
Tversky, A. Features of similarity. Psychol. Rev. 84, 327–352 (1977).
Barsalou, L. W. Context-independent and context-dependent information in concepts. Mem. Cognit. 10, 82–93 (1982).
Maddox, W. T. & Ashby, F. G. Comparing decision bound and exemplar models of categorization. Percept. Psychophys. 53, 49–70 (1993).
Hoyer, P. O. Modeling receptive fields with non-negative sparse coding. Neurocomputing 52, 547–552 (2003).
Murphy, B., Talukdar, P. & Mitchell, T. Learning effective and interpretable semantic models using non-negative sparse embedding. In Proc. of COLING 2012 1933–1950 (2012).
Shepard, R. N. Stimulus and response generalization: a stochastic model relating generalization to distance in psychological space. Psychometrika 22, 325–345 (1957).
Kobak, D. & Berens, P. The art of using t-SNE for single-cell transcriptomics. Nat. Commun. 10, 5416 (2019).
Shelton, J. R., Fouch, E. & Caramazza, A. The selective sparing of body part knowledge: a case study. Neurocase 4, 339–351 (1998).
Pedersen, T., Patwardhan, S. & Michelizzi, J. WordNet::Similarity—measuring the relatedness of concepts. In HLT-NAACL 2004: Demonstration Papers (eds Dumais, S. et al.) 38–41 (ACL Press, 2004).
Warrington, E. K. & Shallice, T. Category specific semantic impairments. Brain 107, 829–853 (1984).
Rips, L. J. in Similarity and Analogical Reasoning (eds Vosniadou, S. & Ortony, A.) 21–59 (Cambridge Univ. Press, 1989).
Smith, E. E. & Sloman, S. A. Similarity- versus rule-based categorization. Mem. Cognit. 22, 377–386 (1994).
Pilehvar, M. T. & Collier, N. De-conflated semantic representations. In 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP) 1680–1690 (2016).
Nosofsky, R. M., Sanders, C. A., Meagher, B. J. & Douglas, B. J. Toward the development of a feature-space representation for a complex natural category domain. Behav. Res. Methods 50, 530–556 (2018).
Nosofsky, R. M., Sanders, C. A., Meagher, B. J. & Douglas, B. J. Search for the missing dimensions: building a feature-space representation for a natural-science category domain. Comput. Brain Behav. 3, 13–33 (2020).
Keil, F. C. Constraints on knowledge and cognitive development. Psychol. Rev. 88, 187–227 (1981).
Shepard, R. N. The analysis of proximities: multidimensional scaling with an unknown distance function. Psychometrika 27, 125–140 (1962).
Torgerson, W. S. Multidimensional scaling: I. Theory and method. Psychometrika 17, 401–419 (1952).
Thurstone, L. L. Multiple factor analysis. Psychol. Rev. 38, 406–427 (1931).
Tranel, D., Logan, C. G., Frank, R. J. & Damasio, A. R. Explaining category-related effects in the retrieval of conceptual and lexical knowledge for concrete entities: operationalization and analysis of factors. Neuropsychologia 35, 1329–1339 (1997).
Shepard, R. N. & Arabie, P. Additive clustering: representation of similarities as combinations of discrete overlapping properties. Psychol. Rev. 86, 87–123 (1979).
Navarro, D. J. & Lee, M. D. Common and distinctive features in stimulus similarity: a modified version of the contrast model. Psychon. Bull. Rev. 11, 961–974 (2004).
Carlson, T. A., Ritchie, J. B., Kriegeskorte, N., Durvasula, S. & Ma, J. Reaction time for object categorization is predicted by representational distance. J. Cogn. Neurosci. 26, 132–142 (2013).
Yee, E. & Thompson-Schill, S. L. Putting concepts into context. Psychon. Bull. Rev. 23, 1015–1027 (2016).
Charest, I., Kievit, R. A., Schmitz, T. W., Deca, D. & Kriegeskorte, N. Unique semantic space in the brain of each beholder predicts perceived similarity. Proc. Natl Acad. Sci. USA 111, 14565–14570 (2014).
De Haas, B., Iakovidis, A. L., Schwarzkopf, D. S. & Gegenfurtner, K. R. Individual differences in visual salience vary along semantic dimensions. Proc. Natl Acad. Sci. USA 116, 11687–11692 (2019).
Peterson, J. C., Abbott, J. T. & Griffiths, T. L. Evaluating (and improving) the correspondence between deep neural networks and human representations. Cogn. Sci. 42, 2648–2669 (2018).
Rajalingham, R. et al. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks. J. Neurosci. 38, 7255–7269 (2018).
Jozwik, K. M., Kriegeskorte, N., Storrs, K. R. & Mur, M. Deep convolutional neural networks outperform feature-based but not categorical models in explaining object similarity judgments. Front. Psychol. 8, 1726 (2017).
Iordan, M. C., Giallanza, T., Ellis, C. T., Beckage, N. & Cohen, J. D. Context matters: recovering human semantic structure from machine learning analysis of large-scale text corpora. Preprint at arXiv https://arxiv.org/abs/1910.06954 (2019).
Bauer, A. J. & Just, M. A. in The Oxford Handbook of Neurolinguistics (eds de Zubicaray, G. I. & Schiller, N. O.) 518–547 (Oxford Univ. Press, 2019).
Binder, J. R. et al. Toward a brain-based componential semantic representation. Cogn. Neuropsychol. 33, 130–174 (2016).
Huth, A. G., Nishimoto, S., Vu, A. T. & Gallant, J. L. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron 76, 1210–1224 (2012).
Cichy, R. M., Kriegeskorte, N., Jozwik, K. M., van den Bosch, J. J. & Charest, I. The spatiotemporal neural dynamics underlying perceived similarity for real-world objects. NeuroImage 194, 12–24 (2019).
Bankson, B. B., Hebart, M. N., Groen, I. I. A. & Baker, C. I. The temporal evolution of conceptual object representations revealed through models of behavior, semantics and deep neural networks. NeuroImage 178, 172–182 (2018).
Zheng, C. Y., Pereira, F., Baker, C. I. & Hebart, M. N. Revealing interpretable object representations from human behavior. Preprint at arXiv https://arxiv.org/abs/1901.02915 (2019).
Abadi, M. et al. TensorFlow: a system for large-scale machine learning. In 12th Symposium on Operating Systems Design and Implementation 265–283 (2016).
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at arXiv https://arxiv.org/abs/1412.6980 (2015).
Acknowledgements
We thank A. Corriveau for help with the data collection in the laboratory experiment, L. Stoinksi and J. Perkuhn for help with finding public-domain images for this publication, and I. Charest, B. Love, A. Martin and P. McClure for helpful discussions and/or comments on the manuscript. This research was supported by the Intramural Research Program of the National Institutes of Health (grant nos ZIA-MH-002909 and ZIC-MH002968), under National Institute of Mental Health Clinical Study Protocol 93-M-1070 (NCT00001360), and by a research group grant awarded to M.N.H. by the Max Planck Society. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
Author information
Authors and Affiliations
Contributions
M.N.H. and C.I.B. conceived and designed the study. M.N.H. collected the data. C.Y.Z., M.N.H. and F.P. designed the computational model. M.N.H., C.Y.Z. and F.P. analysed the data. M.N.H., C.I.B., F.P. and C.Y.Z. wrote the manuscript and provided critical revisions.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Peer review information Primary Handling Editor: Marike Schiffer.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 Reproducibility of dimensions.
Reproducibility of dimensions in the chosen 49-dimensional embedding across 20 random initializations (see Extended Data Fig. 2 for a list of all dimension labels). Shaded areas reflect 95% confidence intervals.
Extended Data Fig. 2 Labels and word clouds for all 49 model dimensions.
Labels for all 49 dimensions, with respective word clouds reflecting the naming frequency across 20 participants. The dimensions appear to reflect both perceptual and conceptual properties of objects. A visual comparison between labels and word clouds indicates a generally good agreement between participant naming and the labels we provided for the dimensions.
Extended Data Fig. 3 Category-typicality correlations.
Detailed results of inferential statistical analyses correlating category-related dimensions with typicality of their category. One-sided p-values were generated using randomization tests and were controlled for false discovery rate (FDR) across multiple tests. 90% confidence intervals were used to complement one-sided tests.
Extended Data Fig. 4 Model performance and dimensionality as a function of training data size.
Model performance and dimensionality varied as a function of the amount of data used for training the model. Models were trained in steps of 100,000 trials. Six models with random initialization and random subsets of data were trained per step and all models applied to the same test data as in the main text, making it a total of 78 trained models. For each step, computation of up to two models did not complete on the compute server for technical reasons, making the total between 4 and 6 models per step. Results for each individual model and the average for each step are shown in the Figure. a. Model performance was already high for 100,000 trials as training data but increased with more data, saturating around the final model performance. b. Dimensionality increased steadily with the amount of training data.
Supplementary information
Rights and permissions
About this article
Cite this article
Hebart, M.N., Zheng, C.Y., Pereira, F. et al. Revealing the multidimensional mental representations of natural objects underlying human similarity judgements. Nat Hum Behav 4, 1173–1185 (2020). https://doi.org/10.1038/s41562-020-00951-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41562-020-00951-3
This article is cited by
-
Large language models predict human sensory judgments across six modalities
Scientific Reports (2024)
-
Distributed representations of behaviour-derived object dimensions in the human visual system
Nature Human Behaviour (2024)
-
Computational reconstruction of mental representations using human behavior
Nature Communications (2024)
-
Centering cognitive neuroscience on task demands and generalization
Nature Neuroscience (2024)
-
Movement trajectories as a window into the dynamics of emerging neural representations
Scientific Reports (2024)