Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Semantic projection recovers rich human knowledge of multiple object features from word embeddings

Nature Human Behaviour volume 6, pages 975–987 (2022)Cite this article

Subjects

Abstract

How is knowledge about word meaning represented in the mental lexicon? Current computational models infer word meanings from lexical co-occurrence patterns. They learn to represent words as vectors in a multidimensional space, wherein words that are used in more similar linguistic contexts—that is, are more semantically related—are located closer together. However, whereas inter-word proximity captures only overall relatedness, human judgements are highly context dependent. For example, dolphins and alligators are similar in size but differ in dangerousness. Here, we use a domain-general method to extract context-dependent relationships from word embeddings: ‘semantic projection’ of word-vectors onto lines that represent features such as size (the line connecting the words ‘small’ and ‘big’) or danger (‘safe’ to ‘dangerous’), analogous to ‘mental scales’. This method recovers human judgements across various object categories and properties. Thus, the geometry of word embeddings explicitly represents a wealth of context-dependent world knowledge.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Schematic illustration of semantic projection.
Fig. 2: Semantic projection predicts human judgements: sample cases.
Fig. 3: Semantic projection predicts human judgements across diverse categories and features.
Fig. 4: Semantic projection predicts human judgements: detailed results.
Fig. 5: Distribution of evaluation scores for semantic projection.
Fig. 6: The fit of semantic projection to human ratings is not driven by outliers.

Similar content being viewed by others

Data availability

All behavioural data and GloVe vectors as reported in the paper are available on the Open Science Framework (https://osf.io/5r2sz/). The full database of GloVe vectors (including many words not used in this study) is available for download from https://nlp.stanford.edu/projects/glove/.

Code availability

Custom MATLAB codes for replicating the analyses are available on the Open Science Framework page referenced above (https://osf.io/5r2sz/). The outputs of these codes include all data visualized in Figs. 2–6.

References

  1. Marr, D. in Vision: A Computational Investigation into the Human Representation and Processing of Visual Information (ed. Marr, D.) 8–38 (MIT Press, 1982).

  2. Goldberg, A. E. Constructions: A Construction Grammar Approach to Argument Structure (Univ. of Chicago Press, 1995).

  3. Jackendoff, R. Foundation of Language: Brain, Meaning, Grammar, Evolution (Oxford Univ. Press, 2002).

  4. Murphy, G. The Big Book of Concepts (MIT Press, 2004).

  5. Jackendoff, R. A User’s Guide to Thought and Meaning (Oxford Univ. Press, 2012).

  6. Steinberg, D. D. & Jakobovits, L. A. Semantics: An Interdisciplinary Reader in Philosophy, Linguistics and Psychology. (Cambridge Univ. Press, 1971).

  7. Richards, M. M. in Language Development, Vol. 1: Syntax and Semantics Vol. 1 (ed. S. Kuczaj) 365–396 (Routledge, 1982).

  8. Pinker, S. & Levin, B. Lexical and Conceptual Semantics (MIT Press, 1991).

  9. Pustejovsky, J. Semantics and the Lexicon Vol. 49 (Springer, 2012).

  10. Quillian, M. R. Semantic Memory. PhD thesis, Carnegie Intitute of Technology (1966).

  11. Tulving, E. in Organization of Memory Vol. 1 (eds Tulving E. & Donaldson W.) 381–403 (Academic, 1972).

  12. Gleitman, L. & Papafragou, A. in The Oxford Handbook of Cognitive Psychology (ed D. Resiberg) 255–275 (Oxford Univ. Press, 2013).

  13. Jackendoff, R. Parts and boundaries. Cognition 41, 9–45 (1991).

    Article  CAS  PubMed  Google Scholar 

  14. Smith, N. J. & Levy, R. The effect of word predictability on reading time is logarithmic. Cognition 128, 302–319 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  15. Skarabela, B., Ota, M., O’Connor, R. & Arnon, I. ‘Clap your hands’ or ‘take your hands’? One-year-olds distinguish between frequent and infrequent multiword phrases. Cognition 211, 104612 (2021).

    Article  PubMed  Google Scholar 

  16. Monsalve, I. F., Frank, S. L. & Vigliocco, G. in Proc. 13th Conference of the European Chapter of the Association for Computational Linguistics, 398–408 (Association for Computational Linguistics, 2012).

  17. Frank, S. & Thompson, R. Early effects of word surprisal on pupil size during reading. In Proc. 34th Annual Conference of the Cognitive Science Society Vol. 34 (eds Miyake, N. et al.) 1554–1559 (Cognitive Science Society, 2012).

  18. Willems, R. M., Frank, S. L., Nijhof, A. D., Hagoort, P. & Van den Bosch, A. Prediction during natural language comprehension. Cereb. Cortex 26, 2506–2516 (2015).

    Article  PubMed  Google Scholar 

  19. McDonald, S. & Ramscar, M. Testing the distributional hypothesis: the influence of context on judgements of semantic similarity. In Proc. 23rd Annual Conference of the Cognitive Science Society https://escholarship.org/uc/item/6959p7b0 (2001).

  20. Ellis, N. C. & Simpson-Vlach, R. Formulaic language in native speakers: triangulating psycholinguistics, corpus linguistics, and education. Corpus Linguist. Linguistic Theory 5, 61–78 (2009).

    Article  Google Scholar 

  21. Louwerse, M. M. Embodied relations are encoded in language. Psychonomic Bull. Rev. 15, 838–844 (2008).

    Article  Google Scholar 

  22. De Saussure, F. Course in General Linguistics (Columbia Univ. Press, 2011).

  23. Wittgenstein, L. Philosophical Investigations. §114–115 (Wiley-Blackwell, 2010).

  24. Harris, Z. S. Distributional structure. Word 10, 146–162 (1954).

    Article  Google Scholar 

  25. Firth, J. R. in Studies in Linguistic Analysis Special volume of the Philological Society (ed. Firth, J. R.) 1–31 (Blackwell, 1957).

  26. Miller, G. A. & Charles, W. G. Contextual correlates of semantic similarity. Lang. Cogn. Process. 6, 1–28 (1991).

    Article  Google Scholar 

  27. Sahlgren, M. The distributional hypothesis. Ital. J. Disabil. Stud. 20, 33–53 (2008).

    Google Scholar 

  28. Bengio, Y., Ducharme, R., Vincent, P. & Jauvin, C. A neural probabilistic language model. J. Mach. Learn. Res. 3, 1137–1155 (2003).

    Google Scholar 

  29. Huang, E. H., Socher, R., Manning, C. D. & Ng, A. Y. in Proc. 50th Annual Meeting of the Association for Computational Linguistics Vol. 1: Long Papers, 873–882 (Association for Computational Linguistics, 2012).

  30. Collobert, R. et al. Natural language processing (almost) from scratch. J. Mach. Learn. Res. 12, 2493–2537 (2011).

    Google Scholar 

  31. Lenci, A. Distributional semantics in linguistic and cognitive research. Ital. J. Ling. 20, 1–31 (2008).

    Google Scholar 

  32. Erk, K. Vector space models of word meaning and phrase meaning: a survey. Lang. Linguist. Compass 6, 635–653 (2012).

    Article  Google Scholar 

  33. Clark, S. in Handbook of Contemporary Semantics (eds Lappin S. & Fox C.) 493–522 (Blackwell, 2015).

  34. Turney, P. D. & Pantel, P. From frequency to meaning: vector space models of semantics. J. Artif. Intell. Res. 37, 141–188 (2010).

    Article  Google Scholar 

  35. Baroni, M., Dinu, G. & Kruszewski, G. in Proc. 52nd Annual Meeting of the Association for Computational Linguistics Vol. 1: Long Papers, 238–247 (Association for Computational Linguistics, 2014).

  36. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. & Dean, J. in Proc. 26th International Conference on Neural Information Processing Systems, 3111–3119 (Curran Associates, Inc., 2013).

  37. Pennington, J., Socher, R. & Manning, C. in Proc. 2014 Conference on Empirical Methods in Natural Language Processing, 1532–1543 (Association for Computational Linguistics, 2014).

  38. Pereira, F., Gershman, S., Ritter, S. & Botvinick, M. A comparative evaluation of off-the-shelf distributed semantic representations for modelling behavioural data. Cogn. Neuropsychol. 33, 175–190 (2016).

    Article  PubMed  Google Scholar 

  39. Levy, O. & Goldberg, Y. in Advances in Neural Information Processing Systems, 2177–2185 (Curran Associates, Inc., 2014).

  40. Lu, H., Wu, Y. N. & Holyoak, K. J. Emergence of analogy from relation learning. Proc. Natl Acad. Sci. U. S. A. 116, 4176–4181 (2019).

  41. Rogers, A., Drozd, A. & Li, B. in Proc. 6th Joint Conference on Lexical and Computational Semantics, 135–148 (Association for Computational Linguistics, 2017).

  42. Peterson, J. C., Chen, D. & Griffiths, T. L. Parallelograms revisited: exploring the limitations of vector space models for simple analogies. Cognition 205, 104440 (2020).

    Article  PubMed  Google Scholar 

  43. Osgood, C. E. The nature and measurement of meaning. Psychol. Bull. 49, 197 (1952).

    Article  CAS  PubMed  Google Scholar 

  44. Osgood, C. E. Semantic differential technique in the comparative study of cultures. Am. Anthropol. 66, 171–200 (1964).

    Article  Google Scholar 

  45. Kozima, H. & Ito, A. Context-sensitive measurement of word distance by adaptive scaling of a semantic space. In Proc. RANLP-95, 161–168 (John Benjamins Publishing Company, 1995).

  46. Bates, D., Mächler, M., Bolker, B. & Walker, S. Fitting linear mixed-effects models using lme4. J. Stat. Softw. 67, 1–48 (2015).

    Article  CAS  Google Scholar 

  47. Peters, M. E. et al. in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2227–2237 (Association for Computational Linguistics, 2018).

  48. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Vol. 1: Long and Short Papers, 4171–4186 (Association for Computational Linguistics, 2019).

  49. Vaswani, A. et al. Attention is all you need. In Proc. Advances in Neural Information Processing Systems 30 (NIPS 2017), 5998–6008 (Curran Associates, Inc., 2017).

  50. Huebner, P. A. & Willits, J. A. Structured semantic knowledge can emerge automatically from predicting word sequences in child-directed speech. Front. Psychol. 9, 133 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  51. Unger, L. & Fisher, A. V. The emergence of richly organized semantic knowledge from simple statistics: a synthetic review. Dev. Rev. 60, 100949 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  52. Caliskan, A., Bryson, J. J. & Narayanan, A. Semantics derived automatically from language corpora contain human-like biases. Science 356, 183–186 (2017).

    Article  CAS  PubMed  Google Scholar 

  53. Lewis, M. & Lupyan, G. Gender stereotypes are reflected in the distributional structure of 25 languages. Nat. Hum. Behav. 4, 1021–1028 (2020).

    Article  PubMed  Google Scholar 

  54. Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V. & Kalai, A. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Proc. 30th International Conference on Neural Information Processing Systems (NIPS 2016), 4356–4364 (Curran Associates, Inc., 2016).

  55. Kozlowski, A. C., Taddy, M. & Evans, J. A. The geometry of culture: analyzing the meanings of class through word embeddings. Am. Sociol. Rev. 84, 905–949 (2019).

    Article  Google Scholar 

  56. Tshitoyan, V. et al. Unsupervised word embeddings capture latent knowledge from materials science literature. Nature 571, 95–98 (2019).

    Article  CAS  PubMed  Google Scholar 

  57. Johns, B. T. & Jones, M. N. Perceptual inference through global lexical similarity. Top. Cog. Sci. 4, 103–120 (2012).

    Article  Google Scholar 

  58. Herbelot, A. & Vecchi, E. M. in Proc. 2015 Conference on Empirical Methods in Natural Language Processing, 22–32 (Association for Computational Linguistics, 2015).

  59. Gupta, A., Boleda, G., Baroni, M. & Padó, S. in Proc. 2015 Conference on Empirical Methods in Natural Language Processing, 12–21 (Association for Computational Linguistics, 2015).

  60. Utsumi, A. Exploring what is encoded in distributional word vectors: a neurobiologically motivated analysis. Cogn. Sci. 44, e12844 (2020).

    Article  PubMed  Google Scholar 

  61. Ichien, N., Lu, H. & Holyoak, K. J. Predicting patterns of similarity among abstract semantic relations. J. Exp. Psychol. Learning Memory Cogn. https://doi.org/10.1037/xlm0001010 (2021).

  62. Iordan, M. C., Giallanza, T., Ellis, C. T., Beckage, N. M. & Cohen, J. D. Context matters: recovering human semantic structure from machine learning analysis of large-scale text corpora. Cogn. Sci. 46, e13085 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  63. Laurence, S. & Margolis, E. in Concepts: Core Readings (eds Laurence, S. & Margolis, E.) 3–81 (MIT Press, 1999).

  64. Markman, A. B. Knowledge Representation (Lawrence Erlbaum, 2013).

  65. Mahon, B. Z. & Hickok, G. Arguments about the nature of concepts: symbols, embodiment, and beyond. Psychon. Bull. Rev. 23, 941–958 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  66. Yee, E., Jones, M. & McRae, K. in Stevens’ Handbook of Experimental Psychology, Memory and Cognitive Processes Vol. 2 (ed Wixted J.) (Wiley, 2014).

  67. Rosch, E. & Mervis, C. B. Family resemblances: studies in the internal structure of categories. Cogn. Psychol. 7, 573–605 (1975).

    Article  Google Scholar 

  68. Smith, E. E. & Medin, D. L. Categories and Concepts (Harvard Univ. Press, 1981).

  69. Rumelhart, D. & Ortony, A. in Schooling and the Acquisition of Knowledge (eds Anderson R. C., Spiro R. J., & Montague W. E.) 99–135 (Lawrence Erlbaum, 1977).

  70. Gopnik, A., Meltzoff, A. N. & Bryant, P. Words, Thoughts, and Theories, Vol. 1 (MIT Press, 1997).

  71. Gopnik, A. in Chomsky and His Critics (eds Antony L. & Hornstein N.) 238–254 (Blackwell, 2003).

  72. Medin, D. L. & Schaffer, M. M. Context theory of classification learning. Psych. Rev. 85, 207 (1978).

    Article  Google Scholar 

  73. Poesio, M. & Almuhareb, A. in Proc. Association for Computational Linguistics SIGLEX Workshop on Deep Lexical Acquisition, 18–27 (Association for Computational Linguistics, 2005).

  74. Barbu, E. in Proc. ESSLLI Workshop on Distributional Lexical Semantics, 9–16 (Association for Logic, Language and Information, 2008).

  75. Baroni, M. & Lenci, A. in Proc. Workshop on Geometrical Models of Natural Language Semantics, 1–8 (Association for Computational Linguistics, 2009).

  76. Baroni, M., Murphy, B., Barbu, E. & Poesio, M. Strudel: a corpus-based semantic model based on properties and types. Cogn. Sci. 34, 222–254 (2010).

    Article  PubMed  Google Scholar 

  77. Rubinstein, D., Levi, E., Schwartz, R. & Rappoport, A. in Proc. 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing Vol. 2: Short Papers, 726–730 (Association for Computational Linguistics, 2015).

  78. Kelly, C., Devereux, B. & Korhonen, A. Automatic extraction of property norm‐like data from large text corpora. Cogn. Sci. 38, 638–682 (2014).

    Article  PubMed  Google Scholar 

  79. Lupyan, G. & Lewis, M. From words-as-mappings to words-as-cues: the role of language in semantic knowledge. Lang. Cogn. Neurosci. 34, 1319–1337 (2019).

    Article  Google Scholar 

  80. Rumelhart, D. E. in Metaphor and Thought (ed. Andrew Ortony) 71–82 (Cambridge Univ. Press, 1979).

  81. Landauer, T. K. & Dumais, S. T. A solution to Plato’s problem: the latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychol. Rev. 104, 211–240 (1997).

    Article  Google Scholar 

  82. Elman, J. L. An alternative view of the mental lexicon. Trends Cogn. Sci. 8, 301–306 (2004).

    Article  PubMed  Google Scholar 

  83. Elman, J. L. On the meaning of words and dinosaur bones: lexical knowledge without a lexicon. Cogn. Sci. 33, 547–582 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  84. Lupyan, G. & Bergen, B. How language programs the mind. Top. Cogn. Sci. 8, 408–424 (2016).

    Article  PubMed  Google Scholar 

  85. Clifton, C., Frazier, L. & Connine, C. Lexical expectations in sentence comprehension. J. Verbal Learn. Verbal Behav. 23, 696–708 (1984).

    Article  Google Scholar 

  86. MacDonald, M. C., Pearlmutter, N. J. & Seidenberg, M. S. The lexical nature of syntactic ambiguity resolution. Psychol. Rev. 101, 676–703 (1994).

    Article  CAS  PubMed  Google Scholar 

  87. Trueswell, J. C., Tanenhaus, M. K. & Garnsey, S. M. Semantic influences on parsing: use of thematic role information in syntactic ambiguity resolution. J. Mem. Lang. 33, 285–318 (1994).

    Article  Google Scholar 

  88. Garnsey, S. M., Pearlmutter, N. J., Myers, E. & Lotocky, M. A. The contributions of verb bias and plausibility to the comprehension of temporarily ambiguous sentences. J. Mem. Lang. 37, 58–93 (1997).

    Article  Google Scholar 

  89. Hale, J. in Proc. Second Meeting of the North American Chapter of the Association for Computational Linguistics on Language Technologies, 1–8 (Association for Computational Linguistics, 2001).

  90. Traxler, M. J., Morris, R. K. & Seely, R. E. Processing subject and object relative clauses: evidence from eye movements. J. Mem. Lang. 47, 69–90 (2002).

    Article  Google Scholar 

  91. Gennari, S. P. & MacDonald, M. C. Semantic indeterminacy in object relative clauses. J. Mem. Lang. 58, 161–187 (2008).

    Article  PubMed  PubMed Central  Google Scholar 

  92. Levy, R. P. Expectation-based syntactic comprehension. Cognition 106, 1126–1177 (2008).

    Article  PubMed  Google Scholar 

  93. Marmor, G. S. Age at onset of blindness and the development of the semantics of color names. J. Exp. Child Psych. 25, 267–278 (1978).

    Article  CAS  Google Scholar 

  94. Landau, B. & Gleitman, L. R. Language and Experience: Evidence from the Blind Child, Vol. 8 (Harvard Univ. Press, 2009).

  95. Shepard, R. N. & Cooper, L. A. Representation of colors in the blind, color-blind, and normally sighted. Psychol. Sci. 3, 97–104 (1992).

    Article  Google Scholar 

  96. Noppeney, U., Friston, K. J. & Price, C. J. Effects of visual deprivation on the organization of the semantic system. Brain 126, 1620–1627 (2003).

    Article  PubMed  Google Scholar 

  97. Bedny, M., Caramazza, A., Pascual-Leone, A. & Saxe, R. Typical neural representations of action verbs develop without vision. Cereb. Cortex 22, 286–293 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  98. Bedny, M., Koster-Hale, J., Elli, G., Yazzolino, L. & Saxe, R. There’s more to “sparkle” than meets the eye: knowledge of vision and light verbs among congenitally blind and sighted individuals. Cognition 189, 105–115 (2019).

    Article  PubMed  Google Scholar 

  99. Louwerse, M. & Connell, L. A taste of words: linguistic context and perceptual simulation predict the modality of words. Cogn. Sci. 35, 381–398 (2011).

    Article  PubMed  Google Scholar 

  100. Baroni, M. & Lenci, A. Concepts and properties in word spaces. Ital. J. Ling. 20, 55–88 (2008).

    Google Scholar 

  101. Andrews, M., Vigliocco, G. & Vinson, D. Integrating experiential and distributional data to learn semantic representations. Psych. Rev. 116, 463–498 (2009).

    Article  Google Scholar 

  102. Riordan, B. & Jones, M. N. Redundancy in perceptual and linguistic experience: comparing feature‐based and distributional models of semantic representation. Top. Cogn. Sci. 3, 303–345 (2011).

    Article  PubMed  Google Scholar 

  103. Hill, F., Reichart, R. & Korhonen, A. Simlex-999: eEvaluating semantic models with (genuine) similarity estimation. Comput. Linguist. 41, 665–695 (2015).

    Article  Google Scholar 

  104. Kim, J. S., Elli, G. V. & Bedny, M. Knowledge of animal appearance among sighted and blind adults. Proc. Natl Acad. Sci. U. S. A. 116, 11213–11222 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  105. Kim, J. S., Elli, G. V. & Bedny, M. Reply to Ostarek et al.: Language, but not co-occurrence statistics, is useful for learning animal appearance. Proc. Natl Acad. Sci. U. S. A. 116, 21974–21975 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  106. Kim, J. S., Elli, G. V. & Bedny, M. Reply to Lewis et al.: Inference is key to learning appearance from language, for humans and distributional semantic models alike. Proc. Natl Acad. Sci. U. S. A. 116, 19239–19240 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  107. Ostarek, M., Van Paridon, J. & Montero-Melis, G. Sighted people’s language is not helpful for blind individuals’ acquisition of typical animal colors. Proc. Natl Acad. Sci. U. S. A. 116, 21972–21973 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  108. Lewis, M., Zettersten, M. & Lupyan, G. Distributional semantics as a source of visual knowledge. Proc. Natl Acad. Sci. U. S. A. 116, 19237–19238 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  109. Majid, A. et al. Differential coding of perception in the world’s languages. Proc. Natl Acad. Sci. U. S. A. 115, 11369–11376 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  110. Clark, E. V. in Cognitive Development and the Acquisition of Language (ed. Moskowitz B. A.) 223–260 (Academic, 1973).

  111. Binder, J. R. et al. Toward a brain-based componential semantic representation. Cogn. Neuropsychol. 33, 130–174 (2016).

    Article  PubMed  Google Scholar 

  112. Barsalou, L. W. & Sewell, D. R. Contrasting the representation of scripts and categories. J. Mem. Lang. 24, 646–665 (1985).

    Article  Google Scholar 

  113. Tanaka, J. W. & Taylor, M. Object categories and expertise: is the basic level in the eye of the beholder? Cogn. Psych. 23, 457–482 (1991).

    Article  Google Scholar 

  114. Baroni, M. & Zamparelli, R. in Proc. 2010 Conference on Empirical Methods in Natural Language Processing, 1183–1193 (Association for Computational Linguistics, 2010).

  115. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S. & Dean, J. in NeurIPS (prev. NIPS), 3111–3119 (Curran Associates, Inc., 2013).

  116. Mahowald, K., Isola, P., Fedorenko, E., Gibson, E. & Oliva, A. Memorable words are monogamous: the role of synonymy and homonymy in word recognition memory. Preprint at PsyArxiv https://psyarxiv.com/p6kv9/ (2018).

  117. Paivio, A., Yuille, J. C. & Madigan, S. A. Concreteness, imagery, and meaningfulness values for 925 nouns. J. Exp. Psychol. 76, 1–25 (1968).

    Article  Google Scholar 

  118. Battig, W. F. & Montague, W. E. Category norms of verbal items in 56 categories A replication and extension of the Connecticut category norms. J. Exp. Psychol. 80, 1–46 (1969).

    Article  Google Scholar 

  119. Cree, G. S. & McRae, K. Analyzing the factors underlying the structure and computation of the meaning of chipmunk, cherry, chisel, cheese, and cello (and many other such concrete nouns). J. Exp. Psychol. Gen. 132, 163–201 (2003).

    Article  PubMed  Google Scholar 

  120. McRae, K., Cree, G. S., Seidenberg, M. S. & McNorgan, C. Semantic feature production norms for a large set of living and nonliving things. Behav. Res. Methods 37, 547–559 (2005).

    Article  PubMed  Google Scholar 

  121. Pereira, F., Botvinick, M. & Detre, G. Using Wikipedia to learn semantic feature representations of concrete concepts in neuroimaging experiments. Artif. Intell. 194, 240–252 (2013).

    Article  PubMed  Google Scholar 

  122. Nelson, D. L., McEvoy, C. L. & Schreiber, T. A. The University of South Florida free association, rhyme, and word fragment norms. Behav. Res. Methods 36, 402–407 (2004).

    Article  Google Scholar 

  123. Brysbaert, M. & New, B. Moving beyond Kučera and Francis: a critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Beh. Res. Methods 41, 977–990 (2009).

    Article  Google Scholar 

  124. Buhrmester, M., Kwang, T. & Gosling, S. D. Amazon’s Mechanical Turk: a new source of inexpensive, yet high-quality, data? Perspect. Psychol. Sci. 6, 3–5 (2011).

    Article  PubMed  Google Scholar 

  125. Silver, N. C. & Dunlap, W. P. Averaging correlation coefficients: should Fisher’s z transformation be used? J. Appl. Psychol. 72, 146 (1987).

    Article  Google Scholar 

  126. Benjamini, Y. & Yekutieli, D. The control of the false discovery rate in multiple testing under dependency. Ann. Stat. 29, 1165–1188 (2001).

    Article  Google Scholar 

Download references

Acknowledgements

F.P. was supported by the National Institute of Mental Health Intramural Research Program ZIC-MH002968. E.F. was supported by R01 awards DC016607 and DC016950, U01 award NS6945189 and funds from the McGovern Institute for Brain Research, the Department of Brain and Cognitive Sciences and the Simons Center for the Social Brain. This work was partially supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Air Force Research Laboratory (AFRL), under contract FA8650-14-C-7358 (to I.A.B., F.P. and E.F.). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. The authors are responsible for all aspects of the study, and views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of ODNI, IARPA, AFRL or the US Government. The US Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation thereon.

Author information

Author notes

  1. These authors contributed equally: Gabriel Grand and Idan Asher Blank.

  2. These authors jointly supervised this work: Francisco Pereira and Evelina Fedorenko.

Authors and Affiliations

  1. Department of Electrical Engineering and Computer Science, MIT, Cambridge, MA, USA

    Gabriel Grand

  2. Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA

    Gabriel Grand

  3. Department of Psychology, UCLA, Los Angeles, CA, USA

    Idan Asher Blank

  4. Department of Linguistics, UCLA, Los Angeles, CA, USA

    Idan Asher Blank

  5. National Institute of Mental Health, Bethesda, MD, USA

    Francisco Pereira

  6. Department of Brain and Cognitive Sciences, MIT, Cambridge, MA, USA

    Evelina Fedorenko

  7. McGovern Institute for Brain Research, MIT, Cambridge, MA, USA

    Evelina Fedorenko

Authors
  1. Gabriel Grand
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Idan Asher Blank
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Francisco Pereira
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Evelina Fedorenko
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

G.G. and I.A.B. are co-first authors. F.P. and E.F. are co-senior authors. Conceptualization: G.G. and I.A.B. Methodology (behavioural experiments): G.G., F.P. and E.F. Behavioural data collection: G.G. Data curation: I.A.B. Model implementation: G.G. and I.A.B. Statistical analysis: I.A.B. Visualization: I.A.B. Writing, original draft: I.A.B. Writing, review and editing: G.G., I.A.B., F.P. and E.F. Funding acquisition: F.P. and E.F. Supervision: F.P. and E.F.

Corresponding author

Correspondence to Idan Asher Blank.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Human Behaviour thanks Gary Lupyan, Katrin Erk and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Correspondence between human judgments and semantic projection using FastText.

Conventions are the same as in Fig. 3 in the manuscript. Descriptive statistics across all tested pairs: (a) Pearson’s correlation: med = 0.41 (CI95 = 0.29-0.50, IQR = 0.26-0.57), adjusted med = 0.44 (CI95 = 0.35-0.53, IQR = 0.29-0.60). (b) OCp: med = 64% (CI95 = 61-68%, IQR = 57-73%), adjusted med = 73% (CI95 = 70-78%, IQR = 67-81%).

Extended Data Fig. 2 Correspondence between human judgments and semantic projection using word2vec.

Conventions are the same as in Fig. 3 in the manuscript. Descriptive statistics across all tested pairs: (a) Pearson’s correlation: med = 0.33 (CI95 = 0.27-0.40, IQR = 0.22-0.44), adjusted med = 0.35 (CI95 = 0.28-0.43, IQR = 0.24-0.47). (b) OCp: med = 62% (CI95 = 57-65%, IQR = 55-67%), adjusted med = 71% (CI95 = 65-74%, IQR = 63-78%).

Extended Data Fig. 3 Correspondence between human judgments and semantic projection using ELMo.

Conventions are the same as in Fig. 3 in the manuscript. Descriptive statistics across all tested pairs: (a) Pearson’s correlation: med = 0.26 (CI95 = 0.20-0.36, IQR = 0.14-0.43), adjusted med = 0.31 (CI95 = 0.21-0.41, IQR = 0.15-0.45). (b) OCp: med = 59% (CI95 = 57-63%, IQR = 55-66%), adjusted med = 70% (CI95 = 65-73%, IQR = 63-76%).

Extended Data Fig. 4 Correspondence between human judgments and semantic projection using BERT.

Conventions are the same as in Fig. 3 in the manuscript. Descriptive statistics across all tested pairs: (a) Pearson’s correlation: med = 0.42 (CI95 = 0.35-0.47, IQR = 0.20-0.54), adjusted med = 0.44 (CI95 = 0.40-0.50, IQR = 0.25-0.57). (b) OCp: med = 65% (CI95 = 62-67%, IQR = 56-72%), adjusted med = 74% (CI95 = 72-76%, IQR = 67-80%).

Extended Data Fig. 5 Detailed results of semantic projection using FastText.

Conventions are the same as in Fig. 4 in the manuscript.

Extended Data Fig. 6 Detailed results of semantic projection using word2vec.

Conventions are the same as in Fig. 4 in the manuscript.

Extended Data Fig. 7 Detailed results of semantic projection using ELMo.

Conventions are the same as in Fig. 4 in the manuscript.

Extended Data Fig. 8 Detailed results of semantic projection using BERT.

Conventions are the same as in Fig. 4 in the manuscript.

Extended Data Fig. 9 Evaluating how well different word embeddings capture conceptual category structure.

Each matrix shows Pearson’s correlations between all pairs of word vectors for all items used in our study, grouped by category (indicated on the y-axis), for a different embedding. Color corresponds to correlation strength, with dark blue corresponding to -1 and red corresponding to 1. Qualitatively, all three embeddings capture categorical structure, as is evidenced by the block-diagonal structure of the correlation matrix. Nonetheless, ELMo appears to generate highly similar vectors for words sharing a category (the diagonal blocks are colored in strong red), indicating a poorer ability to distinguish among within-category items, compared to the other two embeddings. In contrast, BERT appears to separate items from across different categories more poorly than the other two embeddings (the color differences between the diagonal blocks and the rest of the matrix are somewhat weak).

Supplementary information

Supplementary Information

Supplementary Methods, Results, Figure and References.

Reporting Summary

Peer Review File

Supplementary Video 1

Schematic animation of the semantic projection procedure, corresponding to Fig. 1.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Grand, G., Blank, I.A., Pereira, F. et al. Semantic projection recovers rich human knowledge of multiple object features from word embeddings. Nat Hum Behav 6, 975–987 (2022). https://doi.org/10.1038/s41562-022-01316-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-022-01316-8

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing