Is perception of the whole based on perception of its parts? There is psychological1 and physiological2,3 evidence for parts-based representations in the brain, and certain computational theories of object recognition rely on such representations4,5. But little is known about how brains or computers might learn the parts of objects. Here we demonstrate an algorithm for non-negative matrix factorization that is able to learn parts of faces and semantic features of text. This is in contrast to other methods, such as principal components analysis and vector quantization, that learn holistic, not parts-based, representations. Non-negative matrix factorization is distinguished from the other methods by its use of non-negativity constraints. These constraints lead to a parts-based representation because they allow only additive, not subtractive, combinations. When non-negative matrix factorization is implemented as a neural network, parts-based representations emerge by virtue of two properties: the firing rates of neurons are never negative and synaptic strengths do not change sign.
This is a preview of subscription content, access via your institution
Subscribe to this journal
Receive 51 print issues and online access
$199.00 per year
only $3.90 per issue
Rent or buy this article
Prices vary by article type
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Palmer,S. E. Hierarchical structure in perceptual representation. Cogn. Psychol. 9, 441–474 ( 1977).
Wachsmuth,E., Oram,M. W. & Perrett, D. I. Recognition of objects and their component parts: responses of single units in the temporal cortex of the macaque. Cereb. Cortex 4, 509–522 (1994).
Logothetis,N. K. & Sheinberg,D. L. Visual object recognition. Annu. Rev. Neurosci. 19, 577 –621 (1996).
Biederman,I. Recognition-by-components: a theory of human image understanding. Psychol. Rev. 94, 115–147 (1987).
Ullman,S. High-Level Vision: Object Recognition and Visual Cognition (MIT Press, Cambridge, MA, 1996).
Turk,M. & Pentland,A. Eigenfaces for recognition. J. Cogn. Neurosci. 3, 71–86 (1991).
Field,D. J. What is the goal of sensory coding? Neural Comput. 6, 559–601 (1994).
Foldiak,P. & Young,M. Sparse coding in the primate cortex. The Handbook of Brain Theory and Neural Networks 895 –898 (MIT Press, Cambridge, MA, 1995 ).
Olshausen,B. A. & Field,D. J. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 ( 1996).
Lee,D. D. & Seung,H. S. Unsupervised learning by convex and conic coding. Adv. Neural Info. Proc. Syst. 9, 515–521 (1997).
Paatero,P. Least squares formulation of robust non-negative factor analysis. Chemometr. Intell. Lab. 37, 23–35 (1997).
Nakayama,K. & Shimojo,S. Experiencing and perceiving visual surfaces. Science 257, 1357– 1363 (1992).
Hinton,G. E., Dayan,P., Frey,B. J. & Neal,R. M. The “wake-sleep” algorithm for unsupervised neural networks. Science 268, 1158–1161 (1995).
Salton,G. & McGill,M. J. Introduction to Modern Information Retrieval (McGraw-Hill, New York, 1983).
Landauer,T. K. & Dumais,S. T. The latent semantic analysis theory of knowledge. Psychol. Rev. 104, 211–240 (1997).
Jutten,C. & Herault,J. Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture. Signal Proc. 24, 1–10 ( 1991).
Bell,A. J. & Sejnowski,T. J. An information maximization approach to blind separation and blind deconvolution. Neural Comput. 7, 1129–1159 ( 1995).
Bartlett,M. S., Lades,H. M. & Sejnowski, T. J. Independent component representations for face recognition. Proc. SPIE 3299, 528–539 (1998).
Shepp,L. A. & Vardi,Y. Maximum likelihood reconstruction for emission tomography. IEEE Trans. Med. Imaging. 2, 113–122 (1982).
Richardson,W. H. Bayesian-based iterative method of image restoration. J. Opt. Soc. Am. 62, 55–59 ( 1972).
Lucy,L. B. An iterative technique for the rectification of observed distributions. Astron. J. 74, 745–754 ( 1974).
Dempster,A. P., Laired,N. M. & Rubin, D. B. Maximum likelihood from incomplete data via the EM algorithm. J. Royal Stat. Soc. 39, 1– 38 (1977).
Saul,L. & Pereira,F. Proceedings of the Second Conference on Empirical Methods n Natural Language Processing (eds Cardie, C. & Weischedel, R.) 81–89 (Morgan Kaufmann, San Francisco, 1997).
We acknowledge the support of Bell Laboratories and MIT. C. Papageorgiou and T. Poggio provided us with the database of faces, and R. Sproat with the Grolier encyclopedia corpus. We thank L. Saul for convincing us of the advantages of EM-type algorithms. We have benefited from discussions with B. Anderson, K. Clarkson, R. Freund, L. Kaufman, E. Rietman, S. Roweis, N. Rubin, J. Tenenbaum, N. Tishby, M. Tsodyks, T. Tyson and M. Wright.
About this article
Cite this article
Lee, D., Seung, H. Learning the parts of objects by non-negative matrix factorization. Nature 401, 788–791 (1999). https://doi.org/10.1038/44565
This article is cited by
BMC Medical Informatics and Decision Making (2023)
Epidural stimulation restores muscle synergies by modulating neural drives in participants with sensorimotor complete spinal cord injuries
Journal of NeuroEngineering and Rehabilitation (2023)
Muscle synergy patterns as altered coordination strategies in individuals with chronic low back pain: a cross-sectional study
Journal of NeuroEngineering and Rehabilitation (2023)
Model selection and robust inference of mutational signatures using Negative Binomial non-negative matrix factorization
BMC Bioinformatics (2023)
Comprehensive transcriptomic analyses identify KDM genes-related subtypes with different TME infiltrates in gastric cancer
BMC Cancer (2023)