Letter | Published:

Learning the parts of objects by non-negative matrix factorization

Naturevolume 401pages788791 (1999) | Download Citation

Subjects

Abstract

Is perception of the whole based on perception of its parts? There is psychological1 and physiological2,3 evidence for parts-based representations in the brain, and certain computational theories of object recognition rely on such representations4,5. But little is known about how brains or computers might learn the parts of objects. Here we demonstrate an algorithm for non-negative matrix factorization that is able to learn parts of faces and semantic features of text. This is in contrast to other methods, such as principal components analysis and vector quantization, that learn holistic, not parts-based, representations. Non-negative matrix factorization is distinguished from the other methods by its use of non-negativity constraints. These constraints lead to a parts-based representation because they allow only additive, not subtractive, combinations. When non-negative matrix factorization is implemented as a neural network, parts-based representations emerge by virtue of two properties: the firing rates of neurons are never negative and synaptic strengths do not change sign.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1

    Palmer,S. E. Hierarchical structure in perceptual representation. Cogn. Psychol. 9, 441–474 ( 1977).

  2. 2

    Wachsmuth,E., Oram,M. W. & Perrett, D. I. Recognition of objects and their component parts: responses of single units in the temporal cortex of the macaque. Cereb. Cortex 4, 509–522 (1994).

  3. 3

    Logothetis,N. K. & Sheinberg,D. L. Visual object recognition. Annu. Rev. Neurosci. 19, 577 –621 (1996).

  4. 4

    Biederman,I. Recognition-by-components: a theory of human image understanding. Psychol. Rev. 94, 115–147 (1987).

  5. 5

    Ullman,S. High-Level Vision: Object Recognition and Visual Cognition (MIT Press, Cambridge, MA, 1996).

  6. 6

    Turk,M. & Pentland,A. Eigenfaces for recognition. J. Cogn. Neurosci. 3, 71–86 (1991).

  7. 7

    Field,D. J. What is the goal of sensory coding? Neural Comput. 6, 559–601 (1994).

  8. 8

    Foldiak,P. & Young,M. Sparse coding in the primate cortex. The Handbook of Brain Theory and Neural Networks 895 –898 (MIT Press, Cambridge, MA, 1995 ).

  9. 9

    Olshausen,B. A. & Field,D. J. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 ( 1996).

  10. 10

    Lee,D. D. & Seung,H. S. Unsupervised learning by convex and conic coding. Adv. Neural Info. Proc. Syst. 9, 515–521 (1997).

  11. 11

    Paatero,P. Least squares formulation of robust non-negative factor analysis. Chemometr. Intell. Lab. 37, 23–35 (1997).

  12. 12

    Nakayama,K. & Shimojo,S. Experiencing and perceiving visual surfaces. Science 257, 1357– 1363 (1992).

  13. 13

    Hinton,G. E., Dayan,P., Frey,B. J. & Neal,R. M. The “wake-sleep” algorithm for unsupervised neural networks. Science 268, 1158–1161 (1995).

  14. 14

    Salton,G. & McGill,M. J. Introduction to Modern Information Retrieval (McGraw-Hill, New York, 1983).

  15. 15

    Landauer,T. K. & Dumais,S. T. The latent semantic analysis theory of knowledge. Psychol. Rev. 104, 211–240 (1997).

  16. 16

    Jutten,C. & Herault,J. Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture. Signal Proc. 24, 1–10 ( 1991).

  17. 17

    Bell,A. J. & Sejnowski,T. J. An information maximization approach to blind separation and blind deconvolution. Neural Comput. 7, 1129–1159 ( 1995).

  18. 18

    Bartlett,M. S., Lades,H. M. & Sejnowski, T. J. Independent component representations for face recognition. Proc. SPIE 3299, 528–539 (1998).

  19. 19

    Shepp,L. A. & Vardi,Y. Maximum likelihood reconstruction for emission tomography. IEEE Trans. Med. Imaging. 2, 113–122 (1982).

  20. 20

    Richardson,W. H. Bayesian-based iterative method of image restoration. J. Opt. Soc. Am. 62, 55–59 ( 1972).

  21. 21

    Lucy,L. B. An iterative technique for the rectification of observed distributions. Astron. J. 74, 745–754 ( 1974).

  22. 22

    Dempster,A. P., Laired,N. M. & Rubin, D. B. Maximum likelihood from incomplete data via the EM algorithm. J. Royal Stat. Soc. 39, 1– 38 (1977).

  23. 23

    Saul,L. & Pereira,F. Proceedings of the Second Conference on Empirical Methods n Natural Language Processing (eds Cardie, C. & Weischedel, R.) 81–89 (Morgan Kaufmann, San Francisco, 1997).

Download references

Acknowledgements

We acknowledge the support of Bell Laboratories and MIT. C. Papageorgiou and T. Poggio provided us with the database of faces, and R. Sproat with the Grolier encyclopedia corpus. We thank L. Saul for convincing us of the advantages of EM-type algorithms. We have benefited from discussions with B. Anderson, K. Clarkson, R. Freund, L. Kaufman, E. Rietman, S. Roweis, N. Rubin, J. Tenenbaum, N. Tishby, M. Tsodyks, T. Tyson and M. Wright.

Author information

Affiliations

  1. Bell Laboratories, Lucent Technologies , Murray Hill, 07974, New Jersey, USA

    • Daniel D. Lee
    •  & H. Sebastian Seung
  2. Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, 02139, Massachusetts, USA

    • H. Sebastian Seung

Authors

  1. Search for Daniel D. Lee in:

  2. Search for H. Sebastian Seung in:

Corresponding author

Correspondence to H. Sebastian Seung.

About this article

Publication history

Received

Accepted

Issue Date

DOI

https://doi.org/10.1038/44565

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.