Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Letter
  • Published:

Hierarchy and adaptivity in segmenting visual scenes

Nature volume 442, pages 810–813 (2006)Cite this article

Abstract

Finding salient, coherent regions in images is the basis for many visual tasks, and is especially important for object recognition. Human observers perform this task with ease, relying on a system in which hierarchical processing seems to have a critical role1. Despite many attempts, computerized algorithms2,3,4,5 have so far not demonstrated robust segmentation capabilities under general viewing conditions. Here we describe a new, highly efficient approach that determines all salient regions of an image and builds them into a hierarchical structure. Our algorithm, segmentation by weighted aggregation, is derived from algebraic multigrid solvers for physical systems6, and consists of fine-to-coarse pixel aggregation. Aggregates of various sizes, which may or may not overlap, are revealed as salient, without predetermining their number or scale. Results using this algorithm are markedly more accurate and significantly faster (linear in data size) than previous approaches.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: SWA.
Figure 2: The multiscale normalized cut graph approach.
Figure 3: Segmentation results for eight challenging images of animals on cluttered backgrounds.
Figure 4: Similarity search by parts.

Similar content being viewed by others

References

  1. Felleman, D. J. & Van Essen, D. C. Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1, 1–47 (1991)

    Article  CAS  Google Scholar 

  2. Pietikainen, M., Rosenfeld, A. & Walter, I. Split-and-link algorithms for image segmentation. Patt. Recog. 15, 287–298 (1982)

    Article  Google Scholar 

  3. Comanicu, D. & Meer, P. Mean shift: A robust approach toward feature space analysis. IEEE Trans. Patt. Anal. Machine Intell. 24, 603–619 (2002)

    Article  Google Scholar 

  4. Malik, J., Belongie, S., Leung, T. & Shi, J. Contour and texture analysis for image segmentation. Int. J. Comp. Vision 43, 7–27 (2001)

    Article  Google Scholar 

  5. Felzenszwalb, P. & Huttenlocher, D. Efficient graph-based image segmentation. Int. J. Comput. Vision 59, 167–181 (2004)

    Article  Google Scholar 

  6. Brandt, A. Algebraic multigrid theory: the symmetric case. Appl. Math. Comput. 19, 23–56 (1986)

    MathSciNet  MATH  Google Scholar 

  7. Sharon, E., Brandt, A. & Basri, R. Fast multiscale image segmentation. Proc. IEEE Conf. Comput. Vision Patt. Recog. 1, 70–77 (2000)

    Google Scholar 

  8. Hubel, D. H. & Wiesel, T. N. Receptive fields and functional architecture of monkey striate cortex. J. Physiol. 195, 215–243 (1968)

    Article  CAS  Google Scholar 

  9. Tanaka, K. Inferotemporal cortex and object vision. Annu. Rev. Neurosci. 19, 109–139 (1996)

    Article  CAS  Google Scholar 

  10. Shi, J. & Malik, J. Normalized cuts and image segmentation. IEEE Trans. Patt. Anal. Machine Intell. 22, 888–905 (2000)

    Article  Google Scholar 

  11. Golub, G. H. & Van Loan, C. F. Matrix Computations (Johns Hopkins Univ. Press, Baltimore, 1989)

    MATH  Google Scholar 

  12. Brandt, A., McCormick, S. & Ruge, J. In Sparsity and its Applications (ed. Evans, D. J.) 257–284 (Cambridge Univ. Press, Cambridge, 1984)

    Google Scholar 

  13. Galun, M., Sharon, E., Basri, R. & Brandt, A. Texture segmentation by multiscale aggregation of filter responses and shape elements. Proc. Int. Conf. Comput. Vision 1, 469–476 (2003)

    Google Scholar 

  14. Julesz, B. Textons, the elements of texture perception, and their interactions. Nature 290, 91–97 (1981)

    Article  ADS  CAS  Google Scholar 

  15. Voorhees, H. & Poggio, T. Computing texture boundaries from images. Nature 333, 364–367 (1988)

    Article  ADS  CAS  Google Scholar 

  16. Sharon, E., Brandt, A. & Basri, R. Segmentation and boundary detection using multiscale intensity measurements. Proc. IEEE Conf. Comput. Vision Patt. Recog. 1, 469–476 (2001)

    Google Scholar 

  17. Stanley, D. A. & Rubin, N. fMRI activation in response to illusory contours and salient regions in the human lateral occipital complex. Neuron 37, 323–331 (2003)

    Article  CAS  Google Scholar 

Download references

Acknowledgements

Research was supported in part by the European Commission Project Aim Shape, the Binational Science foundation, and by the German–Israeli Foundation. D.S. was supported by a grant from the National Institutes of Health. The research was conducted at the Moross Laboratory for Vision and Motor Control at the Weizmann Institute of Science. We thank N. Rubin and D. Jacobs for many useful remarks, and S. Geman for commenting on an earlier version of the manuscript. We are grateful to E. Borenstein for his help with constructing the sunglasses search system. We also thank M. Varma and R. Deitch for help with the comparisons presented in the Supplementary Information and N. Brandt for help with the graphics.

Author information

Authors and Affiliations

  1. Department of Computer Science and Applied Mathematics, The Weizmann Institute of Science, 76100, Rehovot, Israel

    Eitan Sharon, Meirav Galun, Ronen Basri & Achi Brandt

  2. Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown, Massachusetts, 02129, USA

    Dahlia Sharon

Authors
  1. Eitan Sharon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Meirav Galun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dahlia Sharon
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ronen Basri
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Achi Brandt
    View author publications

    You can also search for this author in PubMed Google Scholar

Ethics declarations

Competing interests

Reprints and permissions information is available at npg.nature.com/reprintsandpermissions. The authors declare no competing financial interests. Correspondence and requests for materials should be addressed to E.S. (eitan.sharon@weizmann.ac.il).

Supplementary information

Supplementary Figure 1

Adaptive vs. geometric aggregation. (PDF 24 kb)

Supplementary Figure 2

Comparison of results of recently published segmentation methods for eight challenging images of animals on cluttered backgrounds. (PDF 1321 kb)

Supplementary Figure 3

The full sunglasses database. (PDF 13953 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sharon, E., Galun, M., Sharon, D. et al. Hierarchy and adaptivity in segmenting visual scenes. Nature 442, 810–813 (2006). https://doi.org/10.1038/nature04977

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nature04977

This article is cited by

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing