Letter | Published:

Hierarchy and adaptivity in segmenting visual scenes

Naturevolume 442pages810813 (2006) | Download Citation



Finding salient, coherent regions in images is the basis for many visual tasks, and is especially important for object recognition. Human observers perform this task with ease, relying on a system in which hierarchical processing seems to have a critical role1. Despite many attempts, computerized algorithms2,3,4,5 have so far not demonstrated robust segmentation capabilities under general viewing conditions. Here we describe a new, highly efficient approach that determines all salient regions of an image and builds them into a hierarchical structure. Our algorithm, segmentation by weighted aggregation, is derived from algebraic multigrid solvers for physical systems6, and consists of fine-to-coarse pixel aggregation. Aggregates of various sizes, which may or may not overlap, are revealed as salient, without predetermining their number or scale. Results using this algorithm are markedly more accurate and significantly faster (linear in data size) than previous approaches.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.


All prices are NET prices.


  1. 1

    Felleman, D. J. & Van Essen, D. C. Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1, 1–47 (1991)

  2. 2

    Pietikainen, M., Rosenfeld, A. & Walter, I. Split-and-link algorithms for image segmentation. Patt. Recog. 15, 287–298 (1982)

  3. 3

    Comanicu, D. & Meer, P. Mean shift: A robust approach toward feature space analysis. IEEE Trans. Patt. Anal. Machine Intell. 24, 603–619 (2002)

  4. 4

    Malik, J., Belongie, S., Leung, T. & Shi, J. Contour and texture analysis for image segmentation. Int. J. Comp. Vision 43, 7–27 (2001)

  5. 5

    Felzenszwalb, P. & Huttenlocher, D. Efficient graph-based image segmentation. Int. J. Comput. Vision 59, 167–181 (2004)

  6. 6

    Brandt, A. Algebraic multigrid theory: the symmetric case. Appl. Math. Comput. 19, 23–56 (1986)

  7. 7

    Sharon, E., Brandt, A. & Basri, R. Fast multiscale image segmentation. Proc. IEEE Conf. Comput. Vision Patt. Recog. 1, 70–77 (2000)

  8. 8

    Hubel, D. H. & Wiesel, T. N. Receptive fields and functional architecture of monkey striate cortex. J. Physiol. 195, 215–243 (1968)

  9. 9

    Tanaka, K. Inferotemporal cortex and object vision. Annu. Rev. Neurosci. 19, 109–139 (1996)

  10. 10

    Shi, J. & Malik, J. Normalized cuts and image segmentation. IEEE Trans. Patt. Anal. Machine Intell. 22, 888–905 (2000)

  11. 11

    Golub, G. H. & Van Loan, C. F. Matrix Computations (Johns Hopkins Univ. Press, Baltimore, 1989)

  12. 12

    Brandt, A., McCormick, S. & Ruge, J. In Sparsity and its Applications (ed. Evans, D. J.) 257–284 (Cambridge Univ. Press, Cambridge, 1984)

  13. 13

    Galun, M., Sharon, E., Basri, R. & Brandt, A. Texture segmentation by multiscale aggregation of filter responses and shape elements. Proc. Int. Conf. Comput. Vision 1, 469–476 (2003)

  14. 14

    Julesz, B. Textons, the elements of texture perception, and their interactions. Nature 290, 91–97 (1981)

  15. 15

    Voorhees, H. & Poggio, T. Computing texture boundaries from images. Nature 333, 364–367 (1988)

  16. 16

    Sharon, E., Brandt, A. & Basri, R. Segmentation and boundary detection using multiscale intensity measurements. Proc. IEEE Conf. Comput. Vision Patt. Recog. 1, 469–476 (2001)

  17. 17

    Stanley, D. A. & Rubin, N. fMRI activation in response to illusory contours and salient regions in the human lateral occipital complex. Neuron 37, 323–331 (2003)

Download references


Research was supported in part by the European Commission Project Aim Shape, the Binational Science foundation, and by the German–Israeli Foundation. D.S. was supported by a grant from the National Institutes of Health. The research was conducted at the Moross Laboratory for Vision and Motor Control at the Weizmann Institute of Science. We thank N. Rubin and D. Jacobs for many useful remarks, and S. Geman for commenting on an earlier version of the manuscript. We are grateful to E. Borenstein for his help with constructing the sunglasses search system. We also thank M. Varma and R. Deitch for help with the comparisons presented in the Supplementary Information and N. Brandt for help with the graphics.

Author information


  1. Department of Computer Science and Applied Mathematics, The Weizmann Institute of Science, 76100, Rehovot, Israel

    • Eitan Sharon
    • , Meirav Galun
    • , Ronen Basri
    •  & Achi Brandt
  2. Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown, Massachusetts, 02129, USA

    • Dahlia Sharon


  1. Search for Eitan Sharon in:

  2. Search for Meirav Galun in:

  3. Search for Dahlia Sharon in:

  4. Search for Ronen Basri in:

  5. Search for Achi Brandt in:

Competing interests

Reprints and permissions information is available at npg.nature.com/reprintsandpermissions. The authors declare no competing financial interests. Correspondence and requests for materials should be addressed to E.S. (eitan.sharon@weizmann.ac.il).

Supplementary information

  1. Supplementary Figure 1

    Adaptive vs. geometric aggregation. (PDF 24 kb)

  2. Supplementary Figure 2

    Comparison of results of recently published segmentation methods for eight challenging images of animals on cluttered backgrounds. (PDF 1321 kb)

  3. Supplementary Figure 3

    The full sunglasses database. (PDF 13953 kb)

About this article

Publication history




Issue Date



Further reading


By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.