Finding salient, coherent regions in images is the basis for many visual tasks, and is especially important for object recognition. Human observers perform this task with ease, relying on a system in which hierarchical processing seems to have a critical role1. Despite many attempts, computerized algorithms2,3,4,5 have so far not demonstrated robust segmentation capabilities under general viewing conditions. Here we describe a new, highly efficient approach that determines all salient regions of an image and builds them into a hierarchical structure. Our algorithm, segmentation by weighted aggregation, is derived from algebraic multigrid solvers for physical systems6, and consists of fine-to-coarse pixel aggregation. Aggregates of various sizes, which may or may not overlap, are revealed as salient, without predetermining their number or scale. Results using this algorithm are markedly more accurate and significantly faster (linear in data size) than previous approaches.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
Supervoxel-based segmentation of 3D imagery with optical flow integration for spatiotemporal processing
IPSJ Transactions on Computer Vision and Applications Open Access 19 June 2018
Subscribe to Journal
Get full journal access for 1 year
only $3.90 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
Felleman, D. J. & Van Essen, D. C. Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1, 1–47 (1991)
Pietikainen, M., Rosenfeld, A. & Walter, I. Split-and-link algorithms for image segmentation. Patt. Recog. 15, 287–298 (1982)
Comanicu, D. & Meer, P. Mean shift: A robust approach toward feature space analysis. IEEE Trans. Patt. Anal. Machine Intell. 24, 603–619 (2002)
Malik, J., Belongie, S., Leung, T. & Shi, J. Contour and texture analysis for image segmentation. Int. J. Comp. Vision 43, 7–27 (2001)
Felzenszwalb, P. & Huttenlocher, D. Efficient graph-based image segmentation. Int. J. Comput. Vision 59, 167–181 (2004)
Brandt, A. Algebraic multigrid theory: the symmetric case. Appl. Math. Comput. 19, 23–56 (1986)
Sharon, E., Brandt, A. & Basri, R. Fast multiscale image segmentation. Proc. IEEE Conf. Comput. Vision Patt. Recog. 1, 70–77 (2000)
Hubel, D. H. & Wiesel, T. N. Receptive fields and functional architecture of monkey striate cortex. J. Physiol. 195, 215–243 (1968)
Tanaka, K. Inferotemporal cortex and object vision. Annu. Rev. Neurosci. 19, 109–139 (1996)
Shi, J. & Malik, J. Normalized cuts and image segmentation. IEEE Trans. Patt. Anal. Machine Intell. 22, 888–905 (2000)
Golub, G. H. & Van Loan, C. F. Matrix Computations (Johns Hopkins Univ. Press, Baltimore, 1989)
Brandt, A., McCormick, S. & Ruge, J. In Sparsity and its Applications (ed. Evans, D. J.) 257–284 (Cambridge Univ. Press, Cambridge, 1984)
Galun, M., Sharon, E., Basri, R. & Brandt, A. Texture segmentation by multiscale aggregation of filter responses and shape elements. Proc. Int. Conf. Comput. Vision 1, 469–476 (2003)
Julesz, B. Textons, the elements of texture perception, and their interactions. Nature 290, 91–97 (1981)
Voorhees, H. & Poggio, T. Computing texture boundaries from images. Nature 333, 364–367 (1988)
Sharon, E., Brandt, A. & Basri, R. Segmentation and boundary detection using multiscale intensity measurements. Proc. IEEE Conf. Comput. Vision Patt. Recog. 1, 469–476 (2001)
Stanley, D. A. & Rubin, N. fMRI activation in response to illusory contours and salient regions in the human lateral occipital complex. Neuron 37, 323–331 (2003)
Research was supported in part by the European Commission Project Aim Shape, the Binational Science foundation, and by the German–Israeli Foundation. D.S. was supported by a grant from the National Institutes of Health. The research was conducted at the Moross Laboratory for Vision and Motor Control at the Weizmann Institute of Science. We thank N. Rubin and D. Jacobs for many useful remarks, and S. Geman for commenting on an earlier version of the manuscript. We are grateful to E. Borenstein for his help with constructing the sunglasses search system. We also thank M. Varma and R. Deitch for help with the comparisons presented in the Supplementary Information and N. Brandt for help with the graphics.
Reprints and permissions information is available at npg.nature.com/reprintsandpermissions. The authors declare no competing financial interests. Correspondence and requests for materials should be addressed to E.S. (email@example.com).
Adaptive vs. geometric aggregation. (PDF 24 kb)
Comparison of results of recently published segmentation methods for eight challenging images of animals on cluttered backgrounds. (PDF 1321 kb)
The full sunglasses database. (PDF 13953 kb)
About this article
Cite this article
Sharon, E., Galun, M., Sharon, D. et al. Hierarchy and adaptivity in segmenting visual scenes. Nature 442, 810–813 (2006). https://doi.org/10.1038/nature04977
World Wide Web (2022)
Annals of Operations Research (2020)
Annals of Operations Research (2019)
Machine Learning (2019)
Sensing and Imaging (2019)