Filling-in is a remarkable perceptual phenomenon in which visual features such as colour, brightness, texture and motion of the surrounding area are perceived in a certain part of the visual field even though these features are not physically present.
One extreme possibility is that our visual system simply ignores the lack of visual input and that filling-in is a passive outcome of this. However, various psychophysical experiments suggest that some active processes are involved in the occurrence of filling-in, and that some neural computation occurs in the brain when filling-in occurs.
In the past decade, several single-unit recording experiments in monkeys and functional MRI experiments in humans have examined neural activities related to filling-in in early visual cortical areas. Many of these studies found that neurons are activated in the region of the retinotopic map of the early visual areas that represents the interior of the surface where filling-in occurs.
Neural mechanisms of filling-in investigated in the above-mentioned studies must be distinguished from topographic remapping induced by binocular retinal scotoma. When retinal lesions are made at corresponding positions in both eyes (binocular retinal scotoma), reorganization of the retinotopic map of the visual cortex occurs, but this differs from situations in which other types of filling-in occur.
Traditionally, 'symbolic' and 'isomorphic' theory have been proposed as neural mechanisms of filling-in. Symbolic theory assumes that early visual areas extract only the contrast information, but this contradicts the results of most of the recent neurophysiological and neuroimaging experiments.
Isomorphic theory assumes that when perceptual filling-in occurs, a two-dimensional array of neurons with a point-by-point representation of visual features is activated in the early visual cortex. Although neurons are activated in early visual areas during filling-in, neural responses recorded during filling-in at the blind spot differ from those predicted by isomorphic theory in several important ways.
Selective activation of neurons in deep layers of the visual cortex that represent a particular spatial scale and that are selective for particular features might be involved in the process that mediates perceptual filling-in.
This research is still at an early stage, and many questions remain to be answered about the neural mechanisms of filling-in. Understanding the details of these mechanisms is important, because it might provide answers as to where and how subjective visual experience emerges, a fundamental question about visual perception.
Filling-in is a perceptual phenomenon in which a visual attribute such as colour, brightness, texture or motion is perceived in a region of the visual field even though such an attribute exists only in the surround. Filling-in dramatically reveals the dissociation between the retinal input and the percept, and raises fundamental questions about how these two relate to each other. Filling-in is observed in various situations, and is an essential part of our normal surface perception. Here, I review recent experiments examining brain activities associated with filling-in, and discuss possible neural mechanisms underlying this remarkable perceptual phenomenon. The evidence shows that neuronal activities in early visual cortical areas are involved in filling-in, providing new insights into visual cortical functions.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
Neural correlates of lateral modulation and perceptual filling-in in center-surround radial sinusoidal gratings: an fMRI study
Scientific Reports Open Access 27 September 2022
Philosophical Studies Open Access 28 April 2022
Nature Communications Open Access 12 January 2022
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
Walls, G. L. The filling-in process. Am. J. Optom. Arch. Am. Acad. Optom. 31, 329–341 (1954).
Gerrits, H. J. M. & Vendrik, A. J. H. Simultaneous contrast, filling-in process and information processing in man's visual system. Exp. Brain Res. 11, 411–430 (1970).
Komatsu, H., Murakami, I. & Kinoshita, M. Surface representation in the visual system. Brain Res. Cogn. Brain Res. 5, 97–104 (1996).
Pessoa, L., Thompson, E. & Noe, A. Finding out about filling-in: a guide to perceptual completion for visual science and the philosophy of perception. Behav. Brain Sci. 21, 723–748; discussion 748–802 (1998).
Ramachandran, V. S. & Gregory, R. L. Perceptual filling in of artificially induced scotomas in human vision. Nature 350, 699–702 (1991).
Ramachandran, V. S. Blind spots. Sci. Am. 266, 86–91 (1992). An excellent introduction to the phenomenon of filling-in. Provides fascinating examples of perceptual filling-in at the blind spot.
Gerrits, H. J. M. & Timmerman, G. J. M. E. N. The filling-in process in patients with retinal scotoma. Vision Res. 9, 439–442 (1969).
Gassel, M. M. & Williams, D. Visual function in patients with homonymous hemianopia. III. The completion phenomenon; insight and attitude to the defect; and visual functional efficiency. Brain 86, 229–260 (1963).
Safran, A. B. & Landis, T. Plasticity in the adult visual cortex: implications for the diagnosis of visual field defects and visual rehabilitation. Curr. Opin. Ophthalmol. 7, 53–64 (1996).
Friedman, H. S., Zhou, H. & von der Heydt, R. Color filling-in under steady fixation: behavioral demonstration in monkeys and humans. Perception 28, 1383–1395 (1999).
von der Heydt, R., Friedman, H. & Zhou, H. in Filling-in (eds Pessoa, L. & De Weerd, P.) 106–127 (Oxford Univ. Press, New York, 2003).
Hamburger, K., Prior, H., Sarris, V. & Spillmann, L. Filling-in with colour: different modes of surface completion. Vision Res. 46, 1129–1138 (2006).
Spillmann, L. & Kurtenbach, A. Dynamic noise backgrounds facilitate target fading. Vision Res. 32, 1941–1946 (1992).
De Weerd, P., Desimone, R. & Ungerleider, L. G. Perceptual filling-in: a parametric study. Vision Res. 38, 2721–2734 (1998).
Yarbus, A. L. Eye Movements and Vision (Plenum, New York, 1967).
Krauskopf, J. Effect of retinal image stabilization on the appearance of heterochromatic targets. J. Opt. Soc. Am. 53, 741–744 (1963).
Gerrits, H. J. M., de Haan, B. & Vendrik, A. J. H. Experiments with retinal stabilized images. Relations between the observations and neural data. Vision Res. 6, 427–440 (1966).
van Tuijl, H. F. J. M. & Leeuwenberg, E. L. J. Neon color spreading and structural information measures. Percept. Psychophys. 25, 269–284 (1979).
Redies, C. & Spillmann, L. The neon color effect in the Ehrenstein illusion. Perception 10, 667–681 (1982).
Grossberg, S. & Mingolla, E. Neural dynamics of form perception: boundary completion, illusory figures, and neon color spreading. Psychol. Rev. 92, 173–211 (1985).
Bressan, P., Mingolla, E., Spillmann, L. & Watanabe, T. Neon color spreading: a review. Perception 26, 1353–1366 (1997).
Cornsweet, T. Visual Perception (Academic, New York, 1970).
Grossberg, S. & Todorovic, D. Neural dynamics of 1-D and 2-D brightness perception: a unified model of classical and recent phenomena. Percept. Psychophys. 43, 241–277 (1988).
Tynan, P. & Sekular, R. Moving visual phantoms: a new contour completion effect. Science 188, 951–952 (1975).
Gyoba, J. Stationary phantoms: a completion effect without motion and flicker. Vision Res. 23, 205–211 (1983).
Murakami, I. Motion aftereffect after monocular adaptation to filled-in motion at the blind spot. Vision Res. 35, 1041–1045 (1995).
Weisstein, N., Maguire, W. & Berbaum, K. A phantom-motion aftereffect. Science 198, 955–958 (1977).
Shimojo, S., Kamitani, Y. & Nishida, S. Afterimage of perceptually filled-in surface. Science 293, 1677–1680 (2001). A clear demonstration that filling-in generates afterimage, which indicates that filling-in is accompanied by some active neural processes.
Davis, G. & Driver, J. A functional role for illusory colour spreading in the control of focused visual attention. Perception 26, 1397–1411 (1997).
Zur, D. & Ullman, S. Filling-in of retinal scotomas. Vision Res. 43, 971–982 (2003).
von der Heydt, R., Peterhans, E. & Baumgartner, G. Illusory contours and cortical neuron responses. Science 224, 1260–1262 (1984).
Fiorani, M., Rosa, M. G. P., Gattas, R. & Rocha-Miranda, C. E. Dynamic surrounds of receptive fields in primate striate cortex: a physiological basis for perceptual completion? Proc. Natl Acad. Sci. USA 89, 8547–8551 (1992).
Matsumoto, M. & Komatsu, H. Neural responses in the macaque V1 to bar stimuli with various lengths presented on the blind spot. J. Neurophysiol. 93, 2374–2387 (2005).
Komatsu, H., Kinoshita, M. & Murakami, I. Neural responses in the retinotopic representation of the blind spot in the macaque V1 to stimuli for perceptual filling-in. J. Neurosci. 20, 9310–9319 (2000). Presents clear evidence that some neurons in the V1 region that represents the blind spot are activated when perceptual filling-in occurs at the blind spot. Also shows that these neurons are localized in deep layers.
Roe, A. W., Lu, H. D. & Hung, C. P. Cortical processing of a brightness illusion. Proc. Natl Acad. Sci. USA 102, 3869–3874 (2005).
Kayama, Y., Riso, R. R., Bartlett, J. R. & Doty, R. W. Luxotonic responses of units in macaque striate cortex. J. Neurophysiol. 42, 1495–1517 (1979).
Maguire, W. M. & Baizer, J. S. Luminance coding of briefly presented stimuli in area 17 of the rhesus monkey. J. Neurophysiol. 47, 128–137 (1982).
Rossi, A. F., Rittenhouse, C. D. & Paradiso, M. A. The representation of brightness in primary visual cortex. Science 273, 1104–1107 (1996).
Kinoshita, M. & Komatsu, H. Neural representation of the luminance and brightness of a uniform surface in the macaque primary visual cortex. J. Neurophysiol. 86, 2559–2570 (2001).
De Weerd, P., Gattass, R., Desimone, R. & Ungerleider, L. G. Responses of cells in monkey visual cortex during perceptual filling-in of an artificial scotoma. Nature 377, 731–734 (1995). Shows good correspondence between the time course of the occurrence of texture filling-in and the gradual rise of activation of extrastriate neurons.
Sasaki, Y. & Watanabe, T. The primary visual cortex fills in color. Proc. Natl Acad. Sci. USA 101, 18251–18256 (2004).
Meng, M., Remus, D. A. & Tong, F. Filling-in of visual phantoms in the human brain. Nature Neurosci. 8, 1248–1254 (2005). This fMRI experiment shows that, in human subjects, changes in V1 activity correlate closely with the occurrence of perceptual filling-in of a phantom illusion.
Tong, F. & Engel, S. A. Interocular rivalry revealed in the human cortical blind-spot representation. Nature 411, 195–199 (2001).
Perna, A., Tosetti, M., Montanaro, D. & Morrone, M. C. Neuronal mechanisms for illusory brightness perception in humans. Neuron 47, 645–651 (2005).
Kaas, J. H. et al. Reorganization of retinotopic cortical maps in adult mammals after lesions of the retina. Science 248, 229–231 (1990).
Gilbert, C. D. & Wiesel, T. N. Receptive field dynamics in adult primary visual cortex. Nature 356, 150–152 (1992).
Smirnakis, S. M. et al. Lack of long-term cortical reorganization after macaque retinal lesions. Nature 435, 300–307 (2005).
Ramachandran, V. S. Behavioral and magnetoencephalographic correlates of plasticity in the adult human brain. Proc. Natl Acad. Sci. USA 90, 10413–10420 (1993).
Gilbert, C. D. Circuitry, architecture, and functional dynamics of visual cortex. Cereb. Cortex 3, 373–386 (1993).
Schmid, L. M., Rosa, M. G., Calford, M. B. & Ambler, J. S. Visuotopic reorganization in the primary visual cortex of adult cats following monocular and binocular retinal lesions. Cereb. Cortex 6, 388–405 (1996).
Baker, C. I., Peli, E., Knouf, N. & Kanwisher, N. G. Reorganization of visual processing in macular degeneration. J. Neurosci. 25, 614–618 (2005).
Tripathy, S. P., Levi, D. M., Ogmen, H. & Harden, C. Perceived length across the physiological blind spot. Vis. Neurosci. 12, 385–402 (1995).
Cumming, G. & Friend, H. Perception at the blind spot and tilt aftereffect. Perception 9, 233–238 (1980).
Awater, H., Kerlin, J. R., Evans, K. K. & Tong, F. Cortical representation of space around the blind spot. J. Neurophysiol. 94, 3314–3324 (2005).
Cohen, M. A. & Grossberg, S. Neural dynamics of brightness perception: features, boundaries, diffusion, and resonance. Percept. Psychophys. 36, 428–456 (1984). This computational modelling study presents important concepts of two parallel contour-sensitive processes, one sensitive to orientation and the other sensitive to features. The interaction that occurs between these signals is an important question that needs to be addressed.
Arrington, K. F. The temporal dynamics of brightness filling-in. Vision Res. 34, 3371–3387 (1994).
Neumann, H., Pessoa, L. & Hansen, T. Visual filling-in for computing perceptual surface properties. Biol. Cybern. 85, 355–369 (2001).
Friedman, H. S., Zhou, H. & von der Heydt, R. The coding of uniform colour figures in monkey visual cortex. J. Physiol. (Lond.) 548, 593–613 (2003).
Paradiso, M. A. & Nakayama, K. Brightness perception and filling-in. Vision Res. 31, 1221–1236 (1991). Using a masking paradigm, this study shows the temporal dynamics of brightness perception and indicates that some process of filling-in occurs even when we see a uniform surface in the normal visual field.
Paradiso, M. A. & Hahn, S. Filling-in percepts produced by luminance modulation. Vision Res. 36, 2657–2663 (1996).
Nishina, S., Okada, M. & Kawato, M. Spatio-temporal dynamics of depth propagation on uniform region. Vision Res. 43, 2493–2503 (2003).
Davey, M. P., Maddess, T. & Srinivasan, M. V. The spatiotemporal properties of the Craik–O'Brien–Cornsweet effect are consistent with 'filling-in'. Vision Res. 38, 2037–2046 (1998).
Tootell, R. B., Silverman, M. S. & De Valois, R. L. Spatial frequency columns in primary visual cortex. Science 214, 813–815 (1981).
Hubener, M., Shoham, D., Grinvald, A. & Bonhoeffer, T. Spatial relationships among three columnar systems in cat area 17. J. Neurosci. 17, 9270–9284 (1997).
Everson, R. M. et al. Representation of spatial frequency and orientation in the visual cortex. Proc. Natl Acad. Sci. USA 95, 8334–8338 (1998).
Issa, N. P., Trepel, C. & Stryker, M. P. Spatial frequency maps in cat visual cortex. J. Neurosci. 20, 8504–8514 (2000).
Gilbert, C. D. Laminar differences in receptive field properties of cells in cat primary visual cortex. J. Physiol. (Lond.) 268, 391–421 (1977).
Gur, M., Kagan, I. & Snodderly, D. M. Orientation and direction selectivity of neurons in V1 of alert monkeys: functional relationships and laminar distributions. Cereb. Cortex 15, 1207–1221 (2005).
Raiguel, S., Van Hulle, M. M., Xiao, D. K., Marcar, V. L. & Orban, G. A. Shape and spatial distribution of receptive fields and antagonistic motion surrounds in the middle temporal area (V5) of the macaque. Eur. J. Neurosci. 7, 2064–2082 (1995).
Komatsu, H. Surface representation by population coding. Behav. Brain Sci. 21, 761–762 (1998).
Mendola, J. D., Dale, A. M., Fischl, B., Liu, A. K. & Tootell, R. B. The representation of illusory and real contours in human cortical visual areas revealed by functional magnetic resonance imaging. J. Neurosci. 19, 8560–8572 (1999).
Sherman, S. & Guillery, R. Exploring the Thalamus (Academic, San Diego, 2001).
Sillito, A. M., Jones, H. E., Gerstein, G. L. & West, D. C. Feature-linked synchronization of thalamic relay cell firing induced by feedback from the visual cortex. Nature 369, 479–482 (1994).
Komatsu, H., Kinoshita, M. & Murakami, I. Neural responses in the primary visual cortex of the monkey during perceptual filling-in at the blind spot. Neurosci. Res. 44, 231–236 (2002).
Haynes, J. D., Deichmann, R. & Rees, G. Eye-specific effects of binocular rivalry in the human lateral geniculate nucleus. Nature 438, 496–499 (2005).
Wunderlich, K., Schneider, K. A. & Kastner, S. Neural correlates of binocular rivalry in the human lateral geniculate nucleus. Nature Neurosci. 8, 1595–1602 (2005).
Kellman, P. J., Yin, C. & Shipley, T. F. A common mechanism for illusory and occluded object completion. J. Exp. Psychol. Hum. Percept. Perform. 24, 859–869 (1998).
Shimojo, S. & Nakayama, K. Amodal representation of occluded surfaces: role of invisible stimuli in apparent motion correspondence. Perception 19, 285–299 (1990).
Watanabe, T. Orientation and color processing for partially occluded objects. Vision Res. 35, 647–655 (1995).
Peterhans, E. & von der Heydt, R. Functional organization of area V2 in the alert macaque. Eur. J. Neurosci. 5, 509–524 (1993).
von der Heydt, R., Zhou, H. & Friedman, H. S. Representation of stereoscopic edges in monkey visual cortex. Vision Res. 40, 1955–1967 (2000).
Brown, J. M. & Weisstein, N. Conflicting figure–ground and depth information reduces moving phantom visibility. Perception 20, 155–165 (1991).
Nakayama, K., Shimojo, S. & Ramachandran, V. S. Transparency: relation to depth, subjective contours, luminance, and neon color spreading. Perception 19, 497–513 (1990). Shows that filling-in is dramatically affected by the three-dimensional interpretation of the scene.
Nakayama, K., Shimonjo, S. & Silverman, G. H. Stereoscopic depth: its relation to image segmentation, grouping, and the recognition of occluded objects. Perception 18, 55–68 (1989).
Bakin, J. S., Nakayama, K. & Gilbert, C. D. Visual responses in monkey areas V1 and V2 to three-dimensional surface configurations. J. Neurosci. 20, 8188–8198 (2000).
Sugita, Y. Grouping of image fragments in primary visual cortex. Nature 401, 269–272 (1999).
Hirsch, J. et al. Illusory contours activate specific regions in human visual cortex: evidence from functional magnetic resonance imaging. Proc. Natl Acad. Sci. USA 92, 6469–6473 (1995).
Ffytche, D. H. & Zeki, S. Brain activity related to the perception of illusory contours. Neuroimage 3, 104–108 (1996).
Kapadia, M. K., Ito, M., Gilbert, C. D. & Westheimer, G. Improvement in visual sensitivity by changes in local context: parallel studies in human observers and in V1 of alert monkeys. Neuron 15, 843–856 (1995).
Polat, U., Mizobe, K., Pettet, M. W., Kasamatsu, T. & Norcia, A. M. Collinear stimuli regulate visual responses depending on cell's contrast threshold. Nature 391, 580–584 (1998).
Lamme, V. A. F. The neurophysiology of figure–ground segregation in primary visual cortex. J. Neurosci. 15, 1605–1615 (1995).
Zipser, K., Lamme, V. A. F. & Schiller, P. H. Contextual modulation in primary visual cortex. J. Neurosci. 16, 7376–7389 (1996).
Zhou, H., Friedman, H. S. & von der Heydt, R. Coding of border ownership in monkey visual cortex. J. Neurosci. 20, 6594–6611 (2000).
Baylis, G. C. & Driver, J. Shape-coding in IT cells generalizes over contrast and mirror reversal, but not figure–ground reversal. Nature Neurosci. 4, 937–942 (2001).
Qiu, F. T. & von der Heydt, R. Figure and ground in the visual cortex: V2 combines stereoscopic cues with gestalt rules. Neuron 47, 155–166 (2005).
Hupe, J. M. et al. Cortical feedback improves discrimination between figure and background by V1, V2 and V3 neurons. Nature 394, 784–787 (1998).
Mumford, D. in Large-Scale Neuronal Theories of the Brain (eds Koch, C. & Davis, J.) 125–152 (MIT Press, Cambridge, Massachusetts, 1994).
Rao, R. P. & Ballard, D. H. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neurosci. 2, 79–87 (1999).
Mendola, J. in Filling-in (eds Pessoa, L. & De Weerd, P.) 38–58 (Oxford Univ. Press, New York, 2003).
He, Z. J. & Nakayama, K. Surfaces versus features in visual search. Nature 359, 231–233 (1992).
Maunsell, J. H. R. & Van, Essen, D. C. The connections of the middle temporal visual area (MT) and their relationship to a cortical hierarchy in the macaque monkey. J. Neurosci. 3, 2563–2586 (1983).
This work was supported by grants from the Ministry of Education, Science, Sports and Culture, the Ministry of Public Management, Home Affairs, Posts and Telecommunications, the Japan Society for the Promotion of Science and the Human Frontier Science Program (HFSP).
The author declares no competing financial interests.
- Surface interpolation
The ability of our visual system to reconstruct a continuous surface from incomplete visual inputs using surrounding information.
- Contrast border
A boundary of a surface generated by spatially discontinuous change (contrast) in brightness or colour. Many neurons in the retina and early visual pathway are sensitive to contrast border.
The intensity of light emitted from a light source or reflected from a surface that can be measured objectively.
- Achromatic grating
A visual stimulus consisting of alternating light and dark bars in shades of grey.
- Motion aftereffect
Also known as the waterfall illusion. Prolonged observation of a moving stimulus leads to an aftereffect in which stationary objects appear to move in the opposite direction.
If a two-dimensional array of neurons in a given area corresponds topographically (in spatial arrangement) to those on the retina, this area is said to have retinotopic organization. Early visual areas have retinotopic organization with different degrees of precision.
- Primary visual cortex
(Also known as V1 or the striate cortex). The cortical area that is the main recipient of visual information from the retinae (by way of the lateral geniculate nucleus).
- Receptive field
The area of sensory space in which stimulus presentation leads to the response of a particular sensory neuron.
The second tier of visual cortical areas, which is adjacent to V1. V2 consists of three compartments that can be visualized by cytochrome oxidase staining; thick stripe, thin stripe and interstripe.
- V2 thin stripe
One of three compartments in V2 where many neurons have sensitivity to the colour or brightness of a visual stimulus.
The third tier of visual cortical areas, which receives its main visual input from V1 and V2. V3 can be divided into two areas because dorsal and ventral parts of V3 have different connections and cell properties.
- Extrastriate areas
A belt of visually responsive areas of cortex surrounding the primary visual cortex.
- Binocular rivalry
A phenomenon that occurs when each of a subject's eyes is shown a different image. This results in a bistable visual experience. For example, perception of horizontal or vertical bars spontaneously alternates when the two bar types are viewed through different eyes simultaneously.
- Macular degeneration
A disease of the retina in which the macula, the central part of the retina, degenerates.
- Pointwise representation
One way to represent surface attributes such as colour is to activate colour-selective neurons that have a small receptive field at each point on a surface. This is an example of pointwise representation.
- Spatial frequency
A variable determined by the width of stripes on a grating. A grating with low spatial frequency has thick stripes, whereas a grating with high spatial frequency has narrow stripes.
- Pulvinar nucleus
The pulvinar is a complex of several nuclei in the thalamus that have strong connections with many visual cortical areas.
- Lateral geniculate nucleus
(LGN). The LGN is a nucleus in the thalamus that acts as a major relay station for visual signals from the retina to area V1. The LGN also receives a massive feedback projection from V1.
Qualities of conscious perceptual experience, or the 'raw' feel of sensation, such as the 'redness' of the colour red.
- Depth assignment
When a retinal image contains multiple surfaces, the depth order of the surfaces is derived by using various monocular as well as binocular visual cues in the image.
A set of two images — one of which is presented to the right eye and the other to the left — that contain binocular stereo-disparity. Different parts of the stereo-image appear to be at different depths.
About this article
Cite this article
Komatsu, H. The neural mechanisms of perceptual filling-in. Nat Rev Neurosci 7, 220–231 (2006). https://doi.org/10.1038/nrn1869
Neural correlates of lateral modulation and perceptual filling-in in center-surround radial sinusoidal gratings: an fMRI study
Scientific Reports (2022)
Nature Communications (2022)
Philosophical Studies (2022)
Genetic pre-screening for glaucoma in population-based epidemiology: protocol for a double-blind prospective screening study within Lifelines (EyeLife)
BMC Ophthalmology (2021)
Nature Human Behaviour (2021)