Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects

Abstract

We describe a model of visual processing in which feedback connections from a higher- to a lower-order visual cortical area carry predictions of lower-level neural activities, whereas the feedforward connections carry the residual errors between the predictions and the actual lower-level activities. When exposed to natural images, a hierarchical network of model neurons implementing such a model developed simple-cell-like receptive fields. A subset of neurons responsible for carrying the residual errors showed endstopping and other extra-classical receptive-field effects. These results suggest that rather than being exclusively feedforward phenomena, nonclassical surround effects in the visual cortex may also result from cortico-cortical feedback as a consequence of the visual system using an efficient hierarchical strategy for encoding natural images.

This is a preview of subscription content

Access options

Buy article

Get time limited or full article access on ReadCube.

$32.00

All prices are NET prices.

Figure 1: Hierarchical network for predictive coding.
Figure 2: Receptive fields of feedforward model neurons after training on natural images.
Figure 6: Nonclassical surround effects in the model.
Figure 3: Endstopping in the model network.
Figure 4: Autocorrelation along dominant orientation directions in natural images.
Figure 5: Predictive feedback and endstopping.

References

  1. Hubel, D. H. & Wiesel, T. N. Receptive fields and functional architecture in two non-striate visual areas (18 and 19) of the cat. J. Neurophysiol. 28, 229–289 (1965).

    CAS  Article  Google Scholar 

  2. Hubel, D. H. & Wiesel, T. N. Receptive fields and functional architecture of monkey striate cortex . J. Physiol. (Lond.) 195, 215–243 (1968).

    CAS  Article  Google Scholar 

  3. Bolz, J. & Gilbert, C. D. Generation of end-inhibition in the visual cortex via interlaminar connections. Nature 320, 362–365 (1986).

    CAS  Article  Google Scholar 

  4. Hubel, D. H. & Livingstone, M. S. Segregation of form, color, and stereopsis in primate area 18. J. Neurosci. 7, 3378–3415 (1987).

    CAS  Article  Google Scholar 

  5. Desimone, R. & Schein, S. J. Visual properties of neurons in area V4 of the macaque: sensitivity to stimulus form. J. Neurophysiol. 57, 835–868 ( 1987).

    CAS  Article  Google Scholar 

  6. Allman, J., Miezin, F. & McGuinness, E. Stimulus specific responses from beyond the classical receptive field: Neurophysiological mechanisms for local-global comparisons in visual neurons. Annu. Rev. Neurosci. 8, 407–429 (1985).

    CAS  Article  Google Scholar 

  7. Dobbins, A., Zucker, S. W. & Cynader, M. S. Endstopped neurons in the visual cortex as a substrate for calculating curvature. Nature 329, 438 –441 (1987).

    CAS  Article  Google Scholar 

  8. Bolz, J., Gilbert, C. D. & Wiesel, T. N. Pharmacological analysis of cortical circuitry. Trends Neurosci. 12, 292–296 (1989).

    CAS  Article  Google Scholar 

  9. Peterhans, E. & von der Heydt, R. in Representations of Vision. Trends and Tacit Assumptions (eds Gorea, A., Frégnac, Y., Kapoulis, Z. & Findlay, J.) 111–124 (Cambridge Univ. Press, Cambridge, UK, 1991).

    Google Scholar 

  10. Grossberg, S., Mingolla, E. & Ross, W. D. Visual brain and visual perception: how does the cortex do perceptual grouping? Trends Neurosci. 20, 106–111 (1997).

    CAS  Article  Google Scholar 

  11. Peterhans, E. & von der Heydt, R. Subjective contours—bridging the gap between psychophysics and physiology. Trends Neurosci. 14, 112–119 ( 1991).

    CAS  Article  Google Scholar 

  12. Rao, R. P. N. & Ballard, D. H. Dynamic model of visual recognition predicts neural response properties in the visual cortex. Neural Comput. 9, 721–763 ( 1997).

    CAS  Article  Google Scholar 

  13. Gallant, J. L., Connor, C. E. & Van Essen, D. C. Neural activity in areas V1, V2 and V4 during free viewing of natural scenes compared to controlled viewing. Neuroreport 9, 2153–2158 ( 1998).

    CAS  Article  Google Scholar 

  14. Attneave, F. Some informational aspects of visual perception. Psychol. Rev. 61, 183–193 ( 1954).

    CAS  Article  Google Scholar 

  15. MacKay, D. M. in Automata Studies (eds Shannon, C. E. & McCarthy, J.) 235– 251 (Princeton Univ. Press, Princeton, NJ, 1956).

    Google Scholar 

  16. Barlow, H. B. in Sensory Communication (ed. Rosenblith, W. A.) 217– 234 (MIT Press, Cambridge, MA, 1961).

    Google Scholar 

  17. Atick, J. J. Could information theory provide an ecological theory of sensory processing? Network Comput. Neural Sys. 3, 213– 251 (1992).

    Article  Google Scholar 

  18. Buchsbaum, G. & Gottschalk, A. Trichromacy, opponent colours coding and optimum colour information transmission in the retina. Proc. R. Soc. Lond. B Biol. Sci. 220, 89– 113 (1983).

    CAS  Article  Google Scholar 

  19. Srinivasan, M. V., Laughlin, S. B. & Dubs A. Predictive coding: A fresh view of inhibition in the retina. Proc. R. Soc. Lond. B Biol. Sci. 216, 427–459 (1982).

    CAS  Article  Google Scholar 

  20. Dan, Y., Atick, J. J. & Reid, R. C. Efficient coding of natural scenes in the lateral geniculate nucleus: experimental test of a computational theory. J. Neurosci. 16, 3351–3362 (1996).

    CAS  Article  Google Scholar 

  21. Dong, D. W. & Atick, J. J. Temporal decorrelation: a theory of lagged and nonlagged responses in the lateral geniculate nucleus. Network Comput. Neural Sys. 6, 159– 178 (1995).

    Article  Google Scholar 

  22. Mumford, D. On the computational architecture of the neocortex. II. The role of cortico-cortical loops. Biol. Cybern. 66, 241– 251 (1992).

    CAS  Article  Google Scholar 

  23. Pece, A. E. C. in Artificial Neural Networks 2 (eds Aleksander, I. & Taylor, J.) 865–868 (Elsevier, Amsterdam, 1992).

    Book  Google Scholar 

  24. Softky, W. R. in Advances in Neural Information Processing Systems 8 (eds Touretzky, D., Mozer, M. & Hasselmo, M.) 809–815 (MIT Press, Cambridge, MA, 1996).

    Google Scholar 

  25. Ullman, S. in Large-Scale Neuronal Theories of the Brain (eds Koch, C. & Davis, J. L.) 257–270 (MIT Press, Cambridge, MA, 1994).

    Google Scholar 

  26. Olshausen, B. A. & Field, D. J. Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Res. 37, 3311–3325 ( 1997).

    CAS  Article  Google Scholar 

  27. Dayan, P., Hinton, G.E., Neal, R.M. & Zemel, R.S. The Helmholtz machine. Neural Comput. 7, 889– 904, (1995).

    CAS  Article  Google Scholar 

  28. Luettgen, M. R. & Willsky, A. S. Likelihood calculation for a class of multiscale stochastic models, with application to texture discrimination. IEEE Trans. Image Proc. 4, 194–207 (1995).

    CAS  Article  Google Scholar 

  29. Felleman, D. J. & Van Essen, D. C. Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1, 1–47 ( 1991).

    CAS  Article  Google Scholar 

  30. Field, D. J. Relations between the statistics of natural images and the response properties of cortical cells. J. Opt. Soc. Am. A 4, 2379–2394 (1987).

    CAS  Article  Google Scholar 

  31. Bell, A. J. & Sejnowski, T. J. The "independent components" of natural scenes are edge filters. Vision Res. 37, 3327–3338 (1997).

    CAS  Article  Google Scholar 

  32. Olshausen, B. A. & Field, D. J. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607– 609 (1996).

    CAS  Article  Google Scholar 

  33. Zipser, K., Lamme, V. A. F. & Schiller, P. H. Contextual modulation in primary visual cortex. J. Neurosci. 16, 7376–7389 (1996).

    CAS  Article  Google Scholar 

  34. Sandell, J. H. & Schiller, P. H. Effect of cooling area 18 on striate cortex cells in the squirrel monkey. J. Neurophysiol. 48, 38–48 (1982).

    CAS  Article  Google Scholar 

  35. Knierim, J. & Van Essen, D. C. Neural responses to static texture patterns in area V1 of the alert macaque monkey. J. Neurophysiol. 67, 961–980 ( 1992).

    CAS  Article  Google Scholar 

  36. Sillito, A. M., Grieve, K. L., Jones, H. E., Cudeiro, J. & Davis, J. Visual cortical mechanisms detecting focal orientation discontinuities. Nature 378, 492–496 (1995).

    CAS  Article  Google Scholar 

  37. Hupé, J. M. et al. Cortical feedback improves discrimination between figure and background by V1, V2 and V3 neurons. Nature 394 , 784–787 (1998).

    Article  Google Scholar 

  38. Murphy, P. C. & Sillito, A. M. Corticofugal feedback influences the generation of length tuning in the visual pathway. Nature 329, 727–729 (1987).

    CAS  Article  Google Scholar 

  39. Gilbert, C. D. Adult cortical dynamics. Physiol. Rev. 78, 467–485 (1998).

    CAS  Article  Google Scholar 

  40. Lee, D. D. & Seung, H. S. in Advances in Neural Information Processing Systems 9 (eds Mozer, M., Jordan, M. & Petsche, T.) 515–521 (MIT Press, Cambridge, MA, 1997).

    Google Scholar 

  41. Heeger, D. J., Simoncelli, E. P. & Movshon, J. A. Computational models of cortical visual processing. Proc. Natl. Acad. Sci. USA 93, 623– 627 (1996).

    CAS  Article  Google Scholar 

  42. Miller, E. K., Li, L. & Desimone, R. A neural mechanism for working and recognition memory in inferior temporal cortex. Science 254, 1377–1379 (1991).

    CAS  Article  Google Scholar 

  43. Gross, C. G. & Sergent, J. Face recognition. Curr. Opin. Neurobiol. 2, 156–161 (1992).

    CAS  Article  Google Scholar 

  44. Logothetis, N. K. & Pauls, J. Psychophysical and physiological evidence for viewer-centered object representations in the primate. Cereb. Cortex 5, 270–288 (1995).

    CAS  Article  Google Scholar 

  45. Poggio, T. & Edelman, S. A network that learns to recognize 3D objects. Nature 343, 263– 266 (1990).

    CAS  Article  Google Scholar 

  46. Bell, C., Bodznick, D., Montgomery, J. & Bastian, J. The generation and subtraction of sensory expectations within cerebellum-like structures. Brain Behav. Evol. 50 Suppl. 1, 17–31 (1997).

    Article  Google Scholar 

  47. Schultz, W. et al. in Models of Information Processing in the Basal Ganglia (eds Houk, J. C., Davis, J. L. & Beiser, D. G.) 233– 248 (MIT Press, Cambridge, MA, 1995).

    Google Scholar 

  48. Creutzfeldt, O. D. Generality of the functional structure of the neocortex. Naturwissenschaften 64, 507–517 ( 1977).

    CAS  Article  Google Scholar 

  49. Rao, R. P. N. An optimal estimation approach to visual perception and learning. Vision Res. (in press).

  50. Rissanen, J. Stochastic Complexity in Statistical Inquiry (World Scientific, Singapore, 1989).

    Google Scholar 

Download references

Acknowledgements

We thank Christof Koch for comments on the manuscript and Mary Hayhoe, Terrence Sejnowski and members of the Computational Neurobiology Lab at the Salk Institute for discussions. This work was supported by research grants from the National Institute of Health (NIH), the National Science Foundation (NSF) and the Alfred P. Sloan Foundation.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Rajesh P. N. Rao.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Rao, R., Ballard, D. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat Neurosci 2, 79–87 (1999). https://doi.org/10.1038/4580

Download citation

  • Received:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1038/4580

Further reading

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing