Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Predictive encoding of motion begins in the primate retina

Nature Neuroscience volume 24, pages 1280–1291 (2021)Cite this article

Subjects

Abstract

Predictive motion encoding is an important aspect of visually guided behavior that allows animals to estimate the trajectory of moving objects. Motion prediction is understood primarily in the context of translational motion, but the environment contains other types of behaviorally salient motion correlation such as those produced by approaching or receding objects. However, the neural mechanisms that detect and predictively encode these correlations remain unclear. We report here that four of the parallel output pathways in the primate retina encode predictive motion information, and this encoding occurs for several classes of spatiotemporal correlation that are found in natural vision. Such predictive coding can be explained by known nonlinear circuit mechanisms that produce a nearly optimal encoding, with transmitted information approaching the theoretical limit imposed by the stimulus itself. Thus, these neural circuit mechanisms efficiently separate predictive information from nonpredictive information during the encoding process.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Ganglion cells show sensitivity to higher-order correlations.
Fig. 2: Information encoding occurs on short timescales.
Fig. 3: Timescales of encoding match higher-order correlations.
Fig. 4: Encoding of predictive information is nearly optimal for certain stimulus classes.
Fig. 5: Ganglion cells optimally encode future information at low contrast.
Fig. 6: Information encoding varies with retinal processing stage.
Fig. 7: Subunit coupling and high output thresholds favor encoding of predictive information.
Fig. 8: Proposed mechanisms contributing to predictive motion encoding.

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Code availability

Software code for data analysis is available from the corresponding authors upon reasonable request. Visual stimulus and data acquisition code are available at https://symphony-das.github.io/ and https://stage-vss.github.io/.

References

  1. Bialek, W., Nemenman, I. & Tishby, N. Predictability, complexity, and learning. Neural Comput. 13, 2409–2463 (2001).

    Article  CAS  PubMed  Google Scholar 

  2. Tishby, N., Pereira, F. C. & Bialek, W. The information bottleneck method. In Proc. 37th Annual Allerton Conference on Communication, Control and Computing (eds. Hajek, B. & Sreenivas, R. S.) 368–377 (Univ. of Illinois, 1999).

  3. Salisbury, J. M. & Palmer, S. E. Optimal prediction in the retina and natural motion statistics. J. Stat. Phys. 162, 1309–1323 (2016).

    Article  Google Scholar 

  4. Clark, D. A. et al. Flies and humans share a motion estimation strategy that exploits natural scene statistics. Nat. Neurosci. 17, 296–303 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Fitzgerald, J. E., Katsov, A. Y., Clandinin, T. R. & Schnitzer, M. J. Symmetries in stimulus statistics shape the form of visual motion estimators. Proc. Natl Acad. Sci. USA 108, 12909–12914 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Nitzany, E. I. & Victor, J. D. The statistics of local motion signals in naturalistic movies. J. Vis. 14, 10 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  7. Nitzany, E. I., Loe, M. E., Palmer, S. E. & Victor, J. D. Perceptual interaction of local motion signals. J. Vis. 16, 22 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  8. Fitzgerald, J. E. & Clark, D. A. Nonlinear circuits for naturalistic visual motion estimation. eLife 4, e09123 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  9. Chen, J., Mandel, H. B., Fitzgerald, J. E. & Clark, D. A. Asymmetric ON–OFF processing of visual motion cancels variability induced by the structure of natural scenes. eLife 8, e47579 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Palmer, S. E., Marre, O., Berry, M. J.2nd & Bialek, W. Predictive information in a sensory population. Proc. Natl Acad. Sci. USA 112, 6908–6913 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  11. Berry, M. J., 2nd, Brivanlou, I. H., Jordan, T. A. & Meister, M. Anticipation of moving stimuli by the retina. Nature 398, 334–338 (1999).

    Article  CAS  PubMed  Google Scholar 

  12. Schwartz, G., Taylor, S., Fisher, C., Harris, R. & Berry, M. J. Synchronized firing among retinal ganglion cells signals motion reversal. Neuron 55, 958–969 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Johnston, J. & Lagnado, L. General features of the retinal connectome determine the computation of motion anticipation. eLife 4, e06250 (2015).

    Article  PubMed Central  Google Scholar 

  14. Leonardo, A. & Meister, M. Nonlinear dynamics support a linear population code in a retinal target-tracking circuit. J. Neurosci. 33, 16971–16982 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Rodieck, R. W. & Watanabe, M. Survey of the morphology of macaque retinal ganglion cells that project to the pretectum, superior colliculus, and parvicellular laminae of the lateral geniculate nucleus. J. Comp. Neurol. 338, 289–303 (1993).

    Article  CAS  PubMed  Google Scholar 

  16. Schiller, P. H., Logothetis, N. K. & Charles, E. R. Functions of the colour-opponent and broad-band channels of the visual system. Nature 343, 68–70 (1990).

    Article  CAS  PubMed  Google Scholar 

  17. Billington, J., Wilkie, R. M., Field, D. T. & Wann, J. P. Neural processing of imminent collision in humans. Proc. Biol. Sci. 278, 1476–1481 (2011).

    PubMed  Google Scholar 

  18. Hu, Q. & Victor, J. D. A set of high-order spatiotemporal stimuli that elicit motion and reverse-phi percepts. J. Vis. 10, 9.1–16 (2010).

    Article  PubMed  Google Scholar 

  19. Yilmaz, M. & Meister, M. Rapid innate defensive responses of mice to looming visual stimuli. Curr. Biol. 23, 2011–2015 (2013).

    Article  CAS  PubMed  Google Scholar 

  20. Field, D. J. Relations between the statistics of natural images and the response properties of cortical cells. J. Opt. Soc. Am. A 4, 2379–2394 (1987).

    Article  CAS  PubMed  Google Scholar 

  21. Dong, D. W. & Atick, J. J. Statistics of natural time varying images. Netw. Comput. Neural Syst. 6, 345–358 (1995).

    Article  Google Scholar 

  22. Manookin, M. B., Patterson, S. S. & Linehan, C. M. Neural mechanisms mediating motion sensitivity in parasol ganglion cells of the primate retina. Neuron 97, 1327–1340.e4 (2018).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  23. Leonhardt, A. et al. Asymmetry of drosophila ON and OFF motion detectors enhances real-world velocity estimation. Nat. Neurosci. 19, 706–715 (2016).

    Article  CAS  PubMed  Google Scholar 

  24. Appleby, T. R. & Manookin, M. B. Selectivity to approaching motion in retinal inputs to the dorsal visual pathway. eLife 9, e51144 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Rhoades, C. E. et al. Unusual physiological properties of smooth monostratified ganglion cell types in primate retina. Neuron 103, 658–672.e6 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  26. Reinagel, P. & Reid, R. C. Temporal coding of visual information in the thalamus. J. Neurosci. 20, 5392–5400 (2000).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  27. Gollisch, T. & Meister, M. Rapid neural coding in the retina with relative spike latencies. Science 319, 1108–1111 (2008).

    Article  CAS  PubMed  Google Scholar 

  28. Uzzell, V. J. & Chichilnisky, E. J. Precision of spike trains in primate retinal ganglion cells. J. Neurophysiol. 92, 780–789 (2004).

    Article  CAS  PubMed  Google Scholar 

  29. Bialek, W., Rieke, F., de Ruyter van Steveninck, R. R. & Warland, D. Reading a neural code. Science 252, 1854–1857 (1991).

    Article  CAS  PubMed  Google Scholar 

  30. Rieke, F., Warland, D., de Ruyter van Steveninck, R. & Bialek, W. Spikes: Exploring the Neural Code (The MIT Press, 1997).

  31. de Ruyter van Steveninck, R. R., Lewen, G. D., Strong, S. P., Koberle, R. & Bialek, W. Reproducibility and variability in neural spike trains. Science 275, 1805–1808 (1997).

    Article  PubMed  Google Scholar 

  32. Panzeri, S., Petersen, R. S., Schultz, S. R., Lebedev, M. & Diamond, M. E. The role of spike timing in the coding of stimulus location in rat somatosensory cortex. Neuron 29, 769–777 (2001).

    Article  CAS  PubMed  Google Scholar 

  33. Bialek, W., De Ruyter Van Steveninck, R. R. & Tishby, N. Efficient representation as a design principle for neural coding and computation. In Proc. 2006 IEEE International Symposium on Information Theory, 659–663 (IEEE, 2006).

  34. Chalk, M., Marre, O. & Tkačik, G. Toward a unified theory of efficient, predictive, and sparse coding. Proc. Natl Acad. Sci. USA 115, 186–191 (2018).

    Article  CAS  PubMed  Google Scholar 

  35. Sederberg, A. J., MacLean, J. N. & Palmer, S. E. Learning to make external sensory stimulus predictions using internal correlations in populations of neurons. Proc. Natl Acad. Sci. USA 115, 1105–1110 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  36. Abbott, L. F. & Dayan, P. The effect of correlated variability on the accuracy of a population code. Neural Comput. 11, 91–101 (1999).

    Article  CAS  PubMed  Google Scholar 

  37. Jacoby, R. A., Wiechmann, A. F., Amara, S. G., Leighton, B. H. & Marshak, D. W. Diffuse bipolar cells provide input to OFF parasol ganglion cells in the macaque retina. J. Comp. Neurol. 416, 6–18 (2000).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  38. Kántor, O. et al. Bipolar cell gap junctions serve major signaling pathways in the human retina. Brain Struct. Funct. 222, 2603–2624 (2017).

    Article  PubMed  CAS  Google Scholar 

  39. Luo, X., Ghosh, K. K., Martin, P. R. & Grünert, U. Analysis of two types of cone bipolar cells in the retina of a new world monkey, the marmoset, Callithrix jacchus. Vis. Neurosci. 16, 707–719 (1999).

    Article  CAS  PubMed  Google Scholar 

  40. Kuo, S. P., Schwartz, G. W. & Rieke, F. Nonlinear spatiotemporal integration by electrical and chemical synapses in the retina. Neuron 90, 320–332 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Turner, M. H. & Rieke, F. Synaptic rectification controls nonlinear spatial integration of natural visual inputs. Neuron 90, 1257–1271 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. van Hateren, J. H. A theory of maximizing sensory information. Biol. Cybern. 68, 23–29 (1992).

    Article  PubMed  Google Scholar 

  43. Atick, J. J. & Redlich, A. N. Towards a theory of early visual processing. Neural Comput. 2, 308–320 (1990).

    Article  Google Scholar 

  44. Srinivasan, M. V., Laughlin, S. B. & Dubs, A. Predictive coding: a fresh view of inhibition in the retina. Proc. R. Soc. Lond. B Biol. Sci. 216, 427–459 (1982).

    Article  CAS  PubMed  Google Scholar 

  45. Dan, Y., Atick, J. J. & Reid, R. C. Efficient coding of natural scenes in the lateral geniculate nucleus: experimental test of a computational theory. J. Neurosci. 16, 3351–3362 (1996).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  46. Fairhall, A. L., Lewen, G. D., Bialek, W. & de Ruyter Van Steveninck, R. R. Efficiency and ambiguity in an adaptive neural code. Nature 412, 787–792 (2001).

    Article  CAS  PubMed  Google Scholar 

  47. Brenner, N., Bialek, W. & de Ruyter van Steveninck, R. Adaptive rescaling maximizes information transmission. Neuron 26, 695–702 (2000).

    Article  CAS  PubMed  Google Scholar 

  48. Sharpee, T. O. et al. Adaptive filtering enhances information transmission in visual cortex. Nature 439, 936–942 (2006).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  49. Frechette, E. S. et al. Fidelity of the ensemble code for visual motion in primate retina. J. Neurophysiol. 94, 119–135 (2005).

    Article  CAS  PubMed  Google Scholar 

  50. Chichilnisky, E. J. & Kalmar, R. S. Temporal resolution of ensemble visual motion signals in primate retina. J. Neurosci. 23, 6681–6689 (2003).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  51. Hosoya, T., Baccus, S. A. & Meister, M. Dynamic predictive coding by the retina. Nature 436, 71–77 (2005).

    Article  CAS  PubMed  Google Scholar 

  52. Rao, R. P. & Ballard, D. H. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 2, 79–87 (1999).

    Article  CAS  PubMed  Google Scholar 

  53. Yildizoglu, T., Riegler, C., Fitzgerald, J. E. & Portugues, R. A neural representation of naturalistic motion-guided behavior in the zebrafish brain. Curr. Biol. 30, 2321–2333.e6 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  54. Zavatone-Veth, J. A., Badwan, B. A. & Clark, D. A. A minimal synaptic model for direction selective neurons in drosophila. J. Vis. 20, 2 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  55. Appleby, T. R. & Manookin, M. B. Neural sensitization improves encoding fidelity in the primate retina. Nat. Commun. 10, 4017 (2019).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  56. Shannon, C. E. A mathematical theory of communication. Bell Syst. Tech. J. 27, 379–423 (1948).

    Article  Google Scholar 

  57. Bialek, W. Biophysics: Searching for Principles (Princeton Univ. Press, 2012).

  58. Chien, J.-F. Encoding the Light Intensity in Retina’s Firing Rate. Master’s thesis, National Taiwan University, Taipei (2017).

  59. Chen, K. S., Chen, C.-C. & Chan, C. K. Characterization of predictive behavior of a retina by mutual information. Front. Comput. Neurosci. 11, 66 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  60. Strong, S. P., de Ruyter van Steveninck, R. R., Bialek, W. & Koberle, R. On the application of information theory to neural spike trains. Pac. Symp. Biocomput. 1998, 621–632 (1998).

    Google Scholar 

  61. Slonim, N. & Tishby, N. in Advances in Neural Information Processing Systems Vol 12 (eds. Solla, S. A. et al.) 617–623 (MIT Press, 2000).

  62. Chichilnisky, E. J. A simple white noise analysis of neuronal light responses. Network 12, 199–213 (2001).

    Article  CAS  PubMed  Google Scholar 

  63. Cafaro, J. & Rieke, F. Noise correlations improve response fidelity and stimulus encoding. Nature 468, 964–967 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  64. Cafaro, J. & Rieke, F. Regulation of spatial selectivity by crossover inhibition. J. Neurosci. 33, 6310–6320 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  65. Fisher, R. A. The arrangement of field experiments. J. Ministry Agric. Great Britain 33, 503–513 (1926).

    Google Scholar 

  66. Boycott, B. B. & Wässle, H. Morphological classification of bipolar cells of the primate retina. Eur. J. Neurosci. 3, 1069–1088 (1991).

    Article  PubMed  Google Scholar 

  67. Tsukamoto, Y. & Omi, N. ON bipolar cells in macaque retina: type-specific synaptic connectivity with special reference to OFF counterparts. Front. Neuroanat. 10, 104 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  68. Tsukamoto, Y. & Omi, N. OFF bipolar cells in macaque retina: type-specific connectivity in the outer and inner synaptic layers. Front. Neuroanat. 9, 122 (2015).

    PubMed  PubMed Central  Google Scholar 

  69. Hassenstein, B. & Reichardt, W. Systemtheoretische Analyse der Zeit-, Reihenfolgen- und Vorzeichenauswertung bei der Bewegungsperzeption des Rüsselkäfers Chlorosphanus. Z. Naturforsch. B 11, 513–524 (1956).

    Article  Google Scholar 

Download references

Acknowledgements

We thank S. Cunnington for technical assistance. Tissue was provided by the Tissue Distribution Program at the Washington National Primate Research Center (WaNPRC; supported through NIH grant P51 OD-010425) and we thank the WaNPRC staff, particularly C. English and A. Baldessari, for making these experiments possible. C. Chen assisted in tissue preparation. We thank S. Palmer, S. Wang and G. Gutierrez for helpful discussions, and C.-C. Chiao for supporting B.L. and A.H. This work was supported in part by grants from the NIH (NEI R01-EY027323 to M.B.M.; NEI R01-EY029247 to E.J. Chichilnisky, F.R., and M.B.M.; NEI R01-EY028542 to F.R.; NEI P30-EY001730 to the Vision Core), Research to Prevent Blindness Unrestricted Grant (to the University of Washington Department of Ophthalmology), Latham Vision Research Innovation Award (to M.B.M.), the Alcon Young Investigator Award (to M.B.M.), the Taiwanese Ministry of Science and Technology (108-2813-C-007-085-B to A.H.) and a travel award to B.L. and A.H. from the Taiwanese Ministry of Education (C.-C. Chiao, principal investigator).

Author information

Author notes

  1. These authors contributed equally: Belle Liu, Arthur Hong.

Authors and Affiliations

  1. Department of Physiology and Biophysics, University of Washington, Seattle, WA, USA

    Belle Liu, Arthur Hong & Fred Rieke

  2. Institute of Systems Neuroscience, National Tsing Hua University, Hsinchu, Taiwan

    Belle Liu & Arthur Hong

  3. Vision Science Center, University of Washington, Seattle, WA, USA

    Fred Rieke & Michael B. Manookin

  4. Department of Ophthalmology, University of Washington, Seattle, WA, USA

    Michael B. Manookin

Authors
  1. Belle Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arthur Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fred Rieke
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael B. Manookin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

M.B.M. conceived and designed the study. F.R. and M.B.M. performed the experiments. B.L., A.H., F.R. and M.B.M. designed the analytical techniques. M.B.M. wrote the analysis code and constructed the computational models. B.L., A.H., F.R. and M.B.M. interpreted the results. M.B.M wrote the original version of the manuscript. B.L., A.H., F.R. and M.B.M. reviewed and edited the manuscript.

Corresponding author

Correspondence to Michael B. Manookin.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Neuroscience thanks Botond Roska, Gregory Schwartz and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Overview of time-shifted mutual information calculations.

To quantify the amount of information that a neuron’s response at a particular time (Rt) contained about the past and future stimulus trajectory (St+Δt), we computed the time-shifted mutual information. The stimulus sequence was shifted relative to the response and mutual information was computed between the stimulus and response for each time shift (Δt). This was done for time shifts between ± 0.5 seconds.

Extended Data Fig. 2 Information encoding occurs on short timescales.

a, Example spike train for a one-second portion of a stimulus containing diverging positive correlations (top). To measure the relationship between spike timing and information transmission, each spike was randomly shifted forward or backward in time (±Δ, bottom). The process was repeated for each spike train in each cell at several time shifts (0–100 ms). b, Spike timing shift at which the sensitivity index increased by 5% relative to the condition with spike time unaltered (Δ, 0 ms). Data are shown across six stimulus conditions and six ganglion cell types. Number of cells in the population are indicated in parentheses. c, Error in information estimates determined by randomly shuffling spike times (open circles) or by using the spike times from the uncorrelated traces in the same cell (closed circles). Error estimates were not significantly different between these techniques (p-value: two-positive, 0.40; two-negative, 0.78; diverging positive, 0.11; converging positive, 0.34; diverging negative, 0.47; converging negative, 0.78; two-sided Wilcoxon signed rank test; n = 73 cells). Circles and error bars indicate mean ± SEM.

Extended Data Fig. 3 Local motion signals show temporally shifted profiles.

Local motion signals show temporally shifted profiles. Third-order spatiotemporal correlations were extracted directly from stimulus sequences. The time-shifted mutual information profile for diverging correlations showed shifts toward future time points, whereas converging information was shifted toward past time lags.

Supplementary information

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, B., Hong, A., Rieke, F. et al. Predictive encoding of motion begins in the primate retina. Nat Neurosci 24, 1280–1291 (2021). https://doi.org/10.1038/s41593-021-00899-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41593-021-00899-1

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing