Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Predictive encoding of motion begins in the primate retina

Abstract

Predictive motion encoding is an important aspect of visually guided behavior that allows animals to estimate the trajectory of moving objects. Motion prediction is understood primarily in the context of translational motion, but the environment contains other types of behaviorally salient motion correlation such as those produced by approaching or receding objects. However, the neural mechanisms that detect and predictively encode these correlations remain unclear. We report here that four of the parallel output pathways in the primate retina encode predictive motion information, and this encoding occurs for several classes of spatiotemporal correlation that are found in natural vision. Such predictive coding can be explained by known nonlinear circuit mechanisms that produce a nearly optimal encoding, with transmitted information approaching the theoretical limit imposed by the stimulus itself. Thus, these neural circuit mechanisms efficiently separate predictive information from nonpredictive information during the encoding process.

Your institute does not have access to this article

Access options

Buy article

Get time limited or full article access on ReadCube.

$32.00

All prices are NET prices.

Fig. 1: Ganglion cells show sensitivity to higher-order correlations.
Fig. 2: Information encoding occurs on short timescales.
Fig. 3: Timescales of encoding match higher-order correlations.
Fig. 4: Encoding of predictive information is nearly optimal for certain stimulus classes.
Fig. 5: Ganglion cells optimally encode future information at low contrast.
Fig. 6: Information encoding varies with retinal processing stage.
Fig. 7: Subunit coupling and high output thresholds favor encoding of predictive information.
Fig. 8: Proposed mechanisms contributing to predictive motion encoding.

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Code availability

Software code for data analysis is available from the corresponding authors upon reasonable request. Visual stimulus and data acquisition code are available at https://symphony-das.github.io/ and https://stage-vss.github.io/.

References

  1. Bialek, W., Nemenman, I. & Tishby, N. Predictability, complexity, and learning. Neural Comput. 13, 2409–2463 (2001).

    CAS  PubMed  Article  Google Scholar 

  2. Tishby, N., Pereira, F. C. & Bialek, W. The information bottleneck method. In Proc. 37th Annual Allerton Conference on Communication, Control and Computing (eds. Hajek, B. & Sreenivas, R. S.) 368–377 (Univ. of Illinois, 1999).

  3. Salisbury, J. M. & Palmer, S. E. Optimal prediction in the retina and natural motion statistics. J. Stat. Phys. 162, 1309–1323 (2016).

    Article  Google Scholar 

  4. Clark, D. A. et al. Flies and humans share a motion estimation strategy that exploits natural scene statistics. Nat. Neurosci. 17, 296–303 (2014).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  5. Fitzgerald, J. E., Katsov, A. Y., Clandinin, T. R. & Schnitzer, M. J. Symmetries in stimulus statistics shape the form of visual motion estimators. Proc. Natl Acad. Sci. USA 108, 12909–12914 (2011).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  6. Nitzany, E. I. & Victor, J. D. The statistics of local motion signals in naturalistic movies. J. Vis. 14, 10 (2014).

    PubMed  PubMed Central  Article  Google Scholar 

  7. Nitzany, E. I., Loe, M. E., Palmer, S. E. & Victor, J. D. Perceptual interaction of local motion signals. J. Vis. 16, 22 (2016).

    PubMed  PubMed Central  Article  Google Scholar 

  8. Fitzgerald, J. E. & Clark, D. A. Nonlinear circuits for naturalistic visual motion estimation. eLife 4, e09123 (2015).

    PubMed  PubMed Central  Article  Google Scholar 

  9. Chen, J., Mandel, H. B., Fitzgerald, J. E. & Clark, D. A. Asymmetric ON–OFF processing of visual motion cancels variability induced by the structure of natural scenes. eLife 8, e47579 (2019).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  10. Palmer, S. E., Marre, O., Berry, M. J.2nd & Bialek, W. Predictive information in a sensory population. Proc. Natl Acad. Sci. USA 112, 6908–6913 (2015).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  11. Berry, M. J., 2nd, Brivanlou, I. H., Jordan, T. A. & Meister, M. Anticipation of moving stimuli by the retina. Nature 398, 334–338 (1999).

    CAS  PubMed  Article  Google Scholar 

  12. Schwartz, G., Taylor, S., Fisher, C., Harris, R. & Berry, M. J. Synchronized firing among retinal ganglion cells signals motion reversal. Neuron 55, 958–969 (2007).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  13. Johnston, J. & Lagnado, L. General features of the retinal connectome determine the computation of motion anticipation. eLife 4, e06250 (2015).

    PubMed Central  Article  Google Scholar 

  14. Leonardo, A. & Meister, M. Nonlinear dynamics support a linear population code in a retinal target-tracking circuit. J. Neurosci. 33, 16971–16982 (2013).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  15. Rodieck, R. W. & Watanabe, M. Survey of the morphology of macaque retinal ganglion cells that project to the pretectum, superior colliculus, and parvicellular laminae of the lateral geniculate nucleus. J. Comp. Neurol. 338, 289–303 (1993).

    CAS  PubMed  Article  Google Scholar 

  16. Schiller, P. H., Logothetis, N. K. & Charles, E. R. Functions of the colour-opponent and broad-band channels of the visual system. Nature 343, 68–70 (1990).

    CAS  PubMed  Article  Google Scholar 

  17. Billington, J., Wilkie, R. M., Field, D. T. & Wann, J. P. Neural processing of imminent collision in humans. Proc. Biol. Sci. 278, 1476–1481 (2011).

    PubMed  Google Scholar 

  18. Hu, Q. & Victor, J. D. A set of high-order spatiotemporal stimuli that elicit motion and reverse-phi percepts. J. Vis. 10, 9.1–16 (2010).

    PubMed  Article  Google Scholar 

  19. Yilmaz, M. & Meister, M. Rapid innate defensive responses of mice to looming visual stimuli. Curr. Biol. 23, 2011–2015 (2013).

    CAS  PubMed  Article  Google Scholar 

  20. Field, D. J. Relations between the statistics of natural images and the response properties of cortical cells. J. Opt. Soc. Am. A 4, 2379–2394 (1987).

    CAS  PubMed  Article  Google Scholar 

  21. Dong, D. W. & Atick, J. J. Statistics of natural time varying images. Netw. Comput. Neural Syst. 6, 345–358 (1995).

    Article  Google Scholar 

  22. Manookin, M. B., Patterson, S. S. & Linehan, C. M. Neural mechanisms mediating motion sensitivity in parasol ganglion cells of the primate retina. Neuron 97, 1327–1340.e4 (2018).

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  23. Leonhardt, A. et al. Asymmetry of drosophila ON and OFF motion detectors enhances real-world velocity estimation. Nat. Neurosci. 19, 706–715 (2016).

    CAS  PubMed  Article  Google Scholar 

  24. Appleby, T. R. & Manookin, M. B. Selectivity to approaching motion in retinal inputs to the dorsal visual pathway. eLife 9, e51144 (2020).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  25. Rhoades, C. E. et al. Unusual physiological properties of smooth monostratified ganglion cell types in primate retina. Neuron 103, 658–672.e6 (2019).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  26. Reinagel, P. & Reid, R. C. Temporal coding of visual information in the thalamus. J. Neurosci. 20, 5392–5400 (2000).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  27. Gollisch, T. & Meister, M. Rapid neural coding in the retina with relative spike latencies. Science 319, 1108–1111 (2008).

    CAS  PubMed  Article  Google Scholar 

  28. Uzzell, V. J. & Chichilnisky, E. J. Precision of spike trains in primate retinal ganglion cells. J. Neurophysiol. 92, 780–789 (2004).

    CAS  PubMed  Article  Google Scholar 

  29. Bialek, W., Rieke, F., de Ruyter van Steveninck, R. R. & Warland, D. Reading a neural code. Science 252, 1854–1857 (1991).

    CAS  PubMed  Article  Google Scholar 

  30. Rieke, F., Warland, D., de Ruyter van Steveninck, R. & Bialek, W. Spikes: Exploring the Neural Code (The MIT Press, 1997).

  31. de Ruyter van Steveninck, R. R., Lewen, G. D., Strong, S. P., Koberle, R. & Bialek, W. Reproducibility and variability in neural spike trains. Science 275, 1805–1808 (1997).

    PubMed  Article  Google Scholar 

  32. Panzeri, S., Petersen, R. S., Schultz, S. R., Lebedev, M. & Diamond, M. E. The role of spike timing in the coding of stimulus location in rat somatosensory cortex. Neuron 29, 769–777 (2001).

    CAS  PubMed  Article  Google Scholar 

  33. Bialek, W., De Ruyter Van Steveninck, R. R. & Tishby, N. Efficient representation as a design principle for neural coding and computation. In Proc. 2006 IEEE International Symposium on Information Theory, 659–663 (IEEE, 2006).

  34. Chalk, M., Marre, O. & Tkačik, G. Toward a unified theory of efficient, predictive, and sparse coding. Proc. Natl Acad. Sci. USA 115, 186–191 (2018).

    CAS  PubMed  Article  Google Scholar 

  35. Sederberg, A. J., MacLean, J. N. & Palmer, S. E. Learning to make external sensory stimulus predictions using internal correlations in populations of neurons. Proc. Natl Acad. Sci. USA 115, 1105–1110 (2018).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  36. Abbott, L. F. & Dayan, P. The effect of correlated variability on the accuracy of a population code. Neural Comput. 11, 91–101 (1999).

    CAS  PubMed  Article  Google Scholar 

  37. Jacoby, R. A., Wiechmann, A. F., Amara, S. G., Leighton, B. H. & Marshak, D. W. Diffuse bipolar cells provide input to OFF parasol ganglion cells in the macaque retina. J. Comp. Neurol. 416, 6–18 (2000).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  38. Kántor, O. et al. Bipolar cell gap junctions serve major signaling pathways in the human retina. Brain Struct. Funct. 222, 2603–2624 (2017).

    PubMed  Article  CAS  Google Scholar 

  39. Luo, X., Ghosh, K. K., Martin, P. R. & Grünert, U. Analysis of two types of cone bipolar cells in the retina of a new world monkey, the marmoset, Callithrix jacchus. Vis. Neurosci. 16, 707–719 (1999).

    CAS  PubMed  Article  Google Scholar 

  40. Kuo, S. P., Schwartz, G. W. & Rieke, F. Nonlinear spatiotemporal integration by electrical and chemical synapses in the retina. Neuron 90, 320–332 (2016).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  41. Turner, M. H. & Rieke, F. Synaptic rectification controls nonlinear spatial integration of natural visual inputs. Neuron 90, 1257–1271 (2016).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  42. van Hateren, J. H. A theory of maximizing sensory information. Biol. Cybern. 68, 23–29 (1992).

    PubMed  Article  Google Scholar 

  43. Atick, J. J. & Redlich, A. N. Towards a theory of early visual processing. Neural Comput. 2, 308–320 (1990).

    Article  Google Scholar 

  44. Srinivasan, M. V., Laughlin, S. B. & Dubs, A. Predictive coding: a fresh view of inhibition in the retina. Proc. R. Soc. Lond. B Biol. Sci. 216, 427–459 (1982).

    CAS  PubMed  Article  Google Scholar 

  45. Dan, Y., Atick, J. J. & Reid, R. C. Efficient coding of natural scenes in the lateral geniculate nucleus: experimental test of a computational theory. J. Neurosci. 16, 3351–3362 (1996).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  46. Fairhall, A. L., Lewen, G. D., Bialek, W. & de Ruyter Van Steveninck, R. R. Efficiency and ambiguity in an adaptive neural code. Nature 412, 787–792 (2001).

    CAS  PubMed  Article  Google Scholar 

  47. Brenner, N., Bialek, W. & de Ruyter van Steveninck, R. Adaptive rescaling maximizes information transmission. Neuron 26, 695–702 (2000).

    CAS  PubMed  Article  Google Scholar 

  48. Sharpee, T. O. et al. Adaptive filtering enhances information transmission in visual cortex. Nature 439, 936–942 (2006).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  49. Frechette, E. S. et al. Fidelity of the ensemble code for visual motion in primate retina. J. Neurophysiol. 94, 119–135 (2005).

    CAS  PubMed  Article  Google Scholar 

  50. Chichilnisky, E. J. & Kalmar, R. S. Temporal resolution of ensemble visual motion signals in primate retina. J. Neurosci. 23, 6681–6689 (2003).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  51. Hosoya, T., Baccus, S. A. & Meister, M. Dynamic predictive coding by the retina. Nature 436, 71–77 (2005).

    CAS  PubMed  Article  Google Scholar 

  52. Rao, R. P. & Ballard, D. H. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 2, 79–87 (1999).

    CAS  PubMed  Article  Google Scholar 

  53. Yildizoglu, T., Riegler, C., Fitzgerald, J. E. & Portugues, R. A neural representation of naturalistic motion-guided behavior in the zebrafish brain. Curr. Biol. 30, 2321–2333.e6 (2020).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  54. Zavatone-Veth, J. A., Badwan, B. A. & Clark, D. A. A minimal synaptic model for direction selective neurons in drosophila. J. Vis. 20, 2 (2020).

    PubMed  PubMed Central  Article  Google Scholar 

  55. Appleby, T. R. & Manookin, M. B. Neural sensitization improves encoding fidelity in the primate retina. Nat. Commun. 10, 4017 (2019).

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  56. Shannon, C. E. A mathematical theory of communication. Bell Syst. Tech. J. 27, 379–423 (1948).

    Article  Google Scholar 

  57. Bialek, W. Biophysics: Searching for Principles (Princeton Univ. Press, 2012).

  58. Chien, J.-F. Encoding the Light Intensity in Retina’s Firing Rate. Master’s thesis, National Taiwan University, Taipei (2017).

  59. Chen, K. S., Chen, C.-C. & Chan, C. K. Characterization of predictive behavior of a retina by mutual information. Front. Comput. Neurosci. 11, 66 (2017).

    PubMed  PubMed Central  Article  Google Scholar 

  60. Strong, S. P., de Ruyter van Steveninck, R. R., Bialek, W. & Koberle, R. On the application of information theory to neural spike trains. Pac. Symp. Biocomput. 1998, 621–632 (1998).

    Google Scholar 

  61. Slonim, N. & Tishby, N. in Advances in Neural Information Processing Systems Vol 12 (eds. Solla, S. A. et al.) 617–623 (MIT Press, 2000).

  62. Chichilnisky, E. J. A simple white noise analysis of neuronal light responses. Network 12, 199–213 (2001).

    CAS  PubMed  Article  Google Scholar 

  63. Cafaro, J. & Rieke, F. Noise correlations improve response fidelity and stimulus encoding. Nature 468, 964–967 (2010).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  64. Cafaro, J. & Rieke, F. Regulation of spatial selectivity by crossover inhibition. J. Neurosci. 33, 6310–6320 (2013).

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  65. Fisher, R. A. The arrangement of field experiments. J. Ministry Agric. Great Britain 33, 503–513 (1926).

    Google Scholar 

  66. Boycott, B. B. & Wässle, H. Morphological classification of bipolar cells of the primate retina. Eur. J. Neurosci. 3, 1069–1088 (1991).

    PubMed  Article  Google Scholar 

  67. Tsukamoto, Y. & Omi, N. ON bipolar cells in macaque retina: type-specific synaptic connectivity with special reference to OFF counterparts. Front. Neuroanat. 10, 104 (2016).

    PubMed  PubMed Central  Article  Google Scholar 

  68. Tsukamoto, Y. & Omi, N. OFF bipolar cells in macaque retina: type-specific connectivity in the outer and inner synaptic layers. Front. Neuroanat. 9, 122 (2015).

    PubMed  PubMed Central  Google Scholar 

  69. Hassenstein, B. & Reichardt, W. Systemtheoretische Analyse der Zeit-, Reihenfolgen- und Vorzeichenauswertung bei der Bewegungsperzeption des Rüsselkäfers Chlorosphanus. Z. Naturforsch. B 11, 513–524 (1956).

    Article  Google Scholar 

Download references

Acknowledgements

We thank S. Cunnington for technical assistance. Tissue was provided by the Tissue Distribution Program at the Washington National Primate Research Center (WaNPRC; supported through NIH grant P51 OD-010425) and we thank the WaNPRC staff, particularly C. English and A. Baldessari, for making these experiments possible. C. Chen assisted in tissue preparation. We thank S. Palmer, S. Wang and G. Gutierrez for helpful discussions, and C.-C. Chiao for supporting B.L. and A.H. This work was supported in part by grants from the NIH (NEI R01-EY027323 to M.B.M.; NEI R01-EY029247 to E.J. Chichilnisky, F.R., and M.B.M.; NEI R01-EY028542 to F.R.; NEI P30-EY001730 to the Vision Core), Research to Prevent Blindness Unrestricted Grant (to the University of Washington Department of Ophthalmology), Latham Vision Research Innovation Award (to M.B.M.), the Alcon Young Investigator Award (to M.B.M.), the Taiwanese Ministry of Science and Technology (108-2813-C-007-085-B to A.H.) and a travel award to B.L. and A.H. from the Taiwanese Ministry of Education (C.-C. Chiao, principal investigator).

Author information

Authors and Affiliations

Authors

Contributions

M.B.M. conceived and designed the study. F.R. and M.B.M. performed the experiments. B.L., A.H., F.R. and M.B.M. designed the analytical techniques. M.B.M. wrote the analysis code and constructed the computational models. B.L., A.H., F.R. and M.B.M. interpreted the results. M.B.M wrote the original version of the manuscript. B.L., A.H., F.R. and M.B.M. reviewed and edited the manuscript.

Corresponding author

Correspondence to Michael B. Manookin.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review informationNature Neuroscience thanks Botond Roska, Gregory Schwartz and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Overview of time-shifted mutual information calculations.

To quantify the amount of information that a neuron’s response at a particular time (Rt) contained about the past and future stimulus trajectory (St+Δt), we computed the time-shifted mutual information. The stimulus sequence was shifted relative to the response and mutual information was computed between the stimulus and response for each time shift (Δt). This was done for time shifts between ± 0.5 seconds.

Extended Data Fig. 2 Information encoding occurs on short timescales.

a, Example spike train for a one-second portion of a stimulus containing diverging positive correlations (top). To measure the relationship between spike timing and information transmission, each spike was randomly shifted forward or backward in time (±Δ, bottom). The process was repeated for each spike train in each cell at several time shifts (0–100 ms). b, Spike timing shift at which the sensitivity index increased by 5% relative to the condition with spike time unaltered (Δ, 0 ms). Data are shown across six stimulus conditions and six ganglion cell types. Number of cells in the population are indicated in parentheses. c, Error in information estimates determined by randomly shuffling spike times (open circles) or by using the spike times from the uncorrelated traces in the same cell (closed circles). Error estimates were not significantly different between these techniques (p-value: two-positive, 0.40; two-negative, 0.78; diverging positive, 0.11; converging positive, 0.34; diverging negative, 0.47; converging negative, 0.78; two-sided Wilcoxon signed rank test; n = 73 cells). Circles and error bars indicate mean ± SEM.

Extended Data Fig. 3 Local motion signals show temporally shifted profiles.

Local motion signals show temporally shifted profiles. Third-order spatiotemporal correlations were extracted directly from stimulus sequences. The time-shifted mutual information profile for diverging correlations showed shifts toward future time points, whereas converging information was shifted toward past time lags.

Supplementary information

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Liu, B., Hong, A., Rieke, F. et al. Predictive encoding of motion begins in the primate retina. Nat Neurosci 24, 1280–1291 (2021). https://doi.org/10.1038/s41593-021-00899-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41593-021-00899-1

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing