Flies and humans share a motion estimation strategy that exploits natural scene statistics

Journal name:
Nature Neuroscience
Volume:
17,
Pages:
296–303
Year published:
DOI:
doi:10.1038/nn.3600
Received
Accepted
Published online

Abstract

Sighted animals extract motion information from visual scenes by processing spatiotemporal patterns of light falling on the retina. The dominant models for motion estimation exploit intensity correlations only between pairs of points in space and time. Moving natural scenes, however, contain more complex correlations. We found that fly and human visual systems encode the combined direction and contrast polarity of moving edges using triple correlations that enhance motion estimation in natural environments. Both species extracted triple correlations with neural substrates tuned for light or dark edges, and sensitivity to specific triple correlations was retained even as light and dark edge motion signals were combined. Thus, both species separately process light and dark image contrasts to capture motion signatures that can improve estimation accuracy. This convergence argues that statistical structures in natural scenes have greatly affected visual processing, driving a common computational strategy over 500 million years of evolution.

At a glance

Figures

  1. Multiple correlations signify natural image motion.
    Figure 1: Multiple correlations signify natural image motion.

    Each row presents a comparison between correlational motion signatures. Columns present context for each comparison (1), properties of pairwise motion estimators (2), properties of diverging three-point estimators (3) and properties of converging three-point estimators (4). (a) Motion is approximated by the rigid translation of natural images (1). Cartoons of the correlation structure that each estimator detects are shown (2–4). (b) Example natural image (1)19. Pixel-wise contributions to motion estimation were highly variable and differed across estimators (2–4). Red and blue pixels indicate opposite directions of motion. (c) Examples from an ensemble of natural images (1). The accuracy with which correlations convey motion was examined across this ensemble (2–4). The performance of each estimator was quantified through the Pearson's correlation coefficient between the estimator output and the simulated velocity. We linearly combined estimators to quantify the improvements afforded by multiple correlational signals. The numbers above each bar denote the fractional increases with respect to the two-point estimate. (d) Data presented as in c, but with signals spatiotemporally filtered to match motion processing in Drosophila. Error bars represent s.d. over cross-validating trials (Online Methods).

  2. Triple correlations distinguish between light and dark moving edges.
    Figure 2: Triple correlations distinguish between light and dark moving edges.

    (a) A dark point becomes light when a light edge moves across the visual field (rows 1 and 3), and a light point becomes dark when a dark edge moves across the visual field (rows 2 and 4). We decomposed the net pair correlation motion signal into four elements whose frequency of occurrence depends on the motion. This net pair correlation motion signal reflects the direction of motion (compare rows 1 and 2 with rows 3 and 4) and is insensitive to whether the edge was light or dark (compare row 1 with row 2 or row 3 with row 4). (b) We similarly decomposed the net diverging and converging triple correlations into four elements (shown for the diverging triple correlation). The sign of the net diverging triple correlation depends both on the contrast polarity of the edge and on the direction of motion (shown for rightward motion). Thus, triple correlations jointly encode the direction and contrast polarity of a moving edge. (c) Natural motion comprised both moving light edges and moving dark edges. Motion signals were associated with each moving edge, but only the three-point motion signatures distinguished between edge contrast polarities.

  3. Drosophila responds to triple correlations.
    Figure 3: Drosophila responds to triple correlations.

    (a) Binary spatiotemporal patterns, glider stimuli with two- and three-point contrast correlations, were presented to flies. Space-time plots for each of the six gliders, and an uncorrelated stimulus, are shown. (b) During the presentation, we measured flies' turning in response to each glider. Positive rotational velocities represent turning in the direction of the 'centroid' of the pattern (to the right in the space-time plots in a). (c) We interleaved 1-s periods of glider stimuli with uncorrelated (Unc) stimuli; the timing of the presentation of the gliders is denoted by the thick black bar. Response curves show the mean (solid line) and s.e.m. (shading) over flies. (d) Mean turning velocities were computed for each glider by averaging over 0.5 s of the stimulus (gray bar in c). Turning responses are presented for wild-type Drosophila, alongside the predicted response of an HRC to each glider (n = 12 in c and d). ** denotes a difference from 0 at the P < 0.01 level (two-tailed t test); from right to left, the marked P values are 4.4 × 10−3 (t11 = 3.6), 6.0 × 10−7 (t11 = 10.2), 8.2 × 10−4 (t11 = 4.6) and 3.7 × 10−6 (t11 = 8.5). Significance was not tested for the two-point gliders. Error bars represent s.e.m.

  4. Detection of triple correlations associated with specific pathways in Drosophila.
    Figure 4: Detection of triple correlations associated with specific pathways in Drosophila.

    (a) Left, schematic of the inputs to the fly motion processing pathways. Signals from photoreceptors are relayed through the lamina monopolar cells L1, L2 and L3. Right, a temperature-inducible dominant-negative suppressor of synaptic transmission (shits) was used to silence L1, L2 and L3 using cell-specific expression of Gal4 (L1 shown, red). (b) We examined the responses of these disrupted motion detectors to three-point gliders. Responses are plotted relative to the two-point positive glider response. The two control genotypes (Gal4/+ and +/shits) have all input pathways intact, but contain the genetic constructs for the experimental genotype (Gal4/shits). For the genotypes Gal4/shits, Gal4/+ and +/shits, from top to bottom, N = (19, 14, 19), (18, 13, 19), (29, 16, 19), (22, 14, 19) and (17, 15, 19). Error bars represent ±s.e.m. *P < 0.01, **P < 0.001, different from both control genotypes (two-tailed t test).

  5. Triple correlation responses predict the edge selectivity of motion pathways.
    Figure 5: Triple correlation responses predict the edge selectivity of motion pathways.

    (a) The frequency of correlational elements in a moving edge depends on its contrast polarity and direction (Fig. 2), and we computed the relative abundance of each correlational element from the difference in frequency of each element in rightward versus leftward motion (Supplementary Fig. 6). The relative abundances of the four triple correlation elements differed between light and dark edges. (b) We used the relative abundance of each correlational element in each edge type (Fig. 2 and Supplementary Fig. 6) to weight and sum the response of each genotype to each correlational element (Fig. 4). This generated the glider-predicted responses to each edge type, from which we computed the predicted edge selectivity for each genotype. It correlated highly with the behaviorally measured edge selectivity (Online Methods). Edge selectivity was computed to be the light minus dark edge responses divided by their sum. Error bars represent ±s.e.m.

  6. Humans differentially adapt to moving light and dark edges.
    Figure 6: Humans differentially adapt to moving light and dark edges.

    (a) Schematic of adaptor and probe stimulus procedure (Supplementary Fig. 7). Black box denotes the time interval used for analysis. (b) Scalp topography of the amplitude of the response at the A′/B′ alternation rate (3 Hz). The amplitude peaked near the occipital pole. (c) Time average of the response from the peak electrode to the probe stimulus under the two adaptation regimes. Response to the unadapted state obtained by probe presentation without the adapting stimulus has been subtracted from this signal (Supplementary Fig. 7). The response to the probe revealed complementary modulation by the adapting stimuli at the frequency of probe alternation (3 Hz). Gray area represents ±1 s.e.m. (d) The within-subject difference of phase and amplitude at 3 Hz between the two adapting conditions. Ellipse represents 1 s.e.m. and the shaded wedge indicates the 95% confidence interval for the phase (n = 7 subjects in c and d).

  7. Adaptation to moving light and dark edges differentially affects the perception of specific three-point gliders.
    Figure 7: Adaptation to moving light and dark edges differentially affects the perception of specific three-point gliders.

    Subjects were presented all combinations of four types of adaptor stimuli (left column) and eight gliders (right), and asked to report the direction of perceived glider motion. Results for each of the eight glider stimuli are shown, grouped by glider. The color of the bar corresponds to the adapting stimulus: static (black), opposing edges (gray), light edges only (green) or dark edges only (magenta). All stimuli were presented mirror-symmetrically and responses were aligned to the direction shown in the left hand column. *P = 1.6 × 10−3 (t16 = 5.6), **P = 8.8 × 10−4 (t14 = 6.2) and ***P = 2.8 × 10−6 (t14 = 10.2) (differences between conditions, two-tailed t test, Bonferroni corrected for 40 comparisons). N = 9 subjects for static and opposing edge adaptation, N = 7 for light and dark edge adaptation conditions. Error bars represent ± s.e.m.

  8. Models of motion estimation.
    Supplementary Fig. 1: Models of motion estimation.

    (a) The classical Hassenstein-Reichardt Correlator (HRC). In this paper, h1(x) and h2(x) were modeled as Gaussian spatial acceptance filters (centered on different points in visual space), f(t) was a low-pass filter, and g(t) was a high-pass filter. In essence, the HRC multiplies delayed values of the contrast with current contrast values across two spatial points. The subtraction stage results in mirror symmetry, thereby enabling responses to both rightward and leftward motion. (b) The classical motion energy (ME) model applies several oriented spatiotemporal filters to the visual input. These filtered signals are subsequently squared and linearly combined to compute the 'motion energy.' (c) To compute the 2- and 3-point correlation images, we computed the product with the rightward orientation (left) and then subtracted the mirror symmetric product with a leftward orientation (center). This results in the images on the right, which are also displayed in Figure 1b. (d) HRC-like diagrams for the diverging and converging 3-point correlators used in Figure 1d.

  9. Triple correlations only signify motion when the stimulus is light-dark asymmetric.
    Supplementary Fig. 2: Triple correlations only signify motion when the stimulus is light-dark asymmetric.

    Each row presents a comparison between correlational motion signatures. Columns present: (i) context for each comparison; (ii) properties of pairwise motion estimators; (iii) properties of diverging 3-point estimators; and (iv) properties of converging 3-point estimators. (ai) Motion is approximated by the rigid translation of images. (aii-aiv) Cartoon of the correlation structure that each estimator detects. (bi) Example sinusoidal grating. (bii) Pair correlations signified motion across the image. (biii-biv) Triple correlations depended on the local phase of the sinusoidal grating and spatially averaged to zero. (ci) Example asymmetric grating. The luminance at each point in space was the luminance of the example sinusoidal grating raised to the tenth power. (cii) Pair correlations varied across the image. (ciii-iciv) Triple correlations still depended on the local phase of the grating, but their spatial average was nonzero. (di) Cartoon of an ensemble of sinusoidal gratings that vary in period and phase. (dii-div) The accuracy with which correlations convey motion was examined across this ensemble. The performance of each estimator was quantified through the Pearson's correlation between the estimator output and the simulated velocity. We linearly combined estimators to quantify the improvements afforded by multiple correlational signals. Neither spatial averaging nor triple correlations improved the motion estimate. (e) Same as (d), but for asymmetric gratings. In this case, both spatial averaging and triple correlations improved the accuracy of motion estimation. The numbers above each bar denote the fractional increase with respect to the 2-point estimate. Error bars are standard deviations over cross-validating trials (see Online Methods).

  10. Correlational motion estimation with spatially varying contrast gain.
    Supplementary Fig. 3: Correlational motion estimation with spatially varying contrast gain.

    Rows a-c present a comparison between correlational motion signatures, when (a) the contrast gain is set locally by considering the average luminance over one degree squares of pixels, (b) the contrast gain is set locally by considering the average luminance over five degree squares of pixels, (c) the contrast gain is set globally by considering the average luminance over the full image. Columns apply to rows a-c and present: (i) example local luminance average, which sets the contrast gain; (ii) accuracy of pairwise motion estimators; (iii) accuracy of diverging 3-point estimators; and (iv) accuracy of converging 3-point estimators. Columns (ii)-(iv) are of the same format as in Figs. 1 and S2, and Figs. S3cii-civ are identical to Figs. 1cii-iv. Error bars are standard deviations over cross-validating trials (see Online Methods). (d) Contrast histograms when the contrast gain was determined by averaging over one degree squares of pixels (left), five degree squares of pixels (center), or the full visual field (right). The mean (i.e. c̄ = left fencecright fence), variance (i.e. left fence(c−)2right fence), and third central moment (i.e. left fence(c−)3right fence) are shown alongside each histogram.

  11. Glider construction and model responses.
    Supplementary Fig. 4: Glider construction and model responses.

    (a) Diagrams of the update rules that generate glider stimuli (see also Online Methods). Given a seed row and a seed column of pixel contrasts (upper left), the glider update rules fill in all remaining pixels, one row at a time, to generate an instantiation of the glider. The red points in the diagrams exemplify the update rule for each glider. The illustrative choices are not special, as any such pixel combinations will obey the update rule by construction. (b) Within a 2-point glider, all 3-point correlations average to 0. Similarly, within a 3-point glider, 2-point correlations (and the other 3-point correlations) average to 0. (c) Example space-time plots of the glider stimuli. (d) The ON/OFF model proposed in Eichner et al.27 correctly predicted the signs of the 2-point glider responses but did not predict the observed 3-point glider responses. Error bars are SEM, as in Fig. 3.

  12. Drosophila respond to several triple correlations involving two points in space and three points in time.
    Supplementary Fig. 5: Drosophila respond to several triple correlations involving two points in space and three points in time.

    The correlation structures for each glider and sample space-time intensity plots are shown at top, and the behavioral responses (relative to the positive 2-pt glider response) are shown below. We found that flies respond less strongly to these gliders than to the diverging and converging gliders (compare to Fig. 3d). We measured statistically significant responses in only 2 of 6 cases (two-tailed t-test, '*' corresponds to p=2.0×10-2 (t15=2.6) and '**' corresponds to p=3.7×10-3 (t15=3.4)). Error bars are SEM and N=16.

  13. We computed the abundance of each correlational element in each edge type.
    Supplementary Fig. 6: We computed the abundance of each correlational element in each edge type.

    We first counted the number of times that each correlational element appeared in right and left-moving edges of the same polarity. The difference between the rightward and leftward counts provided an edge-specific directional signal (denoted “Net” in the figure). As expected, these directional signals depended on edge type for 3-point correlational elements but not for 2-point correlational elements. In particular, the sign of each 3-point directional signal inverted when the edge polarity inverted. In Figure 5, we used these directional signals as linear weighting coefficients to predict Drosophila behavioral responses to moving edges from measured glider responses.

  14. EEG experimental details.
    Supplementary Fig. 7: EEG experimental details.

    (a) Space-time plots of the two mirror-symmetric opposing edge adapters. The light and dark edges moved in opposite directions and are highlighted by the green and purple lines, respectively. Each presentation of the adapters had a random spatial phase. See also movie M1 for examples of the adapters. The probe temporally alternated between adapter A and adapter B. The spatial period of the adapting stimuli was doubled in the probe stimulus, but the speed of the edges remained the same. Green and purple lines highlight moving edges and show that the directions of the light and dark edges invert during the two halves of the probe. The probe was always presented with the same spatial phase. See also movie M2 for an example of the probe. (b) Evoked response waveforms for the probe stimuli under unadapted and adapted conditions. Data were only from the time interval shown in Figure 6a (and in panel c below). The data show that responses to identical stimuli were differentially affected by the identity of the adapter. (c) Strength of the first harmonic response as a function of time after the end of the adapting period. Both adapted responses were above baseline for approximately 5 seconds. Error patches on lines represent 1 SEM. (d) Phase-amplitude plots for the adapted EEG responses shown in Figure 6c. Ellipses represent 1 SEM.

  15. Human psychophysics experimental details.
    Supplementary Fig. 8: Human psychophysics experimental details.

    (a-b) Space-time intensity plots of the adapters and gliders. (a) Subjects' visual systems were adapted with a static adapter, with opposing edge motion, with light edge motion, and with dark edge motion. Each presentation of the adapters had a random spatial phase. See also movie M1 for an example opposing edge stimulus. (b) Space-time plots corresponding to a single row of each glider stimulus (shown after adaptation). All centroids move to the right in the top row and to the left in the bottom row. See also movie M3 for example glider stimuli. (c) Individual subject responses to the glider stimuli following adaptation to the static adapter. Individual subjects' responses are coded by color. The underlying bar plot shows subject means and SEMs for each glider. (d) Two out of nine subjects perceived an overwhelming motion after-effect after adaptation to light or dark edge motion. Shown here is one of these subject's glider responses to the gliders after adaptation to rightward-moving light edges. All responses were to the left, a result that we interpreted as a motion after-effect resulting from net motion in the adapter. The opposing edge adapter avoided this problem.

Videos

  1. Opposing edges.
    Video 1: Opposing edges.
    This stimulus was designed to be equiluminant in time. Light edges move to the right, and dark edges move to the left. In the EEG experiment, this stimulus was full screen. See Fig. S7 for a space-time diagram of this stimulus.
  2. Probe edges.
    Video 2: Probe edges.
    This stimulus consisted of two alternating versions of the opposing edge stimuli. Light edges moved to the right for half of the probe and to the left for the other half. Dark and light edges always moved in opposite directions. See Fig. S7 for a space-time diagram of this stimulus.
  3. Glider stimuli.
    Video 3: Glider stimuli.
    Here each 3-pt glider stimulus is presented sequentially: positive diverging, negative diverging, positive converging, and negative converging. All glider centroids moved to the right. Most subjects perceived leftward motion in the second glider and rightward motion in the first, third, and fourth gliders (see Figure 7 and S8).

References

  1. Field, D.J. Relations between the statistics of natural images and the response properties of cortical cells. J. Opt. Soc. Am. A 4, 23792394 (1987).
  2. Ruderman, D.L. & Bialek, W. Statistics of natural images: scaling in the woods. Phys. Rev. Lett. 73, 814817 (1994).
  3. Simoncelli, E.P. & Olshausen, B.A. Natural image statistics and neural representation. Annu. Rev. Neurosci. 24, 11931216 (2001).
  4. Hassenstein, B. & Reichardt, W. Systemtheoretische Analyse der Zeit-, Reihenfolgen-und Vorzeichenauswertung bei der Bewegungsperzeption des Rüsselkäfers Chlorophanus. Z. Naturforsch. 11, 513524 (1956).
  5. Adelson, E.H. & Bergen, J. Spatiotemporal energy models for the perception of motion. J. Opt. Soc. Am. A 2, 284299 (1985).
  6. Fitzgerald, J.E., Katsov, A.Y., Clandinin, T.R. & Schnitzer, M.J. Symmetries in stimulus statistics shape the form of visual motion estimators. Proc. Natl. Acad. Sci. USA 108, 1290912914 (2011).
  7. Anderson, J.M.M. & Giannakis, G.B. Image motion estimation algorithms using cumulants. IEEE Trans. Image Process. 4, 346357 (1995).
  8. Sayrol, E., Gasull, A. & Fonollosa, J.R. Motion estimation using higher order statistics. IEEE Trans. Image Process. 5, 10771084 (1996).
  9. Rust, N.C., Mante, V., Simoncelli, E.P. & Movshon, J.A. How MT cells analyze the motion of visual patterns. Nat. Neurosci. 9, 14211431 (2006).
  10. Borst, A., Haag, J. & Reiff, D.F. Fly motion vision. Annu. Rev. Neurosci. 33, 4970 (2010).
  11. van Santen, J.P.H. & Sperling, G. Elaborated reichardt detectors. J. Opt. Soc. Am. A 2, 300321 (1985).
  12. Theobald, J.C., Duistermars, B.J., Ringach, D.L. & Frye, M.A. Flies see second-order motion. Curr. Biol. 18, R464R465 (2008).
  13. Zanker, J.M. Theta motion: a paradoxical stimulus to explore higher order motion extraction. Vision Res. 33, 553569 (1993).
  14. Hu, Q. & Victor, J.D. A set of high-order spatiotemporal stimuli that elicit motion and reverse-phi percepts. J. Vis. 10, 9.19.16 (2010).
  15. Lu, Z.L. & Sperling, G. Three-systems theory of human visual motion perception: review and update. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 18, 23312370 (2001).
  16. Victor, J.D. & Conte, M.M. Evoked potential and psychophysical analysis of Fourier and non-Fourier motion mechanisms. Vis. Neurosci. 9, 105123 (1992).
  17. Geisler, W.S. Visual perception and the statistical properties of natural scenes. Annu. Rev. Psychol. 59, 167192 (2008).
  18. Dror, R.O., O'Carroll, D.C. & Laughlin, S.B. Accuracy of velocity estimation by Reichardt correlators. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 18, 241252 (2001).
  19. van Hateren, J.H. & van der Schaaf, A. Independent component filters of natural images compared with simple cells in primary visual cortex. Proc. Biol. Sci. 265, 359366 (1998).
  20. Stocker, A.A. & Simoncelli, E.P. Noise characteristics and prior expectations in human visual speed perception. Nat. Neurosci. 9, 578585 (2006).
  21. Katsov, A.Y. & Clandinin, T. Motion processing streams in Drosophila are behaviorally specialized. Neuron 59, 322335 (2008).
  22. Joesch, M., Schnell, B., Raghu, S., Reiff, D. & Borst, A. ON and OFF pathways in Drosophila motion vision. Nature 468, 300304 (2010).
  23. Clark, D.A., Bursztyn, L., Horowitz, M.A., Schnitzer, M.J. & Clandinin, T.R. Defining the computational structure of the motion detector in Drosophila. Neuron 70, 11651177 (2011).
  24. Silies, M. et al. Modular use of peripheral input channels tunes motion-detecting circuitry. Neuron 79, 111127 (2013).
  25. Buchner, E. Elementary movement detectors in an insect visual system. Biol. Cybern. 24, 85101 (1976).
  26. Tuthill, J.C., Chiappe, M.E. & Reiser, M.B. Neural correlates of illusory motion perception in Drosophila. Proc. Natl. Acad. Sci. USA 108, 96859690 (2011).
  27. Eichner, H., Joesch, M., Schnell, B., Reiff, D.F. & Borst, A. Internal structure of the fly elementary motion detector. Neuron 70, 11551164 (2011).
  28. Tuthill, J.C., Nern, A., Holtz, S.L., Rubin, G.M. & Reiser, M.B. Contributions of the 12 neuron classes in the fly lamina to motion vision. Neuron 79, 128140 (2013).
  29. Westheimer, G. The ON-OFF dichotomy in visual processing: From receptors to perception. Prog. Retin. Eye Res. 26, 636648 (2007).
  30. Ales, J.M. & Norcia, A.M. Assessing direction-specific adaptation using the steady-state visual evoked potential: results from EEG source imaging. J. Vis. 9, 8 (2009).
  31. Moulden, B. & Begg, H. Some tests of the Marr-Ullman model of movement detection. Perception 15, 139 (1986).
  32. Mather, G., Moulden, B. & O'Halloran, A. Polarity-specific adaptation to motion in the human visual system. Vision Res. 31, 10131019 (1991).
  33. Emerson, R.C., Bergen, J.R. & Adelson, E.H. Directionally selective complex cells and the computation of motion energy in cat visual cortex. Vision Res. 32, 203218 (1992).
  34. Nishida, S.y. Advancement of motion psychophysics: review 2001–2010. J. Vis. 11, 11 (2011).
  35. Quenzer, T. & Zanker, J. Visual detection of paradoxical motion in flies. J. Comp. Physiol. A Neuroethol. Sens. Neural Behav. Physiol. 169, 331340 (1991).
  36. Ilg, U.J. & Churan, J. Motion perception without explicit activity in areas MT and MST. J. Neurophysiol. 92, 15121523 (2004).
  37. Hedges, J.H. et al. Dissociation of neuronal and psychophysical responses to local and global motion. Curr. Biol. 21, 20232028 (2011).
  38. Ratliff, C.P., Borghuis, B.G., Kao, Y.-H., Sterling, P. & Balasubramanian, V. Retina is structured to process an excess of darkness in natural scenes. Proc. Natl. Acad. Sci. USA 107, 1736817373 (2010).
  39. Barlow, H.B. & Levick, W.R. The mechanism of directionally selective units in rabbit's retina. J. Physiol. (Lond.) 178, 477 (1965).
  40. Ibn-elhaj, E., Aboutajdine, D., Pateux, S. & Morin, L. HOS-based method of global motion estimation for noisy image sequences. Electron. Lett. 35, 13201322 (1999).
  41. Schiller, P.H., Finlay, B.L. & Volman, S.F. Quantitative studies of single-cell properties in monkey striate cortex. I. Spatiotemporal organization of receptive fields. J. Neurophysiol. 39, 1288 (1976).
  42. Potetz, B. & Lee, T.S. Statistical correlations between two-dimensional images and three-dimensional structures in natural scenes. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 20, 12921303 (2003).
  43. Wiederman, S.D., Shoemaker, P.A. & O'Carroll, D.C. A model for the detection of moving targets in visual clutter inspired by insect physiology. PLoS ONE 3, e2784 (2008).
  44. Poggio, T. & Reichardt, W. Visual control of orientation behaviour in the fly. Q. Rev. Biophys. 9, 377438 (1976).
  45. Card, G. & Dickinson, M.H. Visually mediated motor planning in the escape response of Drosophila. Curr. Biol. 18, 13001307 (2008).
  46. Harris, J.M. & Parker, A.J. Independent neural mechanisms for bright and dark information in binocular stereopsis. Nature 374, 808811 (1995).
  47. Samonds, J.M., Potetz, B.R. & Lee, T.S. Relative luminance and binocular disparity preferences are correlated in macaque primary visual cortex, matching natural scene statistics. Proc. Natl. Acad. Sci. USA 109, 63136318 (2012).
  48. Bialek, W. Physical limits to sensation and perception. Annu. Rev. Biophys. Biophys. Chem. 16, 455478 (1987).
  49. Laughlin, S.B. Energy as a constraint on the coding and processing of sensory information. Curr. Opin. Neurobiol. 11, 475480 (2001).
  50. Laughlin, S.B. & Sejnowski, T.J. Communication in neuronal networks. Science 301, 18701874 (2003).
  51. Potters, M. & Bialek, W. Statistical mechanics and visual signal processing. J. Phys. I 4, 17551775 (1994).
  52. Juusola, M., Uusitalo, R. & Weckström, M. Transfer of graded potentials at the photoreceptor-interneuron synapse. J. Gen. Physiol. 105, 117 (1995).
  53. Gohl, D.M. et al. A versatile in vivo system for directed dissection of gene expression patterns. Nat. Methods 8, 231237 (2011).
  54. Kitamoto, T. Conditional modification of behavior in Drosophila by targeted expression of a temperature-sensitive shibire allele in defined neurons. J. Neurobiol. 47, 8192 (2001).
  55. Stavenga, D.G. Angular and spectral sensitivity of fly photoreceptors. II. Dependence on facet lens F-number and rhabdomere type in Drosophila. J. Comp. Physiol. A Neuroethol. Sens. Neural Behav. Physiol. 189, 189202 (2003).
  56. Palomares, M., Ales, J.M., Wade, A.R., Cottereau, B.R. & Norcia, A.M. Distinct effects of attention on the neural responses to form and motion processing: A SSVEP source-imaging study. J. Vis. 12, 15 (2012).
  57. Berens, P. CircStat: A MATLAB toolbox for circular statistics. J. Stat. Softw. 31, 121 (2009).
  58. Ales, J.M., Farzin, F., Rossion, B. & Norcia, A.M. An objective method for measuring face detection thresholds using the sweep steady-state visual evoked response. J. Vis. 12, 18 (2012).
  59. Gardner, M. Mathematical games: the fantastic combinations of John Conway's new solitaire game, 'life'. Sci. Am. 223, 120123 (1970).

Download references

Author information

  1. These authors contributed equally to this work.

    • Damon A Clark,
    • James E Fitzgerald &
    • Justin M Ales

Affiliations

  1. Department of Neurobiology, Stanford University, Stanford, California, USA.

    • Damon A Clark,
    • Daryl M Gohl,
    • Marion A Silies &
    • Thomas R Clandinin
  2. Department of Physics, Stanford University, Stanford, California, USA.

    • James E Fitzgerald
  3. Department of Psychology, Stanford University, Stanford, California, USA.

    • Justin M Ales &
    • Anthony M Norcia
  4. Present addresses: Department of Molecular, Cellular and Developmental Biology, Yale University, New Haven, Connecticut, USA (D.A.C.), Center for Brain Science, Harvard University, Cambridge, Massachusetts, USA (J.E.F.), School of Psychology and Neuroscience, University of St Andrews, St Andrews, UK (J.M.A.).

    • Damon A Clark,
    • James E Fitzgerald &
    • Justin M Ales

Contributions

D.A.C., J.E.F. and J.M.A. designed and carried out the experiments. J.E.F. performed the natural scenes analysis and developed the theoretical modeling framework. D.A.C. performed the fly experiments. D.A.C. and J.M.A. performed the human experiments. D.M.G. and M.A.S. contributed reagents and fly edge selectivity measurements. D.A.C., J.E.F., J.M.A., A.M.N. and T.R.C. interpreted the data and wrote the paper.

Competing financial interests

The authors declare no competing financial interests.

Corresponding authors

Correspondence to:

Author details

Supplementary information

Supplementary Figures

  1. Supplementary Figure 1: Models of motion estimation. (331 KB)

    (a) The classical Hassenstein-Reichardt Correlator (HRC). In this paper, h1(x) and h2(x) were modeled as Gaussian spatial acceptance filters (centered on different points in visual space), f(t) was a low-pass filter, and g(t) was a high-pass filter. In essence, the HRC multiplies delayed values of the contrast with current contrast values across two spatial points. The subtraction stage results in mirror symmetry, thereby enabling responses to both rightward and leftward motion. (b) The classical motion energy (ME) model applies several oriented spatiotemporal filters to the visual input. These filtered signals are subsequently squared and linearly combined to compute the 'motion energy.' (c) To compute the 2- and 3-point correlation images, we computed the product with the rightward orientation (left) and then subtracted the mirror symmetric product with a leftward orientation (center). This results in the images on the right, which are also displayed in Figure 1b. (d) HRC-like diagrams for the diverging and converging 3-point correlators used in Figure 1d.

  2. Supplementary Figure 2: Triple correlations only signify motion when the stimulus is light-dark asymmetric. (193 KB)

    Each row presents a comparison between correlational motion signatures. Columns present: (i) context for each comparison; (ii) properties of pairwise motion estimators; (iii) properties of diverging 3-point estimators; and (iv) properties of converging 3-point estimators. (ai) Motion is approximated by the rigid translation of images. (aii-aiv) Cartoon of the correlation structure that each estimator detects. (bi) Example sinusoidal grating. (bii) Pair correlations signified motion across the image. (biii-biv) Triple correlations depended on the local phase of the sinusoidal grating and spatially averaged to zero. (ci) Example asymmetric grating. The luminance at each point in space was the luminance of the example sinusoidal grating raised to the tenth power. (cii) Pair correlations varied across the image. (ciii-iciv) Triple correlations still depended on the local phase of the grating, but their spatial average was nonzero. (di) Cartoon of an ensemble of sinusoidal gratings that vary in period and phase. (dii-div) The accuracy with which correlations convey motion was examined across this ensemble. The performance of each estimator was quantified through the Pearson's correlation between the estimator output and the simulated velocity. We linearly combined estimators to quantify the improvements afforded by multiple correlational signals. Neither spatial averaging nor triple correlations improved the motion estimate. (e) Same as (d), but for asymmetric gratings. In this case, both spatial averaging and triple correlations improved the accuracy of motion estimation. The numbers above each bar denote the fractional increase with respect to the 2-point estimate. Error bars are standard deviations over cross-validating trials (see Online Methods).

  3. Supplementary Figure 3: Correlational motion estimation with spatially varying contrast gain. (181 KB)

    Rows a-c present a comparison between correlational motion signatures, when (a) the contrast gain is set locally by considering the average luminance over one degree squares of pixels, (b) the contrast gain is set locally by considering the average luminance over five degree squares of pixels, (c) the contrast gain is set globally by considering the average luminance over the full image. Columns apply to rows a-c and present: (i) example local luminance average, which sets the contrast gain; (ii) accuracy of pairwise motion estimators; (iii) accuracy of diverging 3-point estimators; and (iv) accuracy of converging 3-point estimators. Columns (ii)-(iv) are of the same format as in Figs. 1 and S2, and Figs. S3cii-civ are identical to Figs. 1cii-iv. Error bars are standard deviations over cross-validating trials (see Online Methods). (d) Contrast histograms when the contrast gain was determined by averaging over one degree squares of pixels (left), five degree squares of pixels (center), or the full visual field (right). The mean (i.e. c̄ = left fencecright fence), variance (i.e. left fence(c−)2right fence), and third central moment (i.e. left fence(c−)3right fence) are shown alongside each histogram.

  4. Supplementary Figure 4: Glider construction and model responses. (359 KB)

    (a) Diagrams of the update rules that generate glider stimuli (see also Online Methods). Given a seed row and a seed column of pixel contrasts (upper left), the glider update rules fill in all remaining pixels, one row at a time, to generate an instantiation of the glider. The red points in the diagrams exemplify the update rule for each glider. The illustrative choices are not special, as any such pixel combinations will obey the update rule by construction. (b) Within a 2-point glider, all 3-point correlations average to 0. Similarly, within a 3-point glider, 2-point correlations (and the other 3-point correlations) average to 0. (c) Example space-time plots of the glider stimuli. (d) The ON/OFF model proposed in Eichner et al.27 correctly predicted the signs of the 2-point glider responses but did not predict the observed 3-point glider responses. Error bars are SEM, as in Fig. 3.

  5. Supplementary Figure 5: Drosophila respond to several triple correlations involving two points in space and three points in time. (135 KB)

    The correlation structures for each glider and sample space-time intensity plots are shown at top, and the behavioral responses (relative to the positive 2-pt glider response) are shown below. We found that flies respond less strongly to these gliders than to the diverging and converging gliders (compare to Fig. 3d). We measured statistically significant responses in only 2 of 6 cases (two-tailed t-test, '*' corresponds to p=2.0×10-2 (t15=2.6) and '**' corresponds to p=3.7×10-3 (t15=3.4)). Error bars are SEM and N=16.

  6. Supplementary Figure 6: We computed the abundance of each correlational element in each edge type. (105 KB)

    We first counted the number of times that each correlational element appeared in right and left-moving edges of the same polarity. The difference between the rightward and leftward counts provided an edge-specific directional signal (denoted “Net” in the figure). As expected, these directional signals depended on edge type for 3-point correlational elements but not for 2-point correlational elements. In particular, the sign of each 3-point directional signal inverted when the edge polarity inverted. In Figure 5, we used these directional signals as linear weighting coefficients to predict Drosophila behavioral responses to moving edges from measured glider responses.

  7. Supplementary Figure 7: EEG experimental details. (239 KB)

    (a) Space-time plots of the two mirror-symmetric opposing edge adapters. The light and dark edges moved in opposite directions and are highlighted by the green and purple lines, respectively. Each presentation of the adapters had a random spatial phase. See also movie M1 for examples of the adapters. The probe temporally alternated between adapter A and adapter B. The spatial period of the adapting stimuli was doubled in the probe stimulus, but the speed of the edges remained the same. Green and purple lines highlight moving edges and show that the directions of the light and dark edges invert during the two halves of the probe. The probe was always presented with the same spatial phase. See also movie M2 for an example of the probe. (b) Evoked response waveforms for the probe stimuli under unadapted and adapted conditions. Data were only from the time interval shown in Figure 6a (and in panel c below). The data show that responses to identical stimuli were differentially affected by the identity of the adapter. (c) Strength of the first harmonic response as a function of time after the end of the adapting period. Both adapted responses were above baseline for approximately 5 seconds. Error patches on lines represent 1 SEM. (d) Phase-amplitude plots for the adapted EEG responses shown in Figure 6c. Ellipses represent 1 SEM.

  8. Supplementary Figure 8: Human psychophysics experimental details. (440 KB)

    (a-b) Space-time intensity plots of the adapters and gliders. (a) Subjects' visual systems were adapted with a static adapter, with opposing edge motion, with light edge motion, and with dark edge motion. Each presentation of the adapters had a random spatial phase. See also movie M1 for an example opposing edge stimulus. (b) Space-time plots corresponding to a single row of each glider stimulus (shown after adaptation). All centroids move to the right in the top row and to the left in the bottom row. See also movie M3 for example glider stimuli. (c) Individual subject responses to the glider stimuli following adaptation to the static adapter. Individual subjects' responses are coded by color. The underlying bar plot shows subject means and SEMs for each glider. (d) Two out of nine subjects perceived an overwhelming motion after-effect after adaptation to light or dark edge motion. Shown here is one of these subject's glider responses to the gliders after adaptation to rightward-moving light edges. All responses were to the left, a result that we interpreted as a motion after-effect resulting from net motion in the adapter. The opposing edge adapter avoided this problem.

Video

  1. Video 1: Opposing edges. (2.46 MB, Download)
    This stimulus was designed to be equiluminant in time. Light edges move to the right, and dark edges move to the left. In the EEG experiment, this stimulus was full screen. See Fig. S7 for a space-time diagram of this stimulus.
  2. Video 2: Probe edges. (1 MB, Download)
    This stimulus consisted of two alternating versions of the opposing edge stimuli. Light edges moved to the right for half of the probe and to the left for the other half. Dark and light edges always moved in opposite directions. See Fig. S7 for a space-time diagram of this stimulus.
  3. Video 3: Glider stimuli. (12.16 MB, Download)
    Here each 3-pt glider stimulus is presented sequentially: positive diverging, negative diverging, positive converging, and negative converging. All glider centroids moved to the right. Most subjects perceived leftward motion in the second glider and rightward motion in the first, third, and fourth gliders (see Figure 7 and S8).

PDF files

  1. Supplementary Text and Figures (13,783 KB)

    Supplementary Figures 1–8, Supplementary Table 1, Supplementary Modeling, Supplementary Movies 1–3

Additional data