Subjects

Abstract

How does the mammalian retina detect motion? This classic problem in visual neuroscience has remained unsolved for 50 years. In search of clues, here we reconstruct Off-type starburst amacrine cells (SACs) and bipolar cells (BCs) in serial electron microscopic images with help from EyeWire, an online community of ‘citizen neuroscientists’. On the basis of quantitative analyses of contact area and branch depth in the retina, we find evidence that one BC type prefers to wire with a SAC dendrite near the SAC soma, whereas another BC type prefers to wire far from the soma. The near type is known to lag the far type in time of visual response. A mathematical model shows how such ‘space–time wiring specificity’ could endow SAC dendrites with receptive fields that are oriented in space–time and therefore respond selectively to stimuli that move in the outward direction from the soma.

Main

Compared to cognitive functions such as language, the visual detection of motion may seem trivial, yet the underlying neural mechanisms have remained elusive for half a century1,2. Some retinal outputs (ganglion cells) respond selectively to visual stimuli moving in particular directions, whereas retinal inputs (photoreceptors) lack direction selectivity (DS). How does DS emerge from the microcircuitry connecting inputs to outputs?

Research on this question has converged upon the SAC (Fig. 1a, b). A SAC dendrite is more strongly activated by motion outward from the cell body to the tip of the dendrite, than by motion in the opposite direction3. Therefore a SAC dendrite exhibits DS, and outward motion is said to be its ‘preferred direction’. Note that it is incorrect to assign a single such direction to a SAC, because each of the cell’s dendrites has its own preferred direction (Fig. 1a). DS persists after blocking inhibitory synaptic transmission4, when the only remaining inputs to SACs are BCs, which are excitatory. As the SAC exhibits DS but its BC inputs exhibit little or none5, DS appears to emerge from the BC–SAC circuit.

Figure 1: Starburst amacrine cell and its direction selectivity.
Figure 1

a, b, Off SAC (red) viewed opposite (a) and perpendicular (b) to the light axis. GCL, ganglion cell layer. Greyscale images from the e2198 data set9. Swellings of distal dendrites are presynaptic boutons (inset). Scale bar, 50 μm. c, We propose that a SAC dendrite is wired to pathways with time lags of visual response that differ by an amount τ. d, A previous model invoked the time lag due to signal conduction in a passive dendrite24. e, The previous model predicts an inward preferred direction for the somatic voltage, contrary to empirical observations3.

Mouse BCs have been classified into multiple types6, with different time lags in visual response7,8. Motion is a spatiotemporal phenomenon: an object at one location appears somewhere else after a time delay. Accordingly, DS might arise because different locations on the SAC dendrite are wired to BC types with different time lags. More specifically, we propose that the proximal BCs (wired near the SAC soma) lag the distal BCs (wired far from the soma).

Such ‘space–time wiring specificity’ could lead to DS as follows (Fig. 1c). Motion outward from the soma will activate the proximal BCs followed by the distal BCs. If the stimulus speed is appropriate for the time lag, signals from both BC groups will reach the SAC dendrite simultaneously, summing to produce a large depolarization. For motion inward towards the soma, BC signals will reach the SAC dendrite asynchronously, causing only small depolarizations. Therefore the dendrite will ‘prefer’ outward motion, as observed experimentally3.

Three-dimensional reconstruction by crowd and machine

We tested our hypothesis by reconstructing Off BC–SAC circuitry using e2198, an existing data set of mouse retinal images from serial block-face scanning electron microscopy (SBEM)9. The e2198 data set was oversegmented by an artificial intelligence into groups of neighbouring voxels that were subsets of individual neurons. These ‘supervoxels’ were assembled by humans into accurate three-dimensional (3D) reconstructions of neurons. For this activity, we hired and trained a small number of workers in the laboratory, and also transformed work into play by mobilizing volunteers through EyeWire, a website that turns 3D reconstruction of neurons into a game of colouring serial electron microscopy images.

Through EyeWire, we wanted to enable anyone, anywhere, to participate in our research. The approach is potentially scalable to extremely large numbers of ‘citizen scientists’10. More importantly, the 3D reconstruction of neurons requires highly developed visuospatial abilities, and we wondered whether a game could be more effective11 than traditional methods of recruiting and creating experts.

In gameplay mode, EyeWire shows a 2D slice through a ‘cube’, an e2198 subvolume of 256 × 256 × 256 greyscale voxels (Fig. 2a). Gameplay consists of two activities: colouring the image near a location, or searching for a new location to colour. Colouring is done by clicking at any location in the 2D slice, which causes the supervoxel containing that location to turn blue. Searching is done by translating and orienting the slice within the cube, and interacting with a 3D rendering of the coloured supervoxels.

Figure 2: EyeWire combines crowd and artificial intelligence.
Figure 2

a, 3D and 2D views in the neuron reconstruction game. b, Precision and recall are two measures of accuracy. c, Accuracy of artificial intelligence (AI), 5,881 EyeWirers, and EyeWirer consensus on reconstruction of a ganglion cell. d, EyeWirer precision and recall increase with number of cubes submitted. Solid lines are median values across 208 EyeWirers who submitted at least 500 cubes, and shaded regions indicate 25th to 75th percentile.

When the player first receives a cube, it already comes with a ‘seed’, a contiguous set of coloured supervoxels. The challenge is to colour all the rest of the supervoxels that belong to the same neuron, and avoid colouring other neurons. Gameplay for a cube terminates when the player clicks ‘submit’, receives a numerical score (Extended Data Fig. 1a), and proceeds to the next cube. Because our artificial intelligence is sufficiently accurate, colouring supervoxels is faster than manually colouring voxels, an older approach to 3D reconstruction12.

The scoring system is designed to reward accurate colouring. This is nontrivial because EyeWire does not know the correct colouring. Each cube is assigned to multiple players (typically 5 to 10), and high scores are earned by players who colour supervoxels that other players also colour. In other words, the scoring system rewards agreement between players, which tends to be the same as rewarding accuracy.

Consensus is used not only to incentivize individual players, but also to enhance the accuracy of the entire system. Any player’s colouring is equivalent to a set of supervoxels. Given the colourings of multiple players starting from the same seed in the same cube, a consensus can be computed by voting on each supervoxel. EyeWirer consensus was much more accurate than any individual EyeWirer (Fig. 2b, c).

Colouring a neuron is more challenging than it sounds. Images are corrupted by noise and other artefacts. Neurites take paths that are difficult to predict, and can branch without warning. Careless errors result from lapses in attention. Extensive practice is required to achieve accuracy. The most accurate EyeWirers (Fig. 2c, top right corner) often had experience with thousands of cubes. Improvements in accuracy were observed over the course of hundreds of cubes, corresponding to tens of hours of practice (Fig. 2d). According to subjective reports of EyeWirers, learning continues for much longer than that. By contrast, previous successes at ‘crowdsourcing’ image analysis involved tasks that did not require such extensive training10,13.

Reconstructing an entire neuron requires tracing its branches through thousands of cubes. This process is coordinated by an automatic spawner, which inspects each consensus cube for branches that exit the cube. Each exit generates a new cube and seed, which are added to a queue. EyeWirers are automatically assigned to cubes by an algorithm that attempts to balance the number of plays for each cube.

Over 100,000 registered EyeWirers have been recruited by news reports, social media and the EyeWire blog. Players span a broad range of ages and educational levels, come from over 130 countries, and the great majority have no formal training in neuroscience (Extended Data Figs 2 and 3 and Supplementary Notes). These statistics show that EyeWire indeed widens participation in neuroscience research. At the same time, the most avid players constitute an elite group with disproportionate achievements. For example, the top 100 players have contributed about half of all cubes completed in EyeWire.

Laboratory workers also reconstructed neurons independently of EyeWire, with a more sophisticated version of the user interface (Methods). Their reconstructions were pooled with those of EyeWirers for the analyses reported below. Reconstruction error was quantified (Methods), and was treated like other kinds of experimental error when calculating confidence intervals from our data.

Contact analysis

We reconstructed 195 Off BC axons and 79 Off SACs from e2198 (Fig. 3b and Extended Data Fig. 4). The e2198 retina was stained in an unconventional way that did not mark intracellular structures such as neurotransmitter vesicles9, and reliable morphological criteria for identification of BC presynaptic terminals are unknown. As an indirect measure of connectivity, contact areas were computed for all BC–SAC pairs. The resulting ‘contact matrix’ was analysed through two subsequent steps.

Figure 3: 3D reconstructions of Off BCs and SACs.
Figure 3

a, b, Cells viewed opposite the light axis. BCs alone (a); BCs with SACs (b). Scale bar, 50 μm.

In the first step, Off BC axons were classified into five cell types, following structural criteria14 established to correspond with previous molecular definitions6 (Methods and Extended Data Fig. 5). BC types stratify at characteristic depths in the inner plexiform layer (IPL), and vary in size (Fig. 4a). The BCs of each type formed a ‘mosaic’, meaning that cells were spaced roughly periodically (Extended Data Fig. 6a–e). This is generally accepted as an important defining property of a retinal cell type. Type densities (Extended Data Fig. 6f) were roughly consistent with previous reports6. When the columns of the contact matrix were sorted by BC type (Fig. 4b), it became evident that BC2 and BC3a contact SACs more than other BC types.

Figure 4: BC–SAC contact.
Figure 4

a, Off BCs were divided into five types6,14 on the basis of IPL depth and size. Scale bar, 10 μm. b, Contact areas of BC–SAC pairs, sorted by BC types. c, Pairs were further sorted by the distance of the BC axon from the SAC soma, as measured in the tangential plane. Scale bar, 50 μm. d, Average BC–SAC contact versus distance, normalized to percentage of SAC surface area at that distance (Extended Data Fig. 3b). Standard error is based on the number of pairs for each BC type and distance (see Source Data for sample sizes).

In the second step, we averaged contact area over BC–SAC pairs of the same BC type and similar distance between the BC axon and the SAC soma in the plane tangential to the retina (Fig. 4c). These absolute areas were normalized to convert them into the percentage of SAC surface area covered by BCs of a given type (Methods). The resulting graphs show that BC2 prefers to contact SAC dendrites close to the SAC soma, whereas BC3a prefers to contact far from the soma (Fig. 4d and Extended Data Fig. 7c).

Imaging of intracellular calcium in BC axons7 and extracellular glutamate around BC axons8 indicates that BC2 lags BC3a in visual responses by 50–100 ms. Therefore BC–SAC wiring appears to possess the space–time specificity appropriate for an outward preferred direction, as we proposed (Fig. 1c).

Co-stratification analysis

Off SACs stratify at a particular depth in the IPL (Fig. 1b). Why this depth and not some other? From Fig. 4a, it is obvious that this depth is appropriate for wiring with BC2 and BC3a, as required by our model of DS emergence. Following this logic one step further, we wondered whether the observed dependence of contact on distance from the SAC soma might be reflected in fine aspects of SAC morphology. We hypothesized that SAC dendrites are ‘tilted’, moving deeper into the IPL with distance from the SAC soma. Such a change in depth would be compatible with more overlap with BC2 near the soma, and more overlap with BC3a far from the soma, as BC3a is deeper in the IPL than BC2 (Fig. 4a and Supplementary Video 1).

The hypothesized tilt turns out to exist (Fig. 5a). Very close to the SAC soma, the dendrites dive sharply into the IPL from the inner nuclear layer (INL). Surprisingly, IPL depth continues to increase as distance from the SAC soma in the tangential plane ranges from 20 to 80 μm. The slight increase is not evident in a single dendrite (Fig. 1b), but emerges from statistical averaging.

Figure 5: BC–SAC co-stratification.
Figure 5

a, SAC dendrites move deeper into the IPL (median depth, red line) with increasing distance from the SAC soma in the tangential plane. Stratification profiles of BC types, defined as density of surface area over the depth of the IPL. b, Co-stratification predictions of BC–SAC contact area versus distance from the SAC soma. The curves are normalized by SAC area at each distance, and are therefore directly comparable with those of Fig. 4d.

Could dendritic tilt be the cause of the observed variation in BC–SAC contact with distance (Fig. 4d)? We cannot address causality on the basis of our data, but we can test how well the tilt predicts contact variation. We computed the stratification profiles of BC types (Fig. 5a), defined as the one-dimensional density of BC surface area along the depth of the IPL. We also computed the stratification profile of SAC dendrites at various distances from the SAC soma (quartiles, Fig. 5a). Assuming that BC and SAC arborizations are statistically independent of each other, we estimated contact from ‘co-stratification’, defined as the integral over IPL depth of the product of BC and SAC stratification profiles (Methods).

We found that actual BC2 contact depends more strongly on distance than predicted; the slight change in IPL depth after the initial plunge appears too small to account for the large change in actual BC2 contact. In other failures of contact prediction, BC3a, BC3b and BC4 stratify at the same IPL depths (Fig. 5a), yet BC3a makes much more contact than BC3b or BC4. Also, actual BC3a contact plummets near the tips of SAC dendrites (Fig. 4d), whereas predicted contact does not change at all because the IPL depth of SAC dendrites is constant in this region (Fig. 5b). Overall, the total contact from all BC types seems low in this region (Extended Data Fig. 7d), suggesting that BCs avoid making synaptic inputs to the most distal SAC dendrites. This runs counter to the conventional belief that input synapses are uniformly distributed over the entire length of SAC dendrites15. The unreliability of inferring contact from co-stratification is illustrated by numerous examples of SAC dendrites that pass through BC axonal arborizations without making any contact at all (Extended Data Fig. 8).

Model of the BC-SAC circuit

We mentioned previously that BC2 lags BC3a in visual response. There is another important difference: BC3a responds more transiently to step changes in illumination, whereas BC2 exhibits more sustained responses. The implications of the sustained–transient distinction for DS can be understood using a mathematical model. The activity of a retinal neuron is often approximated as a linear spatiotemporal filtering of the visual stimulus followed by a nonlinearity16,17. Such a ‘linear–nonlinear’ model for the output O(t) of the SAC dendrite can be written asFor simplicity, the dendrite and visual stimulus I(x,t) are restricted to a single spatial dimension x, and the nonlinearity is a half-wave rectification, [z]+ = max{z,0}. We interpret the integral in equation (1) as the summed input from the BCs presynaptic to the SAC. The nonlinearity could arise from various biophysical mechanisms, such as synaptic transmission from SACs to other neurons. The spatiotemporal filter W(x,t) is a sum of two functions,corresponding to contributions from BC2 and BC3a. The sustained temporal filter νs(t) is monophasic, whereas the transient filter νt(t) is biphasic (Fig. 6a). The spatial filter Us(x) represents the entire set of all BC2 inputs to the dendrite, and can be estimated from the BC2 contact area graph in Fig. 4d. Similarly, Ut(x) can be estimated from the BC3a contact area graph. The two spatial filters are displaced relative to each other (Fig. 6a), because BC3a tends to contact SAC dendrites at more distal locations than BC2.

Figure 6: Mathematical model of the BC–SAC circuit.
Figure 6

a, Spatiotemporal filter of equation (2). Green is positive, red is negative, and grey is zero. b, The transient pathway effectively combines a positive channel that leads the sustained pathway by τ and a negative channel that lags by τ. c, Removing the negative channel yields a Reichardt detector (d). e, Removing the positive channel yields a Barlow–Levick detector (f). A moving visual stimulus I(x,t) is oriented in space–time (g, h), and so are the spatiotemporal filters (a, c, e).

Each of the terms in the sum of equation (2) is said to be ‘space–time separable’, because it is the product of a function of space and a function of time. It was previously observed that a spatiotemporal filter W(x,t) of this form can endow a model like equation (1) with DS18,19. This is illustrated by Fig. 6 using the fact that the convolution in equation (1) is equivalent to ‘sliding’ the spatiotemporal filter W in time over the stimulus I, and computing the overlap at each time. The filter W(x,t) is oriented in space–time (Fig. 6a), and so also is a moving stimulus I(x,t) (Fig. 6g, h). The overlap with a rightward-moving stimulus (Fig. 6h) is greater than for a leftward one (Fig. 6g), so the model exhibits DS with a rightward preferred direction.

How is DS affected by the biphasic shape of the transient temporal filter, νt(t)? If we remove the negative lobe (Fig. 6c), then νt(t) will become monophasic like νs(t) and their relation closer to a simple time lag (Fig. 6d). We will refer to this model as a ‘Reichardt detector’, in honour of the pioneering researcher Werner Reichardt, although it more closely resembles a subunit of his model20. On the other hand, removing the positive lobe of νt(t) makes it monophasic but with inverted sign relative to the sustained filter (Fig. 6e). The result (Fig. 6f) resembles a DS model originally proposed by Barlow and Levick21.

Both modified models (Fig. 6d, f) exhibit DS. In the Reichardt detector, the inputs from the two arms enhance each other for motion in the preferred direction. In the Barlow–Levick detector, the two inputs cancel each other for motion in the null direction. As our sustained–transient model (Fig. 6b) uses both mechanisms, it should exhibit more DS than either detector. Our model is related to versions of the Reichardt detector with low-pass and high-pass filters on the two arms22.

In the original Barlow–Levick model, the negative filter corresponded to synaptic inhibition. As BCs are believed to be excitatory, negative BC input in our model represents a reduction of excitation relative to the resting level, rather than true inhibition. Signalling by reduced excitation may be possible, at least for low-contrast stimuli, as BC ribbon synapses may have a significant resting rate of transmitter release23.

The model of equations (1) and (2) is a useful starting point for many theoretical investigations that are outside the scope of this article. For example, DS dependency on the spatial and temporal frequencies of a sinusoidal travelling wave stimulus is calculated in Supplementary Equations, and DS dependence on stimulus speed is graphed in Extended Data Fig. 9.

Discussion

In our DS model, SAC dendrites are wired to BC types with different time lags. A previous model did not distinguish between BC types, and instead relied on the time lag of signal conduction within the SAC dendrite itself24 (Fig. 1d). Like most other amacrine cells, SACs lack an axon; their output synapses are found in the distal zones of their dendrites15 (Fig. 1a, inset). Owing to dendritic conduction delay, proximal BC inputs should take longer to reach the output synapses than distal BC inputs (Fig. 1d). Therefore this time lag is also consistent with the empirical finding of an outward preferred direction. To summarize the novelty of our hypothesis, we place the time lag before BC–SAC synapses, whereas the previous model places it after BC–SAC synapses.

The postsynaptic delay model has a major weakness. If dendritic conduction were the only source of time lag, the somatic voltage would exhibit DS with an inward preferred direction, but this is inconsistent with intracellular recordings3 (Fig. 1e). By contrast, the presynaptic delay model is compatible with approximating a SAC dendrite as isopotential (Fig. 1c), so preferred direction is predicted to be independent of the location of the voltage measurement, consistent with empirical data3. It may also be possible to make the postsynaptic delay model consistent with experiments by adding active dendritic conductances4.

The presynaptic and postsynaptic delay models are not mutually exclusive. If they work together, passive cable theory suggests that presynaptic delay dominates, because estimated postsynaptic delay is much shorter than the time lag between BC2 and BC3a (Supplementary Equations). Can we gauge the relative importance of the delays empirically rather than theoretically? One way would be intracellular recording at the SAC soma of responses to visual stimulation at various dendritic locations. If postsynaptic delay dominates, then response latency will grow with distance of the visual stimulus from the soma. If presynaptic delay dominates, then distal stimulation will evoke somatic responses with shorter latency than proximal stimulation. This prediction may seem counterintuitive, but is an obvious outcome of our model.

Many other models of DS emergence in SACs invoke inhibition as well as excitation25,26,27,28. We have focused on excitatory mechanisms, as blocking inhibition does not abolish DS3. However, inhibition may have the effect of enhancing DS, and its role should be investigated further.

This work focused on Off BC–SAC circuitry. An analogous sustained–transient distinction can also be made for On BC types7,8. It remains to be seen whether their connectivity with On SACs depends on distance from the soma. If this turns out to be the case, then the model of Fig. 6 could serve as a general theory of motion detection by both On and Off SACs. The model filter of Fig. 6a also resembles the spatiotemporal receptive field of the J type of ganglion cell (see Fig. 3b of ref. 29).

Neural activity imaging30 and connectomic analysis31 have recently identified a plausible candidate for the site of DS emergence in the fly visual system. If our theory is correct, then the analogies between insect and mammalian motion detection1 are more far-reaching than previously suspected, with fly T4 and T5 cells corresponding to On and Off SAC dendrites in both connectivity and function.

A glimmer of space–time wiring specificity can even be seen in the structure of the SAC itself. As BC types with different time lags arborize at different IPL depths, IPL depth can be regarded as a time axis. Therefore, the slight tilt of the SAC dendrites in the IPL (Fig. 5a) could be related to the orientation of the SAC receptive field in space–time (Fig. 6a). However, dendritic tilt alone is not sufficient to predict our model, as co-stratification sometimes fails to predict contact (Figs 4d and 5b). For example, co-stratification predicts strong BC4 connectivity to distal SAC dendrites. This would favour an inward preferred direction, contrary to what is observed, because BC2 leads (not lags) BC4 in visual responses7.

The idea that contact (or connectivity) can be inferred from co-stratification is sometimes known as Peters’ rule32, and has also been applied to estimate neocortical connectivity33,34,35. The present work shows that fairly subtle violations of Peters’ rule may be important for visual function. Previous research suggests that On–Off direction-selective ganglion cells inherit their DS from SAC inputs owing to a strong violation of Peters’ rule9,36,37,38.

Our findings were made possible by using artificial intelligence to reduce the amount of human effort required for 3D reconstruction of neurons. Even after the labour savings, our research required great human effort from a handful of paid workers in the laboratory and a large number of volunteers through EyeWire. Our experiences do not support claims that the ‘wisdom of the crowd’ should replace experts39. Instead, EyeWire depends on cooperation between laboratory experts and online amateurs (Methods). Furthermore, some amateurs developed remarkable expertise and were promoted to increasingly sophisticated roles within the EyeWire community (Supplementary Notes). We believe that crowd wisdom requires amplifying the expert voices within the crowd, and also empowering individuals to become experts. Fortunately, such goals are well-matched to the game format.

The EyeWire artificial intelligence was based on a deep convolutional network40,41. Similar networks have been successfully applied to serial electron microscopy images obtained using conventional staining techniques that mark intracellular organelles42. Extending EyeWire to such images, in which synapses are clearly visible, would enable a true connection analysis that goes beyond the contact and co-stratification analyses used here.

Our work demonstrates that reconstructing a neural circuit can provide surprising insights into its function. Much more will be learned as reconstruction speed grows. The combination of crowd and artificial intelligence promises a continuous upward path of improvement, as human input from the crowd is not only useful for generating neuroscience discoveries, but also for making the artificial intelligence more capable through machine learning.

Note added in proof: Further evidence that BC axons exhibit little or no DS appeared while this paper was in press43.

Methods Summary

A convolutional network was trained to detect neural boundaries via the MALIS procedure40 and CNPKG (https://github.com/srinituraga/cnpkg/), which is based on Cortical Network Simulator44. The convolutional network was applied to the e2198 data set, which was then segmented into supervoxels by a modified version of the watershed algorithm. Paid workers and volunteer EyeWirers reconstructed neurons in 3D by assembling supervoxels. The retina was computationally flattened, reconstructed neurons were classified by their structural properties, and contact and co-stratification were analysed by custom Matlab and C++ code.

Online Methods

We worked with the e2198 data set9 rather than the e2006 data set14 because e2198 is large enough to encompass entire SAC dendrites (150 μm). All dimensions are uncorrected for tissue shrinkage, which was previously estimated at 14% by comparison of two-photon and serial electron microscopy images14.

Machine learning

The boundaries between neurons in subvolumes of the e2198 and e2006 data sets were manually traced. Using this as ground truth, a convolutional network was trained to detect boundaries between neurons using the MALIS method40. The convolutional network had the same architecture as one used previously14, and produced as output an affinity graph connecting nearest neighbour voxels41. Any subvolume of e2198 could be oversegmented by applying a modified watershed algorithm to the appropriate subgraph. The regions of the oversegmentation are called supervoxels.

Reconstruction by workers

A team of part-time workers, numbering about half a dozen at any given time, reconstructed neurons using a more sophisticated version of the EyeWire interface. Workers were hired on the basis of an interview and a test of software use passed by three-quarters of the applicants. They were trained for 40–50 h before generating reconstructions used for research. Their skills typically improved for months or even years after the initial training period, and were superior to those of professional neuroscientists without reconstruction experience.

As with EyeWire, the task of reconstructing an entire neuron was divided into subtasks, each of which involved reconstructing the neuron within a subvolume starting from a supervoxel ‘seed’. However, the subvolumes were roughly 100 times larger than EyeWire cubes, and only two workers were assigned to each subvolume.

In the first stage of error correction, disagreements were detected by computer, and resolved by one of the two workers, or a third worker. The third occasionally detected and corrected errors that were not disagreements between the first two. Most disagreements were the result of careless errors, and were easily resolved. More rarely, there were disagreements caused by fundamental ambiguities in the image. These locations were noted for later examination in a further stage of error correction.

This second stage relied on 3D reconstructions of entire neurons assembled from multiple subvolumes and inspected by one of the authors (J.S.K.). Suspicious branches or terminations, as well as overlaps between reconstructions of different neurons were detected. The original image was re-examined at these locations to check for errors. The process was repeated until no further errors could be detected.

The precision of our final reconstruction relative to the truth is probably comparable to the precision of the penultimate reconstruction relative to the final reconstruction, 0.99 for SACs and 0.96 for BCs. Recall is probably somewhat poorer, because missing branches are more difficult to detect than superfluous branches. Recall must be reasonably good for SACs, as missing branches would be detected by deviations from the typical SAC shape and radius.

Reconstruction by EyeWirers

Some reconstruction errors slip past the consensus mechanism. These are detected through visual inspection of an ‘overview’ mode, which displays 3D renderings of entire neurons currently under reconstruction (Extended Data Fig. 1b). False branches become obvious once they are long enough, and are reported by EyeWirers through chat or email. They are chopped off by GrimReaper, a special EyeWirer played by laboratory experts endowed with the superpower of overruling the consensus. GrimReaper also extends branches that have terminated prematurely. Correction by GrimReaper is similar to the second stage of error correction described above, so the final reconstruction presumably has similar accuracy.

SAC reconstructions are extremely difficult for two reasons: (1) SAC dendrites are very thin and may falsely appear to terminate, owing to limited spatial resolution and imperfect staining; and (2) the interiors of many SAC boutons contained irregular darkenings, which could falsely appear like cellular boundaries. (The reason for the darkening is unclear, as the extracellular staining procedure was not intended to mark intracellular structures.)

Novices tend to prematurely terminate SAC dendrites. Experts know that most cubes do not contain termination points, and therefore try harder to find continuations, using a variety of sophisticated search strategies. GrimReaper is also allowed to view how the cube fits into the entire reconstructed neuron. This additional spatial context can be used to disambiguate difficult cubes, given knowledge of the typical appearance of a SAC.

Before learning in normal gameplay (Fig. 2d), all EyeWirers are required to go through a training session immediately after registering for the site. This consists of a sequence of tutorial cubes, each of which was previously coloured by an expert (Extended Data Fig. 1c). Each cube teaches through instructions and per-click feedback about accuracy based on comparing the EyeWirer’s selections with those of the expert. After submitting a tutorial cube, the EyeWirer is given a chance to view mistakes.

Accuracy is monitored on a weekly basis by computing the precision and recall of each EyeWirer with respect to the truth, defined as neuron reconstructions based on EyeWire consensus followed by GrimReaper corrections. Less accurate EyeWirers are given less weight in the vote.

Players’ daily, weekly and monthly scores are publicly displayed on a leaderboard (Extended Data Fig. 1b, right), motivating players to excel through competition. Players communicate with each other through online ‘chat’ (Extended Data Fig. 1b, left) and discussion forums.

A ‘beta test’ version of EyeWire was deployed in February 2012 and attracted a small group of users, who helped guide software development. EyeWire officially launched in December 2012.

Reconstruction of Off SACs

Off SACs were recognized by their somata in the INL, narrow IPL stratification at roughly one third of the depth from the INL to the ganglion cell layer, and characteristic ‘starburst’ appearance (Fig. 1a).

Off SACs were reconstructed by: (1) forward tracing from the soma to dendritic tips; and (2) backward tracing from varicosities on candidate SAC dendrites to the soma. In the forward method, a candidate SAC soma was identified as a supervoxel with a characteristic pattern of dendritic stubs bearing spiny protrusions. By the time reconstruction progressed to approximately half of the average SAC radius, an Off SAC could be conclusively recognized by its starburst shape and narrow stratification at the appropriate IPL depth. More than 90% of candidates turned out to be SACs.

In the backward method, we located a thin dendrite with varicosities at the appropriate IPL depth. This was reconstructed back to the soma, and then the rest of the dendrites were reconstructed from the soma to the tips. The cell could be discarded at any point during this process, if its dendrites escaped from the appropriate IPL depth or failed to exhibit the proper morphological characteristics. Less than 25% of initial candidates ended up confirmed as SACs.

In total, 79 Off SACs were reconstructed, 39 by forward tracing and 52 by backward tracing. This is more than half the entire population in e2198, judging from the published density45. After candidates were identified by one of the authors (J.S.K.), reconstructions were performed by laboratory workers (59 cells) or by EyeWirers (29 cells). Both pairs of numbers sum to more than 79, because the sets overlapped (12 for forward/backward, 9 for workers/EyeWirers).

In March 2012, laboratory workers began reconstruction of SACs. In March 2013, EyeWirers were invited to the ‘Starburst Challenge’, a sequence of tutorial cubes drawn from SACs. Those who passed with sufficient accuracy were an elite group allowed to reconstruct SACs (Supplementary Information). EyeWirers eventually shouldered most of the burden of SAC reconstruction, with only 8% of SAC cubes needing correction by GrimReaper. This enabled laboratory workers to shift their focus to BCs, as described below.

Reconstruction of Off BCs

The somata of Off BCs were generally outside e2198, which extended only partially into the INL (Fig. 1 of ref. 9). The trunks of candidate BC axons were located in the interstices of the INL, and followed into the IPL. If the axons arborized in the Off region of the INL, they were fully reconstructed. Cells that violated known BC structures were identified as amacrine cells and discarded14.

BC axons were difficult to reconstruct owing to poor staining, and their highly irregular shapes. They could not be accurately reconstructed (either by online volunteers or laboratory experts) within the 256 × 256 × 256 cubes of EyeWire, which were too small to provide sufficient spatial context. Therefore BCs were reconstructed only by laboratory workers using the large subvolumes mentioned above.

Coordinate system

For more precise quantification of structural properties, a new coordinate system was defined by applying a nonlinear transformation to neurons so as to flatten the IPL and make it perpendicular to one of the coordinate axes. The nonlinear transformation was found by the following steps. First a global planar approximation to the Off SAC surface was computed. Then the centroid of all the SACs was projected onto this global plane to define the origin of the coordinate system. The projection was along the coordinate axis of the e2198 volume closest in direction to the light axis.

To correct for curvature, an azimuthal equidistant projection46 of the Off SAC surface onto the global plane was made about the origin. Then local planar approximations to the SAC surface were computed in the neighbourhoods of every node in a triangular lattice. At each point in a triangle, the SAC surface was approximated by computing the mean of the planar approximations (as quaternions with yaw constrained to be zero) for the triangle’s vertices, weighted by distance of the point from the vertices.

The Off SACs were defined as 32% IPL depth. We also reconstructed a few On SACs, and defined them as 62%. These choices placed the edge of the INL at 0%. Structural properties of all cells were computed on the basis of locations of their surface voxels after transformation into the new coordinates.

Classification of Off bipolar cells

BC stratification profiles were computed by dividing surface voxels into 100 bins spanning 0 to 100% IPL depth. Classification into cell types was done by using methods similar to those described previously14. The BCs were split into shallow (BC1/2) and deep (BC3/4) clusters using the 75th percentile depth of the stratification profile. The BC1/2 cluster was further subdivided into two clusters by stratification width, defined as the difference between 75th and 25th percentile depths. On the basis of cells per square millimetre (Extended Data Fig. 6f), we inferred that the wider cluster was BC2 and the narrower cluster was BC1. These two types were originally defined by molecular criteria6, and our inferred correspondence with structural definitions is transposed relative to a previous report14. The BC3/4 cluster was subdivided into BC4 and BC3 by the 10th percentile depth, because the molecularly defined BC4 stratifies closer to the INL6. Finally, BC3 was subdivided into BC3a and BC3b on the basis of axonal arborization volume, with BC3a having the larger axonal volume. Each of the above subdivision steps was based on a feature with a roughly bimodal histogram (Extended Data Fig. 5).

The result still contained a small number of classification errors, detected when adjacent BCs of the same type overlapped enough to violate the mosaic property. Corrections were made by an automatic algorithm that greedily swapped cells from one cluster to another such that the total overlap between convex hulls of cells of a given type was minimized. Two swaps were vetoed by an expert (J.S.K.) on the basis of morphological features. In all, six cells were swapped within BC1/2 and 13 within BC3/4. In the final classification, 41, 56, 29, 35 and 34 BCs were identified as types 1, 2, 3a, 3b and 4, respectively (Extended Data Fig. 6). Cells that violated the mosaic of all types (7) or had irregular stratification profiles (9) were discarded as possible reconstruction errors or amacrine cells.

Contact analysis

Edges of the affinity graph connecting BC with SAC voxels were defined as BC–SAC contact edges. For each pair, the sum of the edges yielded an estimate of contact area. The Euclidean distance separating each BC–SAC pair was computed after projecting their centres onto the SAC plane. Centres of SAC somata were manually annotated, and centres of BC arborizations were computed as the centroids of their surface voxels. The pairs were binned by distance of the BC from the SAC soma. For every pair in a bin, the fraction of SAC surface area devoted to BC–SAC contact within the convex hull of the BC was computed as the ratio of BC–SAC contact edges to SAC surface edges within the convex hull. The latter was estimated by the number of SAC surface voxels multiplied by a geometric conversion factor of 1.4 SAC surface edges per surface voxel. (This factor was estimated by dividing the total number of SAC surface edges by the total number of SAC surface voxels in the volume.) BC–SAC pairs with fewer than 10,000 SAC surface voxels inside the hull were excluded from the computation to reduce the effect of fluctuations. The ratios for BCs of the same type were averaged for each distance bin and multiplied by a mosaic overlap factor to yield the values in Fig. 4d. The mosaic overlap factor represents the extent to which neighbouring convex hulls overlap one another, which varies by cell type. This factor was computed by dividing the sum of the hull areas for each cell by the area of the union of hulls for each cell type. For absolute rather than fractional areas, edges in the affinity graph were converted to area in μm2, using the conversion factor of 291.5 μm2 per edge. This factor averages over the different edge orientations and compensates for voxelization effects. A result very similar to Fig. 4d can also be obtained by an alternative method that is simpler but does not yield error bars (Extended Data Fig. 7c).

Co-stratification analysis

All SAC surface voxels were binned by distance from the soma centre in the SAC plane. Within each bin, the stratification profile was computed as for the BCs. The quartiles (median and 25th and 75th percentiles) are graphed in Fig. 5a. The prediction of contact from co-stratification is based on the following formalism.

We define the arborization density ρa(r) as the surface area per unit volume at location r of a type a cell with soma centred at the origin. Its integral is the total surface area of the arborization. We assume that the contact density received by one cell of type a from all cells of type b is equal toThe sum over the b mosaic can be approximated by a function that is independent of x and y,where σb is the number of type b neurons per retinal area andis the stratification profile of a cell of type b. The SAC arborization density is assumed radially symmetric,where ρSAC(r) can be regarded (up to normalization) as the SAC stratification profile as a function of distance from the SAC soma. Integrating the contact density (3) and normalizing yields the fraction ϕb(r) of SAC area contacted by cell type b as a function of r,

References

  1. 1.

    & Seeing things in motion: models, circuits, and mechanisms. Neuron 71, 974–994 (2011)

  2. 2.

    , & Direction selectivity in the retina: symmetry and asymmetry in structure and function. Nature Rev. Neurosci. 13, 194–208 (2012)

  3. 3.

    , & Directionally selective calcium signals in dendrites of starburst amacrine cells. Nature 418, 845–852 (2002)

  4. 4.

    , , & A dendrite-autonomous mechanism for direction selectivity in retinal starburst amacrine cells. PLoS Biol. 5, e185 (2007)

  5. 5.

    et al. The first stage of cardinal direction selectivity is localized to the dendrites of retinal ganglion cells. Neuron 79, 1078–1085 (2013)

  6. 6.

    , , & Cone contacts, mosaics, and territories of bipolar cells in the mouse retina. J. Neurosci. 29, 106–117 (2009)

  7. 7.

    , , & Spikes in mammalian bipolar cells support temporal layering of the inner retina. Curr. Biol. 23, 48–52 (2013)

  8. 8.

    , , & Two-photon imaging of nonlinear glutamate release dynamics at bipolar cell synapses in the mouse retina. J. Neurosci. 33, 10972–10985 (2013)

  9. 9.

    , & Wiring specificity in the direction-selectivity circuit of the retina. Nature 471, 183–188 (2011)

  10. 10.

    et al. Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Mon. Not. R. Astron. Soc. 389, 1179–1189 (2008)

  11. 11.

    et al. Predicting protein structures with a multiplayer online game. Nature 466, 756–760 (2010)

  12. 12.

    Reconstruct: a free editor for serial section microscopy. J. Microsc. 218, 52–61 (2005)

  13. 13.

    & Labeling images with a computer game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 319–326 (ACM, 2004)

  14. 14.

    et al. Connectomic reconstruction of the inner plexiform layer in the mouse retina. Nature 500, 168–174 (2013)

  15. 15.

    Synaptic organization of starburst amacrine cells in rabbit retina: analysis of serial thin sections by electron microscopy and graphic reconstruction. J. Comp. Neurol. 309, 40–70 (1991)

  16. 16.

    & Refractoriness and neural precision. J. Neurosci. 18, 2200–2211 (1998)

  17. 17.

    & Fast and slow contrast adaptation in retinal circuitry. Neuron 36, 909–919 (2002)

  18. 18.

    & Model of human visual-motion sensing. J. Opt. Soc. Am. A 2, 322–341 (1985)

  19. 19.

    & Spatiotemporal energy models for the perception of motion. J. Opt. Soc. Am. A 2, 284–299 (1985)

  20. 20.

    in Sensory Communication (ed. ) 303–317 (MIT Press, 1961)

  21. 21.

    & The mechanism of directionally selective units in rabbit’s retina. J. Physiol. (Lond.) 178, 477–504 (1965)

  22. 22.

    , & Adaptation of response transients in fly motion vision. II: model studies. Vision Res. 43, 1311–1324 (2003)

  23. 23.

    , & Continuous vesicle cycling in the synaptic terminal of retinal bipolar cells. Neuron 17, 957–967 (1996)

  24. 24.

    , & Direction selectivity in a model of the starburst amacrine cell. Vis. Neurosci. 21, 611–625 (2004)

  25. 25.

    & in Single Neuron Computation (eds , & ) Ch. 13 347–76 (Academic San Diego, 1992)

  26. 26.

    , , & Cation–chloride cotransporters mediate neural computation in the retina. Proc. Natl Acad. Sci. USA 100, 16047–16052 (2003)

  27. 27.

    & Symmetric interactions within a homogeneous starburst cell network can lead to robust asymmetries in dendrites of starburst amacrine cells. J. Neurophysiol. 96, 471–477 (2006)

  28. 28.

    & The synaptic mechanism of direction selectivity in distal processes of starburst amacrine cells. Neuron 51, 787–799 (2006)

  29. 29.

    , , , & Molecular identification of a retinal cell type that responds to upward motion. Nature 452, 478–482 (2008)

  30. 30.

    et al. A directional tuning map of Drosophila elementary motion detectors. Nature 500, 212–216 (2013)

  31. 31.

    et al. A visual motion detection circuit suggested by Drosophila connectomics. Nature 500, 175–181 (2013)

  32. 32.

    & Cortex: Statistics and Geometry of Neuronal Connectivity 2nd edn (Springer Berlin, 1998)

  33. 33.

    , & Deriving physical connectivity from neuronal morphology. Biol. Cybern. 88, 210–218 (2003)

  34. 34.

    , & A quantitative map of the circuit of cat primary visual cortex. J. Neurosci. 24, 8441–8453 (2004)

  35. 35.

    & Neurogeometry and potential synaptic connectivity. Trends Neurosci. 28, 387–394 (2005)

  36. 36.

    , & Mechanisms and circuitry underlying directional selectivity in the retina. Nature 420, 411–414 (2002)

  37. 37.

    et al. Spatially asymmetric reorganization of inhibition establishes a motion-sensitive circuit. Nature 469, 407–410 (2010)

  38. 38.

    , , & Development of asymmetric inhibition underlying direction selectivity in the retina. Nature 469, 402–406 (2011)

  39. 39.

    The Wisdom of Crowds (Anchor, 2005)

  40. 40.

    , , , & in Advances in Neural Information Processing Systems 22. 1865–1873 (2009)

  41. 41.

    et al. Convolutional networks can learn to generate affinity graphs for image segmentation. Neural Comput. 22, 511–538 (2010)

  42. 42.

    et al. in Advances in Neural Information Processing Systems 25. 2852–2860 (2012)

  43. 43.

    , , , & Excitatory synaptic inputs to mouse on-off direction-selective retinal ganglion cells lack direction tuning. J. Neurosci. 34, 3976–3981 (2014)

  44. 44.

    , & CNS: a GPU-Based Framework for Simulating Cortically-Organized Networks Tech. Rep. MIT-CSAIL-TR-2010–013/CBCL-286 (MIT, 2010)

  45. 45.

    , & The major cell populations of the mouse retina. J. Neurosci. 18, 8936–8946 (1998)

  46. 46.

    Map Projections–A working Manual 1395 (USGPO, 1987)

Download references

Acknowledgements

This research was made possible by funding from the Gatsby Charitable Foundation, the Howard Hughes Medical Institute, the Human Frontier Science Program, an anonymous donor, and the National Institutes of Health. K.L. was supported by a Samsung Scholarship. Support from the AWS Research Grants Program gave EyeWire global reach through Amazon Cloudfront. We thank K. Briggman for providing the e2198 data set. J. Mutch created the CNS framework on which CNPKG is based. D. Jia, R. Shearer, and B. Warne assisted in early stages of software development, and W. Silversmith with recent modifications. R. Prentki, L. Trawinski, M. Sorek, A. Ostojic, C. David, R. Avery, S. Temple, A. Bost, M. Greenstein and M. Evans worked in the laboratory to reconstruct neurons, and the first six also served as GrimReaper and hosted EyeWire competitions. Additional reconstructions were provided by R. Han, M. Gavrin, G. Lu, A. Ortiz and D. Udvary. All were trained by R. Prentki, who also created training videos for EyeWirers. We are grateful to A. Norton for 3D renderings, and to E. Almeida for EyeWire graphics. We acknowledge discussions with T. Baden, M. Berry, B. Borghuis, A. Borst, E. J. Chichilnisky, D. Chklovskii, D. Clark, J. Demb, T. Euler, M. Helmstaedter, A. Huberman, S. Lee, R. Masland, J. Sanes and Z. Zhou.

Author information

Author notes

    • Jinseop S. Kim
    •  & Matthew J. Greene

    These authors contributed equally to this work.

    • Mark Richardson
    • , Srinivas C. Turaga
    •  & H. Sebastian Seung

    Present addresses: 601 N 42nd Street, Seattle, Washington 98103, USA (M.R.); Princeton Neuroscience Institute and Computer Science Deptartment, Princeton, New Jersey 08544, USA (H.S.S.); Gatsby Computational Neuroscience Unit, London WC1N 3AR, UK (S.C.T.).

Affiliations

  1. Brain & Cognitive Sciences Department, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, USA

    • Jinseop S. Kim
    • , Matthew J. Greene
    • , Kisuk Lee
    • , Mark Richardson
    • , Srinivas C. Turaga
    • , Michael Purcaro
    • , Matthew Balkam
    • , Amy Robinson
    •  & H. Sebastian Seung
  2. Electrical Engineering and Computer Science Department, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, USA

    • Aleksandar Zlateski
  3. Qualcomm Research, 5775 Morehouse Drive, San Diego, California 92121, USA

    • Bardia F. Behabadi
    •  & Michael Campos
  4. Max-Planck Institute for Medical Research, D-69120 Heidelberg, Germany

    • Winfried Denk

Consortia

  1. the EyeWirers

    https://eyewire.org

Authors

  1. Search for Jinseop S. Kim in:

  2. Search for Matthew J. Greene in:

  3. Search for Aleksandar Zlateski in:

  4. Search for Kisuk Lee in:

  5. Search for Mark Richardson in:

  6. Search for Srinivas C. Turaga in:

  7. Search for Michael Purcaro in:

  8. Search for Matthew Balkam in:

  9. Search for Amy Robinson in:

  10. Search for Bardia F. Behabadi in:

  11. Search for Michael Campos in:

  12. Search for Winfried Denk in:

  13. Search for H. Sebastian Seung in:

Contributions

J.S.K. created algorithms, software and procedures for crowd intelligence and learning, and applied them to generate neuron reconstructions. J.S.K. and M.J.G. classified bipolar cells. M.J.G. analysed contact and co-stratification, aided by code from A.Z. and input from W.D. H.S.S. devised the model with help from B.F.B. and M.C. S.C.T. trained the convolutional network. M.P. and M.B. implemented software and algorithms created by A.Z. for interactive segmentation and 3D visualization, with guidance from S.C.T. M.R. created the EyeWire game and M.B. its data infrastructure. K.L. quantified EyeWirer accuracy and learning. A.R. mobilized and studied the EyeWire community. EyeWirers reconstructed neurons and built extensions to EyeWire. H.S.S. wrote the paper with help from J.S.K., M.J.G. and A.R.

Competing interests

W.D. receives license income for SBEM technology from Gatan Inc.

Corresponding author

Correspondence to H. Sebastian Seung.

Extended data

Supplementary information

PDF files

  1. 1.

    Supplementary Information

    This file contains Supplementary Equations showing detailed derivation of the mathematical model of direction selectivity, calculation of the direction selectivity index for a special case that does not depend on the detailed forms of the filters, comparison with experiments, and cable theory estimate of dendritic conduction time. It also contains Supplementary Notes, which include EyeWire demographics, community structure, competitions, and list of EyeWirers who reconstructed SACs.

Videos

  1. 1.

    Off SAC with BC2 and BC3a axons.

    Off SAC with BC2 and BC3a axons. Off SAC in red, BC2 axon in yellow, and BC3a axon in blue.

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nature13240

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.