Article | Open | Published:

# A model of ganglion axon pathways accounts for percepts elicited by retinal implants

## Abstract

Degenerative retinal diseases such as retinitis pigmentosa and macular degeneration cause irreversible vision loss in more than 10 million people worldwide. Retinal prostheses, now implanted in over 250 patients worldwide, electrically stimulate surviving cells in order to evoke neuronal responses that are interpreted by the brain as visual percepts (‘phosphenes’). However, instead of seeing focal spots of light, current implant users perceive highly distorted phosphenes that vary in shape both across subjects and electrodes. We characterized these distortions by asking users of the Argus retinal prosthesis system (Second Sight Medical Products Inc.) to draw electrically elicited percepts on a touchscreen. Using ophthalmic fundus imaging and computational modeling, we show that elicited percepts can be accurately predicted by the topographic organization of optic nerve fiber bundles in each subject’s retina, successfully replicating visual percepts ranging from ‘blobs’ to oriented ‘streaks’ and ‘wedges’ depending on the retinal location of the stimulating electrode. This provides the first evidence that activation of passing axon fibers accounts for the rich repertoire of phosphene shape commonly reported in psychophysical experiments, which can severely distort the quality of the generated visual experience. Overall our findings argue for more detailed modeling of biological detail across neural engineering applications.

## Introduction

Degenerative retinal diseases such as retinitis pigmentosa1 and macular degeneration2 lead to a loss of photoreceptor cells and subsequent remodeling of the neural circuitry in the retina3,4, causing irreversible blindness in more than 10 million people worldwide. Analogous to cochlear implants, the goal of retinal prostheses is to help alleviate these incurable conditions by electrically stimulating surviving cells in the retina (for a recent review, see ref.5). The hope is that electrically evoked neuronal responses will be transmitted to the brain and interpreted by the subject as visual percepts (‘phosphenes’). Two devices are already approved for commercial use: Argus II (epiretinal, Second Sight Medical Products Inc., refs6,7) and Alpha-IMS (subretinal, Retina Implant AG, ref.8), which have been shown to restore vision up to a visual acuity of 20/1,2609 and 20/5468, respectively. In addition, PRIMA (subretinal, Pixium Vision, ref.10) has started clinical trials, with others to follow shortly. In combination with stem cell therapy11,12 and optogenetics13, a range of sight restoration options should be available within a decade14.

However, despite the increasing clinical and commercial use of these devices, the perceptual experience of retinal implant users remains poorly understood. For example, even in response to single-electrode stimulation, the appearance of individual phosphenes is highly variable not only across subjects but also across electrodes within a subject, with subjects typically reporting seeing distorted and often elongated geometric shapes that fade quickly over time15,16,17,18,19,20,21,22. Furthermore, linearly combining these ‘building blocks’ of percepts from individual electrodes often fails to predict the combination of percepts evoked when multiple electrodes are stimulated17,23,24,25. Consequently, most subjects cannot determine the orientation of gratings that are used to measure visual acuity, and those who can recognize letters take more than 40 seconds to do so26,27.

Both computational28,29 and in vitro electrophysiological studies17,30,31 suggest that electrode configurations similar to those implanted in patients do not achieve focal activation, but rather produce significant activation of passing axon fibers, which may result in perceptual distortions in patients. Here, we are the first to directly examine whether axonal stimulation contributes to the rich repertoire of phosphene shapes reported by patients. Our computational model can account for the apparent shape of phosphenes elicited by single-electrode stimulation in two generations of the Argus retinal prosthesis system (Second Sight Medical Products Inc.).

Four subjects suffering from severe retinitis pigmentosa (Table 1) were chronically implanted with an epiretinal prosthesis in the macular region of the retina: one subject was implanted with an Argus I device (16 platinum disc electrodes arranged in a 4x4 checkerboard pattern; see Fig. 1A), and three subjects were implanted with Argus II device (60 platinum disc electrodes in a 6x10 arrangement; Fig. 1B). Electrical stimulation was delivered to a number of pre-selected electrodes in random order (five repetitions each) using square-wave, biphasic, cathodic-first pulse trains with fixed stimulus duration, and we asked subjects to outline perceived phosphene shape either on a grid screen (Argus I; Fig. 1B,C) or a computer touch screen (Argus II; Fig. 1E,F) (see Methods). In a control experiment, we confirmed the reliability of each subject’s tracing (Figs S1S3). We then used a computational model to generate predictions about the apparent shape of the expected visual percepts, and compared the predicted images to patient phosphene drawings. The model assumed that distortions are due to activation of ganglion axon pathways, having estimated the spatial layout of these pathways using traced nerve fiber bundle trajectories extracted from ophthalmic fundus photographs of 55 human eyes32.

## Results

### Phosphene drawings vary across electrodes, but are relatively consistent for a given electrode

All subjects consistently reported seeing phosphenes upon electrical stimulation of the retina. Phosphenes appeared light gray, white, or yellowish in color. However, phosphene drawings varied greatly across subjects and electrodes; representative drawings for each subject are shown in Fig. 2. Whereas stimulation of some electrodes elicited consistent percepts across trials (top row of panels in each subplot), stimulation of other electrodes led to percepts that varied in both size and shape across trials (bottom row of panels in each subplot). Subjects occasionally but rarely reported seeing two distinct shapes (e.g., Subject 1, Electrode D1). Mean images for each electrode were obtained by averaging the drawings across the five stimulation trials, aligned by their centers of mass (column ‘average’). Mean images were then centered over the corresponding electrode in a schematic of the subject's implant to reveal the rich repertoire of elicited percepts across electrodes (large, rightmost panel in each subplot).

As is evident from these data, only a small number of phosphenes could be described as focal spots of light. Subject 1 drew percepts as either curved or straight lines, wedges, or relatively round spots. Subject 2 drew most percepts as ovals or relatively round spots with only a few curved or straight lines of varying thickness, whereas Subject 3 drew all phosphenes as slightly curved or straight, thin lines. Subject 4 predominantly drew ovals, wedges, and triangles, with only few curved or straight lines.

Interestingly, for Subjects 2–4 the percepts produced by electrodes in the first two rows of the array (i.e., Electrodes A1–F1, A2–F2) were much thinner and longer than for other electrodes33,34. It is possible that these electrodes were the ones that were closest to the retinal surface, since the surgical tack used to attach the implant to the retina was located next to the first row of electrodes. However, we did not have access to optical coherence tomography (OCT) or impedance data, which would have allowed us to estimate electrode-retina distance35,36 (see Discussion).

To quantify the similarity and variability of individual phosphene drawings, we calculated three shape descriptors for each collected drawing: phosphene area, orientation, and elongation (see Methods). These parameter-free metrics were based on a set of statistical quantities known as ‘image moments’; that is, particular weighted averages of pixel intensities across an image (Equation 1). Phosphene orientation and elongation were calculated from the eigenvalues and eigenvectors of each drawing’s covariance matrix (Equations 24).

The upper panels of Fig. 3 show distributions of phosphene area (Fig. 3A), orientation (Fig. 3B), and elongation (Fig. 3C) for each subject, across all tested electrodes. The lower panel (Fig. 3D–F) boxplots depicts trial-to-trial variability for each shape descriptor of a given electrode, measured as the standard error of the mean (SEM) calculated across drawings.

To assess whether observed SEM values were smaller than would be predicted from a random sample of phosphene drawings we performed a resampling analysis (1000 iterations). We began by calculating the SEM across all five drawings for each electrode. To assess whether, for an individual subject, drawings were more similar for an individual electrode then across other electrodes in that subjects’ array, resampling was done by randomly sampling (with replacement) phosphene drawings across all the electrodes of that subject. Probability values were estimated by comparing the mean SEM across electrodes of the real distribution to the 1-tailed confidence interval generated by resampling. Detailed results for each shape descriptor are given below.

#### Phosphene area is more consistent within than across electrodes

Phosphene area was calculated as the number of nonzero pixels in the drawing. Since phosphenes were elicited by directly stimulating the electrodes (see Methods), the phosphenes are at an arbitrary distance from the observer and cannot be described in terms of degrees of visual angle. Although we asked subjects to draw phosphenes ‘as if they appeared at arm’s length’, subjects qualitatively reported that phosphenes could appear as close as ‘in front of their face’ to ‘at arm’s length’. As can be seen by the variability in the boxplots in Fig. 3D, estimates of area varied widely across both subjects and electrodes. However, despite the lack of a reference plane in depth, for all but Subject 1 (marginally significant) the observed SEM values for phosphene area were significantly smaller than SEM values from randomly chosen electrodes (Subject 1: p < 0.06, Subject 2: p < 0.01, Subject 3: p < 0.001, Subject 4: p < 0.001). Subject 1 had particularly small percepts, so areal variability may have been more heavily influenced by drawing error.

#### Phosphene orientation is more consistent within than across electrodes

Phosphene orientation was calculated as the angle of the principal eigenvector (in the range [−90°, 90°]). For all subjects, mean SEM values for phosphene orientation were significantly smaller than mean SEM values for our bootstrapped null model (Subject 1: p < 0.001, Subject 2: p < 0.001, Subject 3: p < 0.001, Subject 4: p < 0.001), showing that the variation in phosphene orientation within an individual electrode is less than the variation across electrodes (Fig. 3E).

#### Phosphene elongation is more consistent within than across electrodes

Phosphene elongation was calculated as the relative difference in magnitude of the eigenvalues and normalized to [0, 1], with 0 representing a circle, and 1 representing an infinitesimally thin line. The distribution of elongation values indicates that subjects consistently saw elongated percepts instead of focal spots of light, indeed Subject 3 reported exclusively seeing thin curved and straight lines. These results are in stark contrast to the prevailing assumption in the field that stimulating a single electrode should generate the percept of a small focal spot of light37,38,39,40,41.

Interestingly, phosphene elongation was negatively correlated with phosphene area across subjects (r = −0.411, p < 0.001; see Figure S4); meaning that smaller phosphenes tended to be more elongated than larger ones. This observation also held for each Subject 1 (r = −0.362, p < 0.01), Subject 2 (r = −0.344, p < 0.001), and Subject 4 (r = −0.293, p < 0.001), but not for Subject 3, for whom phosphene elongation was strongly positively correlated with area (r = 0.573, p < 0.001; also see Discussion).

For Subjects 2 and 4, observed SEM values for phosphene elongation were significantly smaller than of our bootstrapped null model (Subject 2: p < 0.001, Subject 4: p < 0.001). For Subjects 1 and 3 results were not significant; percepts were heavily elongated for every electrode, providing little variability in the dataset (Fig. 3F).

#### Drawing accuracy

Our subjects had been lacking tactile-visual feedback for many years. This motivated a control experiment, where we asked subjects to explore various tactile targets (made of felt with a cardboard background) with their hands, and draw the targets on a touch screen (Figure S1). Drawing errors in the tactile experiments were similar to those in our phosphene drawing experiments (Figure S2).

### Phosphene orientation is aligned with retinal nerve fiber bundles

As described above, computational models and electrophysiological evidence from in vitro preparations of rat and rabbit retina suggest that retinal implants may stimulate passing axon fibers17,30,31. Retinal ganglion cells send their axons on highly stereotyped pathways en route to the optic nerve (Fig. 4A–C). Because of this topographic organization, an electrode that stimulates nearby axonal fibers would be expected to antidromically activate cell bodies located peripheral to the point of stimulation. Perceptually, activating an axon fiber that passes under a stimulating electrode is indistinguishable from the percept that would be elicited by activating the corresponding ganglion cell body. The visual percept should appear in the spatial location in visual space for which the corresponding ganglion cell encodes information (receptive field), which could be hundreds of microns away from the stimulation site42. Consequently, percepts elicited from axonal stimulation would be expected to appear elongated along the direction of the underlying nerve bundle trajectory (Fig. 4C).

To test whether the orientation of phosphene drawings were aligned with the underlying nerve fiber bundles, we estimated the relative location and orientation of each subject’s implant with respect to the fovea and the optic disc, using ophthalmic fundus photographs (Fig. 4D; here shown for Subject 4). While yellow screening pigments allow for visualization of macular extent in normal eyes, it is difficult to discriminate the macula-periphery boundary in our subjects because of the characteristic pigmentary deposits associated with retinitis pigmentosa1. We therefore had a retina specialist mark the fovea on a fundus image obtained before surgery (Fig. 4D, top), and subsequently used computer vision techniques (see Methods) to align the presurgical image with a second fundus image obtained after surgery (Fig. 4D, bottom), showing the implant relative to the optic nerve head. This allowed us to estimate the array center with respect to the fovea, the array rotation with respect to the horizontal raphe, and the retinal distance between the fovea and the optic nerve head for each subject (Fig. 4E). The resulting topographic measurements were then used to simulate a map of the ganglion axon pathways32 that was tailored to each subject’s retinal dimensions (Fig. 4F).

Remarkably, we found that for all subjects, phosphene orientation was well aligned with the tangent line of the nerve fiber bundle directly below the stimulating electrode (Fig. 5). Here, insets show mean drawings for representative electrodes, from which phosphene orientation was calculated. This was not only true for line phosphenes, which resembled carbon copies of the underlying fiber bundle topography (e.g., Subject 1: D2, Subject 4: F2), but also for more compact phosphenes, which still tended to be elongated in the direction of the local fiber bundle trajectory (e.g., Subject 2: B9, Subject 4: C10). To get a sense of effect size, we also calculated the fraction of the variance observed in phosphene orientation that the simulated map of nerve fiber bundles could explain (Subject 1: 0.463, Subject 2: 0.567, Subject 3: 0.024, Subject 4: 0.394).

To assess whether these angular errors were smaller than would be predicted from a random placement of the array on the retina, we performed a resampling analysis. First, we calculated the mean absolute angle between the five drawings corresponding to a single electrode and the tangent line of the closest nerve fiber bundle. We repeated this procedure for all electrodes in the array to produce a distribution of mean angular errors (box plots in Fig. 5). We then compared the mean angular error of the real distribution to the 1-tailed confidence interval of values from a resampled null model in order to estimate probability values. We considered two different null models: The first null model (NM1) assumed that phosphene orientation was independent of the axon map (all phosphene orientations sampled from a random uniform distribution $$\in [-\,90^\circ ,90^\circ ]$$, 1000 iterations). The second, more sophisticated null model (NM2) was generated by randomly placing the array on the retina (array center coordinates: $$x\in [-6000,\,4000]\,{\rm{\mu }}m,y\in [-4000,\,4000]\,{\rm{\mu }}m$$, array rotation $$\in [-\,90^\circ ,90^\circ ]$$, 1000 iterations). We found that angular errors were significantly smaller than predicted by either null model (NM1: p < 0.001 for Subjects 1 and 2, p < 0.01 for Subjects 3 and 4; NM2: p < 0.05 for all subjects).

### Predicting phosphene shape based on a simulated map of ganglion axon pathways

We then tested whether the spatial layout of ganglion axon pathways could account for phosphene shape as well as orientation. We assumed that the activation of an axon elicited a percept centered over the receptive field location of that axon’s cell body. The activation sensitivity of a passing axon fiber was assumed to decay exponentially with retinal distance from the stimulation site, with each subject’s data being fit with two parameters: a decay constant λ, which described activation sensitivity along the axon, and a decay constant ρ, which described sensitivity orthogonal to the axon (see Methods). This allowed us to generate a tissue activation map for each stimulating electrode, which we thresholded to arrive at a binary image that could serve as a prediction of a phosphene drawing (small schematic in the center column of Fig. 6).

Alternatively, we considered a simpler but widely used model that treated each electrode in an array as a ‘pixel’ in an image, thus assuming that stimulating a grid of electrodes on the retina would result in the percept of a grid of isolated, focal spots of light37,38,39,40,41. We refer to this model as the ‘scoreboard model’, because much like the large scoreboards found in sports stadiums, an image is created by an array of individual light sources that can be turned off or on. To implement the scoreboard model, we assumed that an electrode would lead to the percept of a Gaussian blob (with spatial decay constant ρ). The resulting intensity profile was again thresholded to obtain a binary image, which was compared to real phosphene drawings (small schematic in the right column of Fig. 6).

To find the parameter values under each model that best predicted phosphene shape, we constructed a cost function from the difference between predicted and observed phosphene area, orientation, and elongation, which we minimized using particle swarm optimization (see Methods). Because scoreboard and axon map models had a different number of parameters (scoreboard model: ρ; axon map model: ρ, λ), we used leave-one-electrode-out cross-validation to allow for fair model comparison, where we repeatedly fit each model to the drawings from all but one electrode in the array. Fitted parameter values were then used to predict the phosphene shapes of the held-out drawings. Note that a single value of ρ and λ was used to describe the drawings of all electrodes in that subject’s array.

The result of this cross-validation procedure is shown in Fig. 6. Here, ground-truth drawings are shown in the left column, and predicted phosphenes (on the test-fold of the cross-validation procedure) are shown in center and right columns. Thus, predicted phosphene shapes represent each model’s ability to generalize to new electrodes. Whereas the axon map model was able to generate both compact and elongated phosphenes that span a range of geometrical shapes such as ‘blobs’, ‘lines’, and ‘wedges’, the scoreboard model exclusively predicted round phosphenes of various size.

The best fitting, cross-validated parameter values are given in Table 2 (averaged across folds). Even though phosphene shape often varied drastically across electrodes (see Discussion), the axon map model recovered similar values for ρ and λ across different folds for a given subject, as indicated by relatively small SEMs. Without adjusting for drawing bias, these results suggest that electrical stimulation influences ganglion cells whose cell bodies are at a distance of approximately ρ = 437 μm (corresponding to ~1.5° of visual angle) orthogonal to the direction of the axon fiber bundle, but as far as λ = 1,420 μm (corresponding to ~5° of visual angle) along a direction parallel to the axon fiber.

To further quantify model performance, we compared cross-validated prediction errors for phosphene area, orientation, and elongation (Equation 13) across axon map and scoreboard models, Fig. 7. Here, each data point in the scatter plots corresponds to the cross-validated prediction error (averaged across every drawing in that fold) for each electrode. Data points almost always lie below the diagonal, indicating that the axon map model was more accurate than the scoreboard model. Indeed, the axon map model often improved cross-validated log prediction error by an order of magnitude (see insets), simply by adding a single parameter λ that accounted for the current spread along axons of passage in the optic nerve fiber layer of the retina.

## Discussion

We show here that the elicited percepts of patients with retinal implants can be accurately predicted using the spatial layout of ganglion axon pathways in the human retina. Model fits to behavioral data suggest that sensitivity to electrical stimulation is not confined to the axon initial segment30, but can be modeled as falling off with different decay constants along (with λ ranging from 500–1,420 μm) and orthogonally from (with ρ ranging from 144–437 μm) the axon, resulting in visual percepts ranging from ‘blobs’ to ‘streaks’ and ‘wedges’ depending on both the relative values of ρ and λ, and the retinal location of the stimulating electrode. These results agree with theoretical work suggesting an anisotropic spread of current in the retinal tissue28 as well as previous animal literature demonstrating that epiretinal stimulation leads to activation of passing axon fibers17,30,31,43, which can severely distort the quality of the generated visual experience15,17,42,44,45. Our findings suggest that the spatial distortions reported by patients are not arbitrary, but rather depend on the topographic organization of optic nerve fiber bundles in each subject’s retina, which can be captured by a computational model. Having an accurate model that generalizes across patients is a crucial first step in developing stimulation strategies for retinal prostheses that can produce complex, perceptually intelligible percepts. Overall our findings argue for more detailed modeling of biological detail across neural engineering applications.

### A rich repertoire of phosphene shapes

The phosphenes elicited by single-electrode stimulation vary dramatically across subjects and electrodes (Figs 2, 3A–C), despite relatively small drawing errors and consistency in drawings within a given electrode (Fig. 3D–F). These results are in agreement with the previous literature that has reported that patients subjectively report a variety of percept shapes22,35,46,47, of which only a small fraction could be described as focal spots of light.

The variability in phosphene shape across subjects that we report (captured by variation in λ and ρ across patients), might be due to several factors, a few of which are outlined below. First, diseases such as retinitis pigmentosa and macular degeneration are characterized by a progressive degeneration of photoreceptors, gradually affecting other layers of the retina4,48,49,50. In severe end-stage retinitis pigmentosa, roughly 95% of photoreceptors, 20% of bipolar cells, and 70% of ganglion cells degenerate51, so that little or no useful vision is retained. Disease progression therefore influences the relative proportion of surviving bipolar and ganglion cell types, which in turn is likely to influence phosphene shape.

Second, with a diameter of 200 μm, each electrode in the Argus II array encompasses the equivalent area of hundreds of photoreceptors. A single electrode therefore inevitably leads to activation of a wide variety of morphologically and functionally distinct retinal cells52,53, including simultaneous activation of both ON and OFF pathways. This is in contrast to natural stimulation, which precisely activates a number of specialized, functionally complementary, parallel processing pathways in the retina (for a recent review see ref.54). Although epiretinal stimulation with relatively short pulses might primarily activate ganglion cells rather than bipolar cells55,56,57,58,59, there is still much to be learned about how the information from these different retinal representations are combined at later stages of processing to form a conscious percept.

Third, electrode-retina distance is known to affect both perceptual thresholds35 and phosphene size47. Figure 8 shows simulations based on the assumption that ρ is primarily determined by current spread, whereas λ is primarily determined by axonal stimulation60,61. In these simulations, electrodes close to the retinal surface have a small ρ compared to λ, and are thus highly elongated (Fig. 8A). When electrodes are further from the retinal surface, ρ increases dramatically, thus resulting in circular percepts (Fig. 8D). These simulations agree with the observation that phosphenes from electrodes close to the retinal tack (and thus likely to be close to the retinal surface) appeared much more elongated than others (Fig. 2). Furthermore, low electrode-retina distances could explain why Subject 3 reported predominantly thin and elongated phosphenes (small errors in the tactile target control task argue against drawing bias; see Figures S2S3). Unfortunately, we did not have access to OCT images or impedance measurements for our subjects, which would have allowed us to infer electrode-retina distances for each electrode35,36. Future studies could use such data to constrain the values of ρ and λ.

Fourth, the mapping of retinal eccentricity to visual field coordinates is nonlinear. Because the foveola contains only photoreceptors, ganglion cell bodies are displaced centrifugally from their cone inputs by several degrees; an effect that extends out as far as 17°62,63.

Finally, phosphene size might be influenced by ganglion cell density and receptive field size. Whereas the receptive field size of retinal ganglion cells only gradually increases with eccentricity64, ganglion cell density decreases rapidly62. Furthermore, retinal degeneration in retinitis pigmentosa tends to progress from the periphery to the macula, thereby having a greater effect on ganglion cell density in the periphery48,51,65. Consequently, more peripheral electrodes would typically stimulate cells with only slightly larger receptive fields, but in much smaller numbers than in the fovea. These two conflicting effects may contribute to our finding of no correlation between phosphene area and retinal eccentricity (data not shown).

### Phosphene shape is mediated by axonal stimulation

Despite the variability in phosphene shape, all subjects reported seeing elongated phosphenes on at least a subset of electrodes (Fig. 3C). Although the electric field generated by a disk electrode is radially symmetric, the neural tissue induces anisotropies in the electric field, and stimulation of axon fibers produces even more striking anisotropies in patterns of neural activation within the retina28. It has long been known that external stimulation of an axon induces an action potential that travels both backward to the cell body and forward to the synaptic terminals66,67.

A number of studies have previously hypothesized that axonal stimulation could lead to phosphenes that are elongated in shape and poorly localized (e.g., ref.42). However, this idea has never been explicitly tested. In the present study we demonstrate that axonal stimulation in the retina leads to predictable distortions of shape in human patients, which can be captured by a computational model (Figs 57).

Axonal stimulation is a concern for other implant technologies as well. Although subretinal prostheses such as Alpha-IMS8 have electrodes in close proximity to bipolar cells, in vitro animal studies have found that subretinal stimulation with 1 ms pulses nonetheless directly activates retinal ganglion cells at thresholds statistically similar to those of inner retinal cells68,69,70. Similarly, axonal stimulation is expected to be an issue for cortical implants, since passing axons from neurons located in distant parts of the brain have been shown to be highly sensitive to electrical stimulation71,72,73.

Several recent studies have tried to identify stimulation protocols that minimize axonal activation, with mixed results. Whereas one in vitro study suggested using short-duration pulses (≤100 μs) to avoid axonal stimulation74, another study did not see any benefits of short pulses, and instead suggested using long-duration pulses (≥25 ms) or low-frequency (<25 Hz) sinusoidal stimulation31. One difficulty with these approaches is that they are likely to limit stimulation to a highly restricted amplitude and/or frequency range, potentially limiting the dynamic range available for the encoding of brightness46,47.

We show here that percepts are highly consistent over time and can potentially be described using an anatomically detailed computational model with a small number of parameters. Thus, an alternative strategy might be to move away from thinking about artificial sight as a linear combination of ‘pixels’, and instead accept the perceptual distortions resulting from axonal stimulation as the fundamental building blocks of prosthetic vision.

## Methods

### Subjects

Participants were four blind subjects (one female and three male) with severe retinitis pigmentosa, ranging from 45 to 70 years in age (Table 1). Subjects were chronically implanted with an epiretinal prosthesis as part of an FDA approved clinical trial (clinicaltrials.gov identifier for Subject 1: NCT00279500, registration date 01/17/2006, completed; Subjects 2–4: NCT00407602, registration date 12/01/2006, active). Surgeries were performed at the Doheny Eye Institute at the University of Southern California (Los Angeles, CA; Subject 1), at the Wilmer Eye Institute at Johns Hopkins School of Medicine (Baltimore, MD; Subject 2), at the Moorfields Eye Hospital (London, UK; Subject 3), and at the Royal Eye Hospital (Manchester, UK; Subject 4). None of the subjects had a recordable visual acuity prior to surgery, scoring worse than 2.9 logMAR (worse than 20/15,887) on a four-alternative forced-choice square-wave grating test18,75.

Due to their geographic location, Subjects 2–4 were not directly examined by the authors of this study. Instead, initial experimental procedures were sent to the clinical site, and trained field clinical engineers performed the experiments as specified. Raw collected data was then sent to the authors for subsequent analysis.

All tests were performed after obtaining informed consent under a protocol approved by the Institutional Review Board (IRB) at each subject’s location and under the principles of the Declaration of Helsinki (Subject 1: University of Southern California IRB at the Keck School of Medicine, Subject 2: Johns Hopkins Medicine IRB, Subject 3: Moorfields Eye Hospital IRB, Subject 4: Royal Eye Hospital IRB). Tests were carried out between four and six years after implantation for Subject 1, and between six months and one year after implantation for Subjects 2–4.

### Implant specification

Subject 1 was implanted with a 16-channel microelectrode array (Argus I; Second Sight Medical Products, Inc., Sylmar, CA) consisting of 260 and 520 μm diameter platinum disc electrodes, subtending 0.9° and 1.8° of visual angle, respectively. Electrodes were spaced 800 μm apart and arranged in a 4 × 4 alternating checkerboard pattern (Fig. 1A). Subjects 2–4 were implanted with a 60-channel microelectrode array (Argus II; Second Sight Medical Products, Inc., Sylmar, CA) consisting of 200 μm diameter platinum disc electrodes, each subtending 0.7° of visual angle. Electrodes were spaced 525 μm apart and arranged in a 6 × 10 grid (Fig. 1B).

All stimuli described in this study were presented in ‘direct stimulation’ mode. Stimuli were programmed in Matlab using custom software, and pulse train parameters (the electrode(s) to be stimulated, current amplitude, pulse width, individual pulse duration, inter-pulse interval, and pulse train duration) were sent directly to an external visual processing unit (VPU), which was used to send stimulus commands to the internal portion of the implant using an inductive coil link. The implanted receiver wirelessly received these data and sent the signals to the electrode array via a small cable.

In day-to-day use, an external unit consisting of a small camera and transmitter mounted on a pair of glasses is worn by the user. The camera captured video and sends the information to the VPU which converts it into pulse trains using pre-specified image processing techniques (‘camera’ mode).

### Psychophysical methods

Perceptual thresholds for individual electrodes were measured using an adaptive yes/no procedure implemented using custom software (see Supplemental Information). All presented stimuli were charge-balanced, square-wave, biphasic, cathodic-first pulse trains with fixed stimulus duration (Argus I: 500 ms, Argus II: 250 ms), current amplitude (set at 2x threshold), stimulus frequency (20 Hz) and pulse duration (0.45 ms/phase, no interphase delay).

Subjects were asked to perform a drawing task with a tactile target (Supplemental Information) or when their retina was electrically stimulated (Fig. 1). In a given experimental run, a total of n stimulus conditions (either tactile or retinal stimulation) were tested. Each condition was repeated for m trials (for a total of mn trials per experimental run). Repeated trials of the same condition were randomized amongst other stimuli to confirm reproducibility of results.

Head movement of the Argus I subject was minimized with a chin rest. After each stimulus presentation, the subject traced the shape on a grid screen (containing 6 inch horizontal and vertical grid lines) with a center location aligned horizontally and vertically with the subject’s head. Drawing was carried out with a pen whose cap was a different color than its body. A head-mounted camera (Misumi CMOS S588-3T), located on the subject’s glasses, was used to record the trials to digital video recorder (DVR). Video files were analyzed off-line to extract shape data using custom built tracking software. In the first stage of processing, the entire image was rotated appropriately using the grid screen background as a reference. In the second stage, vertical and horizontal gridlines, and the distance from the subject to screen were used to set a new coordinate system in visual angle coordinates (since the subject was 16 inches/40.6 cm from the screen, 4 gridlines = 70.0 cm corresponded to 73.8 degrees visual angle). In the third stage, the location of the pen cap was tracked (based on its color) across each frame of the video file. Finally, a binary shape data file was built from pen cap coordinate locations across all frames.

Argus II subjects were placed in a chair at a comfortable distance from a touch screen monitor with its center location aligned horizontally with the subject’s head. The distance from each subject’s eyes to the screen was recorded. After each stimulus presentation, the subject traced the shape on the monitor and the experimenter advanced to the next trial. Touch screen data were instantly recorded by custom software in 2D coordinates to a text file. Text files were analyzed offline to translate vector coordinates to a binary shape data file. The distance recorded from the subject to screen was used to set a new coordinate system in visual angle. Since Subjects 2–4 were 33, 30.0, and 30.5 inches from the screen, this corresponded to a display size of 60, 65 and 64 degrees of visual angle (horizontal screen length), respectively. After translating to the final visual angle coordinate system, the binary image was used in subsequent shape analyses.

### Computational methods

#### Phosphene shape descriptors

Phosphene shape was quantified using three parameter-free shape descriptors commonly used in image processing: area, orientation, and elongation76. (Elongation is sometimes also referred to as eccentricity in the literature. We avoid that usage here to prevent confusion with retinal eccentricity). These descriptors are based on a set of statistical quantities known as ‘image moments’. For an X × Y pixel grayscale image, I(x,y), where $$x\in [1,\,X]$$ and $$y\in [1,\,Y]$$, the raw image moments Mij were calculated as:

$${M}_{ij}=\sum _{x}\sum _{y}{x}^{i}{y}^{j}I(x,y).$$
(1)

Raw image moments were used to compute area (A = M00) and the center of mass ($$\bar{x},\bar{y})=({M}_{10}/{M}_{00},{M}_{01}/{M}_{00}$$) of each phosphene.

Phosphene orientation was calculated from the covariance matrix of an image:

$${\rm{cov}}[I(x,y)]=[\begin{array}{cc}{\mu }_{20}^{\text{'}} & {\mu }_{11}^{\text{'}}\\ {\mu }_{11}^{\text{'}} & {\mu }_{02}^{\text{'}}\end{array}],$$
(2)

where $${\mu }_{20}^{\text{'}}={M}_{20}/{M}_{00}-{\bar{x}}^{2}$$, $${\mu }_{11}^{\text{'}}={M}_{11}/{M}_{00}-\bar{x}\bar{y}$$, and $${\mu }_{02}^{\text{'}}={M}_{02}/{M}_{00}-{\bar{y}}^{2}$$. The eigenvectors of this matrix corresponded to the major and minor axes of the image intensity. Orientation (θ) could thus be extracted from the angle of the eigenvector associated with the largest eigenvalue towards the axis closest to this eigenvector:

$$\theta =\frac{1}{2}\arctan (\frac{2{\mu }_{11}^{\text{'}}}{{\mu }_{20}^{\text{'}}-{\mu }_{02}^{\text{'}}}),$$
(3)

which was valid as long as $${\mu }_{20}^{\text{'}}\ne {\mu }_{02}^{\text{'}}$$, with $$\theta \in \{-\pi /2,\pi /2\}$$. To avoid division by zero, we manually assigned an angle of θ = 0 whenever $${\mu }_{20}^{\text{'}}$$ was equal to $${\mu }_{02}^{\text{'}}$$.

Phosphene elongation (E) was calculated from the eigenvectors of the covariance matrix of Equation (2):

$$E=\sqrt{1-\frac{{\lambda }_{2}}{{\lambda }_{1}}},$$
(4)

where $${\lambda }_{1,2}=({\mu ^{\prime} }_{20}+{\mu ^{\prime} }_{02})/2\pm \sqrt{4{\mu ^{\prime} }_{11}^{2}+{({\mu ^{\prime} }_{20}-{\mu ^{\prime} }_{02})}^{2}}/2$$, and $$E\in [0,1]$$. An elongation of E = 1 represents a circle, and E = 0 represents an infinitesimally thin line.

#### Determination of implant location using fundus photography

Implant location was estimated by analyzing color fundus photographs obtained using systems available at each clinical site. For each subject, we performed the following procedure:

1. 1.

Extract landmarks: On a baseline fundus photograph (before surgery), a retina specialist marked the foveal pit and the center of the optic nerve head. On the most recent fundus photograph (after surgery), we marked the center of the implant.

2. 2.

Combine baseline image with implant image: We performed image registration using feature matching to bring the two images into the same coordinate system.

3. 3.

Adjust for magnification: Pixel distances were converted to retinal distances by using the known electrode-electrode spacing (Argus I: 800 μm, Argus II: 525 μm).

4. 4.

Adjust for rotation: We approximated the horizontal raphe by fitting a parabola to the main vascular arcade, assuming that the center of the optic nerve head lay at the vertex of the parabola, and that the raphe was parallel to the parabola’s axis of symmetry77,78.

5. 5.

Coordinate transform: The registered image was rotated so that the horizontal raphe came to lie on the abscissa, and the foveal pit at the origin of the new coordinate system. We located the coordinates of the center of the optic nerve head as well as the center of the array (from Step 1) in this new coordinate system. Retinal distances (μm) were related to visual space (deg) using a formula that computes the relationship between retinal arc lengths and visual angles from based on the optic axis63.

6. 6.

The extracted landmarks were then used to place a simulated array on a simulated map of ganglion axon pathways using the pulse2percept software60.

This procedure allowed us to estimate each subject’s array location and orientation with respect to the fovea (Table 3). Based on fundus photographs of 104 sighted humans79, the center of the optic disc was expected to be located at 15.5° ± 1.1° nasal, 1.5° ± 0.9° superior with respect to the fovea. For all four subjects, the estimated center of the optic disk was within two standard deviations of these expected values.

#### Scoreboard model

The scoreboard model assumed that electrical stimulation led to the percept of focal dots of light, centered over the visual field location associated with the stimulated retinal field location (xstim, ystim), whose spatial intensity profile decayed with a Gaussian profile40,41:

$${I}_{{\rm{score}}}(x,y;\rho )=\exp (-\frac{{(x-{x}_{{\rm{stim}}})}^{2}+{(y-{y}_{{\rm{stim}}})}^{2}}{2{\rho }^{2}}),$$
(5)

where ρ was the spatial decay constant.

The resulting intensity profile $${I}_{{\rm{score}}}(x,y;\rho )$$ was then thresholded to obtain a binary image. The threshold was chosen as $$1/\sqrt{e}$$, such that points closer than ρ to (xstim, ystim) were assigned a value of 1, and all other points were assigned a value of 0.

#### Axon map model

Following Jansonius et al.32, we assumed that the trajectories of the optic nerve fibers could be described in a modified polar coordinate system (r,ϕ) with its origin located in the center of the optic disc. A nerve fiber was modeled as a spiral:

$$\varphi (r,{\varphi }_{0})={\varphi }_{0}+b(r,{\varphi }_{0}){(r-{r}_{0})}^{c(r,{\varphi }_{0})},$$
(6)

where $${\varphi }_{0}=\varphi (r={r}_{0})$$ is the angular position of the trajectory at its starting point at a circle with radius r0 around the center of the optic disc, b a real number describing the curvature of the spiral,

$$b({\varphi }_{0},r)=\{\begin{array}{c}\exp (-1.9+3.9\,\tanh (\frac{-({\varphi }_{0}-121)}{14})),r\ge 0\\ -\exp (0.7+1.5\,\tanh (\frac{-(-{\varphi }_{0}-90)}{25})),r < 0,\end{array}$$
(7)

and c a positive real number describing the location of the point of maximal curvature,

$$c({\varphi }_{0},r)=\{\begin{array}{c}1.9+1.4\,\tanh (\frac{{\varphi }_{0}-121}{14})\,,r\ge 0\\ 1.0+0.5\,\tanh (\frac{-{\varphi }_{0}-90}{25}),r < 0.\end{array}$$
(8)

Jansonius and colleagues determined parameter values by fitting Equations (68) to the topographical layout of 55 eyes from 55 human subjects for details see ref.32. The attentive reader might notice that Equation (8) above fixes a typo in Equation (3) of ref.32: The tanh numerator should indeed read ϕ0 − 121, not −ϕ0 − 121.

To apply the axon map to the eyes of our subjects, we first transformed the original coordinate system (r, ϕ) to Cartesian coordinates (x, y) with the foveal pit located at (0, 0), and then set the coordinates of the optic disc (xod, yod) to the values estimated from fundus photography (Table 3). The resulting axon maps for each subject can be seen in Fig. 5.

An axon’s sensitivity to electrical stimulation was assumed to decay exponentially with distance from the soma $$({x}_{{\rm{soma}}},\,{y}_{{\rm{soma}}})$$:

$${I}_{{\rm{axon}}}(x,y;\rho ,\lambda )={I}_{{\rm{score}}}(x,y;\rho )\exp (-\frac{{(x-{x}_{{\rm{soma}}})}^{2}+{(y-{y}_{{\rm{soma}}})}^{2}}{2{\lambda }^{2}}),$$
(9)

where λ was the spatial decay constant along the axon. $${I}_{{\rm{score}}}(x,y;\rho )$$ is the same as in Equation (5) and is parameterized by a single parameter, ρ. As in the scoreboard model, the resulting intensity profile $${I}_{{\rm{axon}}}(x,y;\rho ,\lambda )$$ was thresholded to obtain a binary image.

#### Model fitting and evaluation

To fit the scoreboard and axon map models to subject drawings, we first calculated the coefficient of determination (R2) from the predicted binary images and the corresponding ground-truth subject drawings. R2 was calculated from the ratio of the residual sum of squares (SSres) and the total sum of squares (SStot) for each shape descriptor (area, orientation, or elongation):

$${R}^{2}=1-\frac{S{S}_{{\rm{res}}}}{S{S}_{{\rm{tot}}}},$$
(10)
$$S{S}_{res}=\sum _{i}{({s}_{i}-{\hat{s}}_{i})}^{2},$$
(11)
$$S{S}_{tot}=\sum _{i}{({s}_{i}-s)}^{2},$$
(12)

where si was the shape descriptor for the i-th ground-truth image, $${\hat{s}}_{i}$$ was the shape descriptor for the i-th predicted image, and $$\bar{s}$$ was the mean of the shape descriptor averaged over all images. The three quantities $${R}_{{\rm{area}}}^{2}$$, $${R}_{{\rm{orientation}}}^{2}$$, and $${R}_{{\rm{elongation}}}^{2}$$ resulting from this procedure were then combined to construct a cost function that could be iteratively minimized:

$$c=\sum _{d}1-{R}_{d}^{2},$$
(13)

where d = {area, orientation, elongation}. Due to the nonconvexity of this optimization problem, we minimized the cost function using particle swarm optimization80. We set the swarm size at ten times the number of parameters81. We ran every fitting procedure five times with different, randomly chosen initial conditions, and then chose the best run in subsequent analyses.

To allow for fair performance comparison despite the scoreboard and axon map models having different numbers of parameters, we implemented a leave-one-electrode-out cross-validation procedure, where we repeatedly fit each model to the drawings from all but one electrode in the array. This is equivalent to calculating the Akaike Information Criterion that takes into account the difference in number of parameters82. The fitted parameter values were then used to predict the shape descriptors of the held-out drawings (Fig. 7). Note that a single value of ρ and λ were fitted for each subject, and then used for all electrodes in that subject’s array.

## Data Availability

Data are available on the Open Science Framework (https://doi.org/10.17605/osf.io/dw9nz). The software used for analyses was based on the pulse2percept Python package60. Scripts used to fit the scoreboard and axon map models, to analyze the data, and to produce the figures in the paper are available on GitHub (https://github.com/VisCog/ArgusShapes.git, v0.2).

## Ethics declarations

### Competing Interests

Authors M.B., J.D.W., A.R., G.M.B. and I.F. are collaborators with Second Sight Medical Products Inc., the company that develops, manufactures, and markets the Argus II Retinal Prosthesis System referenced within this article. Second Sight had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## References

1. 1.

Hartong, D. T., Berson, E. L. & Dryja, T. P. Retinitis pigmentosa. Lancet 368, 1795–1809, https://doi.org/10.1016/S0140-6736(06)69740-7 (2006).

2. 2.

Jager, R. D., Mieler, W. F. & Miller, J. W. Age-related macular degeneration. N Engl J Med 358, 2606–2617, https://doi.org/10.1056/NEJMra0801537 (2008).

3. 3.

Marc, R. E. & Jones, B. W. Retinal remodeling in inherited photoreceptor degenerations. Mol Neurobiol 28, 139–147, https://doi.org/10.1385/MN:28:2:139 (2003).

4. 4.

Marc, R. E., Jones, B. W., Watt, C. B. & Strettoi, E. Neural remodeling in retinal degeneration. Prog Retin Eye Res 22, 607–655 (2003).

5. 5.

Weiland, J. D., Walston, S. T. & Humayun, M. S. Electrical Stimulation of the Retina to Produce Artificial Vision. Annu Rev Vis Sci 2, 273–294, https://doi.org/10.1146/annurev-vision-111815-114425 (2016).

6. 6.

Rizzo, S. et al. The Argus II Retinal Prosthesis: 12-month outcomes from a single-study center. Am J Ophthalmol 157, 1282–1290, https://doi.org/10.1016/j.ajo.2014.02.039 (2014).

7. 7.

da Cruz, L. et al. Five-Year Safety and Performance Results from the Argus II Retinal Prosthesis System Clinical Trial. Ophthalmology, https://doi.org/10.1016/j.ophtha.2016.06.049 (2016).

8. 8.

Stingl, K. et al. Subretinal Visual Implant Alpha IMS–Clinical trial interim report. Vision Research 111, 149–160, https://doi.org/10.1016/j.visres.2015.03.001 (2015).

9. 9.

Humayun, M. S. et al. Interim results from the international trial of Second Sight’s visual prosthesis. Ophthalmology 119, 779–788, https://doi.org/10.1016/j.ophtha.2011.09.028 (2012).

10. 10.

Lorach, H. et al. Photovoltaic restoration of sight with high visual acuity. Nat Med 21, 476–482, https://doi.org/10.1038/nm.3851 (2015).

11. 11.

Chader, G. J. & Young, M. Preface: Sight Restoration Through Stem Cell Therapy. Invest Ophthalmol Vis Sci 57, ORSFa1–5, https://doi.org/10.1167/iovs.16-19125 (2016).

12. 12.

da Cruz, L. et al. Phase 1 clinical study of an embryonic stem cell-derived retinal pigment epithelium patch in age-related macular degeneration. Nat Biotechnol. https://doi.org/10.1038/nbt.4114 (2018).

13. 13.

Gaub, B. M., Berry, M. H., Holt, A. E., Isacoff, E. Y. & Flannery, J. G. Optogenetic Vision Restoration Using Rhodopsin for Enhanced Sensitivity. Mol Ther 23, 1562–1571, https://doi.org/10.1038/mt.2015.121 (2015).

14. 14.

Fine, I., Cepko, C. L. & Landy, M. S. Vision research special issue: Sight restoration: Prosthetics, optogenetics and gene therapy. Vision Res 111, 115–123, https://doi.org/10.1016/j.visres.2015.04.012 (2015).

15. 15.

Nanduri, D., Humayun, M. S., Greenberg, R. J., McMahon, M. J. & Weiland, J. D. Retinal prosthesis phosphene shape analysis. In 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 1785–1788 (2008).

16. 16.

Luo, Y. H., Zhong, J. J., Clemo, M. & da Cruz, L. Long-term Repeatability and Reproducibility of Phosphene Characteristics in Chronically Implanted Argus(R) II Retinal Prosthesis Subjects. Am J Ophthalmol. https://doi.org/10.1016/j.ajo.2016.07.021 (2016).

17. 17.

Rizzo, J. F. 3rd, Wyatt, J., Loewenstein, J., Kelly, S. & Shire, D. Perceptual efficacy of electrical stimulation of human retina with a microelectrode array during short-term surgical trials. Invest Ophthalmol Vis Sci 44, 5362–5369 (2003).

18. 18.

Caspi, A. et al. Feasibility study of a retinal prosthesis: spatial vision with a 16-electrode implant. Arch Ophthalmol 127, 398–401, https://doi.org/10.1001/archophthalmol.2009.20 (2009).

19. 19.

Wilke, R. et al. Spatial resolution and perception of patterns mediated by a subretinal 16-electrode array in patients blinded by hereditary retinal dystrophies. Invest Ophthalmol Vis Sci 52, 5995–6003, https://doi.org/10.1167/iovs.10-6946 (2011).

20. 20.

Yanai, D. et al. Visual performance using a retinal prosthesis in three subjects with retinitis pigmentosa. Am J Ophthalmol 143, 820–827, https://doi.org/10.1016/j.ajo.2007.01.027 (2007).

21. 21.

Pérez Fornos, A. et al. Temporal Properties of Visual Perception on Electrical Stimulation of the Retina. Investigative Ophthalmology & Visual Science 53, 2720–2731, https://doi.org/10.1167/iovs.11-9344 (2012).

22. 22.

Horsager, A. et al. Predicting visual sensitivity in retinal prosthesis patients. Invest Ophthalmol Vis Sci 50, 1483–1491, https://doi.org/10.1167/iovs.08-2595 (2009).

23. 23.

Horsager, A., Boynton, G. M., Greenberg, R. J. & Fine, I. Temporal interactions during paired-electrode stimulation in two retinal prosthesis subjects. Invest Ophthalmol Vis Sci 52, 549–557, https://doi.org/10.1167/iovs.10-5282 (2011).

24. 24.

Horsager, A., Greenberg, R. J. & Fine, I. Spatiotemporal interactions in retinal prosthesis subjects. Invest Ophthalmol Vis Sci 51, 1223–1233, https://doi.org/10.1167/iovs.09-3746 (2010).

25. 25.

Wilke, R. G., Moghadam, G. K., Lovell, N. H., Suaning, G. J. & Dokos, S. Electric crosstalk impairs spatial resolution of multi-electrode arrays in retinal implants. J Neural Eng 8, 046016, https://doi.org/10.1088/1741-2560/8/4/046016 (2011).

26. 26.

da Cruz, L. et al. The Argus II epiretinal prosthesis system allows letter and word reading and long-term function in patients with profound vision loss. Br J Ophthalmol 97, 632–636, https://doi.org/10.1136/bjophthalmol-2012-301525 (2013).

27. 27.

Zrenner, E. et al. Subretinal electronic chips allow blind patients to read letters and combine them to words. Proceedings. Biological sciences/The Royal Society 278, 1489–1497, https://doi.org/10.1098/rspb.2010.1747 (2011).

28. 28.

Esler, T. B. et al. Minimizing activation of overlying axons with epiretinal stimulation: The role of fiber orientation and electrode configuration. PLoS One 13, e0193598, https://doi.org/10.1371/journal.pone.0193598 (2018).

29. 29.

Tahayori, B. et al. Modelling extracellular electrical stimulation: part 4. Effect of the cellular composition of neural tissue on its spatio-temporal filtering properties. J Neural Eng 11, 065005, https://doi.org/10.1088/1741-2560/11/6/065005 (2014).

30. 30.

Fried, S. I., Lasker, A. C. W., Desai, N. J., Eddington, D. K. & Rizzo, J. F. Axonal Sodium-Channel Bands Shape the Response to Electric Stimulation in Retinal Ganglion Cells. J Neurophysiol 101, 1972–1987, https://doi.org/10.1152/jn.91081.2008 (2009).

31. 31.

Weitz, A. C. et al. Improving the spatial resolution of epiretinal implants by increasing stimulus pulse duration. Sci Transl Med 7, 318ra203, https://doi.org/10.1126/scitranslmed.aac4877 (2015).

32. 32.

Jansonius, N. M. et al. A mathematical description of nerve fiber bundle trajectories and their variability in the human retina. Vision Res 49, 2157–2163 (2009).

33. 33.

Beyeler, M. Biophysical model of axonal stimulation in epiretinal visual prostheses. In 9th International IEEE/EMBS Conference on Neural Engineering (NER), San Francisco, CA, USA, https://doi.org/10.1109/NER.2019.8716969 (2019).

34. 34.

Mueller, J. K. & Grill, W. M. Model-based analysis of multiple electrode array stimulation for epiretinal visual prostheses. J Neural Eng 10, 036002, https://doi.org/10.1088/1741-2560/10/3/036002 (2013).

35. 35.

de Balthasar, C. et al. Factors affecting perceptual thresholds in epiretinal prostheses. Invest Ophthalmol Vis Sci 49, 2303–2314, https://doi.org/10.1167/iovs.07-0696 (2008).

36. 36.

Majdi, J. A., Minnikanti, S., Peixoto, N., Agrawal, A. & Cohen, E. D. Access resistance of stimulation electrodes as a function of electrode proximity to the retina. J Neural Eng 12, 016006, https://doi.org/10.1088/1741-2560/12/1/016006 (2015).

37. 37.

Chen, S. C., Suaning, G. J., Morley, J. W. & Lovell, N. H. Simulating prosthetic vision: I. Visual models of phosphenes. Vision Res 49, 1493–1506 (2009).

38. 38.

Dagnelie, G. et al. Real and virtual mobility performance in simulated prosthetic vision. J Neural Eng 4, S92–101, https://doi.org/10.1088/1741-2560/4/1/S11 (2007).

39. 39.

Freeman, D. K., Rizzo, J. F. 3rd & Fried, S. I. Encoding visual information in retinal ganglion cells with prosthetic stimulation. J Neural Eng 8, 035005, https://doi.org/10.1088/1741-2560/8/3/035005 (2011).

40. 40.

Hayes, J. S. et al. Visually guided performance of simple tasks using simulated prosthetic vision. Artif Organs 27, 1016–1028 (2003).

41. 41.

Thompson, R. W. Jr., Barnett, G. D., Humayun, M. S. & Dagnelie, G. Facial recognition using simulated prosthetic pixelized vision. Invest Ophthalmol Vis Sci 44, 5035–5042 (2003).

42. 42.

Fine, I. & Boynton, G. M. Pulse trains to percepts: the challenge of creating a perceptually intelligible world with sight recovery technologies. Philosophical transactions of the Royal Society of London. Series B, Biological sciences 370, 20140208, https://doi.org/10.1098/rstb.2014.0208 (2015).

43. 43.

Grosberg, L. E. et al. Activation of ganglion cells and axon bundles using epiretinal electrical stimulation. J Neurophysiol 118, 1457–1471, https://doi.org/10.1152/jn.00750.2016 (2017).

44. 44.

Rattay, F. & Resatz, S. Effective electrode configuration for selective stimulation with inner eye prostheses. IEEE Trans Biomed Eng 51, 1659–1664, https://doi.org/10.1109/TBME.2004.828044 (2004).

45. 45.

Beyeler, M., Rokem, A., Boynton, G. M. & Fine, I. Learning to see again: biological constraints on cortical plasticity and the implications for sight restoration technologies. J Neural Eng 14, 051003, https://doi.org/10.1088/1741-2552/aa795e (2017).

46. 46.

Greenwald, S. H. et al. Brightness as a function of current amplitude in human retinal electrical stimulation. Invest Ophthalmol Vis Sci 50, 5017–5025, https://doi.org/10.1167/iovs.08-2897 (2009).

47. 47.

Nanduri, D. et al. Frequency and amplitude modulation have different effects on the percepts elicited by retinal stimulation. Invest Ophthalmol Vis Sci 53, 205–214, https://doi.org/10.1167/iovs.11-8401 (2012).

48. 48.

Humayun, M. S. et al. Morphometric analysis of the extramacular retina from postmortem eyes with retinitis pigmentosa. Invest Ophthalmol Vis Sci 40, 143–148 (1999).

49. 49.

Mazzoni, F., Novelli, E. & Strettoi, E. Retinal ganglion cells survive and maintain normal dendritic morphology in a mouse model of inherited photoreceptor degeneration. J Neurosci 28, 14282–14292, https://doi.org/10.1523/JNEUROSCI.4968-08.2008 (2008).

50. 50.

Jones, B. W. et al. Retinal remodeling triggered by photoreceptor degenerations. The Journal of comparative neurology 464, 1–16, https://doi.org/10.1002/cne.10703 (2003).

51. 51.

Santos, A. et al. Preservation of the inner retina in retinitis pigmentosa. A morphometric analysis. Arch Ophthalmol 115, 511–515 (1997).

52. 52.

Dacey, D. 20 origins of perception: Retinal ganglion cell diversity and the creation of parallel visual pathways. In The Cognitive Neurosciences III (ed. Michael S. Gazzaniga) 281 (MIT Press, 2004).

53. 53.

Field, G. D. & Chichilnisky, E. J. Information processing in the primate retina: circuitry and coding. Annu Rev Neurosci 30, 1–30, https://doi.org/10.1146/annurev.neuro.30.051606.094252 (2007).

54. 54.

Nassi, J. J. & Callaway, E. M. Parallel processing strategies of the primate visual system. Nat Rev Neurosci 10, 360–372, https://doi.org/10.1038/nrn2619 (2009).

55. 55.

Greenberg, R. J. Analysis of electrical stimulation of the vertebrate retina: Work towards a retinal prosthesis, Johns Hopkins University (1998).

56. 56.

Freeman, D. K., Eddington, D. K., Rizzo, J. F. 3rd & Fried, S. I. Selective activation of neuronal targets with sinusoidal electric stimulation. J Neurophysiol 104, 2778–2791, https://doi.org/10.1152/jn.00551.2010 (2010).

57. 57.

Sekirnjak, C. et al. Electrical stimulation of mammalian retinal ganglion cells with multielectrode arrays. J Neurophysiol 95, 3311–3327, https://doi.org/10.1152/jn.01168.2005 (2006).

58. 58.

Sekirnjak, C. et al. High-resolution electrical stimulation of primate retina for epiretinal implant design. J Neurosci 28, 4446–4456, https://doi.org/10.1523/JNEUROSCI.5138-07.2008 (2008).

59. 59.

Fried, S. I., Hsueh, H. A. & Werblin, F. S. A method for generating precise temporal patterns of retinal spiking using prosthetic stimulation. J Neurophysiol 95, 970–978, https://doi.org/10.1152/jn.00849.2005 (2006).

60. 60.

Beyeler, M., Boynton, G. M., Fine, I. & Rokem, A. pulse2percept: A Python-based simulation framework for bionic vision. In Proceedings of the 16th Python in Science Conference. (eds Huff, K., Lippa, D., Niederhut, D. & Pacer, M.) 81–88, https://doi.org/10.25080/shinma-7f4c6e7-00c (2017).

61. 61.

Beyeler, M., Rokem, A., Boynton, G. & Fine, I. Modeling the perceptual experience of retinal prosthesis patients. J Vision 17, 573–573, https://doi.org/10.1167/17.10.573 (2017).

62. 62.

Curcio, C. A. & Allen, K. A. Topography of ganglion cells in human retina. The Journal of comparative neurology 300, 5–25, https://doi.org/10.1002/cne.903000103 (1990).

63. 63.

Watson, A. B. A formula for human retinal ganglion cell receptive field density as a function of visual field location. J Vis 14, https://doi.org/10.1167/14.7.15 (2014).

64. 64.

Peichl, L. & Wassle, H. Size, scatter and coverage of ganglion cell receptive field centres in the cat retina. The Journal of physiology 291, 117–141 (1979).

65. 65.

Stone, J. L., Barlow, W. E., Humayun, M. S., de Juan, E. Jr. & Milam, A. H. Morphometric analysis of macular photoreceptors and ganglion cells in retinas with retinitis pigmentosa. Arch Ophthalmol 110, 1634–1639 (1992).

66. 66.

Bishop, P. O., Burke, W. & Davis, R. Single-unit recording from antidromically activated optic radiation neurones. The Journal of physiology 162, 432–450 (1962).

67. 67.

Lemon, R. Methods for neuronal recording in conscious animals. (John Wiley and Sons, 1984).

68. 68.

Boinagrov, D., Pangratz-Fuehrer, S., Goetz, G. & Palanker, D. Selectivity of direct and network-mediated stimulation of the retinal ganglion cells with epi-, suband intraretinal electrodes. J Neural Eng 11, 026008, https://doi.org/10.1088/1741-2560/11/2/026008 (2014).

69. 69.

Eickenscheidt, M., Jenkner, M., Thewes, R., Fromherz, P. & Zeck, G. Electrical stimulation of retinal neurons in epiretinal and subretinal configuration using a multicapacitor array. J Neurophysiol 107, 2742–2755, https://doi.org/10.1152/jn.00909.2011 (2012).

70. 70.

Tsai, D., Morley, J. W., Suaning, G. J. & Lovell, N. H. Direct activation of retinal ganglion cells with subretinal stimulation. In International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society, 618–621, https://doi.org/10.1109/IEMBS.2009.5333494 (2009).

71. 71.

Ranck, J. B. Jr. Which elements are excited in electrical stimulation of mammalian central nervous system: a review. Brain research 98, 417–440 (1975).

72. 72.

Histed, M. H., Bonin, V. & Reid, R. C. Direct activation of sparse, distributed populations of cortical neurons by electrical microstimulation. Neuron 63, 508–522, https://doi.org/10.1016/j.neuron.2009.07.016 (2009).

73. 73.

Lee, S. W., Fallegger, F., Casse, B. D. & Fried, S. I. Implantable microcoils for intracortical magnetic stimulation. Sci Adv 2, e1600889, https://doi.org/10.1126/sciadv.1600889 (2016).

74. 74.

Jensen, R. J., Ziv, O. R. & Rizzo, J. F. 3rd Thresholds for activation of rabbit retinal ganglion cells with relatively large, extracellular microelectrodes. Invest Ophthalmol Vis Sci 46, 1486–1496, https://doi.org/10.1167/iovs.04-1018 (2005).

75. 75.

Ahuja, A. K. et al. Factors Affecting Perceptual Threshold in Argus II Retinal Prosthesis Subjects. Transl Vis Sci Technol 2, 1, https://doi.org/10.1167/tvst.2.4.1 (2013).

76. 76.

Van der Walt, S. et al. scikit-image: Image processing in Python. PeerJ 2, https://doi.org/10.7717/peerj.453 (2014).

77. 77.

Tobin, K. W., Chaum, E., Govindasamy, V. P. & Karnowski, T. P. Detection of anatomic structures in human retinal imagery. IEEE Trans Med Imaging 26, 1729–1739 (2007).

78. 78.

Chin, K. S., Trucco, E., Tan, L. & Wilson, P. J. Automatic fovea location in retinal images using anatomical priors and vessel density. Pattern Recognition Letters 34, 1152–1158, https://doi.org/10.1016/j.patrec.2013.03.016 (2013).

79. 79.

Rohrschneider, K. Determination of the location of the fovea on the fundus. Invest Ophthalmol Vis Sci 45, 3257–3258, https://doi.org/10.1167/iovs.03-1157 (2004).

80. 80.

Kennedy, J. & Eberhart, R. C. Particle swarm optimization. In IEEE International Conference on Neural Networks. 1942–1948 (1995).

81. 81.

Storn, R. On the usage of differential evolution for function optimization. In Proceedings of North American Fuzzy Information Processing. 519–523 (1996).

82. 82.

Stone, M. An Asymptotic Equivalence of Choice of Model by Cross-Validation and Akaike’s Criterion. Journal of the Royal Statistical Society. Series B (Methodological) 39, 44–47 (1977).

## Acknowledgements

Supported by the Washington Research Foundation Funds for Innovation in Neuroengineering and Data-Intensive Discovery (M.B.), by a grant from the Gordon and Betty Moore Foundation and the Alfred P. Sloan Foundation to the University of Washington eScience Institute Data Science Environment (A.R.), and by the National Institutes of Health (NIH K99 EY-029329 to M.B., EY-12925 to G.M.B., and EY-014645 to I.F.). Research credits for cloud computing were provided by Amazon Web Services. The authors would like to thank retina specialists Drs. Aaron Y. Lee and Cecilia S. Lee in the Department of Ophthalmology at the University of Washington for locating the fovea in fundus photographs.

## Author information

J.W., A.R., G.M.B. and I.F. designed the study. D.N. collected the data. M.B. and A.R. analyzed the data. M.B. wrote the software. M.B., D.N. and I.F. wrote the manuscript.

### Competing Interests

Authors M.B., J.D.W., A.R., G.M.B. and I.F. are collaborators with Second Sight Medical Products Inc., the company that develops, manufactures, and markets the Argus II Retinal Prosthesis System referenced within this article. Second Sight had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Correspondence to Michael Beyeler.