Introduction

Degenerative retinal diseases such as retinitis pigmentosa1 and macular degeneration2 lead to a loss of photoreceptor cells and subsequent remodeling of the neural circuitry in the retina3,4, causing irreversible blindness in more than 10 million people worldwide. Analogous to cochlear implants, the goal of retinal prostheses is to help alleviate these incurable conditions by electrically stimulating surviving cells in the retina (for a recent review, see ref.5). The hope is that electrically evoked neuronal responses will be transmitted to the brain and interpreted by the subject as visual percepts (‘phosphenes’). Two devices are already approved for commercial use: Argus II (epiretinal, Second Sight Medical Products Inc., refs6,7) and Alpha-IMS (subretinal, Retina Implant AG, ref.8), which have been shown to restore vision up to a visual acuity of 20/1,2609 and 20/5468, respectively. In addition, PRIMA (subretinal, Pixium Vision, ref.10) has started clinical trials, with others to follow shortly. In combination with stem cell therapy11,12 and optogenetics13, a range of sight restoration options should be available within a decade14.

However, despite the increasing clinical and commercial use of these devices, the perceptual experience of retinal implant users remains poorly understood. For example, even in response to single-electrode stimulation, the appearance of individual phosphenes is highly variable not only across subjects but also across electrodes within a subject, with subjects typically reporting seeing distorted and often elongated geometric shapes that fade quickly over time15,16,17,18,19,20,21,22. Furthermore, linearly combining these ‘building blocks’ of percepts from individual electrodes often fails to predict the combination of percepts evoked when multiple electrodes are stimulated17,23,24,25. Consequently, most subjects cannot determine the orientation of gratings that are used to measure visual acuity, and those who can recognize letters take more than 40 seconds to do so26,27.

Both computational28,29 and in vitro electrophysiological studies17,30,31 suggest that electrode configurations similar to those implanted in patients do not achieve focal activation, but rather produce significant activation of passing axon fibers, which may result in perceptual distortions in patients. Here, we are the first to directly examine whether axonal stimulation contributes to the rich repertoire of phosphene shapes reported by patients. Our computational model can account for the apparent shape of phosphenes elicited by single-electrode stimulation in two generations of the Argus retinal prosthesis system (Second Sight Medical Products Inc.).

Four subjects suffering from severe retinitis pigmentosa (Table 1) were chronically implanted with an epiretinal prosthesis in the macular region of the retina: one subject was implanted with an Argus I device (16 platinum disc electrodes arranged in a 4x4 checkerboard pattern; see Fig. 1A), and three subjects were implanted with Argus II device (60 platinum disc electrodes in a 6x10 arrangement; Fig. 1B). Electrical stimulation was delivered to a number of pre-selected electrodes in random order (five repetitions each) using square-wave, biphasic, cathodic-first pulse trains with fixed stimulus duration, and we asked subjects to outline perceived phosphene shape either on a grid screen (Argus I; Fig. 1B,C) or a computer touch screen (Argus II; Fig. 1E,F) (see Methods). In a control experiment, we confirmed the reliability of each subject’s tracing (Figs S1–S3). We then used a computational model to generate predictions about the apparent shape of the expected visual percepts, and compared the predicted images to patient phosphene drawings. The model assumed that distortions are due to activation of ganglion axon pathways, having estimated the spatial layout of these pathways using traced nerve fiber bundle trajectories extracted from ophthalmic fundus photographs of 55 human eyes32.

Table 1 Subject details. Columns 3–7 indicate the implant site, gender, preoperative visual acuity (VA) categorized as either bare light perception (BLP) or no light perception (NLP), the age at implantation, and the number of years participants had been blind prior to implantation (self-reported).
Full size table
Figure 1
figure 1

Retinal implants used for the drawing task. (A) Argus I electrode array (4 × 4 electrodes of 260 μm and 520 μm diameter arranged in a checkerboard pattern). (B) Argus I subject drawings on a grid screen were captured by an external camera and recorded to a video file. (C) Video files were analyzed offline by tracking the location of the fingertip frame-by-frame and by translating the drawings to a binary image. (D) Argus II electrode array (6 × 10 electrodes of 200 μm diameter). (E) Argus II subject drawings were recorded by a touch screen monitor. (F) Subject drawings were translated to a binary image. Shapes were closed by automatically connecting the first and last tracked fingertip location, after which a floodfill was applied.

Full size image

Results

Phosphene drawings vary across electrodes, but are relatively consistent for a given electrode

All subjects consistently reported seeing phosphenes upon electrical stimulation of the retina. Phosphenes appeared light gray, white, or yellowish in color. However, phosphene drawings varied greatly across subjects and electrodes; representative drawings for each subject are shown in Fig. 2. Whereas stimulation of some electrodes elicited consistent percepts across trials (top row of panels in each subplot), stimulation of other electrodes led to percepts that varied in both size and shape across trials (bottom row of panels in each subplot). Subjects occasionally but rarely reported seeing two distinct shapes (e.g., Subject 1, Electrode D1). Mean images for each electrode were obtained by averaging the drawings across the five stimulation trials, aligned by their centers of mass (column ‘average’). Mean images were then centered over the corresponding electrode in a schematic of the subject's implant to reveal the rich repertoire of elicited percepts across electrodes (large, rightmost panel in each subplot).

Figure 2
figure 2

Phosphene drawing variation within and across electrodes. Drawings from individual trials are shown for the most consistent (top row in each panel) and least consistent electrodes (bottom row in each panel) for Subjects 1–4. Mean images (labeled ‘average’) were obtained by averaging drawings from individual trials aligned at their center of mass. These averaged drawings were then overlaid over the corresponding electrode in a schematic of each subject’s implant (rightmost column).

Full size image

As is evident from these data, only a small number of phosphenes could be described as focal spots of light. Subject 1 drew percepts as either curved or straight lines, wedges, or relatively round spots. Subject 2 drew most percepts as ovals or relatively round spots with only a few curved or straight lines of varying thickness, whereas Subject 3 drew all phosphenes as slightly curved or straight, thin lines. Subject 4 predominantly drew ovals, wedges, and triangles, with only few curved or straight lines.

Interestingly, for Subjects 2–4 the percepts produced by electrodes in the first two rows of the array (i.e., Electrodes A1–F1, A2–F2) were much thinner and longer than for other electrodes33,34. It is possible that these electrodes were the ones that were closest to the retinal surface, since the surgical tack used to attach the implant to the retina was located next to the first row of electrodes. However, we did not have access to optical coherence tomography (OCT) or impedance data, which would have allowed us to estimate electrode-retina distance35,36 (see Discussion).

To quantify the similarity and variability of individual phosphene drawings, we calculated three shape descriptors for each collected drawing: phosphene area, orientation, and elongation (see Methods). These parameter-free metrics were based on a set of statistical quantities known as ‘image moments’; that is, particular weighted averages of pixel intensities across an image (Equation 1). Phosphene orientation and elongation were calculated from the eigenvalues and eigenvectors of each drawing’s covariance matrix (Equations 2–4).

The upper panels of Fig. 3 show distributions of phosphene area (Fig. 3A), orientation (Fig. 3B), and elongation (Fig. 3C) for each subject, across all tested electrodes. The lower panel (Fig. 3D–F) boxplots depicts trial-to-trial variability for each shape descriptor of a given electrode, measured as the standard error of the mean (SEM) calculated across drawings.

Figure 3
figure 3

Phosphene shape analysis. (A–C) Distribution of phosphene area, orientation, and elongation for each subject (Subject 1: 60 drawings, Subject 2: 110 drawings, Subject 3: 90 drawings, Subject 4: 140 drawings). (D–F) Distribution of the variability of shape descriptors for each subject, measured as the standard error of the mean (SEM) across trials for every electrode. Each box extended from the lower to upper quartile values of the data, with a line at the median. Whiskers extended from the fifth to ninety-fifth percentiles, with data points outside that range considered outliers (‘o’). Area SEM for every electrode was normalized by the mean area of all drawings for that particular electrode.

Full size image

To assess whether observed SEM values were smaller than would be predicted from a random sample of phosphene drawings we performed a resampling analysis (1000 iterations). We began by calculating the SEM across all five drawings for each electrode. To assess whether, for an individual subject, drawings were more similar for an individual electrode then across other electrodes in that subjects’ array, resampling was done by randomly sampling (with replacement) phosphene drawings across all the electrodes of that subject. Probability values were estimated by comparing the mean SEM across electrodes of the real distribution to the 1-tailed confidence interval generated by resampling. Detailed results for each shape descriptor are given below.

Phosphene area is more consistent within than across electrodes

Phosphene area was calculated as the number of nonzero pixels in the drawing. Since phosphenes were elicited by directly stimulating the electrodes (see Methods), the phosphenes are at an arbitrary distance from the observer and cannot be described in terms of degrees of visual angle. Although we asked subjects to draw phosphenes ‘as if they appeared at arm’s length’, subjects qualitatively reported that phosphenes could appear as close as ‘in front of their face’ to ‘at arm’s length’. As can be seen by the variability in the boxplots in Fig. 3D, estimates of area varied widely across both subjects and electrodes. However, despite the lack of a reference plane in depth, for all but Subject 1 (marginally significant) the observed SEM values for phosphene area were significantly smaller than SEM values from randomly chosen electrodes (Subject 1: p < 0.06, Subject 2: p < 0.01, Subject 3: p < 0.001, Subject 4: p < 0.001). Subject 1 had particularly small percepts, so areal variability may have been more heavily influenced by drawing error.

Phosphene orientation is more consistent within than across electrodes

Phosphene orientation was calculated as the angle of the principal eigenvector (in the range [−90°, 90°]). For all subjects, mean SEM values for phosphene orientation were significantly smaller than mean SEM values for our bootstrapped null model (Subject 1: p < 0.001, Subject 2: p < 0.001, Subject 3: p < 0.001, Subject 4: p < 0.001), showing that the variation in phosphene orientation within an individual electrode is less than the variation across electrodes (Fig. 3E).

Phosphene elongation is more consistent within than across electrodes

Phosphene elongation was calculated as the relative difference in magnitude of the eigenvalues and normalized to [0, 1], with 0 representing a circle, and 1 representing an infinitesimally thin line. The distribution of elongation values indicates that subjects consistently saw elongated percepts instead of focal spots of light, indeed Subject 3 reported exclusively seeing thin curved and straight lines. These results are in stark contrast to the prevailing assumption in the field that stimulating a single electrode should generate the percept of a small focal spot of light37,38,39,40,41.

Interestingly, phosphene elongation was negatively correlated with phosphene area across subjects (r = −0.411, p < 0.001; see Figure S4); meaning that smaller phosphenes tended to be more elongated than larger ones. This observation also held for each Subject 1 (r = −0.362, p < 0.01), Subject 2 (r = −0.344, p < 0.001), and Subject 4 (r = −0.293, p < 0.001), but not for Subject 3, for whom phosphene elongation was strongly positively correlated with area (r = 0.573, p < 0.001; also see Discussion).

For Subjects 2 and 4, observed SEM values for phosphene elongation were significantly smaller than of our bootstrapped null model (Subject 2: p < 0.001, Subject 4: p < 0.001). For Subjects 1 and 3 results were not significant; percepts were heavily elongated for every electrode, providing little variability in the dataset (Fig. 3F).

Drawing accuracy

Our subjects had been lacking tactile-visual feedback for many years. This motivated a control experiment, where we asked subjects to explore various tactile targets (made of felt with a cardboard background) with their hands, and draw the targets on a touch screen (Figure S1). Drawing errors in the tactile experiments were similar to those in our phosphene drawing experiments (Figure S2).

Phosphene orientation is aligned with retinal nerve fiber bundles

As described above, computational models and electrophysiological evidence from in vitro preparations of rat and rabbit retina suggest that retinal implants may stimulate passing axon fibers17,30,31. Retinal ganglion cells send their axons on highly stereotyped pathways en route to the optic nerve (Fig. 4A–C). Because of this topographic organization, an electrode that stimulates nearby axonal fibers would be expected to antidromically activate cell bodies located peripheral to the point of stimulation. Perceptually, activating an axon fiber that passes under a stimulating electrode is indistinguishable from the percept that would be elicited by activating the corresponding ganglion cell body. The visual percept should appear in the spatial location in visual space for which the corresponding ganglion cell encodes information (receptive field), which could be hundreds of microns away from the stimulation site42. Consequently, percepts elicited from axonal stimulation would be expected to appear elongated along the direction of the underlying nerve bundle trajectory (Fig. 4C).

Figure 4
figure 4

Model of nerve fiber bundle trajectories. (A–C) The topographic organization of optic nerve fiber bundles is highly stereotyped in the human retina (adapted with permission from ref.32). Fundus images from 55 human eyes (A) were superimposed by translation in order to center the foveola (B), followed by rotation and zooming to align the center of the optic disc (C). Electrical stimulation (red circle) of a nerve fiber bundle could antidromically activate ganglion cell bodies peripheral to the point of stimulation (small black circles), leading to percepts that appear elongated along the direction of the underlying nerve bundle trajectory. (D–E) The location and orientation of each subject’s implant (Subject 4 shown here) was estimated by combining their postsurgical fundus photograph (D, bottom) with a baseline presurgical image in which the fovea had been identified (D, top) to produce a registered image (E; □: foveal pit, o- optic disc). The horizontal raphe (D, white line) was approximated by fitting a parabola to the main vascular arcade and finding the tangent to the parabola inflexion point. (F) The extracted landmarks were then used to place a simulated array on a simulated map of nerve fiber bundles.

Full size image

To test whether the orientation of phosphene drawings were aligned with the underlying nerve fiber bundles, we estimated the relative location and orientation of each subject’s implant with respect to the fovea and the optic disc, using ophthalmic fundus photographs (Fig. 4D; here shown for Subject 4). While yellow screening pigments allow for visualization of macular extent in normal eyes, it is difficult to discriminate the macula-periphery boundary in our subjects because of the characteristic pigmentary deposits associated with retinitis pigmentosa1. We therefore had a retina specialist mark the fovea on a fundus image obtained before surgery (Fig. 4D, top), and subsequently used computer vision techniques (see Methods) to align the presurgical image with a second fundus image obtained after surgery (Fig. 4D, bottom), showing the implant relative to the optic nerve head. This allowed us to estimate the array center with respect to the fovea, the array rotation with respect to the horizontal raphe, and the retinal distance between the fovea and the optic nerve head for each subject (Fig. 4E). The resulting topographic measurements were then used to simulate a map of the ganglion axon pathways32 that was tailored to each subject’s retinal dimensions (Fig. 4F).

Remarkably, we found that for all subjects, phosphene orientation was well aligned with the tangent line of the nerve fiber bundle directly below the stimulating electrode (Fig. 5). Here, insets show mean drawings for representative electrodes, from which phosphene orientation was calculated. This was not only true for line phosphenes, which resembled carbon copies of the underlying fiber bundle topography (e.g., Subject 1: D2, Subject 4: F2), but also for more compact phosphenes, which still tended to be elongated in the direction of the local fiber bundle trajectory (e.g., Subject 2: B9, Subject 4: C10). To get a sense of effect size, we also calculated the fraction of the variance observed in phosphene orientation that the simulated map of nerve fiber bundles could explain (Subject 1: 0.463, Subject 2: 0.567, Subject 3: 0.024, Subject 4: 0.394).

Figure 5
figure 5

Phosphene orientations are aligned with retinal nerve fiber bundles. (A–D) Simulated map of nerve fiber bundles for Subjects 1–4 (scale bar: 1 mm, equivalent to 3.6°; shaded box: area used in null models for random array placement). Phosphene orientation is indicated as oriented bars, overlaid over the corresponding electrode in the array. Insets show example percepts; black bars show their corresponding electrodes. Note that the maps are flipped upside down so that the upper image half corresponds to the upper visual field (inferior retina). Box plots indicate the distribution of mean absolute angular errors between phosphene orientation and the tangent line of the ganglion axon pathway nearest to the corresponding electrode. For all subjects, angular errors were significantly better than would be expected from random array placement.

Full size image

To assess whether these angular errors were smaller than would be predicted from a random placement of the array on the retina, we performed a resampling analysis. First, we calculated the mean absolute angle between the five drawings corresponding to a single electrode and the tangent line of the closest nerve fiber bundle. We repeated this procedure for all electrodes in the array to produce a distribution of mean angular errors (box plots in Fig. 5). We then compared the mean angular error of the real distribution to the 1-tailed confidence interval of values from a resampled null model in order to estimate probability values. We considered two different null models: The first null model (NM1) assumed that phosphene orientation was independent of the axon map (all phosphene orientations sampled from a random uniform distribution \(\in [-\,90^\circ ,90^\circ ]\), 1000 iterations). The second, more sophisticated null model (NM2) was generated by randomly placing the array on the retina (array center coordinates: \(x\in [-6000,\,4000]\,{\rm{\mu }}m,y\in [-4000,\,4000]\,{\rm{\mu }}m\), array rotation \(\in [-\,90^\circ ,90^\circ ]\), 1000 iterations). We found that angular errors were significantly smaller than predicted by either null model (NM1: p < 0.001 for Subjects 1 and 2, p < 0.01 for Subjects 3 and 4; NM2: p < 0.05 for all subjects).

Predicting phosphene shape based on a simulated map of ganglion axon pathways

We then tested whether the spatial layout of ganglion axon pathways could account for phosphene shape as well as orientation. We assumed that the activation of an axon elicited a percept centered over the receptive field location of that axon’s cell body. The activation sensitivity of a passing axon fiber was assumed to decay exponentially with retinal distance from the stimulation site, with each subject’s data being fit with two parameters: a decay constant λ, which described activation sensitivity along the axon, and a decay constant ρ, which described sensitivity orthogonal to the axon (see Methods). This allowed us to generate a tissue activation map for each stimulating electrode, which we thresholded to arrive at a binary image that could serve as a prediction of a phosphene drawing (small schematic in the center column of Fig. 6).

Figure 6
figure 6

Phosphene drawings (left columns) contrasted against cross-validated phosphene predictions of the axon map model (center column) and the scoreboard model (right column), overlaid over a schematic of each subject’s implant. Each predicted phosphene is from the test fold of a leave-one-electrode-out cross-validation.

Full size image

Alternatively, we considered a simpler but widely used model that treated each electrode in an array as a ‘pixel’ in an image, thus assuming that stimulating a grid of electrodes on the retina would result in the percept of a grid of isolated, focal spots of light37,38,39,40,41. We refer to this model as the ‘scoreboard model’, because much like the large scoreboards found in sports stadiums, an image is created by an array of individual light sources that can be turned off or on. To implement the scoreboard model, we assumed that an electrode would lead to the percept of a Gaussian blob (with spatial decay constant ρ). The resulting intensity profile was again thresholded to obtain a binary image, which was compared to real phosphene drawings (small schematic in the right column of Fig. 6).

To find the parameter values under each model that best predicted phosphene shape, we constructed a cost function from the difference between predicted and observed phosphene area, orientation, and elongation, which we minimized using particle swarm optimization (see Methods). Because scoreboard and axon map models had a different number of parameters (scoreboard model: ρ; axon map model: ρ, λ), we used leave-one-electrode-out cross-validation to allow for fair model comparison, where we repeatedly fit each model to the drawings from all but one electrode in the array. Fitted parameter values were then used to predict the phosphene shapes of the held-out drawings. Note that a single value of ρ and λ was used to describe the drawings of all electrodes in that subject’s array.

The result of this cross-validation procedure is shown in Fig. 6. Here, ground-truth drawings are shown in the left column, and predicted phosphenes (on the test-fold of the cross-validation procedure) are shown in center and right columns. Thus, predicted phosphene shapes represent each model’s ability to generalize to new electrodes. Whereas the axon map model was able to generate both compact and elongated phosphenes that span a range of geometrical shapes such as ‘blobs’, ‘lines’, and ‘wedges’, the scoreboard model exclusively predicted round phosphenes of various size.

The best fitting, cross-validated parameter values are given in Table 2 (averaged across folds). Even though phosphene shape often varied drastically across electrodes (see Discussion), the axon map model recovered similar values for ρ and λ across different folds for a given subject, as indicated by relatively small SEMs. Without adjusting for drawing bias, these results suggest that electrical stimulation influences ganglion cells whose cell bodies are at a distance of approximately ρ = 437 μm (corresponding to ~1.5° of visual angle) orthogonal to the direction of the axon fiber bundle, but as far as λ = 1,420 μm (corresponding to ~5° of visual angle) along a direction parallel to the axon fiber.

Table 2 Cross-validated model parameter values, averaged across folds ± uncorrected SE.
Full size table

To further quantify model performance, we compared cross-validated prediction errors for phosphene area, orientation, and elongation (Equation 13) across axon map and scoreboard models, Fig. 7. Here, each data point in the scatter plots corresponds to the cross-validated prediction error (averaged across every drawing in that fold) for each electrode. Data points almost always lie below the diagonal, indicating that the axon map model was more accurate than the scoreboard model. Indeed, the axon map model often improved cross-validated log prediction error by an order of magnitude (see insets), simply by adding a single parameter λ that accounted for the current spread along axons of passage in the optic nerve fiber layer of the retina.

Figure 7
figure 7

Comparison of log mean prediction error for the two models. Prediction error was based on the sum of differences between predicted and observed phosphene area, orientation, and elongation (see Equation 13). Each data point in the scatter plots corresponds to the mean cross-validated prediction error of all drawings associated with a particular held-out electrode. Prediction error was significantly higher for the scoreboard model compared to the axon map model (Subject 1: p < 0.001, N = 12; Subject 2: p < 0.001, N = 22; Subject 3: p < 0.001, N = 18; Subject 4: p < 0.001, N = 28; 2-tailed Wilcoxon signed-rank test). Insets in each panel show the histogram of pair-wise differences in log prediction error.

Full size image

Discussion

We show here that the elicited percepts of patients with retinal implants can be accurately predicted using the spatial layout of ganglion axon pathways in the human retina. Model fits to behavioral data suggest that sensitivity to electrical stimulation is not confined to the axon initial segment30, but can be modeled as falling off with different decay constants along (with λ ranging from 500–1,420 μm) and orthogonally from (with ρ ranging from 144–437 μm) the axon, resulting in visual percepts ranging from ‘blobs’ to ‘streaks’ and ‘wedges’ depending on both the relative values of ρ and λ, and the retinal location of the stimulating electrode. These results agree with theoretical work suggesting an anisotropic spread of current in the retinal tissue28 as well as previous animal literature demonstrating that epiretinal stimulation leads to activation of passing axon fibers17,30,31,43, which can severely distort the quality of the generated visual experience15,17,42,44,45. Our findings suggest that the spatial distortions reported by patients are not arbitrary, but rather depend on the topographic organization of optic nerve fiber bundles in each subject’s retina, which can be captured by a computational model. Having an accurate model that generalizes across patients is a crucial first step in developing stimulation strategies for retinal prostheses that can produce complex, perceptually intelligible percepts. Overall our findings argue for more detailed modeling of biological detail across neural engineering applications.

A rich repertoire of phosphene shapes

The phosphenes elicited by single-electrode stimulation vary dramatically across subjects and electrodes (Figs 2, 3A–C), despite relatively small drawing errors and consistency in drawings within a given electrode (Fig. 3D–F). These results are in agreement with the previous literature that has reported that patients subjectively report a variety of percept shapes22,35,46,47, of which only a small fraction could be described as focal spots of light.

The variability in phosphene shape across subjects that we report (captured by variation in λ and ρ across patients), might be due to several factors, a few of which are outlined below. First, diseases such as retinitis pigmentosa and macular degeneration are characterized by a progressive degeneration of photoreceptors, gradually affecting other layers of the retina4,48,49,50. In severe end-stage retinitis pigmentosa, roughly 95% of photoreceptors, 20% of bipolar cells, and 70% of ganglion cells degenerate51, so that little or no useful vision is retained. Disease progression therefore influences the relative proportion of surviving bipolar and ganglion cell types, which in turn is likely to influence phosphene shape.

Second, with a diameter of 200 μm, each electrode in the Argus II array encompasses the equivalent area of hundreds of photoreceptors. A single electrode therefore inevitably leads to activation of a wide variety of morphologically and functionally distinct retinal cells52,53, including simultaneous activation of both ON and OFF pathways. This is in contrast to natural stimulation, which precisely activates a number of specialized, functionally complementary, parallel processing pathways in the retina (for a recent review see ref.54). Although epiretinal stimulation with relatively short pulses might primarily activate ganglion cells rather than bipolar cells55,56,57,58,59, there is still much to be learned about how the information from these different retinal representations are combined at later stages of processing to form a conscious percept.

Third, electrode-retina distance is known to affect both perceptual thresholds35 and phosphene size47. Figure 8 shows simulations based on the assumption that ρ is primarily determined by current spread, whereas λ is primarily determined by axonal stimulation60,61. In these simulations, electrodes close to the retinal surface have a small ρ compared to λ, and are thus highly elongated (Fig. 8A). When electrodes are further from the retinal surface, ρ increases dramatically, thus resulting in circular percepts (Fig. 8D). These simulations agree with the observation that phosphenes from electrodes close to the retinal tack (and thus likely to be close to the retinal surface) appeared much more elongated than others (Fig. 2). Furthermore, low electrode-retina distances could explain why Subject 3 reported predominantly thin and elongated phosphenes (small errors in the tactile target control task argue against drawing bias; see Figures S2–S3). Unfortunately, we did not have access to OCT images or impedance measurements for our subjects, which would have allowed us to infer electrode-retina distances for each electrode35,36. Future studies could use such data to constrain the values of ρ and λ.

Figure 8
figure 8

Simulated phosphenes as a function of electrode-retina distance, z. An electrode located close to the horizontal meridian (dashed line) is chosen. For small values of z, phosphene shape is dominated by axonal stimulation (λ > ρ) thus appearing elongated. Increasing z leads to an increase in ρ but leaves λ unaffected, thus leading to more compact phosphenes (ρ > λ). (A) z = 0 µm, ρ = 300 µm, λ = 500 µm, elongation: 0.977. (B) z = 200 µm, ρ = 500 µm, λ = 500 µm, elongation: 0.957. (C) z = 500 µm, ρ = 800 µm, λ = 500 µm, elongation: 0.867. (D) z = 1,000 µm, ρ = 1,600 µm, λ = 500 µm, elongation: 0.643.

Full size image

Fourth, the mapping of retinal eccentricity to visual field coordinates is nonlinear. Because the foveola contains only photoreceptors, ganglion cell bodies are displaced centrifugally from their cone inputs by several degrees; an effect that extends out as far as 17°62,63.

Finally, phosphene size might be influenced by ganglion cell density and receptive field size. Whereas the receptive field size of retinal ganglion cells only gradually increases with eccentricity64, ganglion cell density decreases rapidly62. Furthermore, retinal degeneration in retinitis pigmentosa tends to progress from the periphery to the macula, thereby having a greater effect on ganglion cell density in the periphery48,51,65. Consequently, more peripheral electrodes would typically stimulate cells with only slightly larger receptive fields, but in much smaller numbers than in the fovea. These two conflicting effects may contribute to our finding of no correlation between phosphene area and retinal eccentricity (data not shown).

Phosphene shape is mediated by axonal stimulation

Despite the variability in phosphene shape, all subjects reported seeing elongated phosphenes on at least a subset of electrodes (Fig. 3C). Although the electric field generated by a disk electrode is radially symmetric, the neural tissue induces anisotropies in the electric field, and stimulation of axon fibers produces even more striking anisotropies in patterns of neural activation within the retina28. It has long been known that external stimulation of an axon induces an action potential that travels both backward to the cell body and forward to the synaptic terminals66,67.

A number of studies have previously hypothesized that axonal stimulation could lead to phosphenes that are elongated in shape and poorly localized (e.g., ref.42). However, this idea has never been explicitly tested. In the present study we demonstrate that axonal stimulation in the retina leads to predictable distortions of shape in human patients, which can be captured by a computational model (Figs 5–7).

Axonal stimulation is a concern for other implant technologies as well. Although subretinal prostheses such as Alpha-IMS8 have electrodes in close proximity to bipolar cells, in vitro animal studies have found that subretinal stimulation with 1 ms pulses nonetheless directly activates retinal ganglion cells at thresholds statistically similar to those of inner retinal cells68,69,70. Similarly, axonal stimulation is expected to be an issue for cortical implants, since passing axons from neurons located in distant parts of the brain have been shown to be highly sensitive to electrical stimulation71,72,73.

Several recent studies have tried to identify stimulation protocols that minimize axonal activation, with mixed results. Whereas one in vitro study suggested using short-duration pulses (≤100 μs) to avoid axonal stimulation74, another study did not see any benefits of short pulses, and instead suggested using long-duration pulses (≥25 ms) or low-frequency (<25 Hz) sinusoidal stimulation31. One difficulty with these approaches is that they are likely to limit stimulation to a highly restricted amplitude and/or frequency range, potentially limiting the dynamic range available for the encoding of brightness46,47.

We show here that percepts are highly consistent over time and can potentially be described using an anatomically detailed computational model with a small number of parameters. Thus, an alternative strategy might be to move away from thinking about artificial sight as a linear combination of ‘pixels’, and instead accept the perceptual distortions resulting from axonal stimulation as the fundamental building blocks of prosthetic vision.

Methods

Subjects

Participants were four blind subjects (one female and three male) with severe retinitis pigmentosa, ranging from 45 to 70 years in age (Table 1). Subjects were chronically implanted with an epiretinal prosthesis as part of an FDA approved clinical trial (clinicaltrials.gov identifier for Subject 1: NCT00279500, registration date 01/17/2006, completed; Subjects 2–4: NCT00407602, registration date 12/01/2006, active). Surgeries were performed at the Doheny Eye Institute at the University of Southern California (Los Angeles, CA; Subject 1), at the Wilmer Eye Institute at Johns Hopkins School of Medicine (Baltimore, MD; Subject 2), at the Moorfields Eye Hospital (London, UK; Subject 3), and at the Royal Eye Hospital (Manchester, UK; Subject 4). None of the subjects had a recordable visual acuity prior to surgery, scoring worse than 2.9 logMAR (worse than 20/15,887) on a four-alternative forced-choice square-wave grating test18,75.

Due to their geographic location, Subjects 2–4 were not directly examined by the authors of this study. Instead, initial experimental procedures were sent to the clinical site, and trained field clinical engineers performed the experiments as specified. Raw collected data was then sent to the authors for subsequent analysis.

All tests were performed after obtaining informed consent under a protocol approved by the Institutional Review Board (IRB) at each subject’s location and under the principles of the Declaration of Helsinki (Subject 1: University of Southern California IRB at the Keck School of Medicine, Subject 2: Johns Hopkins Medicine IRB, Subject 3: Moorfields Eye Hospital IRB, Subject 4: Royal Eye Hospital IRB). Tests were carried out between four and six years after implantation for Subject 1, and between six months and one year after implantation for Subjects 2–4.

Implant specification

Subject 1 was implanted with a 16-channel microelectrode array (Argus I; Second Sight Medical Products, Inc., Sylmar, CA) consisting of 260 and 520 μm diameter platinum disc electrodes, subtending 0.9° and 1.8° of visual angle, respectively. Electrodes were spaced 800 μm apart and arranged in a 4 × 4 alternating checkerboard pattern (Fig. 1A). Subjects 2–4 were implanted with a 60-channel microelectrode array (Argus II; Second Sight Medical Products, Inc., Sylmar, CA) consisting of 200 μm diameter platinum disc electrodes, each subtending 0.7° of visual angle. Electrodes were spaced 525 μm apart and arranged in a 6 × 10 grid (Fig. 1B).

All stimuli described in this study were presented in ‘direct stimulation’ mode. Stimuli were programmed in Matlab using custom software, and pulse train parameters (the electrode(s) to be stimulated, current amplitude, pulse width, individual pulse duration, inter-pulse interval, and pulse train duration) were sent directly to an external visual processing unit (VPU), which was used to send stimulus commands to the internal portion of the implant using an inductive coil link. The implanted receiver wirelessly received these data and sent the signals to the electrode array via a small cable.

In day-to-day use, an external unit consisting of a small camera and transmitter mounted on a pair of glasses is worn by the user. The camera captured video and sends the information to the VPU which converts it into pulse trains using pre-specified image processing techniques (‘camera’ mode).

Psychophysical methods

Perceptual thresholds for individual electrodes were measured using an adaptive yes/no procedure implemented using custom software (see Supplemental Information). All presented stimuli were charge-balanced, square-wave, biphasic, cathodic-first pulse trains with fixed stimulus duration (Argus I: 500 ms, Argus II: 250 ms), current amplitude (set at 2x threshold), stimulus frequency (20 Hz) and pulse duration (0.45 ms/phase, no interphase delay).

Subjects were asked to perform a drawing task with a tactile target (Supplemental Information) or when their retina was electrically stimulated (Fig. 1). In a given experimental run, a total of n stimulus conditions (either tactile or retinal stimulation) were tested. Each condition was repeated for m trials (for a total of mn trials per experimental run). Repeated trials of the same condition were randomized amongst other stimuli to confirm reproducibility of results.

Head movement of the Argus I subject was minimized with a chin rest. After each stimulus presentation, the subject traced the shape on a grid screen (containing 6 inch horizontal and vertical grid lines) with a center location aligned horizontally and vertically with the subject’s head. Drawing was carried out with a pen whose cap was a different color than its body. A head-mounted camera (Misumi CMOS S588-3T), located on the subject’s glasses, was used to record the trials to digital video recorder (DVR). Video files were analyzed off-line to extract shape data using custom built tracking software. In the first stage of processing, the entire image was rotated appropriately using the grid screen background as a reference. In the second stage, vertical and horizontal gridlines, and the distance from the subject to screen were used to set a new coordinate system in visual angle coordinates (since the subject was 16 inches/40.6 cm from the screen, 4 gridlines = 70.0 cm corresponded to 73.8 degrees visual angle). In the third stage, the location of the pen cap was tracked (based on its color) across each frame of the video file. Finally, a binary shape data file was built from pen cap coordinate locations across all frames.

Argus II subjects were placed in a chair at a comfortable distance from a touch screen monitor with its center location aligned horizontally with the subject’s head. The distance from each subject’s eyes to the screen was recorded. After each stimulus presentation, the subject traced the shape on the monitor and the experimenter advanced to the next trial. Touch screen data were instantly recorded by custom software in 2D coordinates to a text file. Text files were analyzed offline to translate vector coordinates to a binary shape data file. The distance recorded from the subject to screen was used to set a new coordinate system in visual angle. Since Subjects 2–4 were 33, 30.0, and 30.5 inches from the screen, this corresponded to a display size of 60, 65 and 64 degrees of visual angle (horizontal screen length), respectively. After translating to the final visual angle coordinate system, the binary image was used in subsequent shape analyses.

Computational methods

Phosphene shape descriptors

Phosphene shape was quantified using three parameter-free shape descriptors commonly used in image processing: area, orientation, and elongation76. (Elongation is sometimes also referred to as eccentricity in the literature. We avoid that usage here to prevent confusion with retinal eccentricity). These descriptors are based on a set of statistical quantities known as ‘image moments’. For an X × Y pixel grayscale image, I(x,y), where \(x\in [1,\,X]\) and \(y\in [1,\,Y]\), the raw image moments Mij were calculated as:

$${M}_{ij}=\sum _{x}\sum _{y}{x}^{i}{y}^{j}I(x,y).$$
(1)

Raw image moments were used to compute area (A = M00) and the center of mass (\(\bar{x},\bar{y})=({M}_{10}/{M}_{00},{M}_{01}/{M}_{00}\)) of each phosphene.

Phosphene orientation was calculated from the covariance matrix of an image:

$${\rm{cov}}[I(x,y)]=[\begin{array}{cc}{\mu }_{20}^{\text{'}} & {\mu }_{11}^{\text{'}}\\ {\mu }_{11}^{\text{'}} & {\mu }_{02}^{\text{'}}\end{array}],$$
(2)

where \({\mu }_{20}^{\text{'}}={M}_{20}/{M}_{00}-{\bar{x}}^{2}\), \({\mu }_{11}^{\text{'}}={M}_{11}/{M}_{00}-\bar{x}\bar{y}\), and \({\mu }_{02}^{\text{'}}={M}_{02}/{M}_{00}-{\bar{y}}^{2}\). The eigenvectors of this matrix corresponded to the major and minor axes of the image intensity. Orientation (θ) could thus be extracted from the angle of the eigenvector associated with the largest eigenvalue towards the axis closest to this eigenvector:

$$\theta =\frac{1}{2}\arctan (\frac{2{\mu }_{11}^{\text{'}}}{{\mu }_{20}^{\text{'}}-{\mu }_{02}^{\text{'}}}),$$
(3)

which was valid as long as \({\mu }_{20}^{\text{'}}\ne {\mu }_{02}^{\text{'}}\), with \(\theta \in \{-\pi /2,\pi /2\}\). To avoid division by zero, we manually assigned an angle of θ = 0 whenever \({\mu }_{20}^{\text{'}}\) was equal to \({\mu }_{02}^{\text{'}}\).

Phosphene elongation (E) was calculated from the eigenvectors of the covariance matrix of Equation (2):

$$E=\sqrt{1-\frac{{\lambda }_{2}}{{\lambda }_{1}}},$$
(4)

where \({\lambda }_{1,2}=({\mu ^{\prime} }_{20}+{\mu ^{\prime} }_{02})/2\pm \sqrt{4{\mu ^{\prime} }_{11}^{2}+{({\mu ^{\prime} }_{20}-{\mu ^{\prime} }_{02})}^{2}}/2\), and \(E\in [0,1]\). An elongation of E = 1 represents a circle, and E = 0 represents an infinitesimally thin line.

Determination of implant location using fundus photography

Implant location was estimated by analyzing color fundus photographs obtained using systems available at each clinical site. For each subject, we performed the following procedure:

  1. 1.

    Extract landmarks: On a baseline fundus photograph (before surgery), a retina specialist marked the foveal pit and the center of the optic nerve head. On the most recent fundus photograph (after surgery), we marked the center of the implant.

  2. 2.

    Combine baseline image with implant image: We performed image registration using feature matching to bring the two images into the same coordinate system.

  3. 3.

    Adjust for magnification: Pixel distances were converted to retinal distances by using the known electrode-electrode spacing (Argus I: 800 μm, Argus II: 525 μm).

  4. 4.

    Adjust for rotation: We approximated the horizontal raphe by fitting a parabola to the main vascular arcade, assuming that the center of the optic nerve head lay at the vertex of the parabola, and that the raphe was parallel to the parabola’s axis of symmetry77,78.

  5. 5.

    Coordinate transform: The registered image was rotated so that the horizontal raphe came to lie on the abscissa, and the foveal pit at the origin of the new coordinate system. We located the coordinates of the center of the optic nerve head as well as the center of the array (from Step 1) in this new coordinate system. Retinal distances (μm) were related to visual space (deg) using a formula that computes the relationship between retinal arc lengths and visual angles from based on the optic axis63.

  6. 6.

    The extracted landmarks were then used to place a simulated array on a simulated map of ganglion axon pathways using the pulse2percept software60.

This procedure allowed us to estimate each subject’s array location and orientation with respect to the fovea (Table 3). Based on fundus photographs of 104 sighted humans79, the center of the optic disc was expected to be located at 15.5° ± 1.1° nasal, 1.5° ± 0.9° superior with respect to the fovea. For all four subjects, the estimated center of the optic disk was within two standard deviations of these expected values.

Table 3 Estimated locations of the implant and optic disc with respect to the fovea located at (0, 0) using fundus photography.
Full size table

Scoreboard model

The scoreboard model assumed that electrical stimulation led to the percept of focal dots of light, centered over the visual field location associated with the stimulated retinal field location (xstim, ystim), whose spatial intensity profile decayed with a Gaussian profile40,41:

$${I}_{{\rm{score}}}(x,y;\rho )=\exp (-\frac{{(x-{x}_{{\rm{stim}}})}^{2}+{(y-{y}_{{\rm{stim}}})}^{2}}{2{\rho }^{2}}),$$
(5)

where ρ was the spatial decay constant.

The resulting intensity profile \({I}_{{\rm{score}}}(x,y;\rho )\) was then thresholded to obtain a binary image. The threshold was chosen as \(1/\sqrt{e}\), such that points closer than ρ to (xstim, ystim) were assigned a value of 1, and all other points were assigned a value of 0.

Axon map model

Following Jansonius et al.32, we assumed that the trajectories of the optic nerve fibers could be described in a modified polar coordinate system (r,ϕ) with its origin located in the center of the optic disc. A nerve fiber was modeled as a spiral:

$$\varphi (r,{\varphi }_{0})={\varphi }_{0}+b(r,{\varphi }_{0}){(r-{r}_{0})}^{c(r,{\varphi }_{0})},$$
(6)

where \({\varphi }_{0}=\varphi (r={r}_{0})\) is the angular position of the trajectory at its starting point at a circle with radius r0 around the center of the optic disc, b a real number describing the curvature of the spiral,

$$b({\varphi }_{0},r)=\{\begin{array}{c}\exp (-1.9+3.9\,\tanh (\frac{-({\varphi }_{0}-121)}{14})),r\ge 0\\ -\exp (0.7+1.5\,\tanh (\frac{-(-{\varphi }_{0}-90)}{25})),r < 0,\end{array}$$
(7)

and c a positive real number describing the location of the point of maximal curvature,

$$c({\varphi }_{0},r)=\{\begin{array}{c}1.9+1.4\,\tanh (\frac{{\varphi }_{0}-121}{14})\,,r\ge 0\\ 1.0+0.5\,\tanh (\frac{-{\varphi }_{0}-90}{25}),r < 0.\end{array}$$
(8)

Jansonius and colleagues determined parameter values by fitting Equations (6–8) to the topographical layout of 55 eyes from 55 human subjects for details see ref.32. The attentive reader might notice that Equation (8) above fixes a typo in Equation (3) of ref.32: The tanh numerator should indeed read ϕ0 − 121, not −ϕ0 − 121.

To apply the axon map to the eyes of our subjects, we first transformed the original coordinate system (r, ϕ) to Cartesian coordinates (x, y) with the foveal pit located at (0, 0), and then set the coordinates of the optic disc (xod, yod) to the values estimated from fundus photography (Table 3). The resulting axon maps for each subject can be seen in Fig. 5.

An axon’s sensitivity to electrical stimulation was assumed to decay exponentially with distance from the soma \(({x}_{{\rm{soma}}},\,{y}_{{\rm{soma}}})\):

$${I}_{{\rm{axon}}}(x,y;\rho ,\lambda )={I}_{{\rm{score}}}(x,y;\rho )\exp (-\frac{{(x-{x}_{{\rm{soma}}})}^{2}+{(y-{y}_{{\rm{soma}}})}^{2}}{2{\lambda }^{2}}),$$
(9)

where λ was the spatial decay constant along the axon. \({I}_{{\rm{score}}}(x,y;\rho )\) is the same as in Equation (5) and is parameterized by a single parameter, ρ. As in the scoreboard model, the resulting intensity profile \({I}_{{\rm{axon}}}(x,y;\rho ,\lambda )\) was thresholded to obtain a binary image.

Model fitting and evaluation

To fit the scoreboard and axon map models to subject drawings, we first calculated the coefficient of determination (R2) from the predicted binary images and the corresponding ground-truth subject drawings. R2 was calculated from the ratio of the residual sum of squares (SSres) and the total sum of squares (SStot) for each shape descriptor (area, orientation, or elongation):

$${R}^{2}=1-\frac{S{S}_{{\rm{res}}}}{S{S}_{{\rm{tot}}}},$$
(10)
$$S{S}_{res}=\sum _{i}{({s}_{i}-{\hat{s}}_{i})}^{2},$$
(11)
$$S{S}_{tot}=\sum _{i}{({s}_{i}-s)}^{2},$$
(12)

where si was the shape descriptor for the i-th ground-truth image, \({\hat{s}}_{i}\) was the shape descriptor for the i-th predicted image, and \(\bar{s}\) was the mean of the shape descriptor averaged over all images. The three quantities \({R}_{{\rm{area}}}^{2}\), \({R}_{{\rm{orientation}}}^{2}\), and \({R}_{{\rm{elongation}}}^{2}\) resulting from this procedure were then combined to construct a cost function that could be iteratively minimized:

$$c=\sum _{d}1-{R}_{d}^{2},$$
(13)

where d = {area, orientation, elongation}. Due to the nonconvexity of this optimization problem, we minimized the cost function using particle swarm optimization80. We set the swarm size at ten times the number of parameters81. We ran every fitting procedure five times with different, randomly chosen initial conditions, and then chose the best run in subsequent analyses.

To allow for fair performance comparison despite the scoreboard and axon map models having different numbers of parameters, we implemented a leave-one-electrode-out cross-validation procedure, where we repeatedly fit each model to the drawings from all but one electrode in the array. This is equivalent to calculating the Akaike Information Criterion that takes into account the difference in number of parameters82. The fitted parameter values were then used to predict the shape descriptors of the held-out drawings (Fig. 7). Note that a single value of ρ and λ were fitted for each subject, and then used for all electrodes in that subject’s array.