Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Technical Report
  • Published:

A high-performance neural prosthesis enabled by control algorithm design

Nature Neuroscience volume 15, pages 1752–1757 (2012)Cite this article

Subjects

Abstract

Neural prostheses translate neural activity from the brain into control signals for guiding prosthetic devices, such as computer cursors and robotic limbs, and thus offer individuals with disabilities greater interaction with the world. However, relatively low performance remains a critical barrier to successful clinical translation; current neural prostheses are considerably slower, with less accurate control, than the native arm. Here we present a new control algorithm, the recalibrated feedback intention–trained Kalman filter (ReFIT-KF) that incorporates assumptions about the nature of closed-loop neural prosthetic control. When tested in rhesus monkeys implanted with motor cortical electrode arrays, the ReFIT-KF algorithm outperformed existing neural prosthetic algorithms in all measured domains and halved target acquisition time. This control algorithm permits sustained, uninterrupted use for hours and generalizes to more challenging tasks without retraining. Using this algorithm, we demonstrate repeatable high performance for years after implantation in two monkeys, thereby increasing the clinical viability of neural prostheses.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Cursor control with native arm, ReFIT-KF and Velocity-KF.
Figure 2: Performance of ReFIT-KF control across four years.
Figure 3: Performance comparison of native arm versus ReFIT-KF for the pinball task.
Figure 4: Performance comparison of native arm versus ReFIT-KF for the obstacle-avoidance task.
Figure 5: Illustrations of the online neural control paradigm and the ReFIT-KF training methodology.

Similar content being viewed by others

References

  1. Serruya, M.D., Hatsopoulos, N.G., Paninski, L., Fellows, M.R. & Donoghue, J.P. Instant neural control of a movement signal. Nature 416, 141–142 (2002).

    Article  CAS  Google Scholar 

  2. Taylor, D.M., Tillery, S.I. & Schwartz, A.B. Direct cortical control of 3D neuroprosthetic devices. Science 296, 1829–1832 (2002).

    Article  CAS  Google Scholar 

  3. Carmena, J.M. et al. Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biol. 1, E42 (2003).

    Article  Google Scholar 

  4. Velliste, M., Perel, S., Spalding, M.C., Whitford, A.S. & Schwartz, A.B. Cortical control of a prosthetic arm for self-feeding. Nature 453, 1098–1101 (2008).

    Article  CAS  Google Scholar 

  5. Mulliken, G.H., Musallam, S. & Andersen, R.A. Decoding trajectories from posterior parietal cortex ensembles. J. Neurosci. 28, 12913–12926 (2008).

    Article  CAS  Google Scholar 

  6. Shpigelman, L., Lalazar, H. & Vaadia, E. Kernel-arma for hand tracking and brain-machine interfacing during 3D motor control. Adv. Neural Info. Proc. Sys. 21, 1489–1496 (2009).

    Google Scholar 

  7. Ganguly, K. & Carmena, J.M. Emergence of a stable cortical map for neuroprosthetic control. PLoS Biol. 7, e1000153 (2009).

    Article  Google Scholar 

  8. Suminski, A.J., Tkach, D.C., Fagg, A.H. & Hatsopoulos, N.G. Incorporating feedback from multiple sensory modalities enhances brain-machine interface control. J. Neurosci. 30, 16777–16787 (2010).

    Article  CAS  Google Scholar 

  9. Li, Z., O'Doherty, J.E., Lebedev, M.A. & Nicolelis, M.A. Adaptive decoding for brain-machine interfaces through bayesian parameter updates. Neural Comput. 23, 3162–3204 (2011).

    Article  Google Scholar 

  10. Hochberg, L.R. et al. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171 (2006).

    Article  CAS  Google Scholar 

  11. Kim, S.P., Simeral, J.D., Hochberg, L.R., Donoghue, J.P. & Black, M.J. Neural control of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia. J. Neural Eng. 5, 455–476 (2008).

    Article  Google Scholar 

  12. Kim, S.P. et al. Point-and-click cursor control with an intracortical neural interface system in humans with tetraplegia. IEEE Trans. Neural Syst. Rehabil. Eng. 19, 193–203 (2011).

    Article  Google Scholar 

  13. Hochberg, L.R. et al. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485, 372–375 (2012).

    Article  CAS  Google Scholar 

  14. Judy, J. Neural interfaces for upper-limb prosthesis control: opportunities to improve long-term reliability. IEEE Pulse 3, 57–60 (2012).

    Article  Google Scholar 

  15. Chestek, C.A. et al. Long-term stability of neural prosthetic control signals from silicon cortical arrays in rhesus macaque motor cortex. J. Neural Eng. 8, 045005 (2011).

    Article  Google Scholar 

  16. Cunningham, J.P. et al. A closed-loop human simulator for investigating the role of feedback-control in brain-machine interfaces. J. Neurophysiol. 105, 1932–1949 (2011).

    Article  Google Scholar 

  17. Churchland, M.M., Cunningham, J.P., Kaufman, M.T., Ryu, S.I. & Shenoy, K.V. Cortical preparatory activity: representation of movement or first cog in a dynamical machine? Neuron 68, 387–400 (2010).

    Article  CAS  Google Scholar 

  18. Kaufman, M.T. et al. Roles of monkey premotor neuron classes in movement preparation and execution. J. Neurophysiol. 104, 799–810 (2010).

    Article  Google Scholar 

  19. Sadtler, P.T., Ryu, S.I., Yu, B.M., & Batista, A.P. High-performance neural prosthetic control along instructed paths. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2011, 601–604 (2011).

    Google Scholar 

  20. Gage, G.J., Ludwig, K.A., Otto, K.J., Ionides, E.L. & Kipke, D.R. Naive coadaptive cortical control. J. Neural Eng. 2, 52–63 (2005).

    Article  Google Scholar 

  21. Fraser, G.W., Chase, S.M., Whitford, A. & Schwartz, A.B. Control of a brain-computer interface without spike sorting. J. Neural Eng. 6, 055004 (2009).

    Article  Google Scholar 

  22. Chase, S.M., Schwartz, A.B., & Kass, R.E. Bias, optimal linear estimation and the differences between open-loop simulation and closed-loop performance of spiking-based brain-computer interface algorithms. Neural Networks 22, 1203–1213 (2009).

    Article  Google Scholar 

  23. Srinivasan, L., Eden, U.T., Mitter, S.K. & Brown, E.N. General-purpose filter design for neural prosthetic devices. J. Neurophysiol. 98, 2456–2475 (2007).

    Article  Google Scholar 

  24. Yu, B.M. et al. Mixture of trajectory models for neural decoding of goal-directed movements. J. Neurophysiol. 97, 3763–3780 (2007).

    Article  Google Scholar 

  25. Card, S.K., English, W.K. & Burr, B.J. Evaluation of mouse, rate-controlled isometric joystick, step keys, and text keys for text selection on a CRT. Ergonomics 21, 601–613 (1978).

    Article  Google Scholar 

  26. Kalaska, J.F. From intention to action: motor cortex and the control of reaching movements. Adv. Exp. Med. Biol. 629, 139–178 (2009).

    Article  Google Scholar 

  27. Golub, M.D., Yu, B.M. & Chase, S.M. Internal models engaged by brain-computer interface control. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2012, 1327–1330 (2012).

    PubMed  PubMed Central  Google Scholar 

  28. Chestek, C.A. et al. Neural prosthetic systems: current problems and future directions. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2009, 3369–3375 (2009).

    PubMed  Google Scholar 

  29. Santhanam, G. et al. HermesB: a continuous neural recording system for freely behaving primates. IEEE Trans. Biomed. Eng. 54, 2037–2050 (2007).

    Article  Google Scholar 

  30. Hochberg, L.R. Turning thought into action. N. Engl. J. Med. 359, 1175–1177 (2008).

    Article  CAS  Google Scholar 

  31. McFarland, D.J., Sarnacki, W.A. & Wolpaw, J.R. Electroencephalographic (EEG) control of three-dimensional movement. J. Neural Eng. 7, 036007 (2010).

    Article  Google Scholar 

  32. Schalk, G. et al. Two-dimensional movement control using electrocorticographic signals in humans. J. Neural Eng. 5, 75–84 (2008).

    Article  CAS  Google Scholar 

  33. Gilja, V. et al. Challenges and opportunities for next-generation intra-cortically based neural prostheses. IEEE Trans. Biomed. Eng. 58, 1891–1899 (2011).

    Article  Google Scholar 

  34. Santhanam, G., Ryu, S.I., Yu, B.M., Afshar, A. & Shenoy, K.V. A high-performance brain-computer interface. Nature 442, 195–198 (2006).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We thank M. Mazariegos, J. Aguayo, W. Kalkus, S. Kang, E. Morgan and C. Sherman for surgical assistance and veterinary care, D. Haven and B. Oskotsky for information technology support, S. Eisensee, B. Davis and E. Castaneda for administrative assistance, P. Ortega for mathematical insight, and S. Stavisky for data-collection assistance. This work was supported by a US National Defense Science and Engineering Graduate Fellowship (V.G.), National Science Foundation Graduate Research Fellowships (V.G., C.A.C., J.M.F., M.T.K. and J.C.K.), Stanford Medical Scholars Program, Howard Hughes Medical Institute Medical Research Fellows Program, Paul and Daisy Soros Fellowship, Stanford Medical Scientist Training Program (P.N.), Stanford Graduate Fellowship (C.A.C., J.P.C. and J.M.F.), Gatsby Charitable Foundation (B.M.Y.), a Helen Hay Whitney postdoctoral fellowship (M.M.C.), Burroughs Welcome Fund Career Awards in the Biomedical Sciences (M.M.C. and K.V.S.), the Christopher and Dana Reeve Paralysis Foundation (S.I.R. and K.V.S.), Defense Advanced Research Projects Agency Revolutionizing Prosthetics 2009 N66001-06-C-8005, Reorganization and Plasticity to Accelerate Injury Recovery N66001-10-C-2010, US National Institutes of Health, National Institute of Neurological Disorders and Stroke Collaborative Research in Computational Neuroscience grant R01-NS054283, and National Institutes of Health Directors Pioneer Award 1DP1OD006409 (K.V.S.).

Author information

Author notes

  1. Vikash Gilja and Paul Nuyujukian: These authors contributed equally to this work.

Authors and Affiliations

  1. Department of Computer Science, Stanford University, Stanford, California, USA

    Vikash Gilja

  2. Stanford Institute for Neuro-Innovation and Translational Neuroscience, Stanford University, Stanford, California, USA

    Vikash Gilja, Cindy A Chestek & Krishna V Shenoy

  3. Department of Bioengineering, Stanford University, Stanford, California, USA

    Paul Nuyujukian, Joline M Fan & Krishna V Shenoy

  4. School of Medicine, Stanford University, Stanford, California, USA

    Paul Nuyujukian

  5. Department of Electrical Engineering, Stanford University, Stanford, California, USA

    Cindy A Chestek, John P Cunningham, Byron M Yu, Mark M Churchland, Jonathan C Kao, Stephen I Ryu & Krishna V Shenoy

  6. Department of Engineering, University of Cambridge, Cambridge, UK

    John P Cunningham

  7. Neurosciences Program, Stanford University, Stanford, California, USA

    Byron M Yu, Mark M Churchland, Matthew T Kaufman & Krishna V Shenoy

  8. Gatsby Computational Neuroscience Unit, University College London, London, UK

    Byron M Yu

  9. Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA

    Byron M Yu

  10. Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA

    Byron M Yu

  11. Department of Neurosurgery, Palo Alto Medical Foundation, Palo Alto, California, USA

    Stephen I Ryu

  12. Department of Neurobiology, Stanford University, Stanford, California, USA

    Krishna V Shenoy

Authors
  1. Vikash Gilja
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul Nuyujukian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cindy A Chestek
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. John P Cunningham
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Byron M Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Joline M Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Mark M Churchland
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Matthew T Kaufman
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jonathan C Kao
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Stephen I Ryu
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Krishna V Shenoy
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

V.G. and P.N. were responsible for infrastructure development, animal training, and data collection and analysis. V.G. was responsible for algorithm design, supplementary modeling and writing of the paper. P.N. and B.M.Y. participated in algorithm design. C.A.C. participated in animal training, infrastructure development and data analysis. J.P.C. participated in algorithm design and infrastructure development. J.M.F. participated in data collection and analysis. M.M.C. and M.T.K. provided initial animal training for the obstacle avoidance task. J.C.K. participated in data collection. S.I.R. was responsible for surgical implantation. K.V.S. was involved in all aspects of the study.

Corresponding author

Correspondence to Vikash Gilja.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–9, Supplementary Tables 1–2, Supplementary Modeling (PDF 3646 kb)

Supplementary Video 1

Center-out-and-back task (monkey L 1/29/2010). In this task, targets alternate between a central target and one of eight peripheral targets. Peripheral targets are evenly spaced around a ring 8 cm from the central target. Although targets and the cursor are presented as 1.4-cm-diameter spheres, the target acceptance window is a 4 cm by 4 cm square. A trial is successful and a reward is given if the cursor center is held within this acceptance window for 500 ms. The monkey has 2 s to acquire each target. This video was selected as a representative example of performance in this task. The mean time to target for the subset of trials shown in the video is 678 ms. The video was recorded during the experimental session from a monitor mirroring the monitor shown to the monkey. (MP4 2488 kb)

Supplementary Video 2

Center-out-and-back task (monkey J 3/09/2010). This video shows the same task as in Supplementary Video 1, but for the other monkey (J). Task parameters are identical except for the visually presented cursor, and the target diameter is 1.2 cm. This video was selected as a representative example of performance in this task. The mean time to target for the subset of trials shown in the video is 671 ms. The video was recorded during the experimental session from a monitor mirroring the monitor shown to the monkey. (MP4 2486 kb)

Supplementary Video 3

Pinball task (monkey L 4/12/2010). In this task, targets can appear anywhere within a 16 cm by 16 cm workspace. The center of the cursor must be held within the 4 cm by 4 cm acceptance window for 500 ms. The cursor and targets are 1.2 cm in diameter, and the monkey has 2 s to acquire each target. Data from this 100-min neural control session are presented in Figure 3. The video is a reconstruction of the graphics displayed during online control sessions, built from data logged by the behavioral control system. (MP4 101903 kb)

Supplementary Video 4

Pinball task (monkey L 4/12/2010). The video shows 20 seconds of trials taken every 10 min during the 100-min neural control session in Supplementary Video 3 (that is, 0–20 s, 10 m + 0–20 s, 20 m + 0–20 s and so on). The video is a reconstruction of the graphics displayed during online control sessions, built from data logged by the behavioral control system. (MP4 8412 kb)

Supplementary Video 5

Pinball task (monkey J 2/4/2010). This video shows the same task as Supplementary Videos 3 and 4 for the second monkey. Task parameters are identical except the monkey had 3 s to acquire each target; cursor and targets are 1.4 cm in diameter. The video is a reconstruction of the graphics displayed during online control sessions, built from data logged by the behavioral control system. (MP4 82402 kb)

Supplementary Video 6

Maze task (monkey J 3/9/2010). In this obstacle-avoidance task, a magenta barrier appears on screen and turns on at the same time as the target. The monkey must acquire and hold the target (4 cm by 4 cm acceptance window) for 500 ms without hitting the barrier. If the cursor hits the barrier, the trial is aborted. The monkey has 5 s to acquire each target. Cursor and targets are 1.2 cm in diameter. As noted in the main text, this task was challenging for the monkey in both native arm control and neural control modes, with comparable task success rates. Unlike the trials shown in the previous five videos, here the trials selected are not representative. The trials, and associated stretch of video, were selected to show a string of successful trials. The video was recorded during the experimental session from a monitor mirroring the monitor shown to the monkey. (MP4 3689 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gilja, V., Nuyujukian, P., Chestek, C. et al. A high-performance neural prosthesis enabled by control algorithm design. Nat Neurosci 15, 1752–1757 (2012). https://doi.org/10.1038/nn.3265

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn.3265

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing