Letter | Published:

Reach and grasp by people with tetraplegia using a neurally controlled robotic arm

Nature volume 485, pages 372375 (17 May 2012) | Download Citation

Abstract

Paralysis following spinal cord injury, brainstem stroke, amyotrophic lateral sclerosis and other disorders can disconnect the brain from the body, eliminating the ability to perform volitional movements. A neural interface system1,2,3,4,5 could restore mobility and independence for people with paralysis by translating neuronal activity directly into control signals for assistive devices. We have previously shown that people with long-standing tetraplegia can use a neural interface system to move and click a computer cursor and to control physical devices6,7,8. Able-bodied monkeys have used a neural interface system to control a robotic arm9, but it is unknown whether people with profound upper extremity paralysis or limb loss could use cortical neuronal ensemble signals to direct useful arm actions. Here we demonstrate the ability of two people with long-standing tetraplegia to use neural interface system-based control of a robotic arm to perform three-dimensional reach and grasp movements. Participants controlled the arm and hand over a broad space without explicit training, using signals decoded from a small, local population of motor cortex (MI) neurons recorded from a 96-channel microelectrode array. One of the study participants, implanted with the sensor 5 years earlier, also used a robotic arm to drink coffee from a bottle. Although robotic reach and grasp actions were not as fast or accurate as those of an able-bodied person, our results demonstrate the feasibility for people with tetraplegia, years after injury to the central nervous system, to recreate useful multidimensional control of complex devices directly from a small sample of neural signals.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    Bridging the brain to the world: a perspective on neural interface systems. Neuron 60, 511–521 (2008)

  2. 2.

    et al. Challenges and opportunities for next-generation intra-cortically based neural prostheses. IEEE Trans. Biomed. Eng. 58, 1891–1899 (2011)

  3. 3.

    , , & Brain-controlled interfaces: movement restoration with neural prosthetics. Neuron 52, 205–220 (2006)

  4. 4.

    & Principles of neural ensemble physiology underlying the operation of brain-machine interfaces. Nature Rev. Neurosci. 10, 530–540 (2009)

  5. 5.

    & Learning to move machines with the mind. Trends Neurosci. 34, 61–75 (2011)

  6. 6.

    et al. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171 (2006)

  7. 7.

    , , , & Neural control of cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical microelectrode array. J. Neural Eng. 8, 025027 (2011)

  8. 8.

    et al. Point-and-click cursor control with an intracortical neural interface system by humans with tetraplegia. IEEE Trans. Neural Syst. Rehabil. Eng. 19, 193–203 (2011)

  9. 9.

    , , , & Cortical control of a prosthetic arm for self-feeding. Nature 453, 1098–1101 (2008)

  10. 10.

    et al. The DLR lightweight robot: design and control concepts for robots in human environments. Ind. Rob. 34, 376–385 (2007)

  11. 11.

    Research update: VA study to optimize the DEKA Arm. J. Rehabil. Res. Dev. 47, ix–x (2010)

  12. 12.

    , , , & Bayesian population decoding of motor cortical activity using a Kalman filter. Neural Comput. 18, 80–118 (2006)

  13. 13.

    , , , & Reliability of signals from a chronically implanted, silicon-based electrode array in non-human primate primary motor cortex. IEEE Trans. Neural Syst. Rehabil. Eng. 13, 524–541 (2005)

  14. 14.

    et al. Long-term stability of neural prosthetic control signals from silicon cortical arrays in rhesus macaque motor cortex. J. Neural. Eng. 8, 045005 (2011)

  15. 15.

    , , & Seven years of recording from monkey cortex with a chronically implanted multiple microelectrode. Front. Neuroeng. 3, 6 (2010)

  16. 16.

    , , , & Neural control of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia. J. Neural Eng. 5, 455–476 (2008)

  17. 17.

    , , , & in Proc. ICORR ’97: Int. Conf. Rehabilitation Robotics 83–86 (Bath Institute of Medical Engineering, 1997)

  18. 18.

    , , , & Neural decoding of finger movements using Skellam-based maximum-likelihood decoding. IEEE Trans. Biomed. Eng. 57, 754–760 (2010)

  19. 19.

    et al. Decoding complete reach and grasp actions from local primary motor cortex populations. J. Neurosci. 30, 9659–9669 (2010)

  20. 20.

    et al. Inference of hand movements from local field potentials in monkey motor cortex. Nat. Neurosci. 6, 1253–1254 (2003)

  21. 21.

    & Predicting movement from multiunit activity. J. Neurosci. 27, 8387–8394 (2007)

  22. 22.

    , , & Relationships among low-frequency local field potentials, spiking activity, and three-dimensional reach and grasp kinematics in primary motor and ventral premotor cortices. J. Neurophysiol. 105, 1603–1619 (2011)

  23. 23.

    , , , & Cognitive control signals for neural prosthetics. Science 305, 258–262 (2004)

  24. 24.

    , & Decoding trajectories from posterior parietal cortex ensembles. J. Neurosci. 28, 12913–12926 (2008)

  25. 25.

    , , , & A high-performance brain–computer interface. Nature 442, 195–198 (2006)

  26. 26.

    , & Direct control of paralysed muscles by cortical neurons. Nature 456, 639–642 (2008)

  27. 27.

    et al. Toward the restoration of hand use to a paralyzed monkey: brain-controlled functional electrical stimulation of forearm muscles. PLoS One 4, e5924 (2009)

  28. 28.

    et al. Continuous neuronal ensemble control of simulated arm reaching by a human with tetraplegia. J. Neural Eng. 8, 034003 (2011)

  29. 29.

    et al. Targeted reinnervation for enhanced prosthetic arm function in a woman with a proximal amputation: a case study. Lancet 369, 371–380 (2007)

  30. 30.

    , , & Control of a brain–computer interface without spike sorting. J. Neural Eng. 6, 055004 (2009)

  31. 31.

    et al. Multi-state decoding of point-and-click control signals from motor cortical activity in a human with tetraplegia. 3rd Int. IEEE/EMBS Conf. Neural Eng. 486–489 (2007).

  32. 32.

    et al. Soft robotics: from torque feedback controlled light-weight robots to intrinsically compliant systems. Robot. Automat. Mag. 15, 20–30 (2008)

  33. 33.

    et al. Multisensory five-finger dexterous hand: The DLR/HIT Hand II. IEEE/RSJ Int. Conf. Intell. Robots Systems 3692–3697 (2008).

  34. 34.

    , & Requirements for safe robots: measurements, analysis and new insights. Int. J. Robot. Res. 28, 1507–1527 (2009)

  35. 35.

    What is the real shape of extracellular spikes? J. Neurosci. Methods 177, 194–198 (2009)

  36. 36.

    , , & Efficient decoding with steady-state Kalman filter in neural interface systems. IEEE Trans. Neural Syst. Rehabil. Eng. 19, 25–34 (2011)

  37. 37.

    , & Direct cortical control of 3D neuroprosthetic devices. Science 296, 1829–1832 (2002)

  38. 38.

    et al. Functional network reorganization during learning in a brain-computer interface paradigm. Proc. Natl Acad. Sci. USA 105, 19486–19491 (2008)

Download references

Acknowledgements

We thank participants S3 and T2 for their dedication to this research. We thank M. Black for initial guidance in the BrainGate–DLR research. We thank E. Gallivan, E. Berhanu, D. Rosler, L. Barefoot, K. Centrella and B. King for their contributions to this research. We thank G. Friehs and E. Eskandar for their surgical contributions. We thank K. Knoper for assistance with illustrations. We thank D. Van Der Merwe and DEKA Research and Development for their technical support. The contents do not represent the views of the Department of Veterans Affairs or the United States Government. The research was supported by the Rehabilitation Research and Development Service, Office of Research and Development, Department of Veterans Affairs (Merit Review Awards B6453R and A6779I; Career Development Transition Award B6310N). Support was also provided by the National Institutes of Health: NINDS/NICHD (RC1HD063931), NIDCD (R01DC009899), NICHD-NCMRR (N01HD53403 and N01HD10018), NIBIB (R01EB007401), NINDS-Javits (NS25074); a Memorandum of Agreement between the Defense Advanced Research Projects Agency (DARPA) and the Department of Veterans Affairs; the Doris Duke Charitable Foundation; the MGH-Deane Institute for Integrated Research on Atrial Fibrillation and Stroke; Katie Samson Foundation; Craig H. Neilsen Foundation; the European Commission’s Seventh Framework Programme through the project The Hand Embodied (grant 248587). The pilot clinical trial into which participant S3 was recruited was sponsored in part by Cyberkinetics Neurotechnology Systems (CKI).

Author information

Author notes

    • Daniel Bacher
    • , Beata Jarosiewicz
    • , Nicolas Y. Masse
    • , John D. Simeral
    •  & Joern Vogel

    These authors contributed equally to this work.

Affiliations

  1. Rehabilitation Research & Development Service, Department of Veterans Affairs, Providence, Rhode Island 02908, USA

    • Leigh R. Hochberg
    • , Beata Jarosiewicz
    • , John D. Simeral
    • , Jie Liu
    •  & John P. Donoghue
  2. School of Engineering and Institute for Brain Science, Brown University, Providence, Rhode Island 02912, USA

    • Leigh R. Hochberg
    • , Daniel Bacher
    • , John D. Simeral
    • , Jie Liu
    •  & John P. Donoghue
  3. Department of Neurology, Massachusetts General Hospital, Boston, Massachusetts 02114, USA

    • Leigh R. Hochberg
    • , John D. Simeral
    •  & Sydney S. Cash
  4. Harvard Medical School, Boston, Massachusetts 02115, USA

    • Leigh R. Hochberg
    •  & Sydney S. Cash
  5. Department of Neuroscience and Institute for Brain Science, Brown University, Providence, Rhode Island 02912, USA

    • Beata Jarosiewicz
    • , Nicolas Y. Masse
    •  & John P. Donoghue
  6. German Aerospace Center, Institute of Robotics and Mechatronics (DLR, Oberpfaffenhofen) 82230, Germany

    • Joern Vogel
    • , Sami Haddadin
    •  & Patrick van der Smagt

Authors

  1. Search for Leigh R. Hochberg in:

  2. Search for Daniel Bacher in:

  3. Search for Beata Jarosiewicz in:

  4. Search for Nicolas Y. Masse in:

  5. Search for John D. Simeral in:

  6. Search for Joern Vogel in:

  7. Search for Sami Haddadin in:

  8. Search for Jie Liu in:

  9. Search for Sydney S. Cash in:

  10. Search for Patrick van der Smagt in:

  11. Search for John P. Donoghue in:

Contributions

J.P.D. and L.R.H. conceived, planned and directed the BrainGate research and the DEKA sessions. J.P.D., L.R.H. and P.v.d.S. conceived, planned and directed the DLR robot control sessions. J.P.D. and P.v.d.S. are co-senior authors. D.B., B.J., N.Y.M., J.D.S. and J.V. contributed equally to this work and are listed alphabetically. J.D.S., J.V. and D.B. developed the BrainGate–DLR interface. D.B., J.D.S. and J.L. developed the BrainGate–DEKA interface. D.B. and J.V. created the three-dimensional motorized target placement system. B.J., N.Y.M. and D.B. designed the behavioural task, the neural signal processing approach, the filter building approach and the performance metrics. B.J., N.Y.M. and D.B. performed data analysis, further guided by L.R.H., J.D.S. and J.P.D. N.Y.M., L.R.H. and J.P.D. drafted the manuscript, which was further edited by all authors. D.B. and J.D.S. engineered the BrainGate neural interface system/assistive technology system. J.V. and S.H. developed the reactive planner for the Light-Weight Robot III (LWR). S.H. developed the internal control framework of the Light-Weight Robot III. The internal control framework of the DEKA arm was developed by DEKA. L.R.H. is principal investigator of the pilot clinical trial. S.S.C. is clinical co-investigator of the pilot clinical trial and assisted in the clinical oversight of these participants.

Competing interests

J.P.D. is a former chief scientific officer and director of CKI; he held stocks and received compensation. L.R.H. received research support from Massachusetts General and Spaulding Rehabilitation Hospitals, which in turn received clinical trial support from CKI. J.D.S. received compensation as a consultant for CKI. CKI ceased operations in 2009, before the collection of data reported in this manuscript. The BrainGate pilot clinical trials are now administered by Massachusetts General Hospital.

Corresponding authors

Correspondence to Leigh R. Hochberg or John P. Donoghue.

Supplementary information

PDF files

  1. 1.

    Supplementary Information

    This file contains Supplementary Figures 1-9 and Supplementary Table 1 providing additional results, a Supplementary Discussion regarding multidimensional robotic and prosthetic limb control and neural signals, Supplementary Movie Legends demonstrating neural control of reach and grasp by people with tetraplegia and Supplementary References.

Videos

  1. 1.

    Supplementary Movie 1

    This file contains Supplementary Movie 1 which shows neuronal ensemble control of the DLR robot arm and hand for three-dimensional reach and grasp by a woman with tetraplegia, trial day 1959.

  2. 2.

    Supplementary Movie 2

    This file contains Supplementary Movie 2 which shows neuronal ensemble control of the DEKA prosthetic arm and hand by a woman with tetraplegia, trial day 1974.

  3. 3.

    Supplementary Movie 3

    This file contains Supplementary Movie 3 which shows neuronal ensemble control of the DEKA prosthetic arm and hand by a gentleman with tetraplegia (Participant T2), trial day 166.

  4. 4.

    Supplementary Movie 4

    This file contains Supplementary Movie 4 which shows BrainGate-enabled drinking from a bottle by S3, using neurally-controlled 2-D movement and hand state control of the DLR robot arm, trial day 1959.

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nature11076

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.