Brain–computer interface (BCI) neurotechnology has the potential to reduce disability associated with paralysis by translating neural activity into control of assistive devices1,2,3,4,5,6,7,8,9. Surveys of potential end-users have identified key BCI system features10,11,12,13,14, including high accuracy, minimal daily setup, rapid response times, and multifunctionality. These performance characteristics are primarily influenced by the BCI’s neural decoding algorithm1,15, which is trained to associate neural activation patterns with intended user actions. Here, we introduce a new deep neural network16 decoding framework for BCI systems enabling discrete movements that addresses these four key performance characteristics. Using intracortical data from a participant with tetraplegia, we provide offline results demonstrating that our decoder is highly accurate, sustains this performance beyond a year without explicit daily retraining by combining it with an unsupervised updating procedure3,17,18,19,20, responds faster than competing methods8, and can increase functionality with minimal retraining by using a technique known as transfer learning21. We then show that our participant can use the decoder in real-time to reanimate his paralyzed forearm with functional electrical stimulation (FES), enabling accurate manipulation of three objects from the grasp and release test (GRT)22. These results demonstrate that deep neural network decoders can advance the clinical translation of BCI technology.
Subscribe to Journal
Get full journal access for 1 year
only $18.75 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Data used in this study can be made available to qualified individuals for collaboration provided that a written agreement is executed in advance between Battelle Memorial Institute and the requester’s affiliated institution. Such inquiries or requests should be directed to G.S.
Lebedev, M. A. & Nicolelis, M. A. L. Brain–machine interfaces: from basic science to neuroprostheses and neurorehabilitation. Physiol. Rev. 97, 767–837 (2017).
Chaudhary, U., Birbaumer, N. & Ramos-Murguialday, A. Brain–computer interfaces for communication and rehabilitation. Nat. Rev. Neurol. 12, 513–525 (2016).
Jarosiewicz, B. et al. Virtual typing by people with tetraplegia using a self-calibrating intracortical brain–computer interface. Sci. Transl. Med. 7, 313ra179 (2015).
Hochberg, L. R. et al. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485, 372–375 (2012).
Simeral, J. D., Kim, S.-P., Black, M. J., Donoghue, J. P. & Hochberg, L. R. Neural control of cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical microelectrode array. J. Neural. Eng. 8, 025027 (2011).
Collinger, J. L. et al. High-performance neuroprosthetic control by an individual with tetraplegia. Lancet 381, 557–564 (2013).
Gilja, V. et al. Clinical translation of a high-performance neural prosthesis. Nat. Med. 21, 1142–1145 (2015).
Bouton, C. E. et al. Restoring cortical control of functional movement in a human with quadriplegia. Nature 533, 247–250 (2016).
Ajiboye, A. B. et al. Restoration of reaching and grasping movements through brain-controlled muscle stimulation in a person with tetraplegia: a proof-of-concept demonstration. Lancet 389, 1821–1830 (2017).
Snoek, G. J., IJzerman, M. J., Hermens, H. J., Maxwell, D. & Biering-Sorensen, F. Survey of the needs of patients with spinal cord injury: impact and priority for improvement in hand function in tetraplegics. Spinal Cord 42, 526–532 (2004).
Anderson, K. D. Targeting recovery: priorities of the spinal cord-injured population. J. Neurotrauma 21, 1371–1383 (2004).
Collinger, J. L. et al. Functional priorities, assistive technology, and brain-computer interfaces after spinal cord injury. J. Rehabil. Res. Dev. 50, 145–160 (2013).
Huggins, J. E., Moinuddin, A. A., Chiodo, A. E. & Wren, P. A. What would brain-computer interface users want: opinions and priorities of potential users with spinal cord injury. Arch. Phys. Med. Rehabil. 96, S38–S45.e5 (2015).
Huggins, J. E., Wren, P. A. & Gruis, K. L. What would brain-computer interface users want? Opinions and priorities of potential users with amyotrophic lateral sclerosis. Amyotroph. Lateral. Scler. 12, 318–324 (2011).
Kao, J. C., Stavisky, S. D., Sussillo, D., Nuyujukian, P. & Shenoy, K. V. Information systems opportunities in brain-machine interface decoders. Proc. IEEE Ins. Electr. Electron. Eng. 102, 666–682 (2014).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Jarosiewicz, B. et al. Retrospectively supervised click decoder calibration for self-calibrating point-and-click brain-computer interfaces. J. Physiol. Paris 110, 382–391 (2016).
Bishop, W. et al. Self-recalibrating classifiers for intracortical brain–computer interfaces. J. Neural. Eng. 11, 026001 (2014).
Bacher, D. et al. Neural point-and-click communication by a person with incomplete locked-in syndrome. Neurorehabil. Neural. Repair. 29, 462–471 (2015).
Rosenberg, C., Hebert, M. & Schneiderman, H. Semi-supervised self-training of object detection models. in Seventh IEEE Workshop Appl. Comput. Vis. (IEEE, Piscataway, NJ, USA, 2005).
Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? in Advances in Neural Information Processing Systems 27 (eds. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D. & Weinberger, K. Q.) 3320–3328 (Curran Associates, Inc., Red Hook, NY, USA, 2014).
Wuolle, K. S., Van Doren, C. L., Thrope, G. B., Keith, M. W. & Peckham, P. H. Development of a quantitative hand grasp and release test for patients with tetraplegia using a hand neuroprosthesis. J. Hand Surg. Am. 19, 209–218 (1994).
Sharma, G. et al. Using an artificial neural bypass to restore cortical control of rhythmic movements in a human with quadriplegia. Sci. Rep. 6, 33807 (2016).
Friedenberg, D. A. et al. Neuroprosthetic-enabled control of graded arm muscle contraction in a paralyzed human. Sci. Rep. 7, 8386 (2017).
Fernández-Delgado, M., Cernadas, E., Barro, S. & Amorim, D. Do we need hundreds of classifiers to solve real world classification problems? J. Mach. Learn. Res. 15, 3133–3181 (2014).
Colachis, S. C. et al. Dexterous control of seven functional hand movements using cortically-controlled transcutaneous muscle stimulation in a person with tetraplegia. Front. Neurosci. 12, 208 (2018).
Perge, J. A. et al. Intra-day signal instabilities affect decoding performance in an intracortical neural interface system. J. Neural. Eng. 10, 036004 (2013).
Aflalo, T. et al. Neurophysiology. Decoding motor imagery from the posterior parietal cortex of a tetraplegic human. Science 348, 906–910 (2015).
Pandarinath, C. et al. High performance communication by people with paralysis using an intracortical brain-computer interface. eLife 6, e18554 (2017).
Jarosiewicz, B. et al. Advantages of closed-loop calibration in intracortical brain–computer interfaces for people with tetraplegia. J. Neural. Eng. 10, 046012 (2013).
Nuyujukian, P. et al. Performance sustaining intracortical neural prostheses. J. Neural. Eng. 11, 066003 (2014).
Li, Z., O’Doherty, J. E., Lebedev, M. A. & Nicolelis, M. A. L. Adaptive decoding for brain–machine interfaces through Bayesian parameter updates. Neural Comput. 23, 3162–3204 (2011).
Sussillo, D., Stavisky, S. D., Kao, J. C., Ryu, S. I. & Shenoy, K. V. Making brain–machine interfaces robust to future neural variability. Nat. Commun. 7, 13749 (2016).
Flint, R. D., Scheid, M. R., Wright, Z. A., Solla, S. A. & Slutzky, M. W. Long-term stability of motor cortical activity: implications for brain machine interfaces and optimal feedback control. J. Neurosci. 36, 3623–3632 (2016).
Glaser, J. I. et al. Machine learning for neural decoding. Preprint at https://arxiv.org/abs/1708.00909 (2017).
Evans, N., Gale, S., Schurger, A. & Blanke, O. Visual feedback dominates the sense of agency for brain-machine actions. PLoS ONE 10, e0130019 (2015).
Wodlinger, B. et al. Ten-dimensional anthropomorphic arm control in a human brain-machine interface: difficulties, solutions, and limitations. J. Neural. Eng. 12, 016011 (2015).
Wang, F. et al. Enhancing clinical communication assessments using an audiovisual BCI for patients with disorders of consciousness. J. Neural. Eng. 14, 046024 (2017).
Pfurtscheller, G., Guger, C., Müller, G., Krausz, G. & Neuper, C. Brain oscillations control hand orthosis in a tetraplegic. Neurosci. Lett. 292, 211–214 (2000).
Friedenberg, D. A. et al. Big data challenges in decoding cortical activity in a human with quadriplegia to inform a brain computer interface. in 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 3084–3087 (IEEE, Piscataway, NJ, USA, 2016).
Sharma, G. et al. Time stability and coherence analysis of multiunit, single-unit and local field potential neuronal signals in chronically implanted brain electrodes. Bioelectron. Med. 2, 63–71 (2015).
Abadi, M. et al. TensorFlow: large-scale machine learning on heterogeneous distributed systems. Preprint at https://arxiv.org/abs/1605.08695 (2016).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).
Ruder, S. An overview of gradient descent optimization algorithms. Preprint at https://arxiv.org/abs/1609.04747 (2016).
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014).
Reed, S. et al. Training deep neural networks on noisy labels with bootstrapping. Preprint at https://arxiv.org/abs/1412.6596 (2014).
Cleveland, W. S. & Devlin, S. J. Locally weighted regression: an approach to regression analysis by local fitting. J. Am. Stat. Assoc. 83, 596–610 (1988).
Ajemian, R. Neurosurgery: gentler alternatives to chips in the brain. Nature 544, 416 (2017).
Peckham, P. H., Mortimer, J. T. & Marsolais, E. B. Controlled prehension and release in the C5 quadriplegic elicited by functional electrical stimulation of the paralyzed forearm musculature. Ann. Biomed. Eng. 8, 369–388 (1980).
Taylor, P., Esnouf, J. & Hobby, J. The functional impact of the Freehand System on tetraplegic hand function. Clinical Results. Spinal Cord 40, 560–566 (2002).
Müller-Putz, G. R., Scherer, R., Pfurtscheller, G. & Rupp, R. EEG-based neuroprosthesis control: a step towards clinical practice. Neurosci. Lett. 382, 169–174 (2005).
Friedenberg, D. A. & Schwemmer, M. A. Moving a paralyzed hand—a biomedical big data success story. Chance 29, 4–13 (2016).
We thank the study participant and his family for their dedication and support. We thank D. Weber and H. Bresler for assistance in editing the manuscript; M. Zhang, S. Colachis, H. Trivedi, A. Singh, and P. Ganzer for their assistance with the sessions, data collection, and help editing the manuscript; and R. Kittel for help with figure formatting. Financial support for this study came from Battelle Memorial Institute and The Ohio State University Neurological Institute and Department of Physical Medicine & Rehabilitation. M.A.B. also acknowledges the invaluable mentorship of the Rehabilitation Medicine Scientist Training Program at the Association of Academic Physiatrists. M.A.S. would also like to thank his brother E. Schwemmer for being a source of inspiration for this work.
The authors declare competing interests, as they are employed by institutions that provided the funding for this work and/or have filed associated patents. M.A.S., N.D.S., J.E.T., D.A.F., and G.S. are all employed by Battelle Memorial Institute and M.A.B. is employed by the Ohio State University. P.B.S. was also employed by the Ohio State University at the time of this study. D.A.F. and G.S. are listed as inventors on the United States patent application US 2018/0178008 (related WO 2016/196797), and G.S. is listed as an inventor on the United States patent application US 2015/0306373. These are related to the neural bridging BCI technology and stimulation sleeve used in the GRT experiment in the paper.
Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Figures 1–8 and Supplementary Tables 1 and 2
Example cued block in which the participant is in control of the BCI-FES system and performing the three functional grips plus hand open
Example ‘free-time’ block in which the participant is in control of the BCI-FES system and performing the three functional grips plus hand open
Example grasp and release test block (GRT, see Methods)
About this article
Cite this article
Schwemmer, M.A., Skomrock, N.D., Sederberg, P.B. et al. Meeting brain–computer interface user performance expectations using a deep neural network decoding framework. Nat Med 24, 1669–1676 (2018). https://doi.org/10.1038/s41591-018-0171-y
Computational and Mathematical Methods in Medicine (2020)
Intra-cortical brain-machine interfaces for controlling upper-limb powered muscle and robotic systems in spinal cord injury
Clinical Neurology and Neurosurgery (2020)
IEEE Transactions on Neural Systems and Rehabilitation Engineering (2020)
Frontiers in Artificial Intelligence (2020)