Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Letter
  • Published:

Meeting brain–computer interface user performance expectations using a deep neural network decoding framework

Nature Medicine volume 24, pages 1669–1676 (2018)Cite this article

Subjects

Abstract

Brain–computer interface (BCI) neurotechnology has the potential to reduce disability associated with paralysis by translating neural activity into control of assistive devices1,2,3,4,5,6,7,8,9. Surveys of potential end-users have identified key BCI system features10,11,12,13,14, including high accuracy, minimal daily setup, rapid response times, and multifunctionality. These performance characteristics are primarily influenced by the BCI’s neural decoding algorithm1,15, which is trained to associate neural activation patterns with intended user actions. Here, we introduce a new deep neural network16 decoding framework for BCI systems enabling discrete movements that addresses these four key performance characteristics. Using intracortical data from a participant with tetraplegia, we provide offline results demonstrating that our decoder is highly accurate, sustains this performance beyond a year without explicit daily retraining by combining it with an unsupervised updating procedure3,17,18,19,20, responds faster than competing methods8, and can increase functionality with minimal retraining by using a technique known as transfer learning21. We then show that our participant can use the decoder in real-time to reanimate his paralyzed forearm with functional electrical stimulation (FES), enabling accurate manipulation of three objects from the grasp and release test (GRT)22. These results demonstrate that deep neural network decoders can advance the clinical translation of BCI technology.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Experimental set-up, data processing steps, and NN architecture.
Fig. 2: Year-long high-fidelity decoding of movement intentions with NNs.
Fig. 3: Translating gains in NN accuracy to system usability and increasing the number of available functions with transfer learning.
Fig. 4: Real-time control of functional electrical stimulation.

Similar content being viewed by others

Data availability

Data used in this study can be made available to qualified individuals for collaboration provided that a written agreement is executed in advance between Battelle Memorial Institute and the requester’s affiliated institution. Such inquiries or requests should be directed to G.S.

References

  1. Lebedev, M. A. & Nicolelis, M. A. L. Brain–machine interfaces: from basic science to neuroprostheses and neurorehabilitation. Physiol. Rev. 97, 767–837 (2017).

    Article  Google Scholar 

  2. Chaudhary, U., Birbaumer, N. & Ramos-Murguialday, A. Brain–computer interfaces for communication and rehabilitation. Nat. Rev. Neurol. 12, 513–525 (2016).

    Article  Google Scholar 

  3. Jarosiewicz, B. et al. Virtual typing by people with tetraplegia using a self-calibrating intracortical brain–computer interface. Sci. Transl. Med. 7, 313ra179 (2015).

    Article  Google Scholar 

  4. Hochberg, L. R. et al. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485, 372–375 (2012).

    Article  CAS  Google Scholar 

  5. Simeral, J. D., Kim, S.-P., Black, M. J., Donoghue, J. P. & Hochberg, L. R. Neural control of cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical microelectrode array. J. Neural. Eng. 8, 025027 (2011).

    Article  CAS  Google Scholar 

  6. Collinger, J. L. et al. High-performance neuroprosthetic control by an individual with tetraplegia. Lancet 381, 557–564 (2013).

    Article  Google Scholar 

  7. Gilja, V. et al. Clinical translation of a high-performance neural prosthesis. Nat. Med. 21, 1142–1145 (2015).

    Article  CAS  Google Scholar 

  8. Bouton, C. E. et al. Restoring cortical control of functional movement in a human with quadriplegia. Nature 533, 247–250 (2016).

    Article  CAS  Google Scholar 

  9. Ajiboye, A. B. et al. Restoration of reaching and grasping movements through brain-controlled muscle stimulation in a person with tetraplegia: a proof-of-concept demonstration. Lancet 389, 1821–1830 (2017).

    Article  Google Scholar 

  10. Snoek, G. J., IJzerman, M. J., Hermens, H. J., Maxwell, D. & Biering-Sorensen, F. Survey of the needs of patients with spinal cord injury: impact and priority for improvement in hand function in tetraplegics. Spinal Cord 42, 526–532 (2004).

    Article  CAS  Google Scholar 

  11. Anderson, K. D. Targeting recovery: priorities of the spinal cord-injured population. J. Neurotrauma 21, 1371–1383 (2004).

    Article  Google Scholar 

  12. Collinger, J. L. et al. Functional priorities, assistive technology, and brain-computer interfaces after spinal cord injury. J. Rehabil. Res. Dev. 50, 145–160 (2013).

    Article  Google Scholar 

  13. Huggins, J. E., Moinuddin, A. A., Chiodo, A. E. & Wren, P. A. What would brain-computer interface users want: opinions and priorities of potential users with spinal cord injury. Arch. Phys. Med. Rehabil. 96, S38–S45.e5 (2015).

    Article  Google Scholar 

  14. Huggins, J. E., Wren, P. A. & Gruis, K. L. What would brain-computer interface users want? Opinions and priorities of potential users with amyotrophic lateral sclerosis. Amyotroph. Lateral. Scler. 12, 318–324 (2011).

    Article  Google Scholar 

  15. Kao, J. C., Stavisky, S. D., Sussillo, D., Nuyujukian, P. & Shenoy, K. V. Information systems opportunities in brain-machine interface decoders. Proc. IEEE Ins. Electr. Electron. Eng. 102, 666–682 (2014).

    Article  Google Scholar 

  16. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  CAS  Google Scholar 

  17. Jarosiewicz, B. et al. Retrospectively supervised click decoder calibration for self-calibrating point-and-click brain-computer interfaces. J. Physiol. Paris 110, 382–391 (2016).

    Article  Google Scholar 

  18. Bishop, W. et al. Self-recalibrating classifiers for intracortical brain–computer interfaces. J. Neural. Eng. 11, 026001 (2014).

    Article  Google Scholar 

  19. Bacher, D. et al. Neural point-and-click communication by a person with incomplete locked-in syndrome. Neurorehabil. Neural. Repair. 29, 462–471 (2015).

    Article  Google Scholar 

  20. Rosenberg, C., Hebert, M. & Schneiderman, H. Semi-supervised self-training of object detection models. in Seventh IEEE Workshop Appl. Comput. Vis. (IEEE, Piscataway, NJ, USA, 2005).

  21. Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? in Advances in Neural Information Processing Systems 27 (eds. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D. & Weinberger, K. Q.) 3320–3328 (Curran Associates, Inc., Red Hook, NY, USA, 2014).

  22. Wuolle, K. S., Van Doren, C. L., Thrope, G. B., Keith, M. W. & Peckham, P. H. Development of a quantitative hand grasp and release test for patients with tetraplegia using a hand neuroprosthesis. J. Hand Surg. Am. 19, 209–218 (1994).

    Article  CAS  Google Scholar 

  23. Sharma, G. et al. Using an artificial neural bypass to restore cortical control of rhythmic movements in a human with quadriplegia. Sci. Rep. 6, 33807 (2016).

    Article  CAS  Google Scholar 

  24. Friedenberg, D. A. et al. Neuroprosthetic-enabled control of graded arm muscle contraction in a paralyzed human. Sci. Rep. 7, 8386 (2017).

    Article  Google Scholar 

  25. Fernández-Delgado, M., Cernadas, E., Barro, S. & Amorim, D. Do we need hundreds of classifiers to solve real world classification problems? J. Mach. Learn. Res. 15, 3133–3181 (2014).

    Google Scholar 

  26. Colachis, S. C. et al. Dexterous control of seven functional hand movements using cortically-controlled transcutaneous muscle stimulation in a person with tetraplegia. Front. Neurosci. 12, 208 (2018).

    Article  Google Scholar 

  27. Perge, J. A. et al. Intra-day signal instabilities affect decoding performance in an intracortical neural interface system. J. Neural. Eng. 10, 036004 (2013).

    Article  Google Scholar 

  28. Aflalo, T. et al. Neurophysiology. Decoding motor imagery from the posterior parietal cortex of a tetraplegic human. Science 348, 906–910 (2015).

    Article  CAS  Google Scholar 

  29. Pandarinath, C. et al. High performance communication by people with paralysis using an intracortical brain-computer interface. eLife 6, e18554 (2017).

    Article  Google Scholar 

  30. Jarosiewicz, B. et al. Advantages of closed-loop calibration in intracortical brain–computer interfaces for people with tetraplegia. J. Neural. Eng. 10, 046012 (2013).

    Article  Google Scholar 

  31. Nuyujukian, P. et al. Performance sustaining intracortical neural prostheses. J. Neural. Eng. 11, 066003 (2014).

    Article  Google Scholar 

  32. Li, Z., O’Doherty, J. E., Lebedev, M. A. & Nicolelis, M. A. L. Adaptive decoding for brain–machine interfaces through Bayesian parameter updates. Neural Comput. 23, 3162–3204 (2011).

    Article  Google Scholar 

  33. Sussillo, D., Stavisky, S. D., Kao, J. C., Ryu, S. I. & Shenoy, K. V. Making brain–machine interfaces robust to future neural variability. Nat. Commun. 7, 13749 (2016).

    Article  CAS  Google Scholar 

  34. Flint, R. D., Scheid, M. R., Wright, Z. A., Solla, S. A. & Slutzky, M. W. Long-term stability of motor cortical activity: implications for brain machine interfaces and optimal feedback control. J. Neurosci. 36, 3623–3632 (2016).

    Article  CAS  Google Scholar 

  35. Glaser, J. I. et al. Machine learning for neural decoding. Preprint at https://arxiv.org/abs/1708.00909 (2017).

  36. Evans, N., Gale, S., Schurger, A. & Blanke, O. Visual feedback dominates the sense of agency for brain-machine actions. PLoS ONE 10, e0130019 (2015).

    Article  Google Scholar 

  37. Wodlinger, B. et al. Ten-dimensional anthropomorphic arm control in a human brain-machine interface: difficulties, solutions, and limitations. J. Neural. Eng. 12, 016011 (2015).

    Article  CAS  Google Scholar 

  38. Wang, F. et al. Enhancing clinical communication assessments using an audiovisual BCI for patients with disorders of consciousness. J. Neural. Eng. 14, 046024 (2017).

    Article  Google Scholar 

  39. Pfurtscheller, G., Guger, C., Müller, G., Krausz, G. & Neuper, C. Brain oscillations control hand orthosis in a tetraplegic. Neurosci. Lett. 292, 211–214 (2000).

    Article  CAS  Google Scholar 

  40. Friedenberg, D. A. et al. Big data challenges in decoding cortical activity in a human with quadriplegia to inform a brain computer interface. in 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 3084–3087 (IEEE, Piscataway, NJ, USA, 2016).

  41. Sharma, G. et al. Time stability and coherence analysis of multiunit, single-unit and local field potential neuronal signals in chronically implanted brain electrodes. Bioelectron. Med. 2, 63–71 (2015).

    Article  Google Scholar 

  42. Abadi, M. et al. TensorFlow: large-scale machine learning on heterogeneous distributed systems. Preprint at https://arxiv.org/abs/1605.08695 (2016).

  43. Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).

    Article  CAS  Google Scholar 

  44. Ruder, S. An overview of gradient descent optimization algorithms. Preprint at https://arxiv.org/abs/1609.04747 (2016).

  45. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014).

    Google Scholar 

  46. Reed, S. et al. Training deep neural networks on noisy labels with bootstrapping. Preprint at https://arxiv.org/abs/1412.6596 (2014).

  47. Cleveland, W. S. & Devlin, S. J. Locally weighted regression: an approach to regression analysis by local fitting. J. Am. Stat. Assoc. 83, 596–610 (1988).

    Article  Google Scholar 

  48. Ajemian, R. Neurosurgery: gentler alternatives to chips in the brain. Nature 544, 416 (2017).

    Article  CAS  Google Scholar 

  49. Peckham, P. H., Mortimer, J. T. & Marsolais, E. B. Controlled prehension and release in the C5 quadriplegic elicited by functional electrical stimulation of the paralyzed forearm musculature. Ann. Biomed. Eng. 8, 369–388 (1980).

    Article  CAS  Google Scholar 

  50. Taylor, P., Esnouf, J. & Hobby, J. The functional impact of the Freehand System on tetraplegic hand function. Clinical Results. Spinal Cord 40, 560–566 (2002).

    Article  CAS  Google Scholar 

  51. Müller-Putz, G. R., Scherer, R., Pfurtscheller, G. & Rupp, R. EEG-based neuroprosthesis control: a step towards clinical practice. Neurosci. Lett. 382, 169–174 (2005).

    Article  Google Scholar 

  52. Friedenberg, D. A. & Schwemmer, M. A. Moving a paralyzed hand—a biomedical big data success story. Chance 29, 4–13 (2016).

    Article  Google Scholar 

Download references

Acknowledgements

We thank the study participant and his family for their dedication and support. We thank D. Weber and H. Bresler for assistance in editing the manuscript; M. Zhang, S. Colachis, H. Trivedi, A. Singh, and P. Ganzer for their assistance with the sessions, data collection, and help editing the manuscript; and R. Kittel for help with figure formatting. Financial support for this study came from Battelle Memorial Institute and The Ohio State University Neurological Institute and Department of Physical Medicine & Rehabilitation. M.A.B. also acknowledges the invaluable mentorship of the Rehabilitation Medicine Scientist Training Program at the Association of Academic Physiatrists. M.A.S. would also like to thank his brother E. Schwemmer for being a source of inspiration for this work.

Author information

Author notes

  1. These authors contributed equally: Marcia A. Bockbrader, David A. Friedenberg.

Authors and Affiliations

  1. Advanced Analytics, Battelle Memorial Institute, Columbus, OH, USA

    Michael A. Schwemmer, Nicholas D. Skomrock & David A. Friedenberg

  2. Department of Psychology, University of Virginia, Charlottesville, VA, USA

    Per B. Sederberg

  3. Medical Devices and Neuromodulation, Battelle Memorial Institute, Columbus, OH, USA

    Jordyn E. Ting & Gaurav Sharma

  4. Neurological Institute, Ohio State University, Columbus, OH, USA

    Marcia A. Bockbrader

  5. Department of Physical Medicine and Rehabilitation, Ohio State University, Columbus, OH, USA

    Marcia A. Bockbrader

Authors
  1. Michael A. Schwemmer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicholas D. Skomrock
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Per B. Sederberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jordyn E. Ting
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gaurav Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marcia A. Bockbrader
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. David A. Friedenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

D.A.F., G.S., M.A.S., and M.A.B. conceptualized the study; D.A.F., M.A.S., N.D.S., G.S., and M.A.B. designed the experiments; M.A.S., N.D.S., D.A.F., P.B.S., and J.E.T. performed research and data analysis; M.A.S., N.D.S., D.A.F., and M.A.B. wrote the manuscript; all authors contributed to editing the manuscript.

Corresponding author

Correspondence to Michael A. Schwemmer.

Ethics declarations

Competing interests

The authors declare competing interests, as they are employed by institutions that provided the funding for this work and/or have filed associated patents. M.A.S., N.D.S., J.E.T., D.A.F., and G.S. are all employed by Battelle Memorial Institute and M.A.B. is employed by the Ohio State University. P.B.S. was also employed by the Ohio State University at the time of this study. D.A.F. and G.S. are listed as inventors on the United States patent application US 2018/0178008 (related WO 2016/196797), and G.S. is listed as an inventor on the United States patent application US 2015/0306373. These are related to the neural bridging BCI technology and stimulation sleeve used in the GRT experiment in the paper.

Additional information

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–8 and Supplementary Tables 1 and 2

Reporting Summary

Supplementary Video 1

Example cued block in which the participant is in control of the BCI-FES system and performing the three functional grips plus hand open

Supplementary Video 2

Example ‘free-time’ block in which the participant is in control of the BCI-FES system and performing the three functional grips plus hand open

Supplementary Video 3

Example grasp and release test block (GRT, see Methods)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schwemmer, M.A., Skomrock, N.D., Sederberg, P.B. et al. Meeting brain–computer interface user performance expectations using a deep neural network decoding framework. Nat Med 24, 1669–1676 (2018). https://doi.org/10.1038/s41591-018-0171-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41591-018-0171-y

This article is cited by

Search

Advanced search

Quick links

Nature Briefing: Translational Research

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

Get what matters in translational research, free to your inbox weekly. Sign up for Nature Briefing: Translational Research