Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Fully portable and wireless universal brain–machine interfaces enabled by flexible scalp electronics and deep learning algorithm

Nature Machine Intelligence volume 1, pages 412–422 (2019)Cite this article

Subjects

Abstract

Variation in human brains creates difficulty in implementing electroencephalography into universal brain–machine interfaces. Conventional electroencephalography systems typically suffer from motion artefacts, extensive preparation time and bulky equipment, while existing electroencephalography classification methods require training on a per-subject or per-session basis. Here, we introduce a fully portable, wireless, flexible scalp electronic system, incorporating a set of dry electrodes and a flexible membrane circuit. Time-domain analysis using convolutional neural networks allows for accurate, real-time classification of steady-state visually evoked potentials in the occipital lobe. Compared to commercial systems, the flexible electronics show the improved performance in detection of evoked potentials due to significant reduction of noise and electromagnetic interference. The two-channel scalp electronic system achieves a high information transfer rate (122.1 ± 3.53 bits per minute) with six human subjects, allowing for wireless, real-time, universal electroencephalography classification for an electric wheelchair, a motorized vehicle and a keyboard-less presentation.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Overview of the system architecture featuring fully portable and wireless scalp electronics.
Fig. 2: Mechanical flexibility and stretchability of the scalp electronics.
Fig. 3: Comparison of device signal quality and classification accuracy.
Fig. 4: EEG classification with CNNs.
Fig. 5: In vivo demonstration of the SKINTRONICS BMI with human subjects.

Similar content being viewed by others

Data availability

The EEG data recorded for this work are available on the Open Science Framework (https://osf.io/nyfa7/).

Code availability

The code for the CNN models are available on GitLab (https://gitlab.com/musasmahmood/ssvep-cnn-demo).

References

  1. Garcia, J. O., Srinivasan, R. & Serences, J. T. Near-real-time feature-selective modulations in human cortex. Curr. Biol. 23, 515–522 (2013).

    Article  Google Scholar 

  2. Zhu, D., Bieger, J., Garcia Molina, G. & Aarts, R. M. A survey of stimulation methods used in SSVEP-based BCIs. Comput. Intell. Neurosci. 2010, 702357 (2010).

    Article  Google Scholar 

  3. Luo, A. & Sullivan, T. J. A user-friendly SSVEP-based brain-computer interface using a time-domain classifier. J. Neural Eng. 7, 26010 (2010).

    Article  Google Scholar 

  4. Stamps, K. & Hamam, Y. Towards inexpensive BCI control for wheelchair navigation in the enabled environment - a hardware survey. Lect. Notes Artif. Int. 6334, 336–345 (2010).

    Google Scholar 

  5. Guger, C. et al. How many people could use an SSVEP BCI? Front. Neurosci. 6, 169 (2012).

    Google Scholar 

  6. Ramos-Murguialday, A. et al. Brain–machine interface in chronic stroke rehabilitation: a controlled study. Ann. Neurol. 74, 100–108 (2013).

    Article  Google Scholar 

  7. Soekadar, S. R., Birbaumer, N., Slutzky, M. W. & Cohen, L. G. Brain–machine interfaces in neurorehabilitation of stroke. Neurobiol. Dis. 83, 172–179 (2015).

    Article  Google Scholar 

  8. Nicolas-Alonso, L. F. & Gomez-Gil, J. Brain computer interfaces, a review. Sensors 12, 1211–1279 (2012).

    Article  Google Scholar 

  9. Bin, G., Lin, Z., Gao, X., Hong, B. & Gao, S. The SSVEP topographic scalp maps by canonical correlation analysis. In Conf. Proc. IEEE Eng. Med. Biol. Soc. 3759–3762 (IEEE, 2008).

  10. Wang, Y., Wang, R., Gao, X., Hong, B. & Gao, S. A practical VEP-based brain–computer interface. IEEE Trans. Neural Syst. Rehabil. Eng. 14, 234–239 (2006).

    Article  Google Scholar 

  11. Wang, Y., Gao, X., Hong, B., Jia, C. & Gao, S. Brain–computer interfaces based on visual evoked potentials. IEEE Eng. Med. Biol. Mag. 27, 64–71 (2008).

    Article  Google Scholar 

  12. Kelly, S. P., Lalor, E. C., Reilly, R. B. & Foxe, J. J. Visual spatial attention tracking using high-density SSVEP data for independent brain–computer communication. IEEE Trans. Neural Syst. Rehabil. Eng. 13, 172–178 (2005).

    Article  Google Scholar 

  13. Herrmann, C. S. Human EEG responses to 1–100 Hz flicker: resonance phenomena in visual cortex and their potential correlation to cognitive phenomena. Exp. Brain Res. 137, 346–353 (2001).

    Article  Google Scholar 

  14. Wang, Y., Wang, Y. T. & Jung, T. P. Visual stimulus design for high-rate SSVEP BCI. Electron. Lett. 46, 1057–1058 (2010).

    Article  Google Scholar 

  15. Norton, J. J. S. et al. Soft, curved electrode systems capable of integration on the auricle as a persistent brain–computer interface. Proc. Natl Acad. Sci. USA 112, 3920–3925 (2015).

    Article  Google Scholar 

  16. Beverina, F., Palmas, G., Silvoni, S., Piccione, F. & Giove, S. User adaptive BCIs: SSVEP and P300 based interfaces. PsychNology J. 1, 331–354 (2003).

    Google Scholar 

  17. Kronegg, J., Voloshynovskiy, S. & Pun, T. Analysis of bit-rate definitions for brain–computer interfaces. In HCI ‘05: Proc. 2005 International Conference on Human–Computer Interaction 40–46 (HCI International, 2005).

  18. Lin, Z., Zhang, C., Wu, W. & Gao, X. Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs. IEEE Trans. Biomed. Eng. 54, 1172–1176 (2007).

    Article  Google Scholar 

  19. Martinez, P., Bakardjian, H. & Cichocki, A. Fully online multicommand brain–computer interface with visual neurofeedback using SSVEP paradigm. Comput. Intell Neurosci. 2007, 94561 (2007).

    Article  Google Scholar 

  20. Bin, G., Gao, X., Yan, Z., Hong, B. & Gao, S. An online multi-channel SSVEP-based brain–computer interface using a canonical correlation analysis method. J. Neural Eng. 6, 046002 (2009).

    Article  Google Scholar 

  21. Wang, Y. T., Wang, Y., Cheng, C. K. & Jung, T. P. Measuring steady-state visual evoked potentials from non-hair-bearing areas. In Conf. Proc. IEEE Eng. Med. Biol. Soc. 1806–1809 (IEEE, 2012).

  22. Cecotti, H. A time–frequency convolutional neural network for the offline classification of steady-state visual evoked potential responses. Pattern Recognit. Lett. 32, 1145–1153 (2011).

    Article  Google Scholar 

  23. McAdams, E. T., Jossinet, J., Lackermeier, A. & Risacher, F. Factors affecting electrode–gel–skin interface impedance in electrical impedance tomography. Med. Biol. Eng. Comput. 34, 397–408 (1996).

    Article  Google Scholar 

  24. Searle, A. & Kirkup, L. A direct comparison of wet, dry and insulating bioelectric recording electrodes. Physiol. Meas. 21, 271–283 (2000).

    Article  Google Scholar 

  25. Li, G., Wang, S. & Duan, Y. Y. Towards gel-free electrodes: a systematic study of electrode–skin impedance. Sens. Actuators B 241, 1244–1255 (2017).

    Article  Google Scholar 

  26. Lin, C. T. et al. Review of wireless and wearable electroencephalogram systems and brain-computer interfaces - a mini-review. Gerontology 56, 112–119 (2010).

    Article  Google Scholar 

  27. Salvo, P. et al. A 3D printed dry electrode for ECG/EEG recording. Sens. Actuators A 174, 96–102 (2012).

    Article  Google Scholar 

  28. Stauffer, F. et al. Skin conformal polymer electrodes for clinical ECG and EEG recordings. Adv. Healthc. Mater. 7, e1700994 (2018).

    Article  Google Scholar 

  29. Tallgren, P., Vanhatalo, S., Kaila, K. & Voipio, J. Evaluation of commercially available electrodes and gels for recording of slow EEG potentials. Clin. Neurophysiol. 116, 799–806 (2005).

    Article  Google Scholar 

  30. Nakanishi, M. et al. Enhancing detection of SSVEPs for a high-speed brain speller using task-related component analysis. IEEE Trans. Biomed. Eng. 65, 104–112 (2018).

    Article  Google Scholar 

  31. Kwak, N. S., Muller, K. R. & Lee, S. W. A convolutional neural network for steady state visual evoked potential classification under ambulatory environment. PLoS ONE 12, e0172578 (2017).

    Article  Google Scholar 

  32. Dufort y Alvarez, G. et al. Wireless EEG system achieving high throughput and reduced energy consumption through lossless and near-lossless compression. IEEE Trans. Biomed. Circuits Syst. 12, 231–241 (2018).

    Article  Google Scholar 

  33. Lin, C.-T., Chiu, C.-Y., Singh, A. K., King, J.-T. & Wang, Y.-K. A wireless multifunctional SSVEP-based brain computer interface assistive system. IEEE Trans. Cogn. Dev. Syst. 1 (IEEE, 2018).

  34. Schlogl, A., Keinrath, C., Scherer, R. & Furtscheller, P. Information transfer of an EEG-based brain computer interface. In Proc. First International IEEE EMBS Conference on Neural Engineering 641–644 (IEEE, 2003).

  35. Xu, J. & Zhong, B. Review on portable EEG technology in educational research. Comput. Hum. Behav. 81, 340–349 (2018).

    Article  Google Scholar 

  36. Shi, M. et al. Towards portable SSVEP-based brain-computer interface using Emotiv EPOC and mobile phone. In Proc. Tenth International Conference on Advanced Computational Intelligence 249–253 (IEEE, 2018).

  37. Chen, X., Zhao, B., Wang, Y., Xu, S. & Gao, X. Control of a 7-DOF robotic arm system with an SSVEP-based BCI. Int. J. Neural Syst. 28, 1850018 (2018).

    Article  Google Scholar 

  38. Mishra, S. et al. Soft, conformal bioelectronics for a wireless human–wheelchair interface. Biosens. Bioelectron. 91, 796–803 (2017).

    Article  Google Scholar 

  39. Lee, Y. et al. Wireless, intraoral hybrid electronics for real-time quantification of sodium intake toward hypertension management. Proc. Natl Acad. Sci. USA 115, 5377–5382 (2018).

    Article  Google Scholar 

  40. Chi, Y. M. et al. A practical mobile dry EEG system for human computer interfaces. In Proc. International Conference on Augmented Cognition 649–655 (Springer, 2013).

  41. Mullen, T. R. et al. Real-time neuroimaging and cognitive monitoring using wearable dry EEG. IEEE Trans. Biomed. Eng. 62, 2553–2567 (2015).

    Article  Google Scholar 

  42. Lee, Y. et al. Soft electronics enabled ergonomic human–computer interaction for swallowing training. Sci. Rep. 7, 46697 (2017).

    Article  Google Scholar 

  43. Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 1, 1097–1105 (2012).

    Google Scholar 

  44. Chen, X. et al. High-speed spelling with a noninvasive brain–computer interface. Proc. Natl Acad. Sci. USA 112, E6058–E6067 (2015).

    Article  Google Scholar 

  45. Ji, Y., Hwang, J. & Kim, E. Y. An intelligent wheelchair using situation awareness and obstacle detection. Procedia 97, 620–628 (2013).

    Google Scholar 

  46. Ma, T. et al. The hybrid BCI system for movement control by combining motor imagery and moving onset visual evoked potential. J. Neural Eng. 14, 026015 (2017).

    Article  Google Scholar 

  47. Pfurtscheller, G. & Neuper, C. Motor imagery and direct brain–computer communication. Proc. IEEE 89, 1123–1134 (2001).

    Article  Google Scholar 

  48. Delorme, A. & Makeig, S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 134, 9–21 (2004).

    Article  Google Scholar 

  49. Kim, Y. S. et al. Scalable manufacturing of solderable and stretchable physiologic sensing systems. Adv. Mater. 29, 1701312 (2017).

    Article  Google Scholar 

  50. Bideaux, A., Zimmermann, B., Hey, S. & Stork, W. Synchronization in wireless biomedical-sensor networks with Bluetooth Low Energy. Curr. Dir. Biomed. Eng. 1, 73–76 (2015).

    Google Scholar 

  51. Mastinu, E., Ortiz-Catalan, M. & Hakansson, B. Analog front-ends comparison in the way of a portable, low-power and low-cost EMG controller based on pattern recognition. In Conf. Proc. IEEE Eng. Med. Biol. Soc. 2111–2114 (IEEE, 2015).

  52. Zhang, Y., Zhou, G., Jin, J., Wang, X. & Cichocki, A. Frequency recognition in SSVEP-based BCI using multiset canonical correlation analysis. Int. J. Neural Syst. 24, 1450013 (2014).

    Article  Google Scholar 

  53. Xu, R. et al. Fabric-based stretchable electronics with mechanically optimized designs and prestrained composite substrates. Extreme Mech. Lett. 1, 120–126 (2014).

    Article  Google Scholar 

  54. Obermaier, B., Neuper, C., Guger, C. & Pfurtscheller, G. Information transfer rate in a five-classes brain–computer interface. IEEE Trans. Neural Syst. Rehabil. Eng. 9, 283–288 (2001).

    Article  Google Scholar 

  55. Shannon, C. E. & Weaver, W. The Mathematical Theory of Information (Univ. Illinois Press, 1949).

  56. Goldman, R. I., Stern, J. M., Engel, J.Jr & Cohen, M. S. Simultaneous EEG and fMRI of the alpha rhythm. Neuroreport 13, 2487–2492 (2002).

    Article  Google Scholar 

  57. Wang, Y., Wang, R., Gao, X. & Gao, S. Brain-computer interface based on the high-frequency steady-state visual evoked potential. In Proc. First International Conference on Neural Interface and Control 37–39 (IEEE, 2005).

  58. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).

  59. Bevilacqua, V. et al. A novel BCI-SSVEP based approach for control of walking in virtual environment using a convolutional neural network. In Proc. IEEE International Joint Conference on Neural Networks (IJCNN) 4121–4128 (IEEE, 2014).

  60. Volosyak, I. SSVEP-based Bremen–BCI interface—boosting information transfer rates. J. Neural Eng. 8, 036020 (2011).

    Article  Google Scholar 

Download references

Acknowledgements

W.-H.Y. acknowledges a grant from the Fundamental Research Program (project PNK5061) of Korea Institute of Materials Science, funding by the Nano-Material Technology Development Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning (no. 2016M3A7B4900044), and support from the Institute for Electronics and Nanotechnology, a member of the National Nanotechnology Coordinated Infrastructure, which is supported by the National Science Foundation (grant ECCS-1542174).

Author information

Authors and Affiliations

  1. George W. Woodruff School of Mechanical Engineering, Institute for Electronics and Nanotechnology, Georgia Institute of Technology, Atlanta, GA, USA

    Musa Mahmood, Yun-Soung Kim, Saswat Mishra, Robert Herbert & Woon-Hong Yeo

  2. School of Engineering and Digital Arts, University of Kent, Canterbury, UK

    Deogratias Mzurikwao & Chee Siang Ang

  3. Department of Biomedical Engineering, Wichita State University, Wichita, KS, USA

    Yongkuk Lee

  4. School of Psychology, College of Sciences, Georgia Institute of Technology, Atlanta, GA, USA

    Audrey Duarte

  5. Wallace H. Coulter Department of Biomedical Engineering, Parker H. Petit Institute for Bioengineering and Biosciences, Georgia Institute of Technology, Atlanta, GA, USA

    Woon-Hong Yeo

  6. Neural Engineering Center, Center for Flexible and Wearable Electronics Advanced Research, Institute for Materials, Institute for Robotics and Intelligent Machines, Georgia Institute of Technology, Atlanta, GA, USA

    Woon-Hong Yeo

Authors
  1. Musa Mahmood
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Deogratias Mzurikwao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yun-Soung Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yongkuk Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Saswat Mishra
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Robert Herbert
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Audrey Duarte
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Chee Siang Ang
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Woon-Hong Yeo
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

M.M. and W.-H.Y. designed the research project; M.M., D.M., Y.-S.K., Y.L., R.H., A.D., S.M., C.S.A. and W.-H.Y. performed research; M.M., D.M., A.D., C.S.A. and W.-H.Y. analysed data; and M.M., D.M., C.S.A. and W.-H.Y. wrote the paper.

Corresponding author

Correspondence to Woon-Hong Yeo.

Ethics declarations

Competing interests

W.-H.Y. and M.M. are the inventors on a pending US patent application.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs., Tables and notes

Reporting Summary

Supplementary Video 1

Supplementary Video

Supplementary Video 2

Supplementary Video

Supplementary Video 3

Supplementary Video

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mahmood, M., Mzurikwao, D., Kim, YS. et al. Fully portable and wireless universal brain–machine interfaces enabled by flexible scalp electronics and deep learning algorithm. Nat Mach Intell 1, 412–422 (2019). https://doi.org/10.1038/s42256-019-0091-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-019-0091-7

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing