Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

A wearable biosensing system with in-sensor adaptive machine learning for hand gesture recognition

Nature Electronics volume 4, pages 54–63 (2021)Cite this article

Subjects

Abstract

Wearable devices that monitor muscle activity based on surface electromyography could be of use in the development of hand gesture recognition applications. Such devices typically use machine-learning models, either locally or externally, for gesture classification. However, most devices with local processing cannot offer training and updating of the machine-learning model during use, resulting in suboptimal performance under practical conditions. Here we report a wearable surface electromyography biosensing system that is based on a screen-printed, conformal electrode array and has in-sensor adaptive learning capabilities. Our system implements a neuro-inspired hyperdimensional computing algorithm locally for real-time gesture classification, as well as model training and updating under variable conditions such as different arm positions and sensor replacement. The system can classify 13 hand gestures with 97.12% accuracy for two participants when training with a single trial per gesture. A high accuracy (92.87%) is preserved on expanding to 21 gestures, and accuracy is recovered by 9.5% by implementing model updates in response to varying conditions, without additional computation on an external device.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Rent or buy this article

Get just this article for as long as you need it

$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Wearable biosensing system for sEMG.
Fig. 2: Hand gesture classes used in the study and sEMG recording characteristics.
Fig. 3: Hyperdimensional computing algorithm for projecting windows of sEMG data into hypervectors.
Fig. 4: AM operations for training, accessing and contextual updating.
Fig. 5: Real-time, in-sensor classification performance in the baseline context.
Fig. 6: In-sensor training, update and classification results.

Data availability

Both the offline sEMG dataset and the real-time experiment data collected for this study are available at https://github.com/flexemg/flexemg_natelec. NinaPro data were accessed via http://ninapro.hevs.ch (Dataset 4) and CapgMyo data via http://zju-capg.org/myo/data/ (DB-c).

Code availability

The source code used for offline model validation, in-sensor implementation and analysis of results is available at https://github.com/flexemg/flexemg_natelec.

References

  1. Kim, J., Campbell, A. S., de Ávila, B. E. F. & Wang, J. Wearable biosensors for healthcare monitoring. Nat. Biotechnol. 37, 389–406 (2019).

    Article  Google Scholar 

  2. Bandodkar, A. J., Jeerapan, I. & Wang, J. Wearable chemical sensors: present challenges and future prospects. ACS Sens. 1, 464–482 (2016).

    Article  Google Scholar 

  3. Khan, Y., Ostfeld, A. E., Lochner, C. M., Pierre, A. & Arias, A. C. Monitoring of vital signs with flexible and wearable medical devices. Adv. Mater. 28, 4373–4395 (2016).

    Article  Google Scholar 

  4. Patel, S., Park, H., Bonato, P., Chan, L. & Rodgers, M. A review of wearable sensors and systems with application in rehabilitation. J. Neuroeng. Rehabil. 9, 21 (2012).

    Article  Google Scholar 

  5. Nakata, S. et al. A wearable pH sensor with high sensitivity based on a flexible charge-coupled device. Nat. Electron. 1, 596–603 (2018).

    Article  Google Scholar 

  6. Huang, Z. et al. Three-dimensional integrated stretchable electronics. Nat. Electron. 1, 473–480 (2018).

    Article  Google Scholar 

  7. Gao, W. et al. Fully integrated wearable sensor arrays for multiplexed in-situ perspiration analysis. Nature 529, 509–514 (2016).

    Article  Google Scholar 

  8. Imani, S. et al. A wearable chemical-electrophysiological hybrid biosensing system for real-time health and fitness monitoring. Nat. Commun. 7, 11650 (2016).

    Article  Google Scholar 

  9. Lochner, C. M., Khan, Y., Pierre, A. & Arias, A. C. All-organic optoelectronic sensor for pulse oximetry. Nat. Commun. 5, 5745 (2014).

    Article  Google Scholar 

  10. Khan, Y. et al. A flexible organic reflectance oximeter array. Proc. Natl Acad. Sci. USA 115, E11015–E11024 (2018).

    Article  Google Scholar 

  11. Benatti, S. et al. A versatile embedded platform for EMG acquisition and gesture recognition. IEEE Trans. Biomed. Circuits Syst. 9, 620–630 (2015).

    Article  Google Scholar 

  12. Liu, X. et al. The virtual trackpad: an electromyography-based, wireless, real-time, low-power, embedded hand-gesture-recognition system using an event-driven artificial neural network. IEEE Trans. Circuits Syst. II 64, 1257–1261 (2017).

    Article  Google Scholar 

  13. Gu, Y., Yang, D., Huang, Q., Yang, W. & Liu, H. Robust EMG pattern recognition in the presence of confounding factors: features, classifiers and adaptive learning. Expert Syst. Appl. 96, 208–217 (2018).

    Article  Google Scholar 

  14. Young, A. J., Hargrove, L. J. & Kuiken, T. A. The effects of electrode size and orientation on the sensitivity of myoelectric pattern recognition systems to electrode shift. IEEE Trans. Biomed. Eng. 58, 2537–2544 (2011).

    Article  Google Scholar 

  15. Zhang, X. & Huang, H. A real-time, practical sensor fault-tolerant module for robust EMG pattern recognition. J. Neuroeng. Rehabil. 12, 18 (2015).

    Article  Google Scholar 

  16. Hargrove, L., Englehart, K. & Hudgins, B. A training strategy to reduce classification degradation due to electrode displacements in pattern recognition based myoelectric control. Biomed. Signal Process. Control 3, 175–180 (2008).

    Article  Google Scholar 

  17. Tkach, D., Huang, H. & Kuiken, T. A. Study of stability of time-domain features for electromyographic pattern recognition. J. Neuroeng. Rehabil. 7, 21 (2010).

    Article  Google Scholar 

  18. Milosevic, B., Farella, E. & Benatti, S. Exploring arm posture and temporal variability in myoelectric hand gesture recognition. Proc. 7th IEEE Int. Conf. on Biomedical Robotics and Biomechatronics 2018, 1032–1037 (2018).

    Google Scholar 

  19. Krishnan, K. S., Saha, A., Ramachandran, S. & Kumar, S. Recognition of human arm gestures using Myo armband for the game of hand cricket. In IEEE International Symposium on Robotics and Intelligent Sensors (IRIS) 389–394 (IEEE, 2017); https://doi.org/10.1109/IRIS.2017.8250154

  20. Ploengpit, Y. & Phienthrakul, T. Rock–paper–scissors with Myo armband pose detection. In 2016 International Computer Science and Engineering Conference 1–5 (IEEE, 2017).

  21. Nyomen, K., Romarheim Haugen, M. & Jensenius, A. R. MuMYO—evaluating and exploring the MYO armband for musical interaction. In Proc. International Conference on New Interfaces For Musical Expression 215–218 (Louisiana State Univ., 2015).

  22. Abreu, J. G., Teixeira, J. M., Figueiredo, L. S. & Teichrieb, V. Evaluating sign language recognition using the Myo armband. In 2016 XVIII Symposium on Virtual and Augmented Reality 64–70 (IEEE, 2016).

  23. Morais, G. D., Neves, L. C., Masiero, A. A. & Castro, M. C. F. Application of Myo armband system to control a robot interface. In Proc. BIOSTEC 2016 227–231 (2016).

  24. Xu, Y., Yang, C., Liang, P., Zhao, L. & Li, Z. Development of a hybrid motion capture method using MYO armband with application to teleoperation. In 2016 IEEE International Conference on Mechatronics and Automation 1179–1184 (IEEE, 2016).

  25. Chen, X. & Wang, Z. J. Pattern recognition of number gestures based on a wireless surface EMG system. Biomed. Signal Process. Control 8, 184–192 (2013).

    Article  Google Scholar 

  26. Amma, C., Krings, T., Böer, J. & Schultz, T. Advancing muscle–computer interfaces with high-density electromyography. In Proc. 33rd Annual ACM Conference on Human Factors in Computing Systems 929–938 (2015).

  27. Geng, W. et al. Gesture recognition by instantaneous surface EMG images. Sci. Rep. 6, 36571 (2016).

    Article  Google Scholar 

  28. Cerone, G. L., Botter, A. & Gazzoni, M. A modular, smart and wearable system for high density sEMG detection. IEEE Trans. Biomed. Eng. 66, 3371–3380 (2019).

    Article  Google Scholar 

  29. Pancholi, S. & Joshi, A. M. Electromyography-based hand gesture recognition system for upper limb amputees. IEEE Sens. Lett. 3, 1–4 (2019).

    Google Scholar 

  30. Benatti, S. et al. Online learning and classification of EMG-based gestures on a parallel ultra-low power platform using hyperdimensional computing. IEEE Trans. Biomed. Circuits Syst. 13, 516–528 (2019).

    Article  Google Scholar 

  31. Kanerva, P. Hyperdimensional computing: an introduction to computing in distributed representation with high-dimensional random vectors. Cogn. Comput. 1, 139–159 (2009).

    Article  Google Scholar 

  32. Moin, A. et al. An EMG gesture recognition system with flexible high-density sensors and brain-inspired high-dimensional classifier. In 2018 IEEE International Symposium on Circuits and Systems (ISCAS) 1–5 (IEEE, 2018); https://doi.org/10.1109/ISCAS.2018.8351613

  33. Rahimi, A., Tchouprina, A., Kanerva, P., Millán, J. D. R. & Rabaey, J. M. Hyperdimensional computing for blind and one-shot classification of EEG error-related potentials. Mob. Netw. Appl. 25, 1958–1969 (2020).

    Article  Google Scholar 

  34. Burrello, A., Schindler, K., Benini, L. & Rahimi, A. One-shot learning for iEEG seizure detection using end-to-end binary operations: local binary patterns with hyperdimensional computing. In Proc. 2018 IEEE Biomedical Circuits and Systems Conference (BioCAS) 1–4 (IEEE, 2018); https://doi.org/10.1109/BIOCAS.2018.8584751

  35. Johnson, B. C. et al. An implantable 700-μW 64-channel neuromodulation IC for simultaneous recording and stimulation with rapid artifact recovery. In IEEE 2017 Symposium on VLSI Circuits, Digest of Technical Papers C48–C49 (IEEE, 2017); https://doi.org/10.23919/VLSIC.2017.8008543

  36. Cui, H. W. et al. Ultra-fast photonic curing of electrically conductive adhesives fabricated from vinyl ester resin and silver micro-flakes for printed electronics. RSC Adv. 4, 15914–15922 (2014).

    Article  Google Scholar 

  37. Lapatki, B. G., Van Dijk, J. P., Jonas, I. E., Zwarts, M. J. & Stegeman, D. F. A thin, flexible multielectrode grid for high-density surface EMG. J. Appl. Physiol. 96, 327–336 (2004).

    Article  Google Scholar 

  38. Dick, F. S., Bert, U. K., Bernd, G. L. & Johannes, P. V. D. High-density surface EMG: techniques and applications at a motor unit level. Biocybern. Biomed. Eng. 32, 3–27 (2012).

    Article  Google Scholar 

  39. Atzori, M. et al. Electromyography data for non-invasive naturally-controlled robotic hand prostheses. Sci. Data 1, 1–13 (2014).

    Article  Google Scholar 

  40. Pizzolato, S. et al. Comparison of six electromyography acquisition setups on hand movement classification tasks. PLoS ONE 12, e0186132 (2017).

    Article  Google Scholar 

  41. Atzori, M. et al. Building the NinaPro database: a resource for the biorobotics community. In Proc. 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob) 1258–1265 (IEEE, 2012); https://doi.org/10.1109/BioRob.2012.6290287

  42. Du, Y., Jin, W., Wei, W., Hu, Y. & Geng, W. Surface EMG-based inter-session gesture recognition enhanced by deep domain adaptation. Sensors 17, 6–9 (2017).

    Google Scholar 

  43. Rahimi, A. et al. High-dimensional computing as a nanoscalable paradigm. IEEE Trans. Circuits Syst. 64, 2508–2521 (2017).

    Article  Google Scholar 

  44. Smith, L. H., Hargrove, L. J., Lock, B. A. & Kuiken, T. A. Determining the optimal window length for pattern recognition-based myoelectric control: balanÿcing the competing effects of classification error and controller delay. IEEE Trans. Neural Syst. Rehabil. Eng. 19, 186–192 (2011).

    Article  Google Scholar 

  45. Englehart, K. & Hudgins, B. A robust, real-time control scheme for multifunction myoelectric control. IEEE Trans. Biomed. Eng. 50, 848–854 (2003).

    Article  Google Scholar 

  46. Lecun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  Google Scholar 

  47. Liu, J. Adaptive myoelectric pattern recognition toward improved multifunctional prosthesis control. Med. Eng. Phys. 37, 424–430 (2015).

    Article  Google Scholar 

  48. Diehl, C. P. & Cauwenberghs, G. SVM incremental learning, adaptation and optimization. In Proc. International Joint Conference on Neural Networks Vol. 4, 2685–2690 (IEEE, 2003); https://doi.org/10.1109/IJCNN.2003.1223991

  49. Zhu, X., Liu, J., Zhang, D., Sheng, X. & Jiang, N. Cascaded adaptation framework for fast calibration of myoelectric control. IEEE Trans. Neural Syst. Rehabil. Eng. 25, 254–264 (2017).

    Article  Google Scholar 

  50. Zhang, H. et al. An adaptation strategy of using LDA classifier for EMG pattern recognition. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. 2013, 4267–4270 (2013).

    Google Scholar 

  51. Schmuck, M., Benini, L. & Rahimi, A. Hardware optimizations of dense binary hyperdimensional computing: rematerialization of hypervectors, binarized bundling and combinational associative memory. ACM J. Emerg. Technol. Comput. Syst. 15, 1550–4832 (2019).

    Article  Google Scholar 

  52. Kanerva, P. Binary spatter-coding of ordered K-tuples. in Artificial Neural Networks — ICANN 96 (eds von der Malsburg, C. et al.) 869–873 (Lecture Notes in Computer Science Vol. 1112, Springer, 1996); https://doi.org/10.1007/3-540-61510-5_146

Download references

Acknowledgements

We thank E. Alon, A. Araujo, R. Muller, K. Lutz, H. Wu, M. Sadeghi, Cortera Neurotechnologies and Novacentrix. This work was supported in part by the CONIX Research Center, one of six centres in JUMP, a Semiconductor Research Corporation (SRC) programme sponsored by DARPA. This material is based, in part, on research sponsored by the Air Force Research Laboratory under agreement no. FA8650-15-2-5401, as conducted through the flexible hybrid electronics manufacturing innovation institute, NextFlex. The US Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of Air Force Research Laboratory or the US Government. This research used the Savio computational cluster resource provided by the Berkeley Research Computing program at the University of California, Berkeley. Support was also received from sponsors of Berkeley Wireless Research Center, NSF Graduate Research Fellowship under grant no. 1106400, ETH Zurich Postdoctoral Fellowship programme and the Marie Curie Actions for People COFUND Program.

Author information

Author notes

  1. These authors contributed equally: Ali Moin, Andy Zhou.

Authors and Affiliations

  1. Department of Electrical Engineering and Computer Sciences, University of California at Berkeley, Berkeley, CA, USA

    Ali Moin, Andy Zhou, Alisha Menon, George Alexandrov, Senam Tamakloe, Jonathan Ting, Natasha Yamamoto, Yasser Khan, Fred Burghardt, Ana C. Arias & Jan M. Rabaey

  2. IBM Research Zurich, Ruschlikon, Switzerland

    Abbas Rahimi

  3. Department of Electrical, Electronic and Information Engineering, University of Bologna, Bologna, Italy

    Simone Benatti & Luca Benini

  4. Department of Information Technology and Electrical Engineering, ETH Zürich, Zurich, Switzerland

    Luca Benini

Authors
  1. Ali Moin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andy Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Abbas Rahimi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alisha Menon
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Simone Benatti
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. George Alexandrov
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Senam Tamakloe
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jonathan Ting
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Natasha Yamamoto
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Yasser Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Fred Burghardt
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Luca Benini
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Ana C. Arias
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Jan M. Rabaey
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

A. Moin, A.Z., A. Menon, S.B., G.A., S.T., J.T., N.Y., Y.K. and F.B. designed the hardware. A. Moin, A.Z. and A.R. developed and implemented the learning algorithm. A. Moin and A.Z. performed the experiments and analysis. L.B., A.C.A. and J.M.R. oversaw the project. A. Moin, A.Z., A.C.A. and J.M.R. wrote and edited the manuscript.

Corresponding authors

Correspondence to Ali Moin or Jan M. Rabaey.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Notes 1–5, Figs. 1–7 and Tables 1–5.

Reporting Summary

Supplementary Video

Electrode array fabrication process, wearing the device, training and inference for hand gestures in action.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Moin, A., Zhou, A., Rahimi, A. et al. A wearable biosensing system with in-sensor adaptive machine learning for hand gesture recognition. Nat Electron 4, 54–63 (2021). https://doi.org/10.1038/s41928-020-00510-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41928-020-00510-8

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing