Wearable devices that monitor muscle activity based on surface electromyography could be of use in the development of hand gesture recognition applications. Such devices typically use machine-learning models, either locally or externally, for gesture classification. However, most devices with local processing cannot offer training and updating of the machine-learning model during use, resulting in suboptimal performance under practical conditions. Here we report a wearable surface electromyography biosensing system that is based on a screen-printed, conformal electrode array and has in-sensor adaptive learning capabilities. Our system implements a neuro-inspired hyperdimensional computing algorithm locally for real-time gesture classification, as well as model training and updating under variable conditions such as different arm positions and sensor replacement. The system can classify 13 hand gestures with 97.12% accuracy for two participants when training with a single trial per gesture. A high accuracy (92.87%) is preserved on expanding to 21 gestures, and accuracy is recovered by 9.5% by implementing model updates in response to varying conditions, without additional computation on an external device.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
Soft Electronics for Health Monitoring Assisted by Machine Learning
Nano-Micro Letters Open Access 15 March 2023
Machine Learning-Enhanced Flexible Mechanical Sensing
Nano-Micro Letters Open Access 17 February 2023
Efficient emotion recognition using hyperdimensional computing with combinatorial channel encoding and cellular automata
Brain Informatics Open Access 27 June 2022
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 per month
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Rent or buy this article
Get just this article for as long as you need it
Prices may be subject to local taxes which are calculated during checkout
Both the offline sEMG dataset and the real-time experiment data collected for this study are available at https://github.com/flexemg/flexemg_natelec. NinaPro data were accessed via http://ninapro.hevs.ch (Dataset 4) and CapgMyo data via http://zju-capg.org/myo/data/ (DB-c).
The source code used for offline model validation, in-sensor implementation and analysis of results is available at https://github.com/flexemg/flexemg_natelec.
Kim, J., Campbell, A. S., de Ávila, B. E. F. & Wang, J. Wearable biosensors for healthcare monitoring. Nat. Biotechnol. 37, 389–406 (2019).
Bandodkar, A. J., Jeerapan, I. & Wang, J. Wearable chemical sensors: present challenges and future prospects. ACS Sens. 1, 464–482 (2016).
Khan, Y., Ostfeld, A. E., Lochner, C. M., Pierre, A. & Arias, A. C. Monitoring of vital signs with flexible and wearable medical devices. Adv. Mater. 28, 4373–4395 (2016).
Patel, S., Park, H., Bonato, P., Chan, L. & Rodgers, M. A review of wearable sensors and systems with application in rehabilitation. J. Neuroeng. Rehabil. 9, 21 (2012).
Nakata, S. et al. A wearable pH sensor with high sensitivity based on a flexible charge-coupled device. Nat. Electron. 1, 596–603 (2018).
Huang, Z. et al. Three-dimensional integrated stretchable electronics. Nat. Electron. 1, 473–480 (2018).
Gao, W. et al. Fully integrated wearable sensor arrays for multiplexed in-situ perspiration analysis. Nature 529, 509–514 (2016).
Imani, S. et al. A wearable chemical-electrophysiological hybrid biosensing system for real-time health and fitness monitoring. Nat. Commun. 7, 11650 (2016).
Lochner, C. M., Khan, Y., Pierre, A. & Arias, A. C. All-organic optoelectronic sensor for pulse oximetry. Nat. Commun. 5, 5745 (2014).
Khan, Y. et al. A flexible organic reflectance oximeter array. Proc. Natl Acad. Sci. USA 115, E11015–E11024 (2018).
Benatti, S. et al. A versatile embedded platform for EMG acquisition and gesture recognition. IEEE Trans. Biomed. Circuits Syst. 9, 620–630 (2015).
Liu, X. et al. The virtual trackpad: an electromyography-based, wireless, real-time, low-power, embedded hand-gesture-recognition system using an event-driven artificial neural network. IEEE Trans. Circuits Syst. II 64, 1257–1261 (2017).
Gu, Y., Yang, D., Huang, Q., Yang, W. & Liu, H. Robust EMG pattern recognition in the presence of confounding factors: features, classifiers and adaptive learning. Expert Syst. Appl. 96, 208–217 (2018).
Young, A. J., Hargrove, L. J. & Kuiken, T. A. The effects of electrode size and orientation on the sensitivity of myoelectric pattern recognition systems to electrode shift. IEEE Trans. Biomed. Eng. 58, 2537–2544 (2011).
Zhang, X. & Huang, H. A real-time, practical sensor fault-tolerant module for robust EMG pattern recognition. J. Neuroeng. Rehabil. 12, 18 (2015).
Hargrove, L., Englehart, K. & Hudgins, B. A training strategy to reduce classification degradation due to electrode displacements in pattern recognition based myoelectric control. Biomed. Signal Process. Control 3, 175–180 (2008).
Tkach, D., Huang, H. & Kuiken, T. A. Study of stability of time-domain features for electromyographic pattern recognition. J. Neuroeng. Rehabil. 7, 21 (2010).
Milosevic, B., Farella, E. & Benatti, S. Exploring arm posture and temporal variability in myoelectric hand gesture recognition. Proc. 7th IEEE Int. Conf. on Biomedical Robotics and Biomechatronics 2018, 1032–1037 (2018).
Krishnan, K. S., Saha, A., Ramachandran, S. & Kumar, S. Recognition of human arm gestures using Myo armband for the game of hand cricket. In IEEE International Symposium on Robotics and Intelligent Sensors (IRIS) 389–394 (IEEE, 2017); https://doi.org/10.1109/IRIS.2017.8250154
Ploengpit, Y. & Phienthrakul, T. Rock–paper–scissors with Myo armband pose detection. In 2016 International Computer Science and Engineering Conference 1–5 (IEEE, 2017).
Nyomen, K., Romarheim Haugen, M. & Jensenius, A. R. MuMYO—evaluating and exploring the MYO armband for musical interaction. In Proc. International Conference on New Interfaces For Musical Expression 215–218 (Louisiana State Univ., 2015).
Abreu, J. G., Teixeira, J. M., Figueiredo, L. S. & Teichrieb, V. Evaluating sign language recognition using the Myo armband. In 2016 XVIII Symposium on Virtual and Augmented Reality 64–70 (IEEE, 2016).
Morais, G. D., Neves, L. C., Masiero, A. A. & Castro, M. C. F. Application of Myo armband system to control a robot interface. In Proc. BIOSTEC 2016 227–231 (2016).
Xu, Y., Yang, C., Liang, P., Zhao, L. & Li, Z. Development of a hybrid motion capture method using MYO armband with application to teleoperation. In 2016 IEEE International Conference on Mechatronics and Automation 1179–1184 (IEEE, 2016).
Chen, X. & Wang, Z. J. Pattern recognition of number gestures based on a wireless surface EMG system. Biomed. Signal Process. Control 8, 184–192 (2013).
Amma, C., Krings, T., Böer, J. & Schultz, T. Advancing muscle–computer interfaces with high-density electromyography. In Proc. 33rd Annual ACM Conference on Human Factors in Computing Systems 929–938 (2015).
Geng, W. et al. Gesture recognition by instantaneous surface EMG images. Sci. Rep. 6, 36571 (2016).
Cerone, G. L., Botter, A. & Gazzoni, M. A modular, smart and wearable system for high density sEMG detection. IEEE Trans. Biomed. Eng. 66, 3371–3380 (2019).
Pancholi, S. & Joshi, A. M. Electromyography-based hand gesture recognition system for upper limb amputees. IEEE Sens. Lett. 3, 1–4 (2019).
Benatti, S. et al. Online learning and classification of EMG-based gestures on a parallel ultra-low power platform using hyperdimensional computing. IEEE Trans. Biomed. Circuits Syst. 13, 516–528 (2019).
Kanerva, P. Hyperdimensional computing: an introduction to computing in distributed representation with high-dimensional random vectors. Cogn. Comput. 1, 139–159 (2009).
Moin, A. et al. An EMG gesture recognition system with flexible high-density sensors and brain-inspired high-dimensional classifier. In 2018 IEEE International Symposium on Circuits and Systems (ISCAS) 1–5 (IEEE, 2018); https://doi.org/10.1109/ISCAS.2018.8351613
Rahimi, A., Tchouprina, A., Kanerva, P., Millán, J. D. R. & Rabaey, J. M. Hyperdimensional computing for blind and one-shot classification of EEG error-related potentials. Mob. Netw. Appl. 25, 1958–1969 (2020).
Burrello, A., Schindler, K., Benini, L. & Rahimi, A. One-shot learning for iEEG seizure detection using end-to-end binary operations: local binary patterns with hyperdimensional computing. In Proc. 2018 IEEE Biomedical Circuits and Systems Conference (BioCAS) 1–4 (IEEE, 2018); https://doi.org/10.1109/BIOCAS.2018.8584751
Johnson, B. C. et al. An implantable 700-μW 64-channel neuromodulation IC for simultaneous recording and stimulation with rapid artifact recovery. In IEEE 2017 Symposium on VLSI Circuits, Digest of Technical Papers C48–C49 (IEEE, 2017); https://doi.org/10.23919/VLSIC.2017.8008543
Cui, H. W. et al. Ultra-fast photonic curing of electrically conductive adhesives fabricated from vinyl ester resin and silver micro-flakes for printed electronics. RSC Adv. 4, 15914–15922 (2014).
Lapatki, B. G., Van Dijk, J. P., Jonas, I. E., Zwarts, M. J. & Stegeman, D. F. A thin, flexible multielectrode grid for high-density surface EMG. J. Appl. Physiol. 96, 327–336 (2004).
Dick, F. S., Bert, U. K., Bernd, G. L. & Johannes, P. V. D. High-density surface EMG: techniques and applications at a motor unit level. Biocybern. Biomed. Eng. 32, 3–27 (2012).
Atzori, M. et al. Electromyography data for non-invasive naturally-controlled robotic hand prostheses. Sci. Data 1, 1–13 (2014).
Pizzolato, S. et al. Comparison of six electromyography acquisition setups on hand movement classification tasks. PLoS ONE 12, e0186132 (2017).
Atzori, M. et al. Building the NinaPro database: a resource for the biorobotics community. In Proc. 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob) 1258–1265 (IEEE, 2012); https://doi.org/10.1109/BioRob.2012.6290287
Du, Y., Jin, W., Wei, W., Hu, Y. & Geng, W. Surface EMG-based inter-session gesture recognition enhanced by deep domain adaptation. Sensors 17, 6–9 (2017).
Rahimi, A. et al. High-dimensional computing as a nanoscalable paradigm. IEEE Trans. Circuits Syst. 64, 2508–2521 (2017).
Smith, L. H., Hargrove, L. J., Lock, B. A. & Kuiken, T. A. Determining the optimal window length for pattern recognition-based myoelectric control: balanÿcing the competing effects of classification error and controller delay. IEEE Trans. Neural Syst. Rehabil. Eng. 19, 186–192 (2011).
Englehart, K. & Hudgins, B. A robust, real-time control scheme for multifunction myoelectric control. IEEE Trans. Biomed. Eng. 50, 848–854 (2003).
Lecun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Liu, J. Adaptive myoelectric pattern recognition toward improved multifunctional prosthesis control. Med. Eng. Phys. 37, 424–430 (2015).
Diehl, C. P. & Cauwenberghs, G. SVM incremental learning, adaptation and optimization. In Proc. International Joint Conference on Neural Networks Vol. 4, 2685–2690 (IEEE, 2003); https://doi.org/10.1109/IJCNN.2003.1223991
Zhu, X., Liu, J., Zhang, D., Sheng, X. & Jiang, N. Cascaded adaptation framework for fast calibration of myoelectric control. IEEE Trans. Neural Syst. Rehabil. Eng. 25, 254–264 (2017).
Zhang, H. et al. An adaptation strategy of using LDA classifier for EMG pattern recognition. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. 2013, 4267–4270 (2013).
Schmuck, M., Benini, L. & Rahimi, A. Hardware optimizations of dense binary hyperdimensional computing: rematerialization of hypervectors, binarized bundling and combinational associative memory. ACM J. Emerg. Technol. Comput. Syst. 15, 1550–4832 (2019).
Kanerva, P. Binary spatter-coding of ordered K-tuples. in Artificial Neural Networks — ICANN 96 (eds von der Malsburg, C. et al.) 869–873 (Lecture Notes in Computer Science Vol. 1112, Springer, 1996); https://doi.org/10.1007/3-540-61510-5_146
We thank E. Alon, A. Araujo, R. Muller, K. Lutz, H. Wu, M. Sadeghi, Cortera Neurotechnologies and Novacentrix. This work was supported in part by the CONIX Research Center, one of six centres in JUMP, a Semiconductor Research Corporation (SRC) programme sponsored by DARPA. This material is based, in part, on research sponsored by the Air Force Research Laboratory under agreement no. FA8650-15-2-5401, as conducted through the flexible hybrid electronics manufacturing innovation institute, NextFlex. The US Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of Air Force Research Laboratory or the US Government. This research used the Savio computational cluster resource provided by the Berkeley Research Computing program at the University of California, Berkeley. Support was also received from sponsors of Berkeley Wireless Research Center, NSF Graduate Research Fellowship under grant no. 1106400, ETH Zurich Postdoctoral Fellowship programme and the Marie Curie Actions for People COFUND Program.
The authors declare no competing interests.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Notes 1–5, Figs. 1–7 and Tables 1–5.
Electrode array fabrication process, wearing the device, training and inference for hand gestures in action.
Rights and permissions
About this article
Cite this article
Moin, A., Zhou, A., Rahimi, A. et al. A wearable biosensing system with in-sensor adaptive machine learning for hand gesture recognition. Nat Electron 4, 54–63 (2021). https://doi.org/10.1038/s41928-020-00510-8
This article is cited by
Stretchable e-skin and transformer enable high-resolution morphological reconstruction for soft robots
Nature Machine Intelligence (2023)
A skin sensor that can rapidly recognize hand-based tasks with limited training
Nature Electronics (2023)
Cloud-based smartphone-assisted chemiluminescent assay for rapid screening of electroactive bacteria
Science China Technological Sciences (2023)
Machine Learning-Enhanced Flexible Mechanical Sensing
Nano-Micro Letters (2023)
Scorpion-inspired dual-bionic, microcrack-assisted wrinkle based laser induced graphene-silver strain sensor with high sensitivity and broad working range for wireless health monitoring system
Nano Research (2023)