Wearable devices that monitor muscle activity based on surface electromyography could be of use in the development of hand gesture recognition applications. Such devices typically use machine-learning models, either locally or externally, for gesture classification. However, most devices with local processing cannot offer training and updating of the machine-learning model during use, resulting in suboptimal performance under practical conditions. Here we report a wearable surface electromyography biosensing system that is based on a screen-printed, conformal electrode array and has in-sensor adaptive learning capabilities. Our system implements a neuro-inspired hyperdimensional computing algorithm locally for real-time gesture classification, as well as model training and updating under variable conditions such as different arm positions and sensor replacement. The system can classify 13 hand gestures with 97.12% accuracy for two participants when training with a single trial per gesture. A high accuracy (92.87%) is preserved on expanding to 21 gestures, and accuracy is recovered by 9.5% by implementing model updates in response to varying conditions, without additional computation on an external device.
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
The source code used for offline model validation, in-sensor implementation and analysis of results is available at https://github.com/flexemg/flexemg_natelec.
Kim, J., Campbell, A. S., de Ávila, B. E. F. & Wang, J. Wearable biosensors for healthcare monitoring. Nat. Biotechnol. 37, 389–406 (2019).
Bandodkar, A. J., Jeerapan, I. & Wang, J. Wearable chemical sensors: present challenges and future prospects. ACS Sens. 1, 464–482 (2016).
Khan, Y., Ostfeld, A. E., Lochner, C. M., Pierre, A. & Arias, A. C. Monitoring of vital signs with flexible and wearable medical devices. Adv. Mater. 28, 4373–4395 (2016).
Patel, S., Park, H., Bonato, P., Chan, L. & Rodgers, M. A review of wearable sensors and systems with application in rehabilitation. J. Neuroeng. Rehabil. 9, 21 (2012).
Nakata, S. et al. A wearable pH sensor with high sensitivity based on a flexible charge-coupled device. Nat. Electron. 1, 596–603 (2018).
Huang, Z. et al. Three-dimensional integrated stretchable electronics. Nat. Electron. 1, 473–480 (2018).
Gao, W. et al. Fully integrated wearable sensor arrays for multiplexed in-situ perspiration analysis. Nature 529, 509–514 (2016).
Imani, S. et al. A wearable chemical-electrophysiological hybrid biosensing system for real-time health and fitness monitoring. Nat. Commun. 7, 11650 (2016).
Lochner, C. M., Khan, Y., Pierre, A. & Arias, A. C. All-organic optoelectronic sensor for pulse oximetry. Nat. Commun. 5, 5745 (2014).
Khan, Y. et al. A flexible organic reflectance oximeter array. Proc. Natl Acad. Sci. USA 115, E11015–E11024 (2018).
Benatti, S. et al. A versatile embedded platform for EMG acquisition and gesture recognition. IEEE Trans. Biomed. Circuits Syst. 9, 620–630 (2015).
Liu, X. et al. The virtual trackpad: an electromyography-based, wireless, real-time, low-power, embedded hand-gesture-recognition system using an event-driven artificial neural network. IEEE Trans. Circuits Syst. II 64, 1257–1261 (2017).
Gu, Y., Yang, D., Huang, Q., Yang, W. & Liu, H. Robust EMG pattern recognition in the presence of confounding factors: features, classifiers and adaptive learning. Expert Syst. Appl. 96, 208–217 (2018).
Young, A. J., Hargrove, L. J. & Kuiken, T. A. The effects of electrode size and orientation on the sensitivity of myoelectric pattern recognition systems to electrode shift. IEEE Trans. Biomed. Eng. 58, 2537–2544 (2011).
Zhang, X. & Huang, H. A real-time, practical sensor fault-tolerant module for robust EMG pattern recognition. J. Neuroeng. Rehabil. 12, 18 (2015).
Hargrove, L., Englehart, K. & Hudgins, B. A training strategy to reduce classification degradation due to electrode displacements in pattern recognition based myoelectric control. Biomed. Signal Process. Control 3, 175–180 (2008).
Tkach, D., Huang, H. & Kuiken, T. A. Study of stability of time-domain features for electromyographic pattern recognition. J. Neuroeng. Rehabil. 7, 21 (2010).
Milosevic, B., Farella, E. & Benatti, S. Exploring arm posture and temporal variability in myoelectric hand gesture recognition. Proc. 7th IEEE Int. Conf. on Biomedical Robotics and Biomechatronics 2018, 1032–1037 (2018).
Krishnan, K. S., Saha, A., Ramachandran, S. & Kumar, S. Recognition of human arm gestures using Myo armband for the game of hand cricket. In IEEE International Symposium on Robotics and Intelligent Sensors (IRIS) 389–394 (IEEE, 2017); https://doi.org/10.1109/IRIS.2017.8250154
Ploengpit, Y. & Phienthrakul, T. Rock–paper–scissors with Myo armband pose detection. In 2016 International Computer Science and Engineering Conference 1–5 (IEEE, 2017).
Nyomen, K., Romarheim Haugen, M. & Jensenius, A. R. MuMYO—evaluating and exploring the MYO armband for musical interaction. In Proc. International Conference on New Interfaces For Musical Expression 215–218 (Louisiana State Univ., 2015).
Abreu, J. G., Teixeira, J. M., Figueiredo, L. S. & Teichrieb, V. Evaluating sign language recognition using the Myo armband. In 2016 XVIII Symposium on Virtual and Augmented Reality 64–70 (IEEE, 2016).
Morais, G. D., Neves, L. C., Masiero, A. A. & Castro, M. C. F. Application of Myo armband system to control a robot interface. In Proc. BIOSTEC 2016 227–231 (2016).
Xu, Y., Yang, C., Liang, P., Zhao, L. & Li, Z. Development of a hybrid motion capture method using MYO armband with application to teleoperation. In 2016 IEEE International Conference on Mechatronics and Automation 1179–1184 (IEEE, 2016).
Chen, X. & Wang, Z. J. Pattern recognition of number gestures based on a wireless surface EMG system. Biomed. Signal Process. Control 8, 184–192 (2013).
Amma, C., Krings, T., Böer, J. & Schultz, T. Advancing muscle–computer interfaces with high-density electromyography. In Proc. 33rd Annual ACM Conference on Human Factors in Computing Systems 929–938 (2015).
Geng, W. et al. Gesture recognition by instantaneous surface EMG images. Sci. Rep. 6, 36571 (2016).
Cerone, G. L., Botter, A. & Gazzoni, M. A modular, smart and wearable system for high density sEMG detection. IEEE Trans. Biomed. Eng. 66, 3371–3380 (2019).
Pancholi, S. & Joshi, A. M. Electromyography-based hand gesture recognition system for upper limb amputees. IEEE Sens. Lett. 3, 1–4 (2019).
Benatti, S. et al. Online learning and classification of EMG-based gestures on a parallel ultra-low power platform using hyperdimensional computing. IEEE Trans. Biomed. Circuits Syst. 13, 516–528 (2019).
Kanerva, P. Hyperdimensional computing: an introduction to computing in distributed representation with high-dimensional random vectors. Cogn. Comput. 1, 139–159 (2009).
Moin, A. et al. An EMG gesture recognition system with flexible high-density sensors and brain-inspired high-dimensional classifier. In 2018 IEEE International Symposium on Circuits and Systems (ISCAS) 1–5 (IEEE, 2018); https://doi.org/10.1109/ISCAS.2018.8351613
Rahimi, A., Tchouprina, A., Kanerva, P., Millán, J. D. R. & Rabaey, J. M. Hyperdimensional computing for blind and one-shot classification of EEG error-related potentials. Mob. Netw. Appl. 25, 1958–1969 (2020).
Burrello, A., Schindler, K., Benini, L. & Rahimi, A. One-shot learning for iEEG seizure detection using end-to-end binary operations: local binary patterns with hyperdimensional computing. In Proc. 2018 IEEE Biomedical Circuits and Systems Conference (BioCAS) 1–4 (IEEE, 2018); https://doi.org/10.1109/BIOCAS.2018.8584751
Johnson, B. C. et al. An implantable 700-μW 64-channel neuromodulation IC for simultaneous recording and stimulation with rapid artifact recovery. In IEEE 2017 Symposium on VLSI Circuits, Digest of Technical Papers C48–C49 (IEEE, 2017); https://doi.org/10.23919/VLSIC.2017.8008543
Cui, H. W. et al. Ultra-fast photonic curing of electrically conductive adhesives fabricated from vinyl ester resin and silver micro-flakes for printed electronics. RSC Adv. 4, 15914–15922 (2014).
Lapatki, B. G., Van Dijk, J. P., Jonas, I. E., Zwarts, M. J. & Stegeman, D. F. A thin, flexible multielectrode grid for high-density surface EMG. J. Appl. Physiol. 96, 327–336 (2004).
Dick, F. S., Bert, U. K., Bernd, G. L. & Johannes, P. V. D. High-density surface EMG: techniques and applications at a motor unit level. Biocybern. Biomed. Eng. 32, 3–27 (2012).
Atzori, M. et al. Electromyography data for non-invasive naturally-controlled robotic hand prostheses. Sci. Data 1, 1–13 (2014).
Pizzolato, S. et al. Comparison of six electromyography acquisition setups on hand movement classification tasks. PLoS ONE 12, e0186132 (2017).
Atzori, M. et al. Building the NinaPro database: a resource for the biorobotics community. In Proc. 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob) 1258–1265 (IEEE, 2012); https://doi.org/10.1109/BioRob.2012.6290287
Du, Y., Jin, W., Wei, W., Hu, Y. & Geng, W. Surface EMG-based inter-session gesture recognition enhanced by deep domain adaptation. Sensors 17, 6–9 (2017).
Rahimi, A. et al. High-dimensional computing as a nanoscalable paradigm. IEEE Trans. Circuits Syst. 64, 2508–2521 (2017).
Smith, L. H., Hargrove, L. J., Lock, B. A. & Kuiken, T. A. Determining the optimal window length for pattern recognition-based myoelectric control: balanÿcing the competing effects of classification error and controller delay. IEEE Trans. Neural Syst. Rehabil. Eng. 19, 186–192 (2011).
Englehart, K. & Hudgins, B. A robust, real-time control scheme for multifunction myoelectric control. IEEE Trans. Biomed. Eng. 50, 848–854 (2003).
Lecun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Liu, J. Adaptive myoelectric pattern recognition toward improved multifunctional prosthesis control. Med. Eng. Phys. 37, 424–430 (2015).
Diehl, C. P. & Cauwenberghs, G. SVM incremental learning, adaptation and optimization. In Proc. International Joint Conference on Neural Networks Vol. 4, 2685–2690 (IEEE, 2003); https://doi.org/10.1109/IJCNN.2003.1223991
Zhu, X., Liu, J., Zhang, D., Sheng, X. & Jiang, N. Cascaded adaptation framework for fast calibration of myoelectric control. IEEE Trans. Neural Syst. Rehabil. Eng. 25, 254–264 (2017).
Zhang, H. et al. An adaptation strategy of using LDA classifier for EMG pattern recognition. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. 2013, 4267–4270 (2013).
Schmuck, M., Benini, L. & Rahimi, A. Hardware optimizations of dense binary hyperdimensional computing: rematerialization of hypervectors, binarized bundling and combinational associative memory. ACM J. Emerg. Technol. Comput. Syst. 15, 1550–4832 (2019).
Kanerva, P. Binary spatter-coding of ordered K-tuples. in Artificial Neural Networks — ICANN 96 (eds von der Malsburg, C. et al.) 869–873 (Lecture Notes in Computer Science Vol. 1112, Springer, 1996); https://doi.org/10.1007/3-540-61510-5_146
We thank E. Alon, A. Araujo, R. Muller, K. Lutz, H. Wu, M. Sadeghi, Cortera Neurotechnologies and Novacentrix. This work was supported in part by the CONIX Research Center, one of six centres in JUMP, a Semiconductor Research Corporation (SRC) programme sponsored by DARPA. This material is based, in part, on research sponsored by the Air Force Research Laboratory under agreement no. FA8650-15-2-5401, as conducted through the flexible hybrid electronics manufacturing innovation institute, NextFlex. The US Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of Air Force Research Laboratory or the US Government. This research used the Savio computational cluster resource provided by the Berkeley Research Computing program at the University of California, Berkeley. Support was also received from sponsors of Berkeley Wireless Research Center, NSF Graduate Research Fellowship under grant no. 1106400, ETH Zurich Postdoctoral Fellowship programme and the Marie Curie Actions for People COFUND Program.
The authors declare no competing interests.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Moin, A., Zhou, A., Rahimi, A. et al. A wearable biosensing system with in-sensor adaptive machine learning for hand gesture recognition. Nat Electron (2020). https://doi.org/10.1038/s41928-020-00510-8
Nature Electronics (2021)