Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Real-time finger motion recognition using skin-conformable electronics

Nature Electronics volume 6, pages 619–629 (2023)Cite this article

Subjects

Abstract

Interpreting and tracking finger motion in free space is of use in the development of control interfaces for augmented and virtual reality systems. One approach to create human–machine interfaces capable of accurate finger motion recognition is to use wearable sensors with integrated neuromorphic computing. Here we show that an integrated titanium-oxide-based artificial synapse array and organic motion sensor can be conformably attached to a finger and provide real-time motion recognition. The synaptic device and sensor exhibit well-defined synaptic and light-responsive electrical properties, respectively, as well as flexibility and mechanical robustness. The integrated synapses–sensor enables optical–electrical signal conversion and summation of post-synaptic current. Finger motions for time-resolved digit patterns (0–9) can be learned and recognized with an accuracy of up to 95% at varying strains and up to 100 strain cycles.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: TOAS device setup, imaging and initial characterization.
Fig. 2: Evaluation of TOAS device.
Fig. 3: Characterization of OPS and principles of OPS–TOAS devices.
Fig. 4: Finger-writing motion recognition and learning capability.

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from the corresponding authors upon reasonable request.

References

  1. Nguyen, H. T. & Kendon, A. Gesture: visible action as utterance. Crit. Inq. Lang. Stud. 5, 72–77 (2008).

    Article  Google Scholar 

  2. Morris, D. Gestures: Their Origins and Distribution (Jonathan Cape, 1979).

  3. Postolache, O. et al. Remote monitoring of physical rehabilitation of stroke patients using IoT and virtual reality. IEEE J. Sel. Areas Commun. 39, 562–573 (2021).

    Article  Google Scholar 

  4. Jung, W. & Lee, H. G. Energy–accuracy aware finger gesture recognition for wearable IoT devices. Sensors 22, 4801 (2022).

    Article  Google Scholar 

  5. Moin, A. et al. A wearable biosensing system with in-sensor adaptive machine learning for hand gesture recognition. Nat. Electron. 4, 54–63 (2021).

    Article  Google Scholar 

  6. Alam, M. S. et al. Trajectory-based air-writing recognition using deep neural network and depth sensor. Sensors 20, 376 (2020).

    Article  Google Scholar 

  7. Zhou, Z. et al. Sign-to-speech translation using machine-learning-assisted stretchable sensor arrays. Nat. Electron. 3, 571–578 (2020).

    Article  Google Scholar 

  8. Linderman, M., Lebedev, M. A. & Erlichman, J. S. Recognition of handwriting from electromyography. PLoS ONE 4, e6791 (2009).

    Article  Google Scholar 

  9. Mystakidis, S. Metaverse. Encyclopedia 2, 486–497 (2022).

    Article  Google Scholar 

  10. Zhu, M. et al. Haptic-feedback smart glove as a creative human-machine interface (HMI) for virtual/augmented reality applications. Sci. Adv. 6, eaaz8693 (2020).

    Article  Google Scholar 

  11. Shneiderman, B. Direct manipulation: a step beyond programming languages. Computer 16, 57–69 (1983).

    Article  Google Scholar 

  12. Sinha, G., Shahi, R. & Shankar, M. Human computer interaction. In Proc. 2010 3rd International Conference on Emerging Trends in Engineering and Technology 1–4 (IEEE Computer Society, 2010).

  13. Park, Y. & Bae, J. A three-dimensional finger motion measurement system of a thumb and an index finger without a calibration process. Sensors 20, E756 (2020).

    Article  Google Scholar 

  14. Joseph, V., Talpade, A., Suvarna, N. & Mendonca, Z. Visual gesture recognition for text writing in air. In 2018 Second International Conference on Intelligent Computing and Control Systems (ICICCS) 23–26 (IEEE, 2018).

  15. Yana, B. & Onoye, T. Real-time air-writing recognition in motion stream. In International Workshop on Advanced Image Technology (IWAIT) 2019 11049, 69–74 (SPIE, 2019).

  16. Amma, C., Georgi, M. & Schultz, T. Airwriting: hands-free mobile text input by spotting and continuous recognition of 3D-space handwriting with inertial sensors. In 2012 16th International Symposium on Wearable Computers 52–59 (IEEE, 2012).

  17. Guo, L., Lu, Z. & Yao, L. Human-machine interaction sensing technology based on hand gesture recognition: a review. IEEE Trans. Human-Mach. Syst. 51, 300–309 (2021).

    Article  Google Scholar 

  18. Oh, S. et al. Flexible artificial Si-In-Zn-O/ion gel synapse and its application to sensory-neuromorphic system for sign language translation. Sci. Adv. 7, eabg9450 (2021).

    Article  Google Scholar 

  19. Lee, Y. et al. Stretchable organic optoelectronic sensorimotor synapse. Sci. Adv. 4, eaat7387 (2018).

    Article  Google Scholar 

  20. Araromi, O. A. et al. Ultra-sensitive and resilient compliant strain gauges for soft machines. Nature 587, 219–224 (2020).

    Article  Google Scholar 

  21. Wang, M. et al. Gesture recognition using a bioinspired learning architecture that integrates visual data with somatosensory data from stretchable sensors. Nat. Electron. 3, 563–570 (2020).

    Article  Google Scholar 

  22. Sengupta, D., Mastella, M., Chicca, E. & Kottapalli, A. G. P. Skin-inspired flexible and stretchable electrospun carbon nanofiber sensors for neuromorphic sensing. ACS Appl. Electron. Mater. 4, 308–315 (2022).

    Article  Google Scholar 

  23. Kim, K. K. et al. A deep-learned skin sensor decoding the epicentral human motions. Nat. Commun. 11, 2149 (2020).

    Article  Google Scholar 

  24. Yamada, T. et al. A stretchable carbon nanotube strain sensor for human-motion detection. Nat. Nanotechnol. 6, 296–301 (2011).

    Article  Google Scholar 

  25. Lü, X. et al. High sensitivity flexible electronic skin based on graphene film. Sensors 19, 794 (2019).

    Article  Google Scholar 

  26. Zhao, J., Zhang, M., Wan, S., Yang, Z. & Hwang, C. S. Highly flexible resistive switching memory based on the electronic switching mechanism in the Al/TiO2/Al/polyimide structure. ACS Appl. Mater. Interfaces 10, 1828–1835 (2018).

    Article  Google Scholar 

  27. Molina-Lopez, F. et al. Inkjet-printed stretchable and low voltage synaptic transistor array. Nat. Commun. 10, 2676 (2019).

    Article  Google Scholar 

  28. Yokota, T. et al. Ultraflexible organic photonic skin. Sci. Adv. 2, e1501856 (2016).

    Article  Google Scholar 

  29. Jeong, J.-W. et al. Materials and optimized designs for human-machine interfaces via epidermal electronics. Adv. Mater. 25, 6839–6846 (2013).

    Article  Google Scholar 

  30. Yang, J. J., Strukov, D. B. & Stewart, D. R. Memristive devices for computing. Nat. Nanotechnol. 8, 13–24 (2013).

    Article  Google Scholar 

  31. Jang, J. et al. A learning-rate modulable and reliable TiOx memristor array for robust, fast, and accurate neuromorphic computing. Adv. Sci. 9, 2201117 (2022).

    Article  Google Scholar 

  32. Kim, S. H. et al. A bioinspired stretchable sensory-neuromorphic system. Adv. Mater. 33, 2104690 (2021).

    Article  Google Scholar 

  33. Abbott, L. F. & Regehr, W. G. Synaptic computation. Nature 431, 796–803 (2004).

    Article  Google Scholar 

  34. Neves, G., Cooke, S. F. & Bliss, T. V. P. Synaptic plasticity, memory and the hippocampus: a neural network approach to causality. Nat. Rev. Neurosci. 9, 65–75 (2008).

    Article  Google Scholar 

  35. Kim, D.-H. et al. Epidermal electronics. Science 333, 838–843 (2011).

    Article  Google Scholar 

  36. Liu, Y., Pharr, M. & Salvatore, G. A. Lab-on-skin: a review of flexible and stretchable electronics for wearable health monitoring. ACS Nano 11, 9614–9635 (2017).

    Article  Google Scholar 

  37. Kim, S., Lim, M., Kim, Y., Kim, H.-D. & Choi, S.-J. Impact of synaptic device variations on pattern recognition accuracy in a hardware neural network. Sci. Rep. 8, 2638 (2018).

    Article  Google Scholar 

  38. Choi, S. et al. A self-rectifying TaOy/nanoporous TaOx memristor synaptic array for learning and energy-efficient neuromorphic systems. npg Asia Mater. 10, 1097–1106 (2018).

    Article  Google Scholar 

  39. Cho, B., Song, S., Ji, Y., Kim, T.-W. & Lee, T. Organic resistive memory devices: performance enhancement, integration, and advanced architectures. Adv. Funct. Mater. 21, 2806–2829 (2011).

    Article  Google Scholar 

  40. Pang, C. L. Strain and stress effects on single crystal-supported titania and related nanostructures. Semicond. Sci. Technol. 35, 113001 (2020).

    Article  Google Scholar 

  41. Gao, W., Ota, H., Kiriya, D., Takei, K. & Javey, A. Flexible electronics toward wearable sensing. Acc. Chem. Res. 52, 523–533 (2019).

    Article  Google Scholar 

  42. He, Z. et al. Single-junction polymer solar cells with high efficiency and photovoltage. Nat. Photon. 9, 174–179 (2015).

    Article  Google Scholar 

  43. Hall, R. N. Electron-hole recombination in germanium. Phys. Rev. 87, 387–387 (1952).

    Article  Google Scholar 

  44. Shockley, W. & Read, W. T. Statistics of the recombinations of holes and electrons. Phys. Rev. 87, 835–842 (1952).

    Article  MATH  Google Scholar 

  45. Park, S. et al. Ultraflexible near-infrared organic photodetectors for conformal photoplethysmogram sensors. Adv. Mater. 30, 1802359 (2018).

    Article  Google Scholar 

  46. Fukuhara, T., Tamai, Y. & Ohkita, H. Nongeminate charge recombination in organic photovoltaics. Sustain. Energy Fuels 4, 4321–4351 (2020).

    Article  Google Scholar 

  47. Proctor, C. M., Kuik, M. & Nguyen, T.-Q. Charge carrier recombination in organic solar cells. Prog. Polym. Sci. 38, 1941–1960 (2013).

    Article  Google Scholar 

  48. Liu, X. et al. Regular organic solar cells with efficiency over 10% and promoted stability by ligand- and thermal annealing-free Al-doped ZnO cathode interlayer. Adv. Sci. 4, 1700053 (2017).

    Article  Google Scholar 

  49. Lee, H. et al. Stretchable organic optoelectronic devices: design of materials, structures, and applications. Mater. Sci. Eng. R. Rep. 146, 100631 (2021).

    Article  Google Scholar 

  50. Fukuda, K., Yu, K. & Someya, T. The future of flexible organic solar cells. Adv. Energy Mater. 10, 2000765 (2020).

    Article  Google Scholar 

  51. Wu, H. & Gu, X. Towards dropout training for convolutional neural networks. Neural Netw. 71, 1–10 (2015).

    Article  Google Scholar 

  52. Karystinos, G. N. & Pados, D. A. On overfitting, generalization, and randomly expanded training sets. IEEE Trans. Neural Netw. 11, 1050–1057 (2000).

    Article  Google Scholar 

  53. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014).

    MathSciNet  MATH  Google Scholar 

  54. Vincent, P., Larochelle, H., Bengio, Y. & Manzagol, P.-A. Extracting and composing robust features with denoising autoencoders. In Proc. 25th International Conference on Machine Learning 1096–1103 (ACM, 2008).

  55. Lotfi, A. & Benyettou, A. Over-fitting avoidance in probabilistic neural networks. In 2015 World Congress on Information Technology and Computer Applications (WCITCA) 1–6 (IEEE, 2015).

  56. Park, S. et al. Self-powered ultra-flexible electronics via nano-grating-patterned organic photovoltaics. Nature 561, 516–521 (2018).

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Research Foundation of Korea (NRF-2022R1A2B5B02001455, NRF-2022M3H4A1A01009526, NRF-2022K2A9A1A01098066, RS-2023-00220077, RS-2023-00213089 and 2022R1I1A1A01073911); the KU-KIST Research Fund, the Korea University Research Grant, the MSIT, Korea, under the ITRC (Information Technology Research Center) support program (grant no. IITP-2023-2020-0-01461) supervised by the Institute for Information & Communications Technology Planning & Evaluation (IITP); and the Technology Innovation Program (grant no. RS-2022-00154781) funded by the Ministry of Trade, Industry & Energy (MOTIE, Korea). We would like to express our gratitude to B.-G. Lee from Gwangju Institute of Science and Technology and S. Gi from Korea Electronics Technology Institute for their valuable contributions in reviewing and providing insightful feedback on circuit designs.

Author information

Author notes

  1. These authors contributed equally: Haein Cho, Inho Lee, Jingon Jang.

Authors and Affiliations

  1. KU-KIST Graduate School of Converging Science and Technology, Korea University, Seoul, Republic of Korea

    Haein Cho, Jingon Jang & Gunuk Wang

  2. Department of Intelligence Semiconductor Engineering, Ajou University, Suwon, Republic of Korea

    Inho Lee, Jae-Hyun Kim & Sungjun Park

  3. Department of Electrical and Computer Engineering, Ajou University, Suwon, Republic of Korea

    Hanbee Lee & Sungjun Park

  4. Department of Integrative Energy Engineering, Korea University, Seoul, Republic of Korea

    Gunuk Wang

  5. Center for Neuromorphic Engineering, Korea Institute of Science and Technology, Seoul, Republic of Korea

    Gunuk Wang

Authors
  1. Haein Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Inho Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jingon Jang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jae-Hyun Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hanbee Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sungjun Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Gunuk Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

G.W. and S.P. conceived the idea for this work. H.C., I.L., J.J., S.P. and G.W. designed the experiments and wrote the manuscript. H.C. fabricated the TOAS sample and conducted the related electrical and mechanical experiments. I.L., H.L. and J.-H.K. fabricated the ultrathin substrate and OPS sample and conducted the related experiments, including acquiring the digit patterns. J.J. conducted the neural network simulations. G.W. and S.P. oversaw the project, revised the manuscript and led the effort to completion.

Corresponding authors

Correspondence to Sungjun Park or Gunuk Wang.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Electronics thanks Tse Nga Ng and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Notes 1–5, Figs. 1–43, Tables 1 and 2 and references.

Supplementary Video 1

Detachment of the ultrathin TOAS array device from glass support.

Supplementary Video 2

Ultraflexibility of the TOAS array device.

Supplementary Video 3

Transfer of TOAS array device on the elastomer.

Supplementary Video 4

Mechanical endurance test of the ultraflexible TOAS array device.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cho, H., Lee, I., Jang, J. et al. Real-time finger motion recognition using skin-conformable electronics. Nat Electron 6, 619–629 (2023). https://doi.org/10.1038/s41928-023-01012-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41928-023-01012-z

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing