Abstract
Haptic feedback can be used to improve immersion in virtual and augmented reality systems. However, wearable haptic devices typically provide limited feedback, such as simple vibration, and are restricted to a narrow bandwidth of tuneable frequency or to single tactile perceptions. Here, we report a skin-integrated wireless haptic interface that is based on actuator arrays and offers multimodal and complex feedback mechanisms. Different feedback modes—mechanical, electrotactile and thermal—in the interface are used to selectively activate different cutaneous receptors, providing users with diverse haptic sensations. The selective stimulating mechanism is inspired by the neural tactile sensing system and can reproduce tactile information of fine roughness, macro roughness, slipperiness, force and temperature.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The data that support the plots within this paper and other findings of this study are available from the corresponding authors on reasonable request.
Code availability
Custom code used in the study is available from the corresponding authors upon reasonable request.
Change history
13 December 2023
A Correction to this paper has been published: https://doi.org/10.1038/s41928-023-01115-7
References
Keef, C. V. et al. Virtual texture generated using elastomeric conductive block copolymer in a wireless multimodal haptic glove. Adv. Intell. Syst. 2, 2000018 (2020).
Flesher, S. N. et al. A brain-computer interface that evokes tactile sensations improves robotic arm control. Science 372, 831 (2021).
Zhu, M. et al. Haptic-feedback smart glove as a creative human-machine interface (HMI) for virtual/augmented reality applications. Sci. Adv. 6, eaaz8693 (2020).
Steed, A., Friston, S., Pawar, V. & Swapp, D. Docking haptics: extending the reach of haptics by dynamic combinations of grounded and worn devices. In Proc. 26th ACM Symposium on Virtual Reality Software and Technology Article 2 (ACM, 2020).
Ilkhani, G., Aziziaghdam, M. & Samur, E. Data-driven texture rendering on an electrostatic tactile display. Int. J. Hum. Comput. Interact. 33, 756–770 (2017).
De Paolis, L. T. & De Luca, V. The impact of the input interface in a virtual environment: the Vive controller and the Myo armband. Virtual Real. 24, 483–502 (2020).
Yu, X. et al. Skin-integrated wireless haptic interfaces for virtual and augmented reality. Nature 575, 473–479 (2019).
Jung, Y. H., Kim, J.-H. & Rogers, J. A. Skin-integrated vibrohaptic interfaces for virtual and augmented reality. Adv. Funct. Mater. 31, 2008805 (2020).
Yang, T.-H. et al. Recent advances and opportunities of active materials for haptic technologies in virtual and augmented reality. Adv. Funct. Mater. 31, 2008831 (2021).
Augustine, G. J. et al. The Somatic Sensory System (eds Dale Purves et al.). In Neuroscience. Ch. 8, 189–208 (Sinauer Associates, Inc., 2004).
Vallbo, A. B. & Johansson, R. S. Properties of cutaneous mechanoreceptors in the human hand related to touch sensation. Hum. Neurobiol 3, 3–14 (1984).
Saal, H. P., Delhaye, B. P., Rayhaun, B. C. & Bensmaia, S. J. Simulating tactile signals from the whole hand with millisecond precision. Proc. Natl Acad. Sci. USA 114, E5693 (2017).
Jin, H. et al. Highly pixelated, untethered tactile interfaces for an ultra-flexible on-skin telehaptic system. NPJ Flex. Electron. 6, 82 (2022).
Yem, V. & Kajimoto, H. Wearable tactile device using mechanical and electrical stimulation for fingertip interaction with virtual world. In Proc. 2017 IEEE Virtual Reality (VR) 99–104 (IEEE, 2017).
Yin, J., Hinchet, R., Shea, H. & Majidi, C. Wearable soft technologies for haptic sensing and feedback. Adv. Funct. Mater. 31, 2007428 (2020).
You, I. et al. Artificial multimodal receptors based on ion relaxation dynamics. Science 370, 961–965 (2020).
Lee, J., Kim, D., Sul, H. & Ko, S. H. Thermo-haptic materials and devices for wearable virtual and augmented reality. Adv. Funct. Mater. 31, 2007376 (2021).
Hong, S. et al. Wearable thermoelectrics for personalized thermoregulation. Sci. Adv. 5, eaaw0536 (2019).
Nam, J., Kim, Y. & Jang, G. Resonant piezoelectric vibrator with high displacement at haptic frequency for smart devices. IEEE ASME Trans. Mechatron. 21, 394–401 (2016).
Pyo, D., Yang, T.-H., Ryu, S. & Kwon, D.-S. Novel linear impact-resonant actuator for mobile applications. Sens. Actuator A Phys. 233, 460–471 (2015).
Jung, Y. H. et al. A wireless haptic interface for programmable patterns of touch across large areas of the skin. Nat. Electron. 5, 374–385 (2022).
Delazio, A. et al. Force jacket: pneumatically-actuated jacket for embodied haptic experiences. In Proc. 2018 CHI Conference on Human Factors in Computing Systems Paper 320 (Association for Computing Machinery, 2018).
Sonar, H. A. & Paik, J. Soft pneumatic actuator skin with piezoelectric sensors for vibrotactile feedback. Front. Robot. AI 2, 38 (2016).
Yoshimoto, S., Kuroda, Y., Kagiyama, Y., Kuroda, T. & Oshiro, O. Tactile mapping approach using electrical stimulus pattern. In Proc. RO-MAN 2009. The 18th IEEE International Symposium on Robot and Human Interactive Communication 460–465 (IEEE, 2009).
Chun, S. et al. An artificial neural tactile sensing system. Nat. Electron. 4, 429–438 (2021).
Tsai, D. R. & Hsu, W. A new type of a large-area multi-touch tactile device operated by electrotactile stimulation. In Proc. 2019 IEEE World Haptics Conference (WHC) 313–318 (IEEE, 2019).
Yao, K. et al. Encoding of tactile information in hand via skin-integrated wireless haptic interface. Nat. Mach. Intell. 4, 893–903 (2022).
Kajimoto, H., Kawakami, N. & Tachi, S. Psychophysical evaluation of receptor selectivity in electro-tactile display. In Proc. of 13th International Symposium on Measurement and Control in Robotics (ISMCR). 13, 83–86 (IEEE, 2003).
Kaczmarek, K. A., Webster, J. G., Bach-y-Rita, P. & Tompkins, W. J. Electrotactile and vibrotactile displays for sensory substitution systems. IEEE Trans. Biomed. Eng. 38, 1–16 (1991).
Shi, Y. et al. Self-powered electro-tactile system for virtual tactile experiences. Sci. Adv. 7, eabe2943 (2021).
Xu, B. et al. An epidermal stimulation and sensing platform for sensorimotor prosthetic control, management of lower back exertion, and electrical muscle activation. Adv. Mater. 28, 4462–4471 (2016).
Zárate, J. J., Tosolini, G., Petroni, S., De Vittorio, M. & Shea, H. Optimization of the force and power consumption of a microfabricated magnetic actuator. Sens. Actuator A Phys. 234, 57–64 (2015).
Li, D. et al. Miniaturization of mechanical actuators in skin-integrated electronics for haptic interfaces. Microsyst. Nanoeng. 7, 85 (2021).
Huang, Y. et al. Recent advances in multi-mode haptic feedback technologies towards wearable interfaces. Mater. Today Phys. 22, 100602 (2021).
Yem, V. & Kajimoto, H. Comparative evaluation of tactile sensation by electrical and mechanical stimulation. IEEE Trans. Haptics 10, 130–134 (2016).
Yang, Y., Zhou, Y., Zeng, J., He, K. & Liu, H. Electrotactile stimulation waveform modulation based on a customized portable stimulator: a pilot study. In Proc. 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC) 1838–1843 (IEEE, 2019).
Kajimoto, H., Kawakami, N., Maeda, T., Tachi, S. Electro-tactile display with tactile primary color approach. In Proceedings of the 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems Vol. 2, 541–546 (IEEE, 2004).
Kenshalo, D. R., Holmes, C. E. & Wood, P. B. Warm and cool thresholds as a function of rate of stimulus temperature change. Percept. Psychophys. 3, 81–84 (1968).
Konyo, M., Tadokoro, S., Yoshida, A. & Saiwaki, N. A tactile synthesis method using multiple frequency vibrations for representing virtual touch. In Proc. 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems 3965–3971 (IEEE, 2005).
Culbertson, H., Unwin, J. & Kuchenbecker, K. J. Modeling and rendering realistic textures from unconstrained tool-surface interactions. IEEE Trans. Haptics 7, 381–393 (2014).
Okamoto, S., Nagano, H. & Yamada, Y. Psychophysical dimensions of tactile perception of textures. IEEE Trans. Haptics 6, 81–93 (2013).
Jia, C. et al. Highly compressible and anisotropic lamellar ceramic sponges with superior thermal insulation and acoustic absorption performances. Nat. Commun. 11, 3732 (2020).
Acknowledgements
This work was supported by Research Grants Council of the Hong Kong Special Administrative Region (grant nos. 11215722, 11213721 and 11211523), City University of Hong Kong (grant nos. 9667221 and 9680322), National Natural Science Foundation of China (grant nos. 62122002, 12072057, 62172346 and 61907037), Shenzhen Science and Technology Innovation Commission (grant nos. SGDX20220530111401011 and JCYJ20200109110201713) and the Innovation and Technology Fund of Innovation and Technology Commission (grant no. GHP/095/20GD), as part of the InnoHK Project on Project 2.2—AI-based 3D ultrasound imaging algorithm at Hong Kong Centre for Cerebro-Cardiovascular Health Engineering, Guangdong Basic and Applied Basic Research Foundation (project no. 2021A1515011893) and the Centre for Applied Computing and Interactive Media of School of Creative Media, City University of Hong Kong; The CityU Contract Research RMGS (project no. 9239092), the CityU Donations for Research Projects RMGS (project no. 9229075); LiaoNing Revitalization Talents Programme (grant no. XLYC2007196) and the Dalian Outstanding Young Talents in Science and Technology (grant no. 2021RJ06). E.S. acknowledges support from the Shanghai Municipal Science and Technology Major Project (no. 2018SHZDZX01), the ZJ Laboratory and Shanghai Centre for Brain Science and Brain-Inspired Technology.
Author information
Authors and Affiliations
Contributions
X.Y. and Y.H. initiated the concept and proposed the project. Y.H. designed the whole system and collected the overall data. Y.H., J.Z., X.Y., Z.X. and Mr. X.G optimized the structure of feedback device. J.Z and Y.H. designed and characterized control circuits. Y.H., C.K.Y., D.L., K.Y., Y.G., Z.C. and Jiyu Li fabricated the feedback devices. P.K. and Y.H. set up the VR scenes. Y.H., J.Z., P.K., K.Z. and S.C. designed and organized user studies. Y.H., X.Y., S.C., P.K., K.Z. and K.Y. analysed the data obtained from user studies. X.G. and Z.X. led the mechanical, thermal, and electrical modelling and related finite element analysis. Prof. X.G. gave comments on mechanical modelling. Y.H., J.Z., P.K., S.C., Mr. X.G., Z.X., K.Z. and X.Y. wrote the manuscript. All other authors contributed to discussing the data and commenting on the final manuscript.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Electronics thanks Hye Jin Kim, Fang Yi and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 Thermal simulation of the temperature changes when finger touch different materials.
a, The schematic diagram of the thermal simulation model where the finger is simplified into a cylinder. The thermoreceptors are set at a depth of 0.4 mm under the skin. The temperature distribution around the contact region (b) and the thermoreceptors (c) for the finger (R = 8 mm) touching a 5 mm thick quartz glass base at 30 s, in which T1 = 30 °C, T2 = Te = 20 °C, L = 70 mm. d, Time-dependent temperature change curves when a finger (6 mm radius, 30 °C) touches a 2 cm-thick base with different materials (20 °C). e, Summary of the temperature changes in (d) at 5 s, 10 s and 30 s after touching. f, The average temperature changes (over the thermoreceptors) of a finger with 8 mm radius touching the base with different materials and thicknesses at 10 s. g, The average temperature changes (over the thermoreceptors) for the fingers with different radii touching a 5 mm-thick base with different materials at 10 s.
Extended Data Fig. 2 The simulation models for electrotactile.
a, Schematic diagram of the electrode geometries and the tissue layers used in simulation using two circular electrode. b, The distribution of current density around two electrodes with different currents values. c, Two positions chosen for analysis of current density distribution, directly under the electrode (red line) and in the middle of two electrodes (white line). d, The current density changes versus depth under the electrode and in the middle of two electrodes. e, Schematic diagram of the electrode geometries and the tissue layers used in simulation using a circular positive electrode and a rectangular negative electrode. f, The distribution of current density under the circular electrode with different distance L between positive and negative electrodes. g, One position directly under the ring electrode (red line) chosen for analysis of current density distribution. h, The current density changes versus depth under the electrode.
Extended Data Fig. 3 The wireless control circuits.
a, Circuit diagram of the wireless control circuits. Photograph of the bare control circuits (both front-view and back-view) (b) and encapsulated control circuits (both front-view and back-view) (c) as a wristband. The photo of (d) the bended circuits and (e) the encapsulated wristband worn on the arm.
Extended Data Fig. 4 Influence of the contact positions between the skin and device on the interface temperature.
a, The schematic diagram of the thermoelectric simulation model. b, The four contact positions (P1, P2, P3 and P4) between skin and the device. c, The cross section and geometrical dimensions of the device. The temperature distributions on the contact surface when the skin touches four positions with HTC = 0.3 mm (d) and HTC = 1.0 mm (e) (1 A, 20 s), respectively. f, The average temperature change over each contact position versus time. The unit of number in this figure is mm.
Extended Data Fig. 5 Simulation model of heat dissipation capability for the device mounted on skin.
a, Schematic diagram for the device attached on skin. b, The geometric dimensions and c, cross section of the thermoelectric part. d, The absolute average temperature and e, the relative average temperature of the surface contacted with skin at the position 4 with the input current 1 A in the cooling mode, respectively. f, The absolute average temperature and g, the relative average temperature of the surface contacted with skin at the position 4 with the input current 1 A in the cooling mode, respectively. The unit of number in this figure is mm.
Extended Data Fig. 6 Roughness simulation related with the desired surface and the finger speed.
a, The schematic diagram of the roughness simulation where finger processed on the desired surface with speed of v. b, The Von Mises Stress distribution when finger pressed on the surface composed of semi-circular SiC bulge (d = 269 µm) with pressure of 1kPa and sliding speed of 1 cm/s. c, The finger presses on three surfaces which composed of bulge of different curvature radius but has the same roughness with pressure of 1kPa and moving velocity of 1 cm/s. d, Stress curves of the finger feel when moving on three surfaces with same roughness but different morphology.
Extended Data Fig. 7 The VR indoor scenes.
The VR scene and actual scene of users touching the cover of a book (a), touching the wood desk (b), holding a glass cup (c) and pouring hot water in the glass cup (d). The temperature change of touching the book is set around -2 °C (e), touching the desk is set around -4 °C (f) and touching the glass bottle is set around -6 °C (g) in virtual world due to the simulation results and pre-user tests. The temperature first decreases due to the lower temperature of the bottle, but the temperature raises drastically (~ + 12 °C) once hot water is poured into the bottle (h). The mechanical stimulus obtained from sliding on the book is mainly fine roughness, and the mechanical stimulus obtained from the book is mainly the hardness, thus the pulse width of touching book (i) is higher than that of touching the desk (j) and the frequency of touching desk (m) is much higher than that of touching the book (l) for getting a pulse-undistinguished pressure. For holding the bottle, five fingers are pressed on the virtual object (k) and the electrotactile activated when holding the bottle is a feeling composed of vibration and pressure reported by the users (n).
Extended Data Fig. 8 The VR outdoor scenes.
The VR scene and actual scene of users experiencing light breeze (a), heavy rain (b), and light rain with her arm (c). The heavy rain leads to an obvious temperature decrease (e) than light breeze (d) and light rain (f). The breeze blows up the sleeves and the arm feel slight force (g). Raindrops drop on the arm and cause impacts on the arm. Due to the randomness of the location of the fall, the activated electrodes are always changing (h) (i). And the stimulation pulse of heavy rain (j) is set stronger and denser compared with that of light rain (k).
Extended Data Fig. 9 The similarity ratings of four virtual scenes.
The photograph of four scenes, a, S1: disinfections with 75% alcohol tissue, b, S2: pouring hot water into a glass bottle, c, S3: strong wind and d, S4: heavy rain, and corresponding average similarity levels in temperature, force and overall obtained from user study. e, Statistic histograms of similarity levels when users only got visual and auditory signals, only got haptic feedback, or got all visual, auditory signals and haptic feedback. Center line, median; bar height, mean; error bars, SD; n = 15 independent tests. Statistical significance was assessed using two-sided unpaired t-tests.
Supplementary information
Supplementary Information
Supplementary Texts 1–7, Figs. 1–31 and video legends/captions.
Supplementary Video 1
The integration of some VR scenes with our haptic feedback interface.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Huang, Y., Zhou, J., Ke, P. et al. A skin-integrated multimodal haptic interface for immersive tactile feedback. Nat Electron 6, 1020–1031 (2023). https://doi.org/10.1038/s41928-023-01074-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41928-023-01074-z
This article is cited by
-
Compliant Iontronic Triboelectric Gels with Phase-Locked Structure Enabled by Competitive Hydrogen Bonding
Nano-Micro Letters (2024)
-
A skin-like multimodal haptic interface
Nature Electronics (2023)
