Skin-integrated wireless haptic interfaces for virtual and augmented reality

Article metrics

Abstract

Traditional technologies for virtual reality (VR) and augmented reality (AR) create human experiences through visual and auditory stimuli that replicate sensations associated with the physical world. The most widespread VR and AR systems use head-mounted displays, accelerometers and loudspeakers as the basis for three-dimensional, computer-generated environments that can exist in isolation or as overlays on actual scenery. In comparison to the eyes and the ears, the skin is a relatively underexplored sensory interface for VR and AR technology that could, nevertheless, greatly enhance experiences at a qualitative level, with direct relevance in areas such as communications, entertainment and medicine1,2. Here we present a wireless, battery-free platform of electronic systems and haptic (that is, touch-based) interfaces capable of softly laminating onto the curved surfaces of the skin to communicate information via spatio-temporally programmable patterns of localized mechanical vibrations. We describe the materials, device structures, power delivery strategies and communication schemes that serve as the foundations for such platforms. The resulting technology creates many opportunities for use where the skin provides an electronically programmable communication and sensory input channel to the body, as demonstrated through applications in social media and personal engagement, prosthetic control and feedback, and gaming and entertainment.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Design and architecture of an epidermal VR system.
Fig. 2: Optimized operation of key electrical and mechanical components of an epidermal VR system.
Fig. 3: Wireless control strategies for epidermal VR systems.
Fig. 4: Examples of applications of epidermal VR systems.

Data availability

All data are contained within the manuscript. Raw data are available from the corresponding authors upon reasonable request.

Code availability

Custom code used in this study is available from the corresponding authors upon reasonable request.

References

  1. 1.

    Kapur, P., Jensen, M., Buxbaum, L. J., Jax, S. A. & Kuchenbecker, K. J. Spatially distributed tactile feedback for kinesthetic motion guidance. In 2010 IEEE Haptics Symp. (ed. Basodogan, C.) 519–526 (IEEE, 2010).

  2. 2.

    Sardini, E., Serpelloni, M. & Pasqui, V. Wireless wearable T-shirt for posture monitoring during rehabilitation exercises. IEEE Trans. Instrum. Meas. 64, 439–448 (2015).

  3. 3.

    Kim, Y. et al. A bioinspired flexible organic artificial afferent nerve. Science 360, 998–1003 (2018).

  4. 4.

    Novich, S. D. & Eagleman, D. M. Using space and time to encode vibrotactile information: toward an estimate of the skin’s achievable throughput. Exp. Brain Res. 233, 2777–2788 (2015).

  5. 5.

    Sundaram, S. et al. Learning the signatures of the human grasp using a scalable tactile glove. Nature 569, 698–702 (2019).

  6. 6.

    Withana, A., Groeger, D. & Steimle, R. Tacttoo: a thin and feel-through tattoo for on-skin tactile output. In Proc. 31st Ann. ACM Symp. on User Interface Software and Technology (ed. Wilson, A. D.) 365–378 (ACM, 2018).

  7. 7.

    Huang, C.-T., Shen, C.-L., Tang, C.-F. & Chang, S.-H. A wearable yarn-based piezo-resistive sensor. Sens. Actuators A 141, 396–403 (2008).

  8. 8.

    Cogan, S. F. Neural stimulation and recording electrodes. Annu. Rev. Biomed. Eng. 10, 275–309 (2008).

  9. 9.

    Song, Y. M. et al. Digital cameras with designs inspired by the arthropod eye. Nature 497, 95–99 (2013).

  10. 10.

    Rogers, J. A., Someya, T. & Huang, Y. Materials and mechanics for stretchable electronics. Science 327, 1603–1607 (2010).

  11. 11.

    Wilska, A. On the vibrational sensitivity in different regions of the body surface. Acta Physiol. Scand. 31, 285–289 (1954).

  12. 12.

    Mahns, D. A., Perkins, N. M., Sahai, V., Robinson, L. & Rowe, M. J. Vibrotactile frequency discrimination in human hairy skin. J. Neurophysiol. 95, 1442–1450 (2006).

  13. 13.

    Han, S. et al. Battery-free, wireless sensors for full-body pressure and temperature mapping. Sci. Transl. Med. 10, eaan4950 (2018).

  14. 14.

    Kim, D.-H. et al. Epidermal electronics. Science 333, 838–843 (2011).

  15. 15.

    Kim, J. et al. Battery-free, stretchable optoelectronic systems for wireless optical characterization of the skin. Sci. Adv. 2, e1600418 (2016).

  16. 16.

    Zhang, Y. et al. Experimental and theoretical studies of serpentine microstructures bonded to prestrained elastomers for stretchable electronics. Adv. Funct. Mater. 24, 2028–2037 (2014).

  17. 17.

    Huang, X. et al. Epidermal radio frequency electronics for wireless power transfer. Microsyst. Nanoeng. 2, 16052 (2016).

  18. 18.

    Chung, H. U. et al. Binodal, wireless epidermal electronic systems with in-sensor analytics for neonatal intensive care. Science 363, eaau0780 (2019).

  19. 19.

    Beh, T. C., Kato, M., Imura, T., Oh, S. & Hori, Y. Automated impedance matching system for robust wireless power transfer via magnetic resonance coupling. IEEE Trans. Ind. Electron. 60, 3689–3698 (2013).

  20. 20.

    Kim, J., Son, H., Kim, K. & Park, Y. Efficiency analysis of magnetic resonance wireless power transfer with intermediate resonant coil. IEEE Antennas Wirel. Propag. Lett. 10, 389–392 (2011).

  21. 21.

    The Electronic Code of Federal Regulations (e-CFR) Title 47, 1.1310. Radiofrequency radiation exposure limits. https://www.ecfr.gov/cgi-bin/retrieveECFR?gp=&SID=81a1c5a13f8d02f165a06af36937d612&mc=true&n=pt47.1.1&r=PART&ty=HTML#se47.1.1_11310 (e-CFR, 2019).

  22. 22.

    Akamatsu, M., Mackenzie, I. S. & Hasbroucq, T. A comparison of tactile, auditory, and visual feedback in a pointing task using a mouse-type device. Ergonomics 38, 816–827 (1995).

  23. 23.

    Lee, Y. RFID Coil Design (Microchip Technology Inc., 1998).

Download references

Acknowledgements

We gratefully acknolwedge C. J. Su, T. Banks, J. H. Kim, Y. G. Xue and J. K. Chang for their efforts in constructing and testing the optimized systems. This work was supported by the Center for Bio-Integrated Electronics at Northwestern University. Z.X. and X.F. acknowledge support from the National Basic Research Program of China (grant number 2015CB351900) and the National Natural Science Foundation of China (grant numbers 11320101001 and 11402134). X.Y., Z.X., D.L. and Y.L. acknowledge support from the City University of Hong Kong (grant number 9610423). R.S. acknowledges support from the Engineering and Physical Sciences Research Council (grant number EP/L016028/1) and the China Scholarship Council. Y.H. acknowledges support from the NSF (grant number 1635443).

Author information

X.Y., Z.X., Y.Y. and J.L. contributed equally to this work. X.Y, Z.X., H.Y. and J.A.R. conceived the ideas, designed the experiments and wrote the manuscript. X.Y., Z.X., J.L., J.R., A.V., H.L., X.N., A.A., D.L., B.J., R.S., Y.L., J.C., Q.H., C.L., S.K., P.G., C.Z., Y.Z., Y.X., Q.G., A.C., P.T., W.L., J.J., Y.Y., J.C., C.T., B.K., K.L., X.F. and J.C. performed experiments and analysed the experimental data. Z.X. led the structural designs and performed electromagnetic and mechanical modelling, with assistance from B.J., Q.H., C.Z. and Y.X.

Correspondence to Yonggang Huang or John A. Rogers.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Peer review information Nature thanks Sriram Subramanian, Xiaoming Tao and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Extended data figures and tables

Extended Data Fig. 1 Study of the magnetic field strength in an actuator.

a, Distribution of the normalized magnetic field strength (H) around the Cu coil (300 turns) used in the actuator, where d is the distance between the Cu coil and the magnet. The dashed white circles correspond to the holes in the Cu coil. b, Normalized magnetic flux (Φ/Φd=0) through the haptic actuator versus the distance to the Cu coil (300 turns) of the actuator. c, Schematic illustration of the distance between a haptic actuator and a Cu coil in the actuator. d, The normalized magnetic flux (Φ/ΦMax) through the magnet versus turn number and outer diameter (Douter) of the Cu coil.

Extended Data Fig. 2 Resonant frequency tuning of the actuator.

a, Top view and cross-sectional view of the actuator design. The parameters presented here are optimized for tuning the resonance frequency to the skin-sensitive range. b–d, Optical images and normalized amplitude–frequency curves of three actuators with different central angles (θ) of 150° (b), 186° (c) and 217° (d), working without any contact. The actuators shown in b and c are 18 mm in diameter and 2.5 mm thick.

Extended Data Fig. 3 Optimization of the actuator.

a, Cross-sectional schematic illustration of an actuator in contact with skin. b, Theoretical results of the resonance frequency of actuators shown in a versus the central angle θ of the PI handling layer. The dashed lines indicate the resonance frequency of 200 Hz at θ = 217°. c, Comparison of experimental (E; data symbols) and simulation (theory, T; lines) results. d, Experimental results of the normalized amplitude–frequency curves of the actuator (θ = 217°) in contact with skin, for different values of skin elastic modulus: 60 kPa, 130 kPa and 200 kPa. e, Travel amplitude of the magnet as a function of the input power (data points) for an actuator in contact with artificial skin samples with elastic moduli of 60 kPa, 130 kPa and 200 kPa. In c, e, error bars correspond to the calculated standard deviation.

Extended Data Fig. 4 Study of the mechanical behaviour of the actuator when in contact with skin.

a, b, Cross-sectional schematic illustration (a) and optical image (b) of an actuator in contact with artificial skin. Here PDMS, serving as artificial skin, had three different values of elastic modulus, and glass was used for artificial bone. c, Comparison of the experimental (E) and simulation (theory, T) results of the resonance frequency of the actuator in contact with different moduli and thicknesses of artificial skin. The error bars correspond to the calculated standard deviation. d, Optical images captured using a high-speed camera and a working actuator travelling up and down, when in contact with 130-kPa artificial skin. e, FEA results for the amplitude of an actuator: left, when separated from skin; right, when in contact with skin. f, Schematic illustration and FEA results (colour-coded amplitude) for mechanical coupling between an array of haptic actuators, with activation in an ‘N’ pattern.

Extended Data Fig. 5 Mutual interference study of an actuator array.

a, Mutual interference of two actuators at different relative angles α at 200 Hz. Two representative cases were studied, with one actuator (no. 1) positioned along (left) or perpendicular to (right) the bisector of the other actuator (no. 2). Here, only actuator 1 was actuated. The amplitude ratio, that is, the amplitude of actuator 2 over the amplitude (induced by mutual interference) of actuator 1, shown in the table demonstrates that α = 45°, 90° and 270° result in relatively small mutual interference for both representative cases simultaneously. b, Mutual interference of two small actuators at different relative angles α at their resonant frequency of 200 Hz. The size of the actuators and the distance between them were scaled to 1/10 of the original design as shown in a, that is, the distance between two actuators was 2.1 mm rather than 21 mm. Here the thickness of the PI disk was set at 1.8 μm to enable the resonant frequency for the small actuator to be 200 Hz. Two representative cases were studied, with one actuator (no. 1) positioned along (left) or perpendicular to (right) the bisector of the other actuator (no. 2). Here only actuator 1 was actuated. The amplitude ratios (actuator 2 to actuator 1) due to mutual interference are shown in the table. c, Optimization of the actuators’ arrangement. The mutual interference among actuators was studied for α = 0°, α = 45° and a combination of 90°/270° referring to the simulation results in a, and for representative cases when actuators A (around the centre), B (near the boundary) and C (near the corner) are actuated separately. The results show that α = 90°/270° yields the smallest mutual interference (see tables under). The number gives the amplitude ratio due to the mutual interference among the actuators—that is, the amplitude of all actuators over the amplitude of the activated actuator—where 1 represents the activated actuator.

Extended Data Fig. 6 Study of the key electrical components of the epidermal VR system.

a, Interference between the primary coil and NFC antennas. Shown are schematic illustrations of a primary coil with (left) or without (right) four NFC antennas along the Z direction of the transmission antenna. b, Comparison of experimental (E) and theoretical (T) results of voltage induced by a single primary coil versus a primary coil with four NFC antennas. c, d, Transmission antennas used for operating the VR devices: c, small size, 318 mm × 338 mm; and d, large size, 620 mm × 852 mm. e, An transmission antenna placed in the XY plane. f, The magnetic field strength (H) in the ZX plane (the middle plane of the coil) for the small (left) and large (right) transmission antennas (RF readers). g, Theoretical results show that the small and large transmission antennas are suitable for short (<24 cm) and long (>24 cm) working distances, respectively. h, i, Output power of the serpentine primary coil of the epidermal VR device as a function of distance to the small transmission antenna (h), and the large transmission antenna (i). The error bars correspond to the calculated standard deviation.

Extended Data Fig. 7 Electrical properties of the intermediate coil.

a, Configuration of an intermediate coil (20 cm × 20 cm, wound with Cu wire with a diameter of 0.2 mm) oriented parallel to the XY plane, at a distance Z from the transmission antenna. b, Computational results for the magnetic field distribution induced by an RF transmission antenna tuned to resonance with a receiver antenna with and without an intermediate coil. c, Comparison of the magnetic field strength along the Z direction of the transmission antenna with (W) and without (W/O) the intermediate coil. d, Amplification factor η along the Z direction of the transmission antenna with and without the intermediate coil. e–g, Simulation and experimental results for the inductance (e), resistance (f) and Q factor (g) as a function of frequency.

Extended Data Fig. 8 Mechanical characterization of the epidermal VR system.

ad, Output power from the primary coil of an epidermal VR device as a function of tilt angle χ (a, geometry; b, data), bending radius, R (c) and bending cycles to an R of 2.8 cm (d). The distance between the device and the antenna was fixed at 20 cm for all measurements. e, Measured phase responses of the antennas used for wireless control over the SoCs as a function of radius of curvature. The resonance frequency is 13.56 MHz before bending. Bending induces only very slight shifts in these curves.

Extended Data Fig. 9 Visualization of system level operation using an array of 32 lasers.

a, Schematic illustration of a custom-built laser array system for real-time visualization of the operation of a complete epidermal VR system. b–d, The array of lasers (b), the corresponding array of beams reflecting from the haptic actuators, each mounted with a reflective disk (diameter of 8 mm), across an entire system (c), and their arrival at a monitoring screen (d). e, Representative frames extracted from video recorded using a high-speed camera to capture oscillatory motions of each of the laser spots. These motions directly determine the motions of the cantilever-based actuators. f, Schematic illustration of a laser spot produced by projection of a reflected beam onto a screen during the operation of the actuator. g, Representative frames extracted from video recorded using a high-speed camera showing the oscillatory motion of laser spot 1. h, Pictures of a laser spot on a reflector that mounts on a haptic actuator (left) and on the monitoring screen (right). The diameter of the laser spot is ~3 mm. i, Calculated displacements of four actuators determined from the measurement setup geometry and the amplitude of motion of the laser spots in g, e and Supplementary Video 4. The traces are offset in the y direction to facilitate visual inspection. The calculated displacements are somewhat smaller than those measured directly from individual actuators using high-speed cameras owing to slight misalignments of the lasers and to shifts in the resonance frequencies due to absence of the PDMS encapsulation layer for the devices measured using the laser technique. The results allow direct visualization and measurement of the vibration amplitudes, direction and frequency of the cantilever beams associated with each actuator across the full array.

Extended Data Fig. 10 Pictures of the operation of an epidermal VR system, visualized with a reflected array of 32 laser beams.

Activation of a given haptic actuator causes the corresponding reflected spot to transform from a circular to an elliptical shape, owing to the vibratory motions (top row). The results in the lower three rows show representative spatial patterns of actuation, including numbers 0 to 9 and letters ‘N’, ‘A’, ‘T’, ‘U’, ‘R’ and ‘E’. We note that the detailed shapes of the laser spots on the screen depend critically on the positioning of each of the beams across the corresponding reflectors mounted on the cantilevered actuator structures.

Supplementary information

Supplementary Figures

This file contains Supplementary Figures S1-S12 with full legends.

Reporting Summary

Video 1

High-speed-camera recordings of an actuator during operation at a power of 1.75 mW.

Video 2

FEA of the key vibrational mode for an actuator mounted on the skin, where U represents the displacement (unit: mm) and U3 represents the displacement (unit: mm) in the normal direction of the skin.

Video 3

Real-time control and visualization of the operation of an epidermal VR system, enabled by the reflection of an array of 32 laser beams off of the device and onto a screen.

Video 4

Visualization of the vibratory motions of the haptic actuators in an epidermal VR system recorded by using reflected laser beams and a high-speed digital camera.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yu, X., Xie, Z., Yu, Y. et al. Skin-integrated wireless haptic interfaces for virtual and augmented reality. Nature 575, 473–479 (2019) doi:10.1038/s41586-019-1687-0

Download citation

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.