Introduction

Behavioral biometric tactile systems are emerging technologies that identify people by analyzing the way they interact with their devices. For example, by measuring how fast the person writes or moves the pen and how much pressure they exert when interacting with the device, behavioral biometric tactile algorithms can authenticate the person signing a document1,2. Because personal habits are unique, behavioral biometric data is difficult to replicate, making these technologies better at preventing theft and fraud. Consequently, there is significant interest in using dynamic pressure sensing technology to identify and measure different types of forces that can be used for behavioral recognition3,4,5,6,7,8,9,10,11,12,13,14.

Optical tactile sensors are an attractive low-cost solution for interactive devices and tactile displays. While typical optical tactile sensors can measure normal force, information on in-plane motion, such as velocity, can only be obtained by positional analysis of two or more sequential images15,16,17,18,19,20,21,22,23. However, the resulting “time-averaged” velocity may not represent the true motion, especially when the directional change of motion is rapid with respect to the available rate of measurement, e.g., frames-per-second (fps) of a camera. (Supplementary Fig. 1). For more accurate information on in-plane motion, the optical tactile sensor should ideally also measure “instantaneous” velocity of the in-plane motion. Because instantaneous slip velocity is highly correlated to the shear force applied to the solid-solid interface in contact mechanics, having an optical tactile sensor capable of instantaneously measuring both normal and shear force would be desirable for studying behavioral biometrics. Optical polymers24,25, photonic crystals26, optical light guide27, quantum dots28, and organic dyes29,30 have been used to build optical tactile sensors. However, these systems cannot decompose dynamic elements of forces because they have slow signal response, high limits of detection, and low spatiotemporal resolution. To analyze fast movements such as handwriting in behavioral biometrics, we need optical tactile sensors that can quantitatively decompose applied force into vertical, normal, and lateral shear forces from a single image.

Here, we report a highly sensitive optical tactile sensor system that can instantaneously decouple dynamic touch signals in a single image for behavioral biometric analysis. Our tactile sensor contains a stress concentration layer that resembles the sensory architecture of the human skin and an optical tactile layer that contains an array of upconversion nanocrystals (UCNs). When force is applied to the sensor, the stress concentration layer amplifies and transfers the dynamic force to the tactile layer that interfaces with a total internal reflection (TIR) dove prism. Our UCNs-based optical tactile sensor generates unique axisymmetric or non-axisymmetric upconversion luminescence signals with spatiotemporal patterns that contain information on the motion of the object, such as direction, velocity, and magnitude of normal and shear force. Unlike conventional optical tactile sensors, our system can quantitatively decompose the applied force into individual components of vertical normal and lateral shear force from a single image in real time. We used our sensor system for surface profiling of objects, high-resolution fingerprint recognition, and anti-counterfeiting applications. Using a machine learning (ML) framework to analyze the luminescence tactile signals, we applied the sensor system in a small-scale Braille-to-Speech (BTS) translation system. Most importantly, our sensor accurately recognized behavioral biometric handwriting patterns. We expect such a simple sensor design to find ample applications in sophisticated behavioral biometric technologies.

Results

Design and principle of UCN-based optical tactile sensor for dynamic force decoupling

Our tactile sensor system includes a tactile sensor pad, a TIR dove prism and a charge-coupled device (CCD) camera (Fig. 1a, and Supplementary Fig. 2). The tactile sensor pad is composed of a stress concentration layer at the top, a thin (20 nm) anti-reflective platinum (Pt) layer in between, and a signal-generating tactile layer at the bottom (Fig. 1b). The stress concentration layer is designed to mimic the undulating interface of hard epidermis and soft dermis in human skin because such an interface is known to amplify applied forces by increasing interfacial area and localizing the applied force31,32. Further, mechanical mismatch between the rigid epidermis and soft dermis enhances force transmission from the skin to sensory mechanoreceptors33,34. To recreate this microarchitecture, we used stiff thermoplastic polyurethane (TPU, 52.5 MPa) and soft polydimethylsiloxane (PDMS, 1.76 MPa) with an undulated two-dimensional (2D) array of micro-hemispheres patterned at the interface (Fig. 1c, and Supplementary Fig. 3). The Pt layer in between serves to suppress diffuse reflection and improve the accuracy of the images (Supplementary Fig. 4). The bottom tactile layer contains an array of PDMS micro-hemispheres embedded with UCNs (Supplementary Fig. 5 and Supplementary Note 1). For our tactile sensor, we synthesized a hexagonal phase yellow luminescent lanthanide-doped UCNs (β-NaYF4:Yb3+/Er3+/Gd3+; 30/2/30 mol%) using a conventional hydrothermal method (Fig. 1d, e, and Methods for details). When homogenously embedded into the PDMS micro-hemispheres (Fig. 1f and Supplementary Fig. 6), their luminescence remained unchanged (Fig. 1g, and Methods for details). UCNs are advantageous for our system because they have a high spectral resolution (full width at half maximum of ~20 nm), low optical signal errors, and large anti-Stoke shifts. Further, UCNs do not require optical filters for luminescence color purification, have negligible background auto-fluorescence35 and are optically stable upon environmental perturbations36. Moreover, their luminescence colors can be easily tuned37 by simply adjusting the stoichiometry of lanthanide dopant ions (Supplementary Figs. 7 and 8, and Methods for details). UCNs are also amenable to large-scale synthesis38.

Fig. 1: Upconversion nanocrystals (UCNs)-based optical tactile sensor resembling human skin.
figure 1

a UCNs-based optical tactile sensor pad generating upconversion tactile signals via frustrated total internal reflection (TIR) of 980 nm near-infrared (NIR) illumination laser at the UM/prism interface. b Schematic of the optical tactile sensor pad consisting of a stress concentration layer made up of stiff thermoplastic urethane (TPU) interlocked with soft polydimethylsiloxane (PDMS) patterned with micro-hemispheres, a thin (20 nm) anti-reflective Pt layer, and a signal-generating tactile layer containing an array of micro-hemispheres (UMs). c Cross-sectional SEM images of the optical tactile sensor pad. (Scale bar, 100 µm). d TEM image of rod-like hexagonal phase lanthanide UCNs. (Scale bar, 100 nm). e Photograph of UCNs in cyclohexane excited by 980 nm NIR light display yellow luminescence. (Scale bar, 1 mm). f Luminescence image of UMs array upon 980 nm NIR excitation. (Scale bar, 200 µm) g, Emission spectra of yellow luminescence of UCNs and UCNs embedding in PDMS micro-hemispheres. h Principle of upconversion luminescence generation depending on the contact area changes between UM and TIR prism from the applied force. i, j Luminescence intensity profile under i static normal (axisymmetric luminescence) and j dynamic shear force (non-axisymmetric luminescence). k Decoupling of normal and shear forces from a single image frame using Machine Learning-based feature engineering. Source data are provided as a Source Data file.

Full size image

In our system, signal generation depends on the contact between the UCNs-embedded micro-hemispheres (UMs) in the tactile layer and the TIR prism. Irradiating the prism with 980 nm near-infrared (NIR) laser generates evanescent waves in the thin (<100 nm) boundary layer close to the interface of the prism and UMs layer. When an object moves across our sensor pad, the stress concentration layer amplifies and transfers the applied dynamic forces to the UMs layer. As the contact area between the UM layer and TIR prism increases, the UMs produce a luminescence signal measurable by the CCD camera and luminescence spectrometer (Fig. 1h). This phenomenon can be ascribed to the transformation of the evanescence wave field into propagating far-fields at contact points between the UM and prism, where TIR is frustrated due to the decrease in the refractive index contrast according to the higher refractive index of PDMS (n = 1.4) compared to air (n = 1.0) (Supplementary Figs. 9 and 10 and Supplementary Note 2). At the contact points, the propagating far-field components can now penetrate the UM, effectively illuminating the UCNs within the UM volume. Consequently, locally compressed UMs with a more extensive contact area with the prism generate a stronger luminescence signal, while those without applied force do not yield measurable luminescence. It is noteworthy that the non-linear luminescence property of UCNs suppresses the initial luminescence caused by the evanescence wave field from the minimal contact between the UM layer and the TIR prism in the absence of applied force, consequently reducing background noise39. Locally pressed UMs with higher contact area with the prism produce higher luminescence signals, while those without applied force do not contact the prism and, therefore, do not produce luminescence (Supplementary Fig. 11a, b). When we applied a normal force to the top layer of the tactile sensor by vertically pressing a spherical indenter (5 mm diameter), the resulting luminescence signal was axisymmetric (Fig. 1i). The tactile sensor we fabricated exhibited uniform signal generation across the entire sensor pad or 10 different sensor pads and demonstrated highly stable signals under continuous laser excitation (Supplementary Figs. 12–14). However, when we simultaneously applied a lateral shear force in addition to the normal force by moving the pressed indenter along the surface of the top layer, the luminescence signal became non-axisymmetric—the shape of the signal is skewed towards the direction of movement (Fig. 1j, and Supplementary Fig. 11c). This suggests that detailed information on the surface-parallel motions of the indenter such as the magnitude of normal force, and the velocity, direction and magnitude of lateral shear force can be extracted by analyzing the non-axisymmetric shape of the luminescence signal. These features induced subtle changes in luminescence intensity, and to effectively observe this aspect, we proposed a tactile sensor based on UCNs instead of the conventional mechanoluminescence-based tactile sensors. Optical tactile sensors using UCNs have better sensitivity and response time compared with mechanoluminescence tactile sensors40,41. By comparing the luminescence shape from the optical tactile sensor, we proposed a method to determine both the direction and magnitude of normal and shear forces in real-time using an ML algorithm for force discrimination (Fig. 1k). By analyzing the luminescence tendency such as axisymmetric and non-axisymmetric for static and dynamic optical tactile sensors, respectively, we proposed ML methodologies to discriminate normal and shear forces in a single image frame in real-time.

Characterization of normal force-induced luminescence for high-sensitivity force measurement

To comprehend the relationship between normal force and luminescence signals, we conducted a systematic analysis of the signal intensity under varying applied normal forces. As the magnitude of the normal force increased from 0 to 5.0 N, we observed a corresponding increase in both luminescence intensity and the pressure distribution area, maintaining an axisymmetric circle (Fig. 2a, b, Supplementary Figs. 15–17, Supplementary Movie 1, and Methods). The stress concentration layer, due to its modulus difference and undulating interface, exhibited a strong and wide luminescence area with enhanced intensity. We established a correlation between grayscale and pressure by analyzing the area beneath the prism (Supplementary Figs. 18a and 19). Our optimized tactile sensor pad (1000 µm thick, including a 200 µm thick stress concentration layer) demonstrated the ability to detect normal forces as low as 0.05 N (Fig. 2b, Supplementary Note 3 for optimization, and Supplementary Figs. 20–22). The characteristic luminescence intensity of yellow UCNs at 530 and 660 nm showed a proportional increase with normal force (Fig. 2c). The consistent relationship between luminescence intensity and the cumulative gray value for each pixel enabled the presentation of a pressure distribution in a color map (Fig. 2d and Supplementary Fig. 23). Our sensor exhibited several notable properties. It showed no hysteresis in luminescence intensity under normal force application, indicating negligible adhesion between the PDMS micro-hemispherical array and the TIR prism (Fig. 2e). Additionally, our sensor demonstrated high stability, maintaining constant luminescence intensity over 5000 cycles of 5.0 N force application, even at high temperatures (Fig. 2f and Supplementary Fig. 24). Our sensor also boasted a fast response time of 9.12 ms, determined using Gaussian fitting. (Fig. 2g, and Supplementary Fig. 25). Remarkably, without the stress concentration layer, our sensor detected pressures as low as 0.02 N, outperforming current optical tactile sensors by an order of magnitude (Supplementary Table 1)15,16,17,18,19,20,21,22,40,41. These results demonstrate the exceptional performance and versatility of our tactile sensor, highlighting its potential for applications requiring precise force measurements and environmental conditions.

Fig. 2: Quantitative measurement of static normal force.
figure 2

a The axisymmetric UCN luminescence image (top) and its pressure distribution profile (bottom) under the normal force using a spherical indenter on the tactile sensor pad. (Scale bar, 2 mm). b Normalized UCN intensity across the centered indented area of the tactile sensor pad with increasing normal force from 0 to 5.0 N. (Scale bar, 2 mm). c Change of upconversion luminescence spectra with increasing normal force from 0 to 5.0 N. d Double y axis graph of integration of gray value of luminescence image (left) and luminescence intensity from 510 to 560 nm wavelength (right). e Loading-unloading hysteresis curves of luminescence signal. f Repeated stability test of luminescence intensity of UCNs-based optical tactile sensor subjected to a 5.0 N normal force over ~5000 cycles. g Response time of our tactile sensor using Gaussian fitting (9.12 ms). Source data are provided as a Source Data file.

Full size image

Characterization of shear force-induced non-axisymmetric luminescence for real-time force decoupling

We further investigated the temporal change of luminescence signal under applied lateral shear force (Methods in details). Initially, a 1.0 N vertical preload was applied to the tactile sensor pad using a spherical indenter, creating a circular axisymmetric luminescence intensity and pressure contour (Supplementary Note 4). We then accelerated the indenter along the sensor pad surface at a constant rate of 1.0 mm/s2 until the velocity reached 1.0 mm/s. As the indenter began sliding, the axisymmetric luminescence circle stretched into a non-axisymmetric oval along the shear force direction (Fig. 3a). This transformation was captured at 10 frames-per-second (Supplementary Movie 2 middle) and analyzed frames-by-frame. Both the pressure contour color map and the luminescence intensity plot confirmed the change from circular to oval profile, with the center maintaining axisymmetric circular luminescence (Fig. 3a, and Supplementary Fig. 26). Note that the non-axisymmetric luminescence continued to form until the indenter reached a specific velocity (Supplementary Fig. 27 and Supplementary Movie 2). Additionally, the higher applied velocity made the more non-axisymmetric luminescence at the same time (Fig. 3b, c). This non-axisymmetric luminescence profile results from shear force-induced friction between the micro-hemisphere array layer and the TIR prism, causing asymmetric lateral deformation of the individual micro-hemispheres (Supplementary Note 4)42. We verified this unique luminescence profile by reducing friction with silicone oil, which eliminated the non-axisymmetric luminescence profile, maintaining an axisymmetric signal even during acceleration (Supplementary Fig. 28). Using a customized force gauge, we quantitatively measured and recorded the corresponding shear force for each non-axisymmetric luminescence image (Fig. 3a and Supplementary Fig. 29). Notably, our sensor allows immediate determination of shear force direction from a single non-axisymmetric luminescence profile, unlike conventional tactile sensors that requires a coordination analysis of multiple images (Fig. 3d)43,44.

Fig. 3: Non-axisymmetric luminescence observation from directional and velocity information.
figure 3

a Luminescence images (left), pressure distribution profile (middle), and luminescence intensity plots (right) show quantitative changes in the shape of the luminescence signal as the velocity of the spherical indenter increases. (Scale bar, 2 mm). b, c Luminescence images (left), pressure distribution profile (right), and luminescence intensity plots (bottom) at (b) 0 and (c) 0.5 s according to the velocity change of the indenter at 1.0 N of the initial normal force. (Scale bar, 2 mm). d UCN luminescence image (left) and pressure contour map (right) change as the indenter moves in eight different directions (Scale bar, 2 mm). e Calculated skewness for a moving indenter matched the measured shear force. f Overlay of calculated skewness with the experimentally measured shear force. Source data are provided as a Source Data file.

Full size image

We further analyzed the friction force using a surface forces apparatus (SFA) (Supplementary Fig. 30, and Methods in detail). The kinetic friction coefficient increased from 0.148 to 0.531 as driving velocity increased from 11.6 to 58.0 μm/s (Supplementary Fig. 31). indicating that higher velocity leads to increased frictional forces and micro-hemisphere deformation, affecting the non-axisymmetric shape of the luminescence signal. These velocity-dependent friction coefficients are commonly in complex systems with soft polymers45 or polymer-like surfaces46, contradicting conventional Amontons’ Laws of Friction47,48.

To quantify the non-axisymmetric luminescence characteristic of shear force, the degree of non-axisymmetric luminescence was statistically described using the concept of skewness (γ), which is obtained by calculating the median and mean of the luminescence intensity profile (Supplementary Note 5). For a moving indenter, the calculated skewness matched well with the measured shear force, indicating that the magnitude of the shear force could be quantitatively estimated directly from the non-axisymmetric luminescence profile (Fig. 3e). When we analyzed the change in shear force at a given skewness for various normal forces, skewness decreased when the normal force increased from 0.5 to 1.5 N (Fig. 3f, and Supplementary Figs. 32–34). This is because an increase in normal force decreases the relative contribution of shear force (and thus, skewness value) to the luminescence signal, which is formed from both the normal force that creates the axisymmetric profile, and shear force that creates the non-axisymmetric profile (Fig. 3f, and Supplementary Fig. 35). Based on the single image frame analysis of luminescence intensity profile for static normal force and the directional skewness for dynamic shear force, we successfully quantified the direction and magnitude of both normal and shear forces in real-time from a single image using force discrimination ML algorithm (Fig. 4a, b, Supplementary Tables 2–4, Supplementary Movies 3, and Supplementary Notes 6 and 7). Our model showed high reliability with r2 values of 0.965 and 0.947 for normal and shear forces, respectively. The algorithm analyzed r, g, and b values of detected images to predict the applied normal force and determine the threshold for normal force region contour, enabling successful decoupling of normal and shear force in real-time (Fig. 4c, d, Supplementary Movie 4). Despite shear force attenuation due to frictional force between the sensor pad and TIR prism, our approach relies on the direct correlation between the measured skewness of the luminescence intensity and actual shear force applied to the sensor pad surface. This relationship has been thoroughly validated and used to train our ML model (Supplementary Notes 8). Importantly, the skewness change remained consistent across different time frame rates, suggesting that a cost-effective 10-fps camera is suitable for normal and shear force decoupling (Supplementary Figs. 36 and 37). This finding enhances the practical applicability of our sensor system.

Fig. 4: Flowchart illustrating the process of machine learning-based force decoupling system within a single image frame.
figure 4

a Captured non-axisymmetric luminescence signal induced by dynamic shear force (top) and pressure distribution profile (bottom) (Scale bar, 2 mm). b The process of feature engineering-based standard threshold estimation for force decoupling. Tendentious features of the RGB color space under applied normal force are quantified and utilized to train an SVR model for normal force estimation. Subsequently, the standard threshold for extracting the normal force region is determined by another SVR, which was trained on intensity profiles corresponding to the applied normal force. The estimated threshold and its half value were employed to delineate the normal and shear force regions, respectively. c Demonstration of decoupled normal (left) and shear force (right) in luminescence images (top) and pressure distribution profile (bottom). d Real-time decoupled estimation from a single frame of a non-axisymmetric luminescence signal. Source data are provided as a Source Data file.

Full size image

Effect of interfacial undulation on enhanced shear force transmission in stress concentration layer

We further examined whether and how the undulated interface between the stiff TPU and soft PDMS in the stress concentration layer of the tactile sensor pad affects the non-axisymmetric luminescence (Supplementary Figs. 38 and 39). When an indenter moved at a velocity of 1.0 mm/s with 1.0 mm/s2 acceleration across a pad without undulation at the TPU-PDMS interface, the maximum value of skewness was 0.1 (Fig. 5a). With undulation, the maximum value of skewness increased to 0.37. When the velocity of the indenter was decreased to velocity of 0.5 mm/s with 0.5 mm/s2, the skewness decreased further to 0.32. Our results indicate that the undulation at the interface and the differences in modulus between TPU and PDMS improved the transmission of the applied shear force to the signal-generating micro-hemisphere layer, which, in turn, increased the non-axisymmetric of the luminescence signal.

Fig. 5: Skewness change and FEA analysis of stress concentration layer.
figure 5

a Experimentally measured Skewness change of four types of the stress concentration layers at 1.0 N of the initial normal force. b Finite element analysis (FEA) calculation of the local stress distribution at the undulating interface and contact width variations on the micro-hemisphere array under applied normal (Fn) and shear (Fs) forces in a 2D model sensor. c Comparison of the experimental luminescence intensity profile (black) and the calculated contact width profile (red) and its fit to the skewed Gaussian (blue). Experimental data was collected when Fs = 0.49 N. d Experimentally calculated skewness values for sensor pads with and without undulating interface under applied normal (Fn = 1.0 N) and shear (Fs = 0.49 N for high Fs and 0.22 N for low Fs) forces. Skewness increased with Fs and was higher in sensor pads with the undulating interface. e Color map at the undulating interface of difference in First Piola–Kirchhoff stress distribution under applied shear force. The dashed line describes the center of applied force. f Normalized plot of First Piola–Kirchhoff stresses of each shear (red) and normal (gray) direction and contact width to experimental intensity when Fs = 0. g FEA calculation of First Piola–Kirchhoff stress distribution between TPU and PDMS interfaces show regions below the interface (black dotted line) is on average 50% higher for the undulated interface. h Comparison plot of first Piola–Kirchhoff stress difference with and without the stress concentration layers at 10 μm below the interface between TPU and PDMS. Source data are provided as a Source Data file.

Full size image

To understand how the TPU-PDMS interface undulation enhances force transmission, we employed finite element analysis (FEA) to examine the stress distribution in 2D model sensors with and without interface undulation under 1.0 N applied normal force (Fn) with and without shear force (Fs) ranging from 0 to 1.0 N (Fig. 5b, Supplementary Figs. 40–43, and Supplementary Note 9). Because the luminescence signal in our tactile sensor is produced only when UMs contact the TIR prism, we expected the intensity of the luminescence signal in the model sensor to scale with the contact width between the individual UMs and the underlying substrate. Consistent with the experimental intensity profile, our FEA analysis revealed that the contact width between UMs and the underlying substrate (which determines luminescence intensity) is symmetric under Fn alone but skewed in the direction of applied Fs (Fig. 5c and Supplementary Figs. 44 and 45). The calculated skewness, γ, increased with the magnitude of Fs and was substantially higher in the models with an undulating interface (Fig. 5d), corroborating the experimental results showing enhanced Fs transmission in sensors with an undulating interface (Supplementary Figs. 38 and 39). We visualized this enhancement by plotting the stress distribution differences (Δσ) between sensors with (σw) and without (σwo) an undulating interface (Fig. 5e). Notably Δσyx increased strongly near the micro-hemispheres array, indicating more effective Fs transmission due to the undulating interface (Supplementary Fig. 46). While σyy remained symmetric even under Fs, σyx became increasingly asymmetric with higher Fs (Fig. 5f, and Supplementary Fig. 47). Further analysis of σyx near the TPU-PDMS interface showed the regions beneath the undulated interface had, on average 50% higher σyx values (Fig. 5g, h). This phenomenon was independent of the indenter shape, with non-axisymmetric luminescence patterns observed across various indenter shapes and geometries (Supplementary Figs. 48–53). Although stress concentration may complicate precise tip shape determination, we utilized deep learning methods to aid in shape identification. This approach allows our sensor to maintain its effectiveness across a range of indenter shapes and applied forces. (Supplementary Fig. 54 and detailed in Supplementary Note 10).

High-resolution surface texture recognition using a refined thin sensor pad

To expand the versatility of our optical tactile sensor, we developed a thin sensor pad (200 µm thickness) without the stress concentration layer, optimized for precise high-resolution surface inspection (Supplementary Figs. 55–57). This refined design enables exceptional spatiotemporal tactile recognition of 2D surface textures, including multiple objects and fingerprints (Fig. 6a, b). Our sensor achieves a maximum spatial resolution of 100 µm, positioning it at the forefront of optical tactile sensing technology (Supplementary Fig. 58, and Supplementary Table 1)15,16,17,18,19,20,21,22,40,41. This high resolution allows for the differentiation and recognition of various surface textures with remarkable precision. For instance, our sensor visualizes intricate details such as the size and shape of dimples on golf balls and sign of wear with high spatiotemporal resolution (Fig. 6c). We further successfully identified small screws with subtle thread differences, imperceptible to the naked eye, using MATLAB-based automated image analysis (Fig. 6d, e, Supplementary Fig. 59, Supplementary Movie 5, and Supplementary Note 11)49. In addition, we developed a compact BTS system that instantly translates microscopic braille patterns into corresponding sounds using ML (Supplementary Note 12). The system converts topographic patterns of micro-sized braille (e.g., Hello World) into luminescence images and then into audio output by extracting 2D dot patterns (Fig. 6f–h, Supplementary Figs. 60 and 61, and Supplementary Movie 6).

Fig. 6: Demonstration of high spatiotemporal optical tactile recognition.
figure 6

a Tactile recognition of alphabets (Scale bar, 2 mm). b Optical (left) and luminescence (right) images of fingerprints for high spatiotemporal tactile recognition (Scale bar, 2 mm). c Photograph of a golf ball (left), the dimples (right) on its surface, and luminescence images of dimples with new and used golf ball (Scale bar, 5 mm). d Optical (top), luminescence and binary images (middle), and the corresponding intensity profile (bottom) showing the threads of different types of screws. (Scale bar, 1 mm). e Luminescence (top) and 3D color map (bottom) recognition of multiple screw threads (Scale bar, 1 mm). f Extraction of braille dot features from contour maps (bottom). g Schematic diagram of a machine learning workflow for real-time braille-to-speech (BTS). h BTS test using the HELLO WORLD sentence (Scale bar, 2 mm). Source data are provided as a Source Data file.

Full size image

ML-based biometric handwriting analysis using decoupled normal and shear force data

Most notably, using the ability to simultaneously visualize the direction of movement and the intensity change of both normal and shear force, we developed a ML framework that distinguishes dynamic biometric handwriting of individuals (Fig. 7a, Supplementary Movie 7, and detailed in Supplementary Notes 13–15). Decoupled images represented normal and shear forces, and related features extracted from videos of three different individuals writing the letter ‘e’ were fed into our ML-based linear discriminant analysis (LDA) for writing style classification. LDA clustering successfully discriminated the three different handwritten letters ‘e’ (Fig. 7b, Supplementary Figs. 62–66, Supplementary Tables 6–9, and Supplementary Note 16). We further extracted the averaged velocity-related features from multiple image frames and combined them with normal and shear force features for handwriting analysis. Our single-frame touch signal decoupling-based ML framework showed outstanding performance when compared to analyzing the handwriting using normal forces alone or normal forces with velocity-related features (Fig. 7c, Supplementary Figs. 67 and 68 and Supplementary Table 8). While the average distance within a specific cluster was the same for all analyses, the average distance between different clusters was greater when the analysis included shear force features than when it included velocity features. Comparing the data cluster analyzed using both velocity and shear force features with those analyzed using velocity only and shear force only, we find that the analysis using combined features had a greater impact on the data clusters of velocity only analysis than on shear force only analysis. Further, between velocity-only analysis and shear force-only analysis, we find that shear-force feature-based handwriting analysis forms a more clear-cut data cluster (Supplementary Fig. 66).

Fig. 7: Behavioral biometric mining of dynamic forces from luminescence images.
figure 7

a Schematic diagram of the workflow for discriminating handwriting using a machine learning-based analysis of shear and normal forces extracted from luminescence profiles. Normal force features (red panels) extracted for analysis include maximum intensity from the intensity profile and diameter of the clustering region with maximum intensity (blue circle). Extracted shear force features (green panels) include the length of the major/minor axis of the oval, the ratio of the major to minor axis of the oval, the angle of the major axis, and the statistically estimated shear force. (Scale bar, 2 mm). b Linear discriminant analysis (LDA) clustering using different combinations of extracted features distinguish the handwritten letter ‘e’ of three individuals with different clarity. (Scale bar, 2 mm). c Mean of intra-cluster distance (gray) and mean of inter-cluster distance (orange) for LDA clustering using different combinations of extracted features. d Comparison of LDA clustering performances in terms of Calinski-Harabasz index and Silhouette coefficient for each case. Source data are provided as a Source Data file.

Full size image

To compare clustering performance, we calculated the Silhouette coefficient, which describes the average distance between points within the same cluster and the average distance between points in the nearest cluster to which the data point does not belong, and the Calinski-Harabasz index, which is the ratio of dispersion within a cluster to dispersion between clusters. Of all the analysis combinations, the normal/shear force combination showed a Silhouette coefficient closest to 1 (0.867) and a remarkable value Calinski-Harabasz index (5.76 × 102) (Fig. 7d, Supplementary Fig. 68, and Supplementary Table 9). Adding the velocity feature to the normal/shear force feature slightly decreased the Silhouette coefficients (0.858) and Calinski-Harabasz index (4.52 × 102), and there is no significant advantage in terms of computational cost. From this point of view, we believe that besides normal force, the shear force feature is a dominant factor for distinguishing writing style. These results demonstrate that normal/shear force-based handwriting analysis represents the most clear-cut cluster and is well classified with clusters of different force combinations. In conventional handwriting authentication, verification is predominantly grounded in the consistency of static handwriting features such as letter size, inclination, and inter-character spacing, rather than dynamic attributes. Remarkably, our study introduces an innovative methodology for handwriting verification, focused on shear force—a parameter not previously explored in this context. This approach encapsulates a wealth of data, encompassing factors such as the exerted pressure during writing and the speed of inscription. These parameters raise the expectation that our framework can offer valuable insights into the writer’s identity, as they exhibit variations correlated with the writer’s age and gender50. Furthermore, our framework possesses the unique capability to decouple normal and shear forces within a single frame, allowing it to provide real-time feedback, unlike previous studies that evaluated handwriting only after all letters were completed. Thus, our system holds the potential to significantly enhance the precision of the handwriting authentication process.

In conclusion, we present an upconversion luminescence-based behavioral biometric optical tactile sensor that simultaneously and quantitatively discriminates velocity, direction, normal, and shear force in a single-frame analysis. Such a unique axisymmetric and non-axisymmetric luminescence signal amplified through the human skin mimetic stress concentration microarchitecture was theoretically studied by FEA simulation, experimentally examined by SFA measurement, and quantitatively analyzed. Our optical sensors are simple to build, and we show they can be used for the tactile recognition of surface textures, fingerprints, and small-scale braille. Most importantly, using ML, our system has enabled accurate dynamic biometric handwriting analysis. We believe our sensor system will open new avenues in facile optical tactile sensors that can intuitively and sensitively display multiple elements of dynamic forces in a single image frame, forming the basis of flexible behavioral biometric optical tactile sensors in wearable devices.

Methods

Materials

NaOH (Aldrich), ethanol (Aldrich, 99%), GdCl3·6H2O (Aldrich, 99.999%), YCl3·6H2O (Aldrich, 99.999%), YbCl3·6H2O (Aldrich, 99.999%), TmCl3·6H2O (Aldrich, 99.9%), NH4F (Aldrich, 99.9%), ErCl3·6H2O (Aldrich, 99.999%), and oleic acid (Aldrich, 90%).

Synthesis of rod-shaped β-NaREF4 (RE = Gd, Y, Yb, Er, Tm) UCNs

Hexagonal rod-shaped UCNs were synthesized using a hydrothermal method38. Briefly, NaOH (0.2 g mL–1), ethanol (10 mL), oleic acid (10 mL), a solution of the lanthanide mixture (0.2 M RECl3, RE = Gd, Y, Yb, Er, and Tm, 4 mL), and 2 M ammonium fluoride (NH4F, 2 mL) were mixed by stirring for 10 min and transferred into a Teflon-lined hydrothermal device. Next, the premixed solution was heated at 200 °C for 3 h. Subsequently, the hydrothermal device was naturally cooled to room temperature (26 °C). The resulting product was collected via centrifugation and washed several times with ethanol and water. Finally, the resulting powder was dispersed in ethanol.

Stamps and screws information

Triangle, rectangle, pentagon, hexagon, alphabet letters, and braille stamps were fabricated using a 3D printer (Form2, Formlabs). The length of one side of the triangle, rectangle, pentagon, and hexagon was 5 mm, and the width of the alphabet stroke on stamps was 400 μm. The diameter and pitch of the dots in braille were 500 and 833.33 μm, respectively. Four types of screw (1/4–20; SS25S050, Thorlabs, M6; SS6MS12, Thorlabs, 8–32; SS8S050, Thorlabs, and M3; SS3S6, Thorlabs) were used for the screening of thread pitches (1/4–20; 1.27 mm, M6; 1.00 mm, 8–32; 0.79 mm, M3; 0.50 mm). Additionally, a golf ball (Titleist Pro V1) was used for surface profiling.

Optical imaging

Optical images for micro-hemisphere structure deformation were captured by a microscopic camera (HA010, Gasworld). To capture these microscopic images through the thick dove prism used for generating TIR, we utilized a long working distance objective lens (Mitutoyo 10×, 0.28 NA) in combination with a macro camera lens (Nikkor 105 mm f/2.8) that served as the tube lens with sCMOS camera (C14440-20UP, Hamamatsu photonics). Luminescence images were captured using a 10-fps CCD camera (DCC3260C, Thorlabs) and a 30-fps CMOS camera (DigiRetina16, Tucsen). Luminescence images were converted to color map contours of the pressure distribution using Origin and MATLAB. The luminescence intensity profile along the centerline was plotted using the ImageJ software.

Measurement of friction force and coefficient of friction using SFA

The friction forces were measured using piezoelectric 3D sensor (upper surface) and biomorph slider (lower surface) attachments in the SFA system (SFA2000, Surforce LLC, USA)51. Supplementary Fig. 31 shows a schematic of the friction force measurement setup. For the upper surface, a PDMS sample with an optical tactile sensor (10 mm × 10 mm) taped onto a flat glass disk was used. For the lower surface, a hemispherical brass with a radius of curvature of 2 cm radius was used. Friction was measured while increasing the load stepwise, at two different driving velocities (11.6 and 58.0 μm/s). The kinetic coefficient of friction was then obtained from the slope of the friction vs. load curve. A constant shear distance (sliding amplitude) (116 μm) and triangular wave signal determined by a function generator was applied during all measurements. The frictional force was measured at least five times for each load.

Characterization of optical signals

Luminescence images of the UMs were taken with a Nikon D810 digital camera on an inverted microscope (Ti-E, Nikon) under irradiation with a 980 nm NIR laser (MDL-F-980-5W, Dragon laser). The UMs were excited using a 980 nm NIR laser. The NIR reflection was removed using a 950 nm cut-on filter (Semrock Inc.). The upconversion luminescence spectra of the UCNs were measured using a modular spectrometer (QE Pro-FL, Ocean Optics). Image frames were extracted using the Adobe Premiere Pro CC.

Characterization of UMs integrated with the stress concentration layer

The geometry of the optical tactile sensor was characterized by field-emission scanning electron microscopy (S-4800, Hitachi) and high-resolution transmission electron microscopy (JEM-1400, JEOL).

Fabrication of the UCN-embedded micro-hemisphere layer assembled with a human skin mimetic stress concentration layer

The synthesized UCNs were dispersed in n-hexane by ultrasonic sonication (VCX 750, Sonics) for 2 h. The UCN solution was added to the PDMS base (Sylgard 184, Dow Corning) in a 2:3 ratio (UCN: PDMS). The UCN/PDMS mixture was dispersed using an ultrasonic processor for 1 h. Subsequently, a PDMS curing agent (1:5 ratio of curing agent to base) was added to the UCN/PDMS mixture depending on the thickness of the sensor. The resulting UCN/PDMS mixture was dispersed using an ultrasonic process for 5 min before pouring into a micro-hemisphere silicon mold (diameter and pitch: 90 and 100 μm, respectively) and cured at 90 °C for 4 h. The dispersion of UCNs and PDMS was observed (Supplementary Fig. 6). The top surface of the UMs array was coated with a Pt layer using an ion sputter (MC1000, Hitachi). To fabricate the stress concentration layer, the pre-cured PDMS solution was coated on a micro-hemisphere silicon mold (diameter and pitch: 20 and 40 μm, respectively) using a spin coater at 750 rpm for 1 min. After curing at 90 °C for 2 h, the obtained undulating microstructured PDMS was attached to a Pt layer. After 1 min of the O2 plasma process (FEMTO SCIENCE, CUTE-1MPR, 100 W, 8 sccm), the TPU solution dissolved in DMF was poured onto the undulating microstructured PDMS surface, and the DMF solvent was evaporated at 60 °C for 12 h.

Normal force and shear force detection device setting

To measure the optical properties of the optical tactile sensor under mechanical forces, the fabricated UMs integrated with a stress concentration layer were placed on the TIR dove prism, where a 980 nm NIR laser (FC-EW-980-40W, Uniotech) was irradiated. A schematic of the normal force detection system is shown in Supplementary Fig. 14. The normal force was measured by a vertically attached force gauge with a spherical indenter to the Z axis liner stage. The force gauge used in our study had a resolution of 0.01 N. A normal force of 0–5.0 N was measured by moving the Z axis linear stage, and movie files of the luminescence profiles were recorded (Supplementary Movie 1). Supplementary Fig. 28 shows a schematic of the shear force detection system. To measure the shear force, a right-angle spherical indenter was fabricated using a 90° tilted digital force gauge, and the shear force applied according to the velocity and acceleration of the spherical indenter was measured. Supplementary Movie 2 shows the change in the luminescence profile as the velocity reaches 1.5, 1.0, and 0.5 mm/s (acceleration: 1.5, 1.0, and 0.5 mm/s2, respectively); the velocity was maintained (1.5 mm/s: 2.3 s, 1.0 mm/s: 4 s, and 0.5 mm/s: 9 s, respectively), and then decelerated to 0 mm/s in 1 s. To obtain the luminescence images and movie file, a CCD camera (DCC3260C, Thorlabs) was placed under a glass dove prism, and a 750 nm cut-off filter (FESH0750, Thorlabs) was used to eliminate 980 nm NIR light.

Handwriting data mining ML

Regression analysis from the normal and shear force predictions for each related feature extraction was conducted based on support vector regression (SVR). SVR is typically used for non-linear systems and predicts the appropriate hyperplane in higher dimensions to fit the input data. Parameters C = 10, degree = 3, epsilon = 0.1, and kernel = ‘rbf’ were selected for the best performance results. The model was trained on 70% of the data randomly split from the data set, and the unused data were used to test the model performance. To evaluate the performance, r2, RMSE, and MAPE parameters were used. LDA, which determines linear combinations of features that characterize or distinguish two or more classes, was used to classify handwriting by lowering the dimensions of the extracted force-related features. In addition, the clustering performance was calculated using the silhouette coefficient and the Calinski-Harabasz index for each class.