Introduction

Wearable electronics experienced enormous development and advancement in the past decades benefiting from its intrinsically superior flexibility and portability1,2,3. To further improve the quality of life, the invention of various wearable sensors based on electrocardiograph4, electromyogram5, body temperature6, heart rate7, strain sensors8, etc., offers opportunities in both fitness service and medical diagnostics by the long-term monitoring of physiological signals. In particular, the detection of the human motions is significantly valuable in creating insights into the user’s health status, activity quantifying, and the establishment of an effective channel between humans and machines9,10. When it comes to the continuous and convenient monitoring of diversified human motion states, cameras and inertial measurement unit (IMU) sensors are the widely adopted devices for smart home applications11,12. However, vision recognition by cameras may cause privacy issues, and the utilization of IMUs are not intrinsically flexible and comfortable enough as the preferable wearable solution. In addition, wearing a few bulky IMUs on human body could cause inevitable interferences to the motions.

With the aid of recent advances in the fifth generation wireless networks and internet of things (IoT), immense widely allocated wearable devices are expected to be wirelessly interconnected at rapid data exchange rates to provide concurrent communication of information about the human body13,14,15,16,17. Among them, wireless sensor networks (WSNs) have become a key technology to analyze information related to identification, healthcare, human–machine interface (HMI), and human activity monitoring. However, a major bottleneck of the WSNs is the overall power consumption for long-term connectivity of the whole system. To solve this foreseeing energy crisis issue, energy harvesting technologies and advanced storage devices have emerged to make the waste energy in our surrounding environment valuable. Motivated by that, substantial efforts have been made for the development of wearable systems equipped with the abovementioned advanced technologies and potential self-sustainability18,19,20,21.

In the past few decades, several of flexible devices have been reported with different device structures and materials22,23,24. Textile as a fundamental part of normal garments has been extensively investigated as a great flexible and stretchable electronics platform25,26,27,28. There are a few research developments of smart textiles based on various mechanisms, aimed to be not only more flexible for an improved comfortability but also multifunctional, i.e., sensation, perception, or integration29,30,31,32,33,34. Triboelectric nanogenerator (TENG) gradually becomes an optimal option for scavenging waste energy, and sensing of both physical and chemical parameters based on the textile platform due to the particular advantages, including various choices of materials, easy fabrication, low-power consumption, and low cost35,36,37,38,39. In this regard, textile-based TENGs (T-TENG) exhibit the superb ability of structural retention and fatigue resistance, during wearing and washing40,41,42,43. Most of the T-TENGs are working under the vertical contact-separation mode, lateral sliding mode, freestanding mode, and single-electrode mode44. He et al. developed a narrow-gap T-TENG, which can harvest mechanical energy from various body parts to power Bluetooth sensors45. Though the narrow-gap textile is in a form most similar to the normal clothes, hence it can be seamlessly integrated with garments, such kind of structure is not suitable for pressure sensing, especially on foot due to its limited sensing range. The devices designed for foot become extremely important because the activities of foot are one of the major sources for collecting kinetic energy from human body46. Besides, gait contains useful information for the application of healthcare and personal identification as well47,48,49. A great amount of effort has been devoted to developing insoles with energy harvesting and gait sensing functions, because the demand for the flexibility and wear comfortability of the insoles is very low50,51,52. Thus, the socks can be an optimal option for gait analysis in terms of suitability. In addition, compared with the insoles, the socks can be more widely used in indoor scenarios, such as wearing sock at home where shoes are not applicable. Zhu et al. reported a self-powered sock, which provides the simple walking pattern recognition and motion-tracking information for smart home applications, but the reported sock can only distinguish gait patterns with large differences and the recognition accuracy is highly affectable by the environmental variations46.

Artificial intelligence (AI) techniques will highly amplify the intelligence of wearable electronics, and provide more reliable yet simpler solutions to more problems and resonating tasks. The conventional method of analyzing sensory information was limited in the ability of handling natural data, which relies on the manual extraction of shallow features from the raw data53,54,55,56. For decades, as a subfield of machine learning, deep learning has shown its great potential in image processing, speech recognition, human activity recognition, and so on, which provides an efficient way to learn higher-level features of the raw input from various sensing signals57,58,59,60. Hence, deep learning methods provide a promising and feasible solution to achieve high accuracy with a low computational cost for wearable sensors. Sundaram et al. developed a low-cost, scalable tactile glove using deep residual networks for recognition of different grasping objects61. The combination of AI technology with the IoT technology will lead to the Artificial Intelligence of Things (AIoT) for supporting more reliable communication between wearable devices and the cloud, which will improve the capability of prediction in the more complicated and dynamic system62,63,64,65. With the advent of the AIoT, the digital twin enables real-time monitoring of systems and processes, seasonable analysis of data to prevent fault diagnosis, and future upgrades and development66,67. The concept and application of the digital twin can be further broadened beyond our imaginations by endowing the same concept with the wearable electronics, i.e., to create digital replicas of the humans in the virtual space. The digital human now becomes a viable future technology with great opportunities for various applications, ranging from personalized medicine and healthcare treatment, social media, and individual AI brain68,69,70.

In this work, we developed the triboelectric smart socks equipped with sensing capabilities that can provide a more comprehensive long-term monitoring of the user’s physical status. In addition, this intelligent sock would be a feasible option for scavenging low-frequency energy from natural human body motions to power a Bluetooth module, and transmit the body temperature value detected by the embedded temperature sensor under IoT framework. The textile-based triboelectric pressure sensors are designed and developed to be embedded on the commercial sock for gait monitoring. The self-generated output voltage spares the triboelectric sensors from external power sources for operation compared to other sensing mechanisms, such as capacitive sensors and resistive sensors71,72. Besides, by leveraging the deep learning technology for data analysis, the intelligent socks are successfully demonstrated to realize the gait identification of group users with the accuracy of 93.54%. The application of the virtual reality (VR) fitness game by using the proposed sock offers the possibility of a complementary control interface besides the common vision and voice control terminal for augmented interactions, which shows good potential on smart home. Hence, this deep learning-enabled triboelectric smart sock provides more comprehensive information to us, which would be advantageous for sports monitoring, healthcare, future smart home applications, etc. More importantly, the privacy issue is not comprised when we use the socks for such purposes. In the future, the entire system could be flexible integrating the flexible print circuit boards (PCBs) with our sock, which would further improve the wearing comfortability of the system73,74,75,76. In short, with the integrated information collected from this sock, and additional wearable sensors and neural electrodes, this platform paves the way to the realization of the digital human technology in near future (Fig. 1).

Fig. 1: The schematics and future prospects of the deep learning-enabled socks.
figure 1

The intelligent socks would assist wearable electronics to move toward digital human for diversified applications: energy harvesting for IoT application, human activity monitoring for VR fitness game, and gait identification for smart home application.

Full size image

Results

Design, sensing mechanism, and characterization

Though a narrow-gap T-TENG is thin and more resemble to the normal clothes, the sensitivity and sensing range of it as a pressure sensor is quite limited. To build up a T-TENG pressure sensor for gait detection which should have a large sensing range up to 200 kPa, a surface-structured T-TENG sensor is proposed and fabricated. Here, we adopted the mm-scale frustum structure to be patterned on the silicone rubber surface with the aid of the 3D-printed mold, which is low cost and highly scalable for mass production in future. The T-TENG sensor which contains four functional layers, including a nitrile thin film, a silicone rubber film with patterned frustum structures (the detailed dimensions of the frustum with a square base as shown in Fig. 2a on the contact surface), and two conductive textiles attached to the back of the two aforementioned contact electrification layers for charge collection. Moreover, two nonconductive textile layers are used to seal the device on the outer surface. The pressure stimulus will induce charges due to the contact-separation mode based on the working mechanism of the T-TENG sensor (Fig. 2b), and further flow in the external circuit to transform the mechanical energy into electricity. To clearly demonstrate the necessity of the patterned surface structure, a small piece of 3 cm × 3 cm of the textile sensor with no surface structure is fabricated and characterized. It can be observed that its output voltage saturates quickly at 72 kPa with a sensitivity of 0.4 V kPa−1 in the first linear range (Fig. 2c). It is advocated that the pressure on foot is in a large range up to 200–300 kPa, hence a large sensing range at least up to 200 kPa is desired to capture the pressure information of foot motions. Correspondingly, the open-circuit voltage of the T-TENG sensor with the designed surface structures under ascending applied pressures is measured and shown in Fig. 2c. Benefiting from the mm-scale frustum structure to be patterned on the silicone rubber surface, a threefold higher sensitivity over the flat-structured textile sensor (first linear range from 10 to 70 kPa) can be observed, and the sensing range is successfully extended to >200 kPa, making it much suitable for gait sensing. The open-circuit voltage waveforms under the high pressure of 244 kPa of the textile sensors with/without frustum structures are provided in Fig. 2d, e. Notably, the absolute voltage value of the frustum-patterned sensor is also increased by three times for the enlarged contact areas and the heightened spacing. The characterizations of the transferred charge (Q) and the short-circuit current (ISC) of the T-TENG sensor are tested and sketched in Fig. 2f, h. The transferred charge of the T-TENG was measured under the load of 13, 38, and 244 kPa with the pressing and releasing speed of 15 mm s−1 (Fig. 2f). It can be observed that the value of the transferred charge increases with the applied pressure as well due to the increased contact areas. Similarly, the short-circuit current of the T-TENG was measured, and the same trend is observed when the load pressure increases (Fig. 2g). Unlike open-circuit voltage and transferred charge, the amplitude of the short-circuit current also depends on the pressing and releasing speed. As characterized in Fig. 2h, fewer peaks are observed when the pressing/releasing speed decreases from 15 to 5 mm s−1, and the amplitude of the peak current is also lowered due to the reduced charge transfer speed. Without the specific illustration, the frustum-patterned sensor is used for all the following testing and demonstrations. The durability and washability tests can be seen in Supplementary Figs. 3 and 4.

Fig. 2: The basic characterization of T-TENG sensors.
figure 2

a The detailed dimensions of the frustum with a square base. b The working mechanism of the T-TENG sensor which works in the contact-separation mode. c The open-circuit voltage versus pressure curve of T-TENG sensors with no structure and frustum structure. d, e The schematics of T-TENG sensor and its open-circuit voltage waveforms at 244 kPa without frustum structures and with frustum structures. f The transferred charge of the T-TENG under the load of 13, 38, and 244 kPa with the pressing and releasing speed of 15 mm s−1. g The short-circuit current of the T-TENG under the load of 13, 78, and 244 kPa with the pressing and releasing speed of 15 mm s−1. h The short-circuit current of the T-TENG with the pressing and releasing speed of 5, 10, and 15 mm s−1 under the load of 244 kPa.

Full size image

A T-TENG sensor with a size similar to the foot of the user is fabricated and attached to the bottom of a cotton sock, to make an intelligent sock for gait analysis. By wearing the single-sensor sock on the right foot, the user’s gait can be detected and analyzed through the triboelectric output voltages. In general, a gait cycle includes four events as indicated in Fig. 3a, i.e., “heel contact”, “toe contact”, “heel leave”, and “toe leave”. Based on the working mechanism of the triboelectric textile sensor, there will be positive peaks generated as the pressure is applied on it, and reversed charge flow can be detected when the pressure is released. Accordingly, when the user walks forward, two positive peaks representing the heal contact and toe contact are observed during the half cycle of gait, and two negative peaks representing heal leave and toe leave are detected thereafter (Fig. 3b). Thus, some preliminary features can be extracted from the output voltages, i.e., frequency of gait cycles, positive/negative peak amplitudes, and the time interval of every event in a gait cycle (more detail can be found in Supplementary Fig. 6). Through the detection of these characteristics, the types of human activities can be preliminarily monitored. Parkinson’s disease (PD), as a progressive and neurodegenerative movement disorder, usually presents with symptoms of gait disturbance and even falls. To show the possible application of healthcare monitoring of our proposed sock, mimetic motions of the PD that are frequently measured among the patients were performed, and the corresponding detected signals are shown in Fig. 3c. Three conditions including the normal walking, loss of stride, and the freezing of gait (FOG) are detected in sequence, resembling the eliciting of the PD. The patient normally experiences the loss of stride ahead of the FOG, which is characterized by the shortened intervals between spikes. During the FOG, the patient couldn’t move the feet even with the intention to do so, which can be reflected by the small and irregular oscillating signals from the sensors. In addition, the smart sock was used to detect fall-down event from the recorded gait signals as shown in Fig. 3d. An abnormal negative peak with an extremely high amplitude against previous normal data can be detected as the falling happens, and there is no positive peak generated thereafter, indicating the feet of the user are still off the ground. Furthermore, there is a real-time remote healthcare monitoring system is illustrated, as a demonstration of the intelligent sock shown in Supplementary Fig. 7. The gait signals will be collected by a microcontroller unit (MCU) and transmitted to a PC via wireless communication. If the gaits from users appear the irregular signal compared with recorded normal walking signals, the emergency medical alert system will display the corresponding warning, such as loss of stride or falling and inform the caregivers (Supplementary Video 1). Hence, the demonstrated real-time system for health condition monitoring and medical emergency alert presents a great prospect of the intelligent sock in multiple healthcare applications to assist both the elderly and patients, which can be further used for smart home, hospitals, and healthcare centers. Meanwhile, this sock could be very useful in studying walking patterns in people affected by illnesses, such as diabetes, musculoskeletal abnormality, and rehabilitation from different injuries.

Fig. 3: The characterization of T-TENG socks and preliminary gait analysis.
figure 3

a Schematics of four phases of a typical contact cycle and b corresponding signals of normal walking (right foot). c, d The real-time healthcare monitoring of mimetic walking pattern of Parkinson’s disease patient and gait signals of a fall-down event. e, f The maximum output powers of a single sock on right foot under 1 Hz walking and 2 Hz running were tested by changing the external load resistances from 0.1 to 100 MΩ. g Charging curve of different capacitors (i.e., 1, 4.7, 10, and 27 μF) were charged to 5 V. h Charging and discharging curve with the socks on foot, where each voltage drop represents a discharging to the Bluetooth module. i Monitoring the temperature on armpit under various exercise intensities by Bluetooth module.

Full size image

Besides of gait sensing, as intelligent sock works under the self-powered triboelectric mechanism, it can also be used as an energy harvester to scavenge waste energy from foot motions. The output power of the single sock on the right foot is measured on different external loads from 0.1 to 100 MΩ. As shown in Fig. 3e, the output power of 0.32 mW is generated from 1 Hz walking on a 44.4 MΩ load. In the case of running with the frequency of 2 Hz, owing to the larger force and frequency that the foot applied on the sock, a maximum power of 3.18 mW was measured on a 21.3 MΩ (Fig. 3f). Hence, the proposed sock is capable of energy harvesting from the gait cycles under various motions. The rectified output voltages from the sock charged different capacitors of 1, 4.7, 10, and 27 μF to be 5 V (Fig. 3g), demonstrating the good charging capability of it as a reliable power source. Within ~300 times of steps in normal walking, the output from the sock can charge a 27 μF capacitor up to 8 V to power up a Bluetooth module, after which the voltage on the capacitor drops to 2 V as shown in Fig. 3h. This discharged power from sock can activate the Bluetooth module to send the humidity and temperature data to the smartphone, which can be used for monitoring the temperature on human body under various exercise intensities. As known, the body temperature is susceptible to exercise intensities, hence it should be taken into account when assessing the body temperature of a subject. The body temperature under armpit of a subject was recorded under different activities states (normal walking, light exercise, moderate exercise, vigorous exercise, and rest after exercise) at five different moments, as shown in Fig. 3i. It can be observed that the armpit temperature increased gradually when the subject was exercising continuously, which was dropped at rest. As a result, the monitored body temperature will provide an insight into the physical state of a person, and the elevated body temperature will be an indication of fever or vigorous exercise. Meanwhile, a TENG device connected in parallel with a high-voltage diode and a switch can effectively create instantaneous discharging output, with an enhanced amplitude. Hence, an approach to create an instantaneous discharging for the socks with the integration of a high-voltage diode is feasible to speed up the charging time (more details can be found in Supplementary Note 1 and Supplementary Figs. 8–10). Through the incorporation of the proposed sock with external circuit design and other low-power consumption sensors of different functionalities, a more comprehensive IoT platform could be feasible in near future.

Development of deep learning-enabled socks

Considering the privacy concern of the face recognition, fingerprint recognition, and acoustics recognition, gait as a personal characteristic has many advantages for identification, i.e., individual gait recognition has fewer environmental limitations for data collection, it is relatively easy and quick to collect data in daily life49,77. Differ from the widely used IMU sensors that require high computing power and high computing power for precise body movement monitoring50, our deep learning-enabled sock with the minimal number of triboelectric sensors offers an alternative solution for the detection and analysis of the individual gait. Previously, analysis strategies for triboelectric outputs are to manually extract the shallow features from a single waveform, such as frequency, amplitude, interval of peaks, and holding time, which cannot achieve recognition of complicated features with subtle differences among them and is very susceptible to environmental variations, resulting in reduced recognition accuracy37,49,78. The technology fusion of the emerging AI with wearable electronics promoted enormous advances to form a whole intelligent system in technologies related to the process of data acquisition, processing/analysis, and transmission79,80. Recently, deep learning technologies due to its high level of performance across many types of data are becoming a very favored subset of machine learning, which offers a promising and feasible solution for triboelectric output analysis to achieve a higher accuracy of complex recognition. For the recent study on the integration of deep learning techniques and triboelectric devices, Zhao at al. has successfully used multiple deep belief network to realize the keystroke dynamic identification by a triboelectric keyboard81. Apart from that method, convolutional neural network (CNN) becomes another promising technology of deep learning for automatically extracting features from triboelectric output by its convolutional layers. In addition, the one-dimensional (1D) CNN-based method for recognizing human activity has been proven to be very effective for the overall dataset, in which the position of features in the segment is not highly correlated; thus, it is suitable for the analysis of time sequences of sensor data82,83,84,85. In our work, the raw data from sock which contains diversified information about a dynamic gait cycle, e.g., gait speed, contacting force between feet and shoes, triggering sequence of sensors and the individual usage manners for a specific user, will be fed into the training 1D CNN model directly without the complicated data processing steps. Figure 4a, b shows the gait pattern identification of five participants wearing the single-sensor sock with the 1D CNN method. A dataset was built from the five participants with proposed sock (45 kg female, 40 kg female, 60 kg male, 70 kg male, and 55 kg male), as they walked normally on a 5 m corridor within ~600 s. Next, the middle 500 s data (16,000 datapoints) representing the stable gait of participants will be segmented by a sliding window without overlap. For each participant, the whole data were divided into the training set (60%), testing set (20%), and validation set (20%). Because individual walking speed is an important consideration for gait analysis, the length of the sliding window needs to be carefully determined to ensure that each sample can cover enough information. In view of the trade-off between sample number and window length, the accuracy rate reaches the highest when the sample length is 4 s (Supplementary Fig. 11). The sample triboelectric outputs for the five participants are shown with the featured patterns, including some important features (Fig. 4b), i.e., amplitude, frequency, interval of events in gait cycle, etc. The proposed network architecture of the intelligent sock system (Fig. 4b) consists of four convolutional layers, four max-pooling layers, and one fully connected layer that outputs the predicted identification of five participants. The accuracy of training set is 100% after 500 epochs training of the neural network and the accuracy of validation set can be up to 98.4%. Then the trained 1D CNN model can assist the testing set of the intelligent sock to achieve 96% accuracy of identity recognition, and the confusion map of the classified result is shown in Fig. 4c. Although the sock with single sensor demonstrates a relatively good accuracy for differentiating the gait patterns of five participants, it could be largely reduced as the number of participants increases for it lacks spatiotemporal detecting of foot pressure. Using a small piece of 3 cm × 3 cm of the textile sensor to test the different outputs of various places around the foot is shown in Supplementary Fig. 12. The larger voltage means the better sensitivity to distinguish the gait pattern of different people; thus, we chose the three locations with the largest pressure for further research. By attaching a multi-pixel sensor array (i.e., three pieces) on the intelligent sock with the locations as shown in Fig. 4d, the intelligent sock should be able to collect more information from foot motions for more complicated gait analysis, and potentially raise the accuracy for a larger group of participants. It should be noted that the sensing outputs show negligible interference from shoe wearing (Supplementary Fig. 13), which demonstrates the good stability and adaptability of the intelligent sock in gait monitoring. In terms of the triple-sensor intelligent sock, after applying the same method of data collection, the triboelectric signals from three channels can form a whole spectrum and enter the input layer of same CNN structure (Fig. 4b) to classify the different users. The final accuracy of the triple-sensor intelligent sock can reach 100% (Fig. 4e), which is a 4% improvement over the single-sensor sock. In general, the larger the number of samples, the higher the classification accuracy of the trained model. To increase the accuracy of the recognition system, here we present a method of 4 s sliding window with 2 s overlap. It will extend the limited training set twice than the previous dataset. As shown in Supplementary Fig. 14, with the increase of participants, models trained with overlapping window datasets have higher accuracy. As a result, a dataset of 13 individual gait patterns was collected and a high accuracy of 93.54% was also achieved with the aforementioned sliding window method, as shown in Fig. 4f. The detailed parameters used for training a 13-class CNN are provided in Supplementary Table 1. Due to the limitation of the number of datasets, the accuracy rate decreases slightly as the number of people increases. This problem can be solved by expanding the training set samples, e.g., collecting more gait data from multiple participants.

Fig. 4: Gait identification of deep learning-enabled socks.
figure 4

a The process of data collection and the 4-s sliding window for the segment of dataset. b Schematics of the process and parameters for constructing the 1D CNN structure. c Confusion map of the prediction with the gait patterns of five participants. d Schematic of the triple-sensor socks and its three-channel output characteristics within 600 s. e, f Confusion maps of the prediction with the triple-sensor socks of 5 and 13 participants.

Full size image

The rapid development of VR and augmented reality (AR) technologies paves a way for the diversified application in social media, personal engagement, surgical training, gaming, and so on. The deep learning-enabled sock as a wearable HMI can provide the gait information of the users to create an immersive experience for VR/AR scenarios, and here we demonstrate a VR fitness game with the proposed intelligent sock as the control interface (Supplementary Video 2). As depicted by the schematics in Fig. 5a, the complete real-time control system includes the triple-sensor sock, signal preprocessing circuit, MCU with wireless transmitter module, PC, and the virtual space in Unity software. First, when the user moves, the triboelectric output signal from the sock will be generated and then enter the MCU after going through the preprocessing circuit. The preprocessing circuit is used for filtering out the ambient noise and removing the crosstalk between different channels, which is mainly composed of circuit blocks with several functions, i.e., bias circuit, amplifier circuit, and low-pass filter (further described in Supplementary Note 2 and Supplementary Fig. 15). Then the MCU will detect the equivalent analog signal of output voltage and send the data of the whole spectrum to the PC through wireless transmission. Based on the received spectrum of signals, the trained machine learning model will then send respective motion commands to Unity. In our demonstrated VR fitness game, there are five default movements available for detection to control the virtual character, including the leaping, running, sliding, jumping, and walking. The corresponding triboelectric outputs are shown in Fig. 5b, with respect to the five different motions in virtual space. After the training model of 1D CNN, the recognition accuracy of human activities reaches 96.67% with 100 training samples from each motion as shown in the confusion map (Fig. 5c). The program of unity can receive the command of the predicted motion and convert it into the motion of the virtual character, as show in Fig. 5d. For example, when the wood pile appears, the user should take the leap motion to pass it, which will be simultaneously synchronized onto the virtual character’s movement in virtual space. Next, users can choose run motion or walk motion in the woods path, and gravel road until the following barrier. When they are facing the rock and the stone cave, they should take the jump and slide motion, respectively. This VR demonstration well indicates the potential of the intelligent sock for the realization of the advanced multifunctional HMIs toward diversified applications. In the future, the intelligent system based on the proposed socks could be further integrated with personal fitness monitoring system, which measures and analyses data on physical performance during the VR gaming.

Fig. 5: Human activities recognition of deep learning-enabled socks.
figure 5

a The process flow from sensory information collection to the real-time prediction in VR fitness game. b 3D plots of the deep learning sock outputs responding to different motions (leap, run, slide, jump, and walk). c The confusion map for deep learning outcome. d The motion of the virtual character corresponding to real motion in a proposed digital human system.

Full size image

Since the deep learning-enabled sock is capable of recognizing identification and monitoring activity, it has a great aspect in the development of the intelligent wearable system, where the whole-body movement and the physical signals can be continuously monitored by the integration of the sock and other wearable sensors. Such a wearable system would facilitate the rapid advancement of the future digital human, where a realistic digital replicate of a human could be created in the virtual space. For a simple demonstration toward the future application of the digital human, a smart home system is developed to enable the automatic distinguishing of the family members from strangers, as well as the real-time monitoring of family member’s indoor activities in a no-camera environment, where the recognition results in the real space can be projected into a virtual space. Figure 6a summarizes the structure of the whole system, including the first stage of identity recognition, second stage of activity monitoring, and real-time VR projection. As shown in Fig. 6b, the server side of the smart home can establish wireless communication protocols with multiple users’ intelligent socks at the same time. The middle segment of the collected gait signals from the users will be sent to the first stage of smart home system separately (Fig. 6c) to recognize the user’s identity. It is known that triboelectric output can be susceptible to environmental variations, such as humidity, which will affect the recognition accuracy of the user’s identity as well. To eliminate the interferences from the environment, here we tried to collect the datasets of the users from multiple days to form a comprehensive training set so as to generalize the data. And it is observed that as the number of training days increases, the accuracy is improved significantly, showing the excellent stability of the deep learning-enabled sock in gait pattern identification (Supplementary Note 3 and Supplementary Fig. 16). These results also indicate a reliable and general method to improve the reliability of the triboelectric sensors, which is considered as one of the great challenges for them to be used for practical applications. After the identity recognition of first stage, the gait signals of family members will enter second stage of real-time activity monitoring to output the final individual activity. Here, three common actions (running, walking, and jumping) are extracted to build individual classifiers for second stage. The corresponding signals from the triple-sensor sock of each family member under walking states are shown in Fig. 6c. Integrating the trained identification model and motion recognition model, the laptop screen displays the signal waveform of gait and the corresponding prediction result. Simultaneously, the virtual space will generate the corresponding human model who will perform the same actions as the predicted motion of the gait in the second stage of the system. The proposed system provides a no-camera environment for the privacy and security aspect of the smart home, as shown in Supplementary Video 3.

Fig. 6: Integrated demonstration of smart home application.
figure 6

a Overview of the two-stage recognition system. b Schematics of multiple users with their gait signals collected by wireless communication. c The gait signal of two family members for gait identification in the first stage of smart home system. d The gait signals of recorded family members responding to different motions (run, walk, and jump) in the second stage of the smart home system.

Full size image

This integrated system of different deep learning models can also be used for future smart classroom setting. From Fig. 7a, b, a check-in system is demonstrated. All the signals obtained from the students who have entered the classroom will be identified and displayed in the system. As shown in Fig. 7c, when an unknown student moves from outside classroom to the classroom, the real-time gait signal will be collected and analyzed by the system to identify the student. And the students entering the classroom will be logged into the system via a trigger signal, as shown in Fig. 7d, i.e., stamping foot twice, and it will be synchronously displayed in the virtual space. After that, it can be detected whether the student is sitting on the seat or keeping moving around in Fig. 7e. The system not only can record the number of people participating in classes, but also can monitor the records of students entering and leaving the classroom (Supplementary Video 4). In addition, the user’s body temperature could be detected with an embedded temperature sensor and uploaded to the server together with the gait information, in which case the body temperature of all check-in students in the classroom will also be monitored, providing a more comprehensive personal status monitoring. Body temperature monitoring is not only beneficial for the user himself, but also helpful in controlling the wide spread of the contagious diseases that may cause a fever, as the detectable symptom. As a result, by leveraging the deep learning techniques, this intelligent sock can eventually turn into a solution for supplying a safer and more intelligent environment, regarding the smart building in no need for supports from cameras and microphones.

Fig. 7: The implementation of deep learning-enabled socks for smart classroom application.
figure 7

a, b The photograph of real classroom, the corresponding screenshot of virtual classroom in Unity and the corresponding demonstration of check-in system interface. c The gait signal from the man outside the classroom for gait identification. d The check-in signal from the woman entering the classroom. e The static sitting signals from the students sitting in the classroom.

Full size image

Discussion

In summary, a deep learning-enabled sock based on the triboelectric textile sensors has been developed with great potential of providing a low cost and low-power consumption solution to the future digital human. The sensing range and sensitivity of the triboelectric textile sensor are highly improved with the aid of the mm-scale frustum structures patterned on the contact surface, making it more suitable for gait sensing. By scavenging energy from normal walking, this sock can charge up a 27 µF capacitor in 3–4 min, which is sufficient to support a Bluetooth IoT sensor module to transmit temperature and humidity data. Furthermore, 96% accuracy of differentiating gait patterns among five participants has been achieved with the single-sensor sock by 1D CNN-based deep learning analytics. To further boost the gait patterns sensing accuracy of the intelligent sock, a triple-sensor sock can enhance the identification accuracy of 5 and 13 participants to 100% and 93.54%, respectively. More importantly, the proposed sock is capable of detecting different human activities with 96.67% accuracy among five predefined motions for an identified user. A VR fitness game with it as a control interface has been demonstrated. Combining the identification and activity monitoring capabilities, we demonstrate the deep learning-enabled sock as a functional part in both smart home and smart classroom applications by creating a virtual figure with replicated motions of the user, showing its great prospect in the development of the digital human in near future. In general, the proposed socks are equipped with diversified functionalities for the application of energy harvesting in IoT framework, and gait sensing for walking pattern recognition and human activity monitoring. Further explorations would obtain more comprehensive information of users ranging from entertainment, social network, healthcare, sports monitoring, and smart home, etc.

Methods

Fabrication of the triboelectric textile sensor

The triboelectric textile sensor contains two layers: a positive charge generation layer and a negative charge generation layer. Firstly, the conductive textile is cut into the desired size and shape, which is made of metalized fabric (polyester Cu) coated with an adhesive. To fabricate the positive charge generation layer, a thin nitrile film is attached to one side of a conductive textile. Another conductive textile is coated with silicone rubber film on the one side as well. The coating process was firstly dispensing required amounts of parts A and B of the EcoFlexTM 00-30 into a mixing container (1A:1B by volume or weight), followed by mixing the blend thoroughly for 3 min, and then the mixed solution was poured into a 3D-printed mold followed by 20-min baking at 70 °C for curing. For the textile sensor without surface structures, the uncured mixture was directly pasted onto the conductive textile to form a flat surface. Lastly, the silicone rubber-coated textile was stitched to the nitrile-coated textile with two nonconductive textiles attached to the outer sides for encapsulation, the detailed fabrication sees the Supplementary Information (Supplementary Figs. 1 and 2).

Experiment measurement and characterization

The signal outputs in the characterization of the T-TENG sensor were measured by an oscilloscope (DSO-X3034A, Agilent), using a high impedance probe of 100 MΩ. Calibrations of output voltage against force for triboelectric sensors were conducted by force gauge (Mecmesin, MultiTest 2.5-i) with the speed of 600 mm min−1. Finite element method simulation of electric potential between two triboelectric layers (Eco-flex and Nitrile) were numerically calculated using the commercial software COMSOL (Supplementary Fig. 5). The open-circuit voltages, transferred charges, and short-circuit currents were measured by an electrometer (Model 6514, Keithley), and the signals were displayed and recorded by an oscilloscope (DSO-X3034A, Agilent). The Bluetooth module for transmitting the data of temperature and humility sensors were made from commercial BLE sensors (CYALKIT-E02), with an integrated temperature and humility sensors (Si7020-A20). Analog voltage signals generated from intelligent socks for real-time monitoring, HMIs, and digital human system are collected and processed by the hardware circuit, consisting of a conditioner PCB and an MCU (ESP32-PICO-KIT V4). After the measurements, MATLAB®2019a was used for data analysis. The CNN models were developed in Python with Keras and Tensorflow backend.