Abstract
Sense of touch is a major part of man’s communication with their environment. Artificial skins can help robots to have the same sense of touch, especially for their social interactions. This paper presents a pressure mapping sensing using piezo-resistive fabric to represent aspects of the sense of touch. In past few years’ electrical impedance tomography (EIT) is considered to be able offer a good alternative for artificial skin in particular for its ease of adaptation for large area skin compared to individual matrix based sensors. The EIT has also very good temporal performance in data collection allowing for monitoring of fast responses to touch stimulation, enabling a truly real time touch sensing. Electromechanical responses of a conductive fabric can be exploited using EIT to create a low cost and large area touch sensing. Such electromechanical properties are often very complex, so to improve the imaging resolution and touch visibility an artificial intelligent (AI) was used in addition to the state of the art spatio-temporal imaging algorithm. This work demonstrates a step towards an integrated seamless skin with large area sensing in dynamical settings, closer to natural human skin’s behaviour. For the first time a dynamical touch sensing are studies by means of a spatio-temporal based electrical impedance tomography (EIT) imaging on a conductive fabric. The experimental results demonstrated the successful results by a combined AI with dynamical EIT imaging results in single and multiple points of touch.
Similar content being viewed by others
Introduction
Over the last two decades, there are various types of artificial skin based on different touch sensing techniques have been developed1. Piezoresistive sensor measures the resistance changes of the elastomer, foam, conductive carbon ink or other conductive material while the pressure force applied directly2,3. This well-structured touch sensor is commonly used on robotic hands, gives high resolution and low cost4,5. However, the high power consumption, low repeatability, and large hysteresis are some of the drawbacks of piezoresistive sensors. At the same hand, capacitive sensors have been widely used in robotic since they have relatively high sensitivity and can give good spatial resolution, but the structure of the sensors is more complicated due to its sandwich design and the capacitors arrays, which easily causes noise and interference6. Optoelectronic sensors use optical technology; the pressure can be detected by changing in light intensity or spectrum. It has the advantages of large sensing range, high spatial resolution, immune to electromagnetic interference and fast response, but they are huge in size, required high power supply7,8. Other tactile sensors using magnetic, piezoelectric, and ultrasonic are introduced and summarised in4,9.
Electrical impedance tomography (EIT) as an imaging technique, can determine the interior conductivity distribution of the object by only using measurements from electrodes placed on its boundary. The utilizations of the EIT have been successfully in biomedical, geophysical and industrial fields, it has been used for such as the brain and thorax monitor, liquid mixing and flow analysis because of the non-destructive property. Instead of the human body or other target, if a conductive material that will respond to local conductivity of permittivity changes is used, a pressure mapping imaging system or a touch sensing sensor can be created. Commonly, a large-scaled sensor array for robotic skin consists numbers of interior wires or electrodes, which can easily cause electromagnetic noise and affect on the flexibility and stretchability of the skin. Where, the EIT based fabric sensor were introduced to overcome this issue. The first tactile sensor based on EIT without wire and sensing elements in its region was presented in10, the rubber response to pressures with local changes in conductivity. To realise the light-weight, low-cost and stretchable artificial skin, single-layered conductive fabrics were used in11 and12. A highly stretchable tactile distribution sensor was used for smooth surfaced humanoids13. A tactile distribution sensor which enables stable measurement under high and dynamic stretch was also14. They demonstrated an improved EIT-based fabric sensor which can map local pressure. However, the image resolution of fabric EIT is experiencing low quality problem. Important information is surrounded by noise and other impact.
In this paper, the EIT is used for mapping touch over time and space. The work presents underlying mathematical framework of EIT-based fabric sensor, analyses some major sources of error and perform various dynamic experiments and improve image quality using deep learning for the first time.
Measurement System
Electrical impedance tomography (EIT) aims to reconstruct image of a conductivity distribution inside the testing object. EIT systems measure boundary voltages according to constant, low frequency and multiple injection currents to reconstruct cross-sectional images of the conductivity distribution15. A typical EIT system is shown in Fig. 1. All the proposed EIT systems are based on the basic methodology which are: (1) the data collection of a set of independent transfer electrical impedances. (2) The stable solutions of forward and inverse problems in order to reconstruct images with high quality16. To reconstruct EIT images in general geometrical settings, all functions are required which are described in details: a system for measuring the surface geometry, a mesh generator to produce a mesh based on the surface information, code for computing the finite element approximation and an algorithm to solve the inverse problem. Also, tools to display the results and also to analyse the images are required17. Therefore, 4 key parts of the EIT system are shown below, The basic EIT system has four main components: (i) EIT-Instrumentation, (ii) Electrode Array or EIT Sensors, (iii) PC with Reconstruction Algorithm, and (iv) Subject under test (SUT).
Method
To generate EIT images one needs to solve the forward model, which is physical model of measurement processes and the inverse problems of identifying the conductivity maps by the boundary measurements. Figure 2 shows these two mathematical problems in diagrammatic format.
The forward problem of EIT is determining the voltage distribution of a known domain by a given current and a specified conductivity distribution. The finite element method (FEM) is a numerical technique for solving practical differential equations, finite element meshing is utilized to discrete the domain and to calculate the voltage as a forward solver18. For an object occupy a two-dimensional region Ω with its boundary ∂Ω, electrodes e1, l = 1, 2, 3, …, L on the boundary. A current I with an angular frequency ω is applied to an adjacent electrodes pair, by calculating the divergence of both sides of the Maxwell equations, the following equation is obtained:
with the boundary condition
The boundary condition was introduced because of electrodes are inevitably mounted on the boundary of the sensing field19. Where σ and u are the distributions of conductivity and potential. And J is the electric current density on the boundary, ∂/∂n is the normal derivative to the boundary.
To get the current j, appropriate boundary condition need to be considered to build an accurate model for EIT. In this study we use the complete electrode model (CEM), which considers both the shunting effect of the electrodes and the contact impedance20. Given the CEM equation:
where zl indicates the contact impedance between the lth electrode and the tissue, n is the outward normal, V is the voltage and I is the current. Equation (5) indicates behaviour of the non-electrode boundary. In order to ensure the existence and the uniqueness of the solution, the law of the conservation of the charge is applied:
The forward operator F is then defined as a map from the conductivity distribution σ to the measured boundary voltage u, that is, \(\,F(\sigma )=u\).
The inverse problem in EIT can be defined as the recovery of a change in conductivity Δσ from a change in measured boundary voltage Δu. To solve this inverse problem, the matrix described the mapping between Δu on the boundary and Δσ in the interior is required. And we call this matrix as Jacobian Matrix, where can be written as \({\rm{\Delta }}u={\rm{A}}\,{\rm{\Delta }}{\rm{\sigma }}\). Based on the Reciprocity theorem, A is computed by the Fréchet derivative of u with respect to σ21. The regularization method used in this study is spatiotemporal total variation (ST-TV)22, as in dynamic fabric EIT, consecutively images are expected to show a strong correlation, where spatial and temporal gradient domains are involved based on total variation23. states that spatiotemporal total variation problem as:
where two terms of the function represent spatial and temporal behaviour, and where Δσ represents a 3D (x, y, t) conductivity distribution and \(\tilde{A}\) is an augmented Jacobian operating on a acts in a frame-by-frame basis24.
Results and Discussion
This work focuses on a Pressure Mapping Imaging based on EIT. The special sensor used is built up by a conductive fabric and a wooden circle frame with 16 electrodes on boundary. The conductive stretchable fabric LTT-SLPA used in the experiment was manufactured by Eeonyx Corp with the ability to stretch in both direction, surface resistivity (conductivity) changes as it is stretched or compressed, made by stretchy nylon/spandex and coated with a long-lasting conductive formulation, which gives a surface resistivity in the range of 104 to 107 ohm/sq. Two of the electrodes are used to input a constant current into the conductive fabric, other electrodes will get an initial voltage reading. And then, pressuring the conductive fabric over the circular sensor area, electrodes on boundary will generate a new voltage reading. Using this new reading minus initial voltage reading to get voltage difference (ΔV). The voltage difference is used to image pressure map by MATLAB processing. We conducted several experiments and two sets of experiments are shown here for comparative study.
One-object test
In this test, one finger was used to apply the pressure on fabric over the circular fabric sensor area (as shown in Fig. 3(a)). The background data was collected first, and then apply force from first test point. The pressure point was moving anticlockwise near boundary and moving towards center after one turn. KHU Mark 2.5 data collection system (Kyung Hee University, South Korea) is used with a 50 frames per second temporal. But when it comes to electromechanical sensing which involves mechanical reposes of the fabric this 50 frames per second cannot be achieved. The temporal resolution becomes around 1 frame/sec. In this case 397 sets of scan data were collected.
Figure 3(b) shows 21 snap of reconstructed images for a single finger touch going anticlockwise and from frame 13 moving towards top left and then through the center.
Two-object test
In this experiment, the same approach as test one, two pressure points applied at the same time and moving clockwise. In this case 252 sets of data were collected. Figure 4 shows clockwise movement of two finger touch points.
Temporal image reconstruction using state of the art TV algorithm performed very well, taking advantage of time correlation between frames of data. This includes ease of detection of pressure points in central area of fabric, which is always seen an issue for large area fabric EIT testing. There are still some artefacts remaining in those images and if we revert to a single frame by frame reconstruction (not taking advantage of temporal correlations), then these noises are more severe. Next section aims to address these issues with help from arterial intelligent.
Image processing – Deep learning
Since the beginning of EIT research, it is known that the shape mismatch and electrode position uncertainty are key sources of errors in imaging reconstruction22. In the most industrial applications and phantom simulations, the electrodes are stably mounted around boundary, reconstruction for these cases will not affected by the movement of electrode. However, in fabric EIT, the natural of fabric causes the shape mismatch when pressure applied.
The arrows in Fig. 5 indicated each electrode’s movement. In the image of phantom data, there is no electrode movement which gives a fairy good reconstruction result. For the fabric data, reconstructed images are damaged by the noise caused by electrode movement. The closer of the pressure point is to the boundary, the more the electrode will be affected around pressure point, the greater amplitude of the movement by these electrodes gives more collapse in image reconstruction. Also, the area around pressure point are blurred due to the net structure of the fabric, especially when there are more than one pressure points, the separation can be difficult like shown in Fig. 5(d).
There is no way to change the elastic structure of the fabric, and this is a great advantage in applications such as artificial skin and wearable monitoring. Therefore, implies post image processing is necessary to improve image quality and visibility caused by these inevitable physical phenomena. Figure 6 shows a reconstructed image and a typical thersholding, the initial image and include noises, but a systematic error such as the ones shown in Fig. 5 due to electrode movement could be removed by AI training.
The aim is to remove the electrodes artefacts which can be done with deep convolutional neural network. In general, a neural network consists of an input layer, a set of hidden layers formed of neurons and an output layer. A neuron contains all input information x, where x is a vector, interact with weights w (learnable parameter). The result of weighted sum xw passes through an activation function f. The output of the neuron is thus \(y=f\,(\sum {x}_{i}{w}_{i})\).
In the case of a classical neural network, each neuron takes all values in previous layer as input like in Fig. 7. This means for a network with an image as input and made up of approximately 10 to 20 layers, there will be millions of weights to learn at the same time and the number of calculations is huge. In fabric EIT images, the size is modest and yet the number of calculations remains very important. Which convolutional neural network is rather good when treating images due to the lower calculation complexity by parameter sharing, neurons do not take into account of all pixels as input. And this type of calculations considers neighborhood in images25.
A learning base is created by an input of a set of noisy images and an output of all cleared images. The network is updated with the information through learning, which is modifying weights w until reaching the desired result, and then, a set of unreleased images can be sent to the network to improve visibility. The learning base has to be as robust as possible. In our case, the input is images representing pressure point that travel all accessible areas of the fabric with a significant noise related to the electrodes. The output is the improved image that we want, in general, the perfect image is not known, that is why in this learning the “perfect” images have been created. An Example of producing a “perfect” image is shown in Fig. 8. A boundary suppression applied to remove electrode’s noise, and large size of blobs were kept only values greater than 55% of the maximum amplitude since even a small pressure point can produce a large blob due to the deformation of the material. In additional, small blobs that less than 15 pixels of area was removed.
In this image training, a U-net convolutional neural network implemented with Tensorflow (https://www.tensorflow.org/) is used, the structure of this network is shown in Fig. 9. This type of network is commonly used in biomedical image segmentation, also has good results during classification problems and image reconstruction26.
Training 1
The one object dataset training from experiment one is composed of 439 pairs of images where one pair of image contains a raw image and the image wanted. Once the training is done, new images with noise artefacts can be sent into the neural network, the output being an image hopefully improved. The proposed network could be simplified to a smaller network as the overall task in this problem of 2D and time is less demanding than some other challenging problems U-net can deal with.
Measurement of image clearing quality is done by two approaches.
-
With the formula of mean squared error:
$$MSE=E[{(image-imag{e}_{wanted})}^{2}]$$ -
And simply by exanimating the final image keeping in mind the dynamic of real experience.
The test of the network was done with the initial training data since almost each possible position is taken in this set, the set is exhaustive. Training results are shown in Fig. 10, electrodes noise is almost suppressed, 21 corresponding frames in Fig. 3b are greatly improved. The pressure points are clearly detected, and they are almost near “perfect” images since the test dataset is the same as training set.
Second test to check the robustness of the learning is to see how two objects images are improved through one object trained network. Test was made on 273 images from experiment two.
Results are shown in Fig. 11, compare to the images before processing in Fig. 4, pressure points are clearly detected, most noise caused by electrodes and fabric deformation are eliminated. Also to be noted that separation between objects is very well observed leading to better resolution. MSEs are calculated in Fig. 12, the error of the output has been reduced by more than half.
Training 2
The other approach is to train the network with two objects images. In this experiment the dataset of training was using the test set in last section, composed of 273 pairs of images, from two-object experiment. Once the training done the first test was to look at the improvement of image quality for two objects images.
Results are again really good, shown in Fig. 13, electrodes noise is mostly removed and separation between objects is well done.
Since the test dataset was the same as training dataset in last experiment, the second experiment is using one object dataset of 439 pairs of images which is the same one as for the one object learning in Training 1.
This is an interesting observation where the neural net was used to train by using two pressure points data and tested against a single point experiment, the results were very good shown in Fig. 14. The MSE of the output is reduced as desired shown in Fig. 15. Compare to the images in Fig. 3(b), there is a huge improvement in visibility, the amplitude and location of the pressure point is more specific, almost all error caused by electrodes movement are eliminated.
A conductive fabric behaviour is complex in both resistive and reactive terms under and applied AC current, because of the capacitances produced by the interactions of the conductive yarns and air gaps. A stretchable conductive fabric shows an even more complex behaviour. Due to complex nature of this sensing the deep learning approach shown in this paper actually does more than an image post processing, it overcome some of these model uncertainties that are extremely hard to physical model.
Conclusion
An ideal material used in EIT system for artificial skin would be light-weight, have continuous and homogeneous conductivity, low-cost, local conductivity changes in response to touch or pressure and will not be affected by stretching. The fabric used in this study is conductive, the conductivity changes as it is stretched, and it is light-weight and low-cast as wanted. This simplicity of sensor comes with some complex electromechanical behaviour which is not easy to model in forward model. Hence the use of the deep learning is justified in this application. In previous study, pressure or touch could be detected by fabric EIT system statically. The dynamic behaviour of the fabric EIT system were presented in this paper for the first time. The experimental shows good results of temporal performance in data collecting and generated a movie like pressure mapping at the same time. There are still some major errors caused by electrode movement and other inevitable shape-shift of the fabric in reconstruction images using spatiotemporal total variation algorithm. The hysteresis behaviour was shown during experiment, motivating future studies in time-related response of the fabric EIT system. In training stage, the results of test data set are going through trained net-work (using independent data set) are impressive in error reduction for all cases shown here. The way to improve image visibility and reduce these errors through neural network is effective and has the advantage of being autonomous.
References
Silvera-Tawil, D., Rye, D., Soleimani, M. & Velonaki, M. Electrical impedance tomography for artificial sensitive robotic skin: A review. ICE Sensors J. 15, 2001–2016 (2015).
Weiß, K. & Worn, H. The working principle of resistive tactile sensor cells. In Mechatronics and Automation, 2005 IEEE International Conference, vol. 1, 471–476 (IEEE, 2005).
Papakostas, T. V., Lima, J. & Lowe, M. A large area force sensor for smart skin applications. In Sensors, 2002. Proceedings of IEEE, vol. 2, 1620–1624 (IEEE, 2002).
Dahiya, R. S., Metta, G., Valle, M. & Sandini, G. Tactile sensing—from humans to humanoids. IEEE transactions on robotics 26, 1–20 (2010).
Choi, B., Lee, S., Choi, H. R. & Kang, S. Development of anthropomorphic robot hand with tactile sensor: Skku hand ii. In Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on, 3779–3784 (IEEE, 2006).
Schmitz, A. et al. Methods and technologies for the implementation of large-scale robot tactile sensors. IEEE Transactions on Robotics 27, 389–400 (2011).
Ohmura, Y., Kuniyoshi, Y. & Nagakubo, A. Conformable and scalable tactile sensor skin for curved surfaces. In Robotics and Automation, 2006. ICRA 2006. Proceedings 2006 IEEE International Conference on, 1348–1353 (IEEE, 2006).
Reimer, E. M. & Danisch, L. Pressure sensor based on illumination of a deformable integrating cavity US Patent 5,917,180 (1999).
Girão, P. S., Ramos, P. M. P., Postolache, O. & Pereira, J. M. D. Tactile sensors for robotic applications. Measurement 46, 1257–1271 (2013).
Kato, Y., Mukai, T., Hayakawa, T. & Shibata, T. Tactile sensor without wire and sensing element in the tactile region based on eit method. In Sensors, 2007 IEEE, 792–795 (IEEE, 2007).
Yao, A. & Soleimani, M. A pressure mapping imaging device based on electrical impedance tomography of conductive fabrics. Sens. Rev. 32, 310–317 (2012).
Yao, A., Yang, C. L., Seo, J. K. & Soleimani, M. Eit-based fabric pressure sensing. Comput. mathematical methods medicine 2013 (2013).
Alirezaei, H, Nagakubo, A & Kuniyoshi, Y A tactile distribution sensor which enables stable measurement under high and dynamic stretch. In 2009 IEEE Symposium on 3D User Interfaces, 87–93 (IEEE, 2009).
Nagakubo, A, Alirezaei, H & Kuniyoshi, Y. A deformable and deformation sensitive tactile distribution sensor. In 2007 IEEE International Conference on Robotics and Biomimetics (ROBIO), 1301–1308 (IEEE, 2007).
Hansen, P. C. Analysis of discrete ill-posed problems by means of the l-curve. SIAM review 34, 561–580 (1992).
Brown, B. H. Electrical impedance tomography (eit): a review. J. medical engineering &. technology 27, 97–108 (2003).
Borsic, A., Graham, B. M., Adler, A. & Lionheart, W. R. In vivo impedance imaging with total variation regularization. IEEE transactions on medical imaging 29, 44–54 (2010).
Vauhkonen, M., Lionheart, W. R., Heikkinen, L. M., Vauhkonen, P. J. & Kaipio, J. P. A matlab package for the eidors project to reconstruct two-dimensional eit images. Physiol. measurement 22, 107 (2001).
Cheng, A. H.-D. & Cheng, D. T. Heritage and early history of the boundary element method. Eng. Analysis with Boundary Elem. 29, 268–302 (2005).
Somersalo, E., Cheney, M. & Isaacson, D. Existence and uniqueness for electrode models for electric current computed tomography. SIAM J. on Appl. Math. 52, 1023–1040 (1992).
Brandstatter, B. Jacobian calculation for electrical impedance tomography based on the reciprocity principle. IEEE transactions on magnetics 39, 1309–1312 (2003).
Montesinos, P., Abascal, J. F. P., Cusso, L., Vaquero, J. J. & Desco, M. Application of the compressed sensing technique to self-gated cardiac cine sequences in small animals. Magn. resonance medicine 72, 369–380 (2014).
Chen, B., Abascal, J. & Soleimani, M. Electrical resistance tomography for visualization of moving objects using a spatiotemporal total variation regularization algorithm. Sensors 18, 1704 (2018).
Barber, D. & Brown, B. Recent developments in applied potential tomography-apt. In Information processing in medical imaging, 106–121 (Springer, 1986).
Jin, K. H., McCann, M. T., Froustey, E. & Unser, M. Deep convolutional neural network for inverse problems in imaging. IEEE Transactions on Image Process. 26, 4509–4522 (2017).
Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, 234–241 (Springer, 2015).
Author information
Authors and Affiliations
Contributions
M.S. conceived the concept; X.D. and M.S. conducted the experiments, reconstructed images and wrote this manuscript. S.T. helped with deep learning training and testing. Results are analyzed jointly by all authors. Research directed by M.S.
Corresponding authors
Ethics declarations
Competing Interests
The authors declare no competing interests.
Additional information
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Duan, X., Taurand, S. & Soleimani, M. Artificial skin through super-sensing method and electrical impedance data from conductive fabric with aid of deep learning. Sci Rep 9, 8831 (2019). https://doi.org/10.1038/s41598-019-45484-6
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41598-019-45484-6
This article is cited by
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.