Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Decoding of facial strains via conformable piezoelectric interfaces

Nature Biomedical Engineering volume 4, pages 954–972 (2020)Cite this article

Subjects

Abstract

Devices that facilitate nonverbal communication typically require high computational loads or have rigid and bulky form factors that are unsuitable for use on the face or on other curvilinear body surfaces. Here, we report the design and pilot testing of an integrated system for decoding facial strains and for predicting facial kinematics. The system consists of mass-manufacturable, conformable piezoelectric thin films for strain mapping; multiphysics modelling for analysing the nonlinear mechanical interactions between the conformable device and the epidermis; and three-dimensional digital image correlation for reconstructing soft-tissue surfaces under dynamic deformations as well as for informing device design and placement. In healthy individuals and in patients with amyotrophic lateral sclerosis, we show that the piezoelectric thin films, coupled with algorithms for the real-time detection and classification of distinct skin-deformation signatures, enable the reliable decoding of facial movements. The integrated system could be adapted for use in clinical settings as a nonverbal communication technology or for use in the monitoring of neuromuscular conditions.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: The system overview accompanying cFaCES.
Fig. 2: In vitro mechanical characterization of the cFaCES on a mock skin.
Fig. 3: In vivo mechanical characterization of the cFaCES on facial skin of healthy individuals and patients with ALS.
Fig. 4: 3D-DIC and theoretical modelling for prediction and validation of cFaCES performance in vivo.
Fig. 5: Sensor placement for RTD informed by analysis of skin strains from 3D-DIC.
Fig. 6: RTD of facial motions and library construction.

Similar content being viewed by others

Data availability

The data supporting the results in this study are available within the paper and its Supplementary Information. The raw patient data are available from the corresponding author, subject to approval from the Institutional Review Board of the Massachusetts Institute of Technology.

Code availability

Code used for addressing and capturing images from the cameras for 3D-DIC is available at GitHub (https://github.com/ConformableDecoders/PT-Grey-Image-Acquisition). Code used for 3D-DIC analysis is available at GitHub (https://github.com/MultiDIC/MultiDIC). Code used for RTD of facial deformations is available at GitHub (https://github.com/ConformableDecoders/cFaCES_RTD).

References

  1. Asheber, W. T., Lin, C.-Y. & Yen, S. H. Humanoid head face mechanism with expandable facial expressions. Int. J. Adv. Robot. Syst. 13, 29 (2016).

    Article  Google Scholar 

  2. Blow, M., Dautenhahn, K., Appleby, A., Nehaniv, C. L. & Lee, D. The art of designing robot faces: dimensions for human–robot interaction. In Proc. 1st ACM SIGCHI/SIGART Conference on Human–Robot Interaction Vol. 6 (Eds Goodrich, M. A. et al.) 331–332 (Association for Computing Machinery, 2006).

  3. Yagi, M. Mathematical modeling of aging effects in adulthood on the basis of smiling motions with skin mechanical properties. In Proc. 2013 4th International Conference on Intelligent Systems, Modelling and Simulation (Eds Al-Dabass, D. et al.) 182–185 (Institute of Electrical and Electronics Engineers, 2013).

  4. Shaw, P. J. Molecular and cellular pathways of neurodegeneration in motor neurone disease. J. Neurol. Neurosurg. Psychiatry 76, 1046–1057 (2005).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. GBD 2016 Motor Neuron Disease Collaborators. Global, regional, and national burden of motor neuron diseases 1990–2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet Neurol. 17, 1083–1097 (2018).

  6. Rong, P. et al. Predicting speech intelligibility decline in amyotrophic lateral sclerosis based on the deterioration of individual speech subsystems. PLoS ONE 11, e0154971 (2016).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  7. Bandini, A. et al. Automatic detection of amyotrophic lateral sclerosis (ALS) from video-based analysis of facial movements: speech and non-speech tasks. In Proc. 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018) (Eds Bhanu, B. et al.) 150–157 (Institute of Electrical and Electronics Engineers, 2018).

  8. Kapur, A., Kapur, S. & Maes, P. AlterEgo: a personalized wearable silent speech interface. In Proc. 23rd International Conference on Intelligent User Interfaces (Eds Berkovsky, S. et al.) 43–53 (Association for Computing Machinery, 2018).

  9. Shao, L. Facial movements recognition using multichannel EMG signals. In Proc. 2019 IEEE Fourth International Conference on Data Science in Cyberspace (DSC) (Eds Zhu, S. et al.) 561–566 (Institute of Electrical and Electronics Engineers, 2019).

  10. Essa, I. A. & Pentland, A. P. Facial expression recognition using a dynamic model and motion energy. In Proc. IEEE International Conference on Computer Vision 360–367 (Institute of Electrical and Electronics Engineers, 1995).

  11. Essa, I., Basu, S., Darrell, T. & Pentland, A. Modeling, tracking and interactive animation of faces and heads using input from video. In Proc. Computer Animation ‘96 Vol. 96 (Eds Thalmann, N. M. & Thalmann, D.) 68–79 (Institute of Electrical and Electronics Engineers, 1996).

  12. La Cascia, M., Valenti, L. & Sclaroff, S. Fully automatic, real-time detection of facial gestures from generic video. In Proc. IEEE 6th Workshop on Multimedia Signal Processing, 2004 (Ed. Barni, M), 175–178 (Institute of Electrical and Electronics Engineers, 2004).

  13. Wilson, A. J., Chin, B. C., Hsu, V. M., Mirzabeigi, M. N. & Percec, I. Digital image correlation: a novel dynamic three-dimensional imaging technique for precise quantification of the dynamic rhytid and botulinum toxin type A efficacy. Plast. Reconstr. Surg. 135, 869e–876e (2015).

    Article  CAS  PubMed  Google Scholar 

  14. Miura, N., Sakamoto, T., Aoyagi, Y. & Yoneyama, S. Visualizing surface strain distribution of facial skin using stereovision. Theor. Appl. Mech. Lett. 6, 167–170 (2016).

    Article  Google Scholar 

  15. Chen, Z. et al. Noninvasive, three-dimensional full-field body sensor for surface deformation monitoring of human body in vivo. J. Biomed. Opt. 22, 095001 (2017).

    Google Scholar 

  16. Dagnes, N. et al. Optimal marker set assessment for motion capture of 3D mimic facial movements. J. Biomech. 93, 86–93 (2019).

    Article  PubMed  Google Scholar 

  17. de Lucena, J. O., Lima, J. P., Thomas, D. & Teichrieb, V. Real-time facial motion capture using RGB-D images under complex motion and occlusions. In Proc. 2019 21st Symposium on Virtual and Augmented Reality (SVR) (Eds Raposo, A. & Trevisan, D.) 120–129 (Institute of Electrical and Electronics Engineers, 2019).

  18. Dagdeviren, C. et al. Conformal piezoelectric systems for clinical and experimental characterization of soft tissue biomechanics. Nat. Mater. 14, 728–736 (2015).

    Article  CAS  PubMed  Google Scholar 

  19. Yuan, J. et al. Computational models for the determination of depth-dependent mechanical properties of skin with a soft, flexible measurement device. Proc. R. Soc. A 472, 20160225 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  20. Feng, X. et al. Stretchable ferroelectric nanoribbons with wavy configurations on elastomeric substrates. ACS Nano 5, 3326–3332 (2011).

    Article  CAS  PubMed  Google Scholar 

  21. Dong, G. et al. Super-elastic ferroelectric single-crystal membrane with continuous electric dipole rotation. Science 366, 475–479 (2019).

    Article  CAS  PubMed  Google Scholar 

  22. Dagdeviren, C. et al. Transient, biocompatible electronics and energy harvesters based on ZnO. Small 9, 3398–3404 (2013).

    Article  CAS  PubMed  Google Scholar 

  23. Persano, L. et al. High performance piezoelectric devices based on aligned arrays of nanofibers of poly(vinylidenefluoride-co-trifluoroethylene). Nat. Commun. 4, 1633 (2013).

    Article  CAS  PubMed  Google Scholar 

  24. Persano, L. et al. Shear piezoelectricity in poly(vinylidenefluoride-co-trifluoroethylene): full piezotensor coefficients by molecular modeling, biaxial transverse response, and use in suspended energy-harvesting nanostructures. Adv. Mater. 28, 7633–7639 (2016).

    Article  CAS  PubMed  Google Scholar 

  25. Dagdeviren, C. et al. Recent progress in flexible and stretchable piezoelectric devices for mechanical energy harvesting, sensing and actuation. Extrem. Mech. Lett. 9, 269–281 (2016).

    Article  Google Scholar 

  26. Dagdeviren, C., Li, Z. & Wang, Z. L. Energy harvesting from the animal/human body for self-powered electronics. Annu. Rev. Biomed. Eng. 19, 85–108 (2017).

    Article  CAS  PubMed  Google Scholar 

  27. Dagdeviren, C. et al. Conformal piezoelectric energy harvesting and storage from motions of the heart, lung, and diaphragm. Proc. Natl Acad. Sci. USA 111, 1927–1932 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  28. Dagdeviren, C. et al. Flexible piezoelectric devices for gastrointestinal motility sensing. Nat. Biomed. Eng. 1, 807–817 (2017).

    Article  CAS  PubMed  Google Scholar 

  29. Niu, S. et al. A wireless body area sensor network based on stretchable passive tags. Nat. Electron. 2, 361–368 (2019).

    Article  Google Scholar 

  30. Dagdeviren, C. et al. Conformable amplified lead zirconate titanate sensors with enhanced piezoelectric response for cutaneous pressure monitoring. Nat. Commun. 5, 4496 (2014).

    Article  CAS  PubMed  Google Scholar 

  31. Yeo, W.-H. et al. Multifunctional epidermal electronics printed directly onto the skin. Adv. Mater. 25, 2773–2778 (2013).

    Article  CAS  PubMed  Google Scholar 

  32. Akiyama, M. et al. Preparation of oriented aluminum nitride thin films on polyimide films and piezoelectric response with high thermal stability and flexibility. Adv. Funct. Mater. 17, 458–462 (2007).

    Article  CAS  Google Scholar 

  33. Fei, C. et al. AlN piezoelectric thin films for energy harvesting and acoustic devices. Nano Energy 51, 146–161 (2018).

    Article  CAS  Google Scholar 

  34. Doll, J. C., Petzold, B. C., Ninan, B., Mullapudi, R. & Pruitt, B. L. Aluminum nitride on titanium for CMOS compatible piezoelectric transducers. J. Micromech. Microeng. 20, 025008 (2009).

    Article  CAS  Google Scholar 

  35. Shelton, S. et al. CMOS-compatible AlN piezoelectric micromachined ultrasonic transducers. In Proc. 2009 IEEE International Ultrasonics Symposium (Ed. Yuhas, M. P.) 402–405 (Institute of Electrical and Electronics Engineers, 2009).

  36. Rödel, J. et al. Transferring lead-free piezoelectric ceramics into application. J. Eur. Ceram. Soc. 35, 1659–1681 (2015).

    Article  CAS  Google Scholar 

  37. Priya, S. & Nahm, S. Lead-Free Piezoelectrics (Springer Science & Business Media, 2011).

  38. Wang, S. et al. Mechanics of epidermal electronics. J. Appl. Mech. 79, 031022 (2012).

    Article  Google Scholar 

  39. Kim, D.-H. et al. Epidermal electronics. Science 333, 838–843 (2011).

    Article  CAS  PubMed  Google Scholar 

  40. Solav, D., Moerman, K. M., Jaeger, A. M., Genovese, K. & Herr, H. M. MultiDIC: an open-source toolbox for multi-view 3D digital image correlation. IEEE Access 6, 30520–30535 (2018).

    Article  Google Scholar 

  41. Solav, D., Moerman, K. M., Jaeger, A. M. & Herr, H. A framework for measuring the time-varying shape and full-field deformation of residual limbs using 3D digital image correlation. IEEE Trans. Biomed. Eng. 66, 2740–2752 (2019).

    Article  PubMed  Google Scholar 

  42. Blaber, J., Adair, B. & Antoniou, A. Ncorr: open-source 2D digital image correlation MATLAB software. Exp. Mech. 55, 1105–1122 (2015).

    Article  Google Scholar 

  43. Pan, B., Qian, K., Xie, H. & Asundi, A. Two-dimensional digital image correlation for in-plane displacement and strain measurement: a review. Meas. Sci. Technol. 20, 062001 (2009).

    Article  CAS  Google Scholar 

  44. Solav, D., Rubin, M. B., Cereatti, A., Camomilla, V. & Wolf, A. Bone pose estimation in the presence of soft tissue artifact using triangular cosserat point elements. Ann. Biomed. Eng. 44, 1181–1190 (2016).

    Article  PubMed  Google Scholar 

  45. Solav, D. et al. Chest wall kinematics using triangular cosserat point elements in healthy and neuromuscular subjects. Ann. Biomed. Eng. 45, 1963–1973 (2017).

    Article  PubMed  Google Scholar 

  46. Zhao, Y. et al. Investigation of mechanical behaviour of amorphous aluminium nitride. Materialia 2, 148–156 (2018).

    Article  Google Scholar 

  47. Ansari, M. & Amin Karami, M. Experimental study on nonlinear thermally buckled piezoelectric energy harvesters for leadless pacemakers. In Proc. Active and Passive Smart Structures and Integrated Systems XII (Eds Erturk, A. & Han, J.-H.) 105951A (Society of Photo-Optical Instrumentation Engineers, 2018).

  48. Ansari, M. H. & Amin Karami, M. Energy harvesting from controlled buckling of piezoelectric beams. Smart Mater. Struct. 24, 115005 (2015).

    Article  CAS  Google Scholar 

  49. Ansari, M. H. & Amin Karami, M. Nonlinear thermally buckled piezoelectric energy harvester. In Proc. ASME 2016 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. Vol. 6: 12th International Conference on Multibody Systems, Nonlinear Dynamics, and Control V006T09A065 (American Society of Mechanical Engineers, 2016).

  50. Karami, M. A., Inman, D. J. & Ansari, M. H. Energy harvesting from constrained buckling of piezoelectric beams. US patent 10447177 (2019).

  51. Ansari, M. H. & Karami, M. A. Energy harvesting from controlled buckling of a horizontal piezoelectric beam. In Proc. ASME 2014 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference V008T11A017 (American Society of Mechanical Engineers Digital Collection, 2015).

  52. Karami, M. A. & Inman, D. J. Controlled buckling of piezoelectric beams for direct energy harvesting from passing vehicles. In Proc. ASME 2012 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference 1231–1236 (American Society of Mechanical Engineers Digital Collection, 2013).

  53. Abou-Rayan, A. M., Nayfeh, A. H., Mook, D. T. & Nayfeh, M. A. Nonlinear response of a parametrically excited buckled beam. Nonlinear Dyn. 4, 499–525 (1993).

    Article  Google Scholar 

  54. Erturk, A. & Inman, D. J. An experimentally validated bimorph cantilever model for piezoelectric energy harvesting from base excitations. Smart Mater. Struct. 18, 025009 (2009).

    Article  CAS  Google Scholar 

  55. Karami, M. A. & Inman, D. J. Equivalent damping and frequency change for linear and nonlinear hybrid vibrational energy harvesting systems. J. Sound Vib. 330, 5583–5597 (2011).

    Article  Google Scholar 

  56. Nayfeh, A. H. & Frank Pai, P. Linear and Nonlinear Structural Mechanics (John Wiley & Sons, 2004).

  57. Virgin, L. N. Vibration of Axially Loaded Structures (Cambridge University Press, 2007).

  58. Sirohi, J. & Chopra, I. Fundamental understanding of piezoelectric strain sensors. In Proc. Smart Structures and Materials 1999: Smart Structures and Integrated Systems (Ed. Wereley, N. M.) 528–542 (Society of Photo-Optical Instrumentation Engineers, 1999).

  59. Varatharajan, R., Manogaran, G., Priyan, M. K. & Sundarasekar, R. Wearable sensor devices for early detection of Alzheimer disease using dynamic time warping algorithm. Clust. Comput. 21, 681–690 (2018).

    Article  Google Scholar 

  60. Zhang, Z. et al. Dynamic time warping under limited warping path length. Inf. Sci. 393, 91–107 (2017).

    Article  Google Scholar 

  61. Wan, Y., Chen, X.-L. & Shi, Y. Adaptive cost dynamic time warping distance in time series analysis for classification. J. Comput. Appl. Math. 319, 514–520 (2017).

    Article  Google Scholar 

  62. Salvador, S. & Chan, P. Toward accurate dynamic time warping in linear time and space. Intell. Data Anal. 11, 561–580 (2007).

    Article  Google Scholar 

  63. Aristidou, A., Cohen-Or, D. & Hodgins, J. K. Self-similarity analysis for motion capture cleaning. Comput. Graph. 37, 297–309 (2018).

    Google Scholar 

  64. Zhao, W. et al. Real-time vehicle motion detection and motion altering for connected vehicle: algorithm design and practical applications. Sensors 19, 4108 (2019).

    Article  PubMed Central  Google Scholar 

  65. Tait, R. N. & Mirfazli, A. Low temperature aluminum nitride deposition on aluminum by rf reactive sputtering. J. Vac. Sci. Technol. A 19, 1586–1590 (2001).

    Article  CAS  Google Scholar 

  66. Iqbal, A. & Mohd-Yasin, F. Reactive sputtering of aluminum nitride (002) thin films for piezoelectric applications: a review. Sensors 18, 1797 (2018).

    Article  CAS  PubMed Central  Google Scholar 

  67. Zhang, Q. M. & Zhao, J. Electromechanical properties of lead zirconate titanate piezoceramics under the influence of mechanical stresses. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 46, 1518–1526 (1999).

    Article  CAS  PubMed  Google Scholar 

  68. Jiang, X. et al. Monolithic ultrasound fingerprint sensor. Microsyst. Nanoeng. 3, 17059 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  69. Tadigadapa, S. & Mateti, K. Piezoelectric MEMS sensors: state-of-the-art and perspectives. Meas. Sci. Technol. 20, 092001 (2009).

    Article  CAS  Google Scholar 

  70. Ruby, R. The ‘how & why’ a deceptively simple acoustic resonator became the basis of a multi-billion dollar industry. In Proc. 2017 IEEE 30th International Conference on Micro Electro Mechanical Systems (MEMS) (Eds Nguyen, C. & Meng, E.) 308–314 (Institute of Electrical and Electronics Engineers, 2017).

  71. Chen, G. & Rinaldi, M. Aluminum nitride combined overtone resonators for the 5G high frequency bands. J. Microelectromech. Syst. 29, 148–159 (2020).

  72. Petroni, S. et al. Tactile multisensing on flexible aluminum nitride. Analyst 137, 5260–5264 (2012).

    Article  CAS  PubMed  Google Scholar 

  73. Petroni, S. et al. Aluminum nitride piezo-MEMS on polyimide flexible substrates. Microelectron. Eng. 88, 2372–2375 (2011).

    Article  CAS  Google Scholar 

  74. Akiyama, M. et al. Flexible piezoelectric pressure sensors using oriented aluminum nitride thin films prepared on polyethylene terephthalate films. J. Appl. Phys. 100, 114318 (2006).

    Article  CAS  Google Scholar 

  75. Jackson, N., Keeney, L. & Mathewson, A. Flexible-CMOS and biocompatible piezoelectric AlN material for MEMS applications. Smart Mater. Struct. 22, 115033 (2013).

    Article  CAS  Google Scholar 

  76. Li, Q. et al. Growth and characterization of polyimide-supported AlN films for flexible surface acoustic wave devices. J. Electron. Mater. 45, 2702–2709 (2016).

    Article  CAS  Google Scholar 

  77. Bi, X., Wu, Y., Wu, J., Li, H. & Zhou, L. A model for longitudinal piezoelectric coefficient measurement of the aluminum nitride thin films. J. Mater. Sci. Mater. Electron. 25, 2435–2442 (2014).

    Article  CAS  Google Scholar 

  78. Yang, J. et al. Growth of AlN films as a function of temperature on Mo films deposited by different techniques. J. Electron. Mater. 43, 369–374 (2014).

    Article  CAS  Google Scholar 

  79. Lu, Y. et al. Surface morphology and microstructure of pulsed DC magnetron sputtered piezoelectric AlN and AlScN thin films. Phys. Status Solidi 215, 1700559 (2018).

    Article  CAS  Google Scholar 

  80. Martin, F., Muralt, P., Dubois, M.-A. & Pezous, A. Thickness dependence of the properties of highly c-axis textured AlN thin films. J. Vac. Sci. Technol. A 22, 361–365 (2004).

    Article  CAS  Google Scholar 

  81. Miyanaga, M. et al. Evaluation of AlN single-crystal grown by sublimation method. J. Cryst. Growth 300, 45–49 (2007).

    Article  CAS  Google Scholar 

  82. Sanz-Hervás, A. et al. Comparative study of c-axis AlN films sputtered on metallic surfaces. Diam. Relat. Mater. 14, 1198–1202 (2005).

    Article  CAS  Google Scholar 

  83. Yarar, E. et al. Low temperature aluminum nitride thin films for sensory applications. AIP Adv. 6, 075115 (2016).

    Article  CAS  Google Scholar 

  84. Singh, A. V., Chandra, S. & Bose, G. Deposition and characterization of c-axis oriented aluminum nitride films by radio frequency magnetron sputtering without external substrate heating. Thin Solid Films 519, 5846–5853 (2011).

    Article  CAS  Google Scholar 

  85. Tay, K.-W., Huang, C.-L., Wu, L. & Lin, M.-S. Performance characterization of thin AlN films deposited on Mo electrode for thin-film bulk acoustic-wave resonators. Jpn. J. Appl. Phys. 43, 5510 (2004).

    Article  CAS  Google Scholar 

  86. Schnable, G. L. & Keen, R. S. Aluminum metallization—advantages and limitations for integrated circuit applications. Proc. IEEE 57, 1570–1580 (1969).

    Article  CAS  Google Scholar 

  87. Obuh, I. E. et al. Low-cost microfabrication for MEMS switches and varactors. IEEE Trans. Compon. Packaging Manuf. Technol. 8, 1702–1710 (2018).

    Article  CAS  Google Scholar 

  88. Baeg, K.-J., Bae, G.-T. & Noh, Y.-Y. Efficient charge injection in p-type polymer field-effect transistors with low-cost molybdenum electrodes through V2O5 interlayer. ACS Appl. Mater. Inter. 5, 5804–5810 (2013).

    Article  CAS  Google Scholar 

  89. Rogers, J. A., Someya, T. & Huang, Y. Materials and mechanics for stretchable electronics. Science 327, 1603–1607 (2010).

    Article  CAS  PubMed  Google Scholar 

  90. Artieda, A., Barbieri, M., Sandu, C. S. & Muralt, P. Effect of substrate roughness on c-oriented AlN thin films. J. Appl. Phys. 105, 024504 (2009).

    Article  CAS  Google Scholar 

  91. Luboz, V., Promayon, E. & Payan, Y. Linear elastic properties of the facial soft tissues using an aspiration device: towards patient specific characterization. Ann. Biomed. Eng. 42, 2369–2378 (2014).

    Article  CAS  PubMed  Google Scholar 

  92. Kim, Y.-S. et al. Regional thickness of facial skin and superficial fat: application to the minimally invasive procedures. Clin. Anat. 32, 1008–1018 (2019).

    Article  PubMed  Google Scholar 

  93. Leo, D. J. Engineering Analysis of Smart Material Systems (John Wiley & Sons, 2007).

  94. Goodno, B. J. & Gere, J. M. Mechanics of Materials SI edn (Cengage Learning, 2017).

  95. Popov, E. P. & Balan, T. A. Engineering Mechanics of Solids 2nd edn (Prentice Hall, 1999).

  96. Bauchau, O. A. & Craig, J. I. Structural Analysis: With Applications to Aerospace Structures (Springer Science & Business Media, 2009).

  97. Rao, S. S. Vibration of Continuous Systems (John Wiley & Sons, 2019).

  98. Lepi, S. Practical Guide to Finite Elements: A Solid Mechanics Approach (CRC Press, 1998).

  99. Inman, D. J. Engineering Vibration (Prentice Hall, 2001).

  100. Rao, S. S. in Vibration of Continuous Systems 393–419 (John Wiley & Sons, 2006).

  101. Tsagkrasoulis, D., Hysi, P., Spector, T. & Montana, G. Heritability maps of human face morphology through large-scale automated three-dimensional phenotyping. Sci. Rep. 7, 45885 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  102. Du, S. et al. A new electrode design method in piezoelectric vibration energy harvesters to maximize output power. Sensor. Actuat. A Phys. 263, 693–701 (2017).

    Article  CAS  Google Scholar 

  103. Erturk, A. & Inman, D. J. Piezoelectric Energy Harvesting (John Wiley & Sons, 2011).

  104. Nayfeh, A. H. & Balachandran, B. Applied Nonlinear Dynamics: Analytical, Computational, and Experimental Methods (John Wiley & Sons, 2008).

  105. Karami, M. A., Varoto, P. S. & Inman, D. J. Experimental study of the nonlinear hybrid energy harvesting system. In Modal Analysis Topics: Proc. Society for Experimental Mechanics Series. Vol. 3 (Ed. Proulx, T.) 461–478 (Springer, 2011).

  106. Strogatz, S. H. Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering (CRC Press, 2018).

  107. Baker, S. & Matthews, I. Lucas-Kanade 20 years on: a unifying framework. Int. J. Comput. Vis. 56, 221–255 (2004).

    Article  Google Scholar 

  108. Pan, B. Reliability-guided digital image correlation for image deformation measurement. Appl. Opt. 48, 1535–1542 (2009).

    Article  PubMed  Google Scholar 

  109. Koydemir, H. C. & Ozcan, A. Wearable and implantable sensors for biomedical applications. Annu. Rev. Anal. Chem. 11, 127–146 (2018).

    Article  CAS  Google Scholar 

  110. Coletta, N. A., Mallette, M. M., Gabriel, D. A., Tyler, C. J. & Cheung, S. S. Core and skin temperature influences on the surface electromyographic responses to an isometric force and position task. PLoS ONE 13, e0195219 (2018).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

Download references

Acknowledgements

C.D. thanks the late S. Hawking for the discussion on 25 April 2016 at the Harvard Society of Fellows, and for inspiring this research since then; Y. Büyükerşen for his suggestions on face painting material selections and wax sculpting; and M. Mercan for his support and discussions during the manuscript preparation. F.T. and C.D. thank D. Roy and D. Beeferman for initial discussions concerning kNN–DTW. F.T. thanks K. Warren for discussions on sensor characterization and R. Wiken for discussions on the design and fabrication of 3D-DIC set-ups. C.D., F.T. and T.S. thank R. Brown of the University of Massachusetts Medical School for helping to recruit the patients with ALS and for discussion on cFaCES application on the patients with ALS. We thank the families of P. Gerber and D. Ceruti for their help and dedication in trials of patients with ALS; and members of the microfabrication facility/cleanroom of the Conformable Decoders research group at the MIT Media Lab, the YellowBox and the Instron Laboratory of the Koch Institute For Integrative Cancer Research at MIT. C.D. acknowledges that this research was supported by MIT Media Lab Consortium funding and the National Science Foundation under NSF award no. 2026344. This work was performed in part at the Center for Nanoscale Systems (CNS), which is a member of the National Nanotechnology Coordinated Infrastructure Network (NNCI), which was supported by the National Science Foundation under NSF award no. 1541959. CNS is part of Harvard University. M.A.K. acknowledges the National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under award no. R21EB023613, which partially supported this work. M.A.K. also acknowledges the National Science Foundation under grant no. 1905252, which partially supported this work. T.S. and Y.G. acknowledge the Institute of Microelectronics (IME), A*STAR, Singapore for funds for initial sensor fabrication and materials.

Author information

Author notes

  1. These authors contributed equally: Tao Sun, Farita Tasnim.

Authors and Affiliations

  1. Media Lab, Massachusetts Institute of Technology, Cambridge, MA, USA

    Tao Sun, Farita Tasnim, Rachel T. McIntosh, Dana Solav, David Sadat, Lin Zhang & Canan Dagdeviren

  2. Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA

    Rachel T. McIntosh

  3. Department of Mechanical and Aerospace Engineering, University at Buffalo, Buffalo, NY, USA

    Nikta Amiri, Mostafa Tavakkoli Anbarani & M. Amin Karami

  4. Faculty of Mechanical Engineering, Technion Israel Institute of Technology, Haifa, Israel

    Dana Solav

  5. Institute of Microelectronics, Agency for Science, Technology and Research (A*STAR), Singapore, Singapore

    Yuandong Gu

Authors
  1. Tao Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Farita Tasnim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rachel T. McIntosh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nikta Amiri
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dana Solav
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mostafa Tavakkoli Anbarani
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. David Sadat
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Lin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Yuandong Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. M. Amin Karami
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Canan Dagdeviren
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

C.D. conceived the overall research goals and aims. C.D., T.S. and F.T. designed the experiments. T.S. and Y.G. fabricated the initial devices. C.D., F.T., D. Sadat and L.Z. assisted with the device fabrication and conducting in vitro device characterization experiments. R.T.M. and F.T. built the DIC set-ups and executed the DIC experiments, performed the in vivo human trials, performed data analysis and organized the results. D. Solav assisted with DIC set-up design, data analysis and interpretation. F.T. designed and built the RTD set-up. F.T. and R.T.M. conducted RTD trials. N.A., M.T.A. and M.A.K. conducted the theoretical calculations and FEM. C.D., F.T. and D. Sadat composed the layout of Supplementary Videos 1–8; and D. Sadat formed all of the videos. All of the authors contributed to writing the manuscript.

Corresponding author

Correspondence to Canan Dagdeviren.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary methods, figures and tables, and captions for Supplementary Videos 1–8.

Reporting Summary

Peer-review information

Supplementary Video 1

Method to make cFaCES visually invisible with the facial skin.

Supplementary Video 2

3D-DIC set-up and an example of results of the trial for the motion OM without the sensor.

Supplementary Video 3

3D-DIC set-up and an example of the results of the trial for the motion PL without the sensor.

Supplementary Video 4

3D-DIC set-up and an example of the results of the trial for the motion SM without the sensor.

Supplementary Video 5

3D-DIC set-up and an example of the results of the trial for the motion OM with the sensor.

Supplementary Video 6

3D-DIC set-up and an example of the results of the trial for the motion PL with the sensor.

Supplementary Video 7

3D-DIC set-up and an example of the results of the trial for the motion SM with the sensor.

Supplementary Video 8

RTD set-up, trials and an example of usage.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, T., Tasnim, F., McIntosh, R.T. et al. Decoding of facial strains via conformable piezoelectric interfaces. Nat Biomed Eng 4, 954–972 (2020). https://doi.org/10.1038/s41551-020-00612-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41551-020-00612-w

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing