Shared human–robot proportional control of a dexterous myoelectric prosthesis

Abstract

Myoelectric prostheses allow users to recover lost functionality by controlling a robotic device with their remaining muscle activity. Such commercial devices can give users a high level of autonomy, but still do not approach the dexterity of the intact human hand. Here we present a method to control a robotic hand, shared between user intention and robotic automation. The algorithm allows user-controlled movements when high dexterity is desired, but also assisted grasping when robustness is paramount. This combination of features is currently lacking in commercial prostheses and can greatly improve prosthesis usability. First, we design and test a myoelectric proportional controller that can predict multiple joint angles simultaneously and with high accuracy. We then implement online control with both able-bodied and amputee subjects. Finally, we present a shared control scheme in which robotic automation aids in object grasping by maximizing the contact area between the hand and the object, greatly increasing grasp success and object hold times in both a virtual and a physical environment. Our results present a viable method of prosthesis control implemented in real time, for reliable articulation of multiple simultaneous degrees of freedom.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Experimental set-up and subjects.
Fig. 2: Analysis of online prediction performance of the MLP.
Fig. 3: Shared control in a virtual environment, set-up and results.
Fig. 4: Shared control results in virtual environment, continued from Fig. 3.
Fig. 5: Shared control in a physical environment, set-up and results.
Fig. 6: EMG analysis with and without shared control.

Data availability

The data that support the findings of this study are available within the paper and its Supplementary Information. All datasets generated for this study (EMG signals, variables recorded from the virtual and real hands during the experiments, and decoding algorithms) are available from the corresponding author upon reasonable request.

Code availability

The MATLAB code used for data analysis and synthesis of results presented in this study is available at https://github.com/KZzizzle/0713.git. Data collection code is available from the corresponding author on reasonable request.

References

  1. 1.

    Ziegler-Graham, K., MacKenzie, E. J., Ephraim, P. L., Travison, T. G. & Brookmeyer, R. Estimating the prevalence of limb loss in the United States: 2005 to 2050. Arch. Phys. Med. Rehabil. 89, 422–429 (2008).

    Article  Google Scholar 

  2. 2.

    Watve, S., Dodd, G., MacDonald, R. & Stoppard, E. R. Upper limb prosthetic rehabilitation. Orthop. Trauma 25, 135–142 (2011).

    Article  Google Scholar 

  3. 3.

    Geethanjali, P. Myoelectric control of prosthetic hands: state-of-the-art review. Med. Devices 9, 247–255 (2016).

    Article  Google Scholar 

  4. 4.

    Biddiss, E. & Chau, T. Upper-limb prosthetics: critical factors in device abandonment. Am. J. Phys. Med. Rehabil. 86, 977–987 (2007).

    Article  Google Scholar 

  5. 5.

    Biddiss, E. A. & Chau, T. T. Upper limb prosthesis use and abandonment: a survey of the last 25 years. Prosthet. Orthot. Int. 31, 236–257 (2007).

    Article  Google Scholar 

  6. 6.

    Farina, D. et al. The extraction of neural information from the surface EMG for the control of upper-limb prostheses: emerging avenues and challenges. IEEE Trans. Neural Syst. Rehabil. Eng. 22, 797–809 (2014).

    Article  Google Scholar 

  7. 7.

    Hioki, M. & Kawasaki, H. Estimation of finger joint angles from sEMG using a neural network including time delay factor and recurrent Structure. ISRN Rehabilitation 2012, 1–13 (2012).

    Article  Google Scholar 

  8. 8.

    Malešević, N. et al. Decoding of individual finger movements from surface EMG signals using vector autoregressive hierarchical hidden Markov models (VARHHMM). In Proc. 2017 International Conference on Rehabilitation Robotics 1518–1523 (IEEE, 2017).

  9. 9.

    Tenore, F. V. G. et al. Decoding of individuated finger movements using surface electromyography. IEEE Trans. Biomed. Eng. 56, 1427–1434 (2009).

    Article  Google Scholar 

  10. 10.

    Smith, R. J., Tenore, F., Huberdeau, D., Etienne-Cummings, R. & Thakor, N. V. Continuous decoding of finger position from surface EMG signals for the control of powered prostheses. In 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society 197–200 (IEEE, 2008).

  11. 11.

    Ngeo, J. G., Tamei, T. & Shibata, T. Continuous and simultaneous estimation of finger kinematics using inputs from an EMG-to-muscle activation model. J. NeuroEng. Rehabil. 11, 122 (2014).

    Article  Google Scholar 

  12. 12.

    Krasoulis, A., Vijayakumar, S. & Nazarpour, K. Evaluation of regression methods for the continuous decoding of finger movement from surface EMG and accelerometry. In Proc. 7th International IEEE/EMBS Conference on Neural Engineering 631–634 (IEEE, 2015).

  13. 13.

    Cipriani, C. et al. Online myoelectric control of a dexterous hand prosthesis by transradial amputees. IEEE Trans. Neural Syst. Rehabil. Eng. 19, 260–270 (2011).

    Article  Google Scholar 

  14. 14.

    Jiang, N., Dosen, S., Muller, K. R. & Farina, D. Myoelectric control of artificial limbs—Is there a need to change focus?. IEEE Signal Process. Mag. 29, 152–150 (2012).

    Article  Google Scholar 

  15. 15.

    Kim, H. K. et al. Continuous shared control for stabilizing reaching and grasping with brain–machine interfaces. IEEE Trans. Biomed. Eng. 53, 1164–1173 (2006).

    Article  Google Scholar 

  16. 16.

    Iturrate, I., Montesano, L. & Minguez, J. Shared-control brain–computer interface for a two dimensional reaching task using EEG error-related potentials. In Proc. 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society 5258–5262 (IEEE, 2013).

  17. 17.

    Chen, X. et al. A shared control policy for center-out movement decoding in motor brain–machine interface. IFAC Proc. Vol. 46, 345–348 (2013).

    Article  Google Scholar 

  18. 18.

    Ciancio, A. L. et al. Control of prosthetic hands via the peripheral nervous system. Front. Neurosci. 10, 116 (2016).

    Article  Google Scholar 

  19. 19.

    Došen, S. et al. Cognitive vision system for control of dexterous prosthetic hands: experimental evaluation. J. NeuroEngin. Rehabil. 7, 42 (2010).

    Article  Google Scholar 

  20. 20.

    Light, C. M., Chappell, P. H., Hudgins, B. & Engelhart, K. Intelligent multifunction myoelectric control of hand prostheses. J. Med. Eng. Technol. 26, 139–146 (2002).

    Article  Google Scholar 

  21. 21.

    Tura, A., Lamberti, C., Davalli, A. & Sacchetti, R. Experimental development of a sensory control system for an upper limb myoelectric prosthesis with cosmetic covering. J. Rehabil. Res. Dev. 35, 14–26 (1998).

    Google Scholar 

  22. 22.

    Fani, S. et al. Assessment of myoelectric controller performance and kinematic behavior of a novel soft synergy-inspired robotic hand for prosthetic applications. Front. Neurorobotics 10, 11 (2016).

    Article  Google Scholar 

  23. 23.

    Sommer, N. & Billard, A. Multi-contact haptic exploration and grasping with tactile sensors. Robot. Auton. Syst. 85, 48–61 (2016).

    Article  Google Scholar 

  24. 24.

    Celadon, N., Došen, S., Binder, I., Ariano, P. & Farina, D. Proportional estimation of finger movements from high-density surface electromyography. J. NeuroEngin. Rehabil. 13, 73 (2016).

    Article  Google Scholar 

  25. 25.

    Segil, J. L., Controzzi, M., Weir, R. F. & Cipriani, C. Comparative study of state-of-the-art myoelectric controllers for multigrasp prosthetic hands. J. Rehabil. Res. Dev. 51, 1439–1454 (2014).

    Article  Google Scholar 

  26. 26.

    Mathiowetz, V., Volland, G., Kashman, N. & Weber, K. Adult norms for the box and block test of manual dexterity. Am. J. Occup. Ther. 39, 386–391 (1985).

    Article  Google Scholar 

  27. 27.

    Park, E. & Meek, S. G. Fatigue compensation of the electromyographic signal for prosthetic control and force estimation. IEEE Trans. Biomed. Eng. 40, 1019–1023 (1993).

    Article  Google Scholar 

  28. 28.

    Tkach, D., Huang, H. & Kuiken, T. A. Study of stability of time-domain features for electromyographic pattern recognition. J. NeuroEngin. Rehabil. 7, 21 (2010).

    Article  Google Scholar 

  29. 29.

    Asghari Oskoei, M. & Hu, H. Myoelectric control systems—A survey. Biomed. Signal Process. Control 2, 275–294 (2007).

    Article  Google Scholar 

  30. 30.

    Micera, S., Carpaneto, J. & Raspopovic, S. Control of hand prostheses using peripheral information. IEEE Rev. Biomed. Eng 3, 48–68 (2010).

    Article  Google Scholar 

  31. 31.

    Wan, B. et al. Study on fatigue feature from forearm SEMG signal based on wavelet analysis. In Proc. IEEE International Conference on Robotics and Biomimetics 1229–1232 (IEEE, 2010).

  32. 32.

    Zardoshti-Kermani, M., Wheeler, B. C., Badie, K. & Hashemi, R. M. EMG feature evaluation for movement control of upper extremity prostheses. IEEE Trans. Rehabil. Eng. 3, 324–333 (1995).

    Article  Google Scholar 

  33. 33.

    Phinyomark, A., Phukpattaranont, P. & Limsakul, C. Feature reduction and selection for EMG signal classification. Expert Syst. Appl. 39, 7420–7431 (2012).

    Article  Google Scholar 

  34. 34.

    Chu, J. U., Moon, I. & Mun, M. S. A real-time EMG pattern recognition system based on linear–nonlinear feature projection for a multifunction myoelectric hand. IEEE Trans. Biomed. Eng. 53, 2232–2239 (2006).

    Article  Google Scholar 

  35. 35.

    Abadi, M. et al. Tensorflow: a system for large-scale machine learning. In 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16) 265–283 (ACM, 2016).

  36. 36.

    Sanderson, C. & Curtin, R. Armadillo: a template-based C++ library for linear algebra. J. Open Source Softw. 1, 26 (2016).

    Article  Google Scholar 

Download references

Acknowledgements

We acknowledge B. Wester, F. Tenore and the Johns Hopkins University Applied Physics Laboratory for providing the Virtual Integration Environment, which was developed on the Defense Advanced Research Projects Agency’s Revolutionizing Prosthetics programme under contract no. N66001-10-C-4056. We also thank F. Iberite for his assistance in conducting experiments and A. Devillars for the development of the Unity model of the hand. This project was partly funded by the Swiss National Competence Center for Research in Robotics, by the Bertarelli Foundation and by the European Union’s Horizon 2020 research and innovation programme under Marie Skłodowska Curie grant agreement no. 750947 (project BIREHAB).

Author information

Affiliations

Authors

Contributions

K.Z.Z. and E.F. designed and carried out experiments 1 and 2, and performed analysis of data. A.B. and S.M. were responsible for planning and supervising the work. N.S. provided code and expertise for the shared controller and contributed greatly to experimental set-up. V.M. and F.A. developed the decoding algorithm for experiment 3. V.M., F.A. and S.A. performed the system integration. V.M. and S.A. performed all of the trials for experiment 3. E.D’A. aided in experimentation, G.G., G.C. and W.R. were clinical liaisons and F.P. supervised experiment 1. K.Z.Z., V.M. and S.A. wrote the manuscript and designed figures. N.S., E.F., E.D’A., A.B., F.A. and S.M. contributed critical feedback to the manuscript.

Corresponding author

Correspondence to Silvestro Micera.

Ethics declarations

Competing interests

S.M. and F.P. are co-founders of Sensars Neuroprosthetics, a small company working on the commericalization of bidirectional hand prostheses.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhuang, K.Z., Sommer, N., Mendez, V. et al. Shared human–robot proportional control of a dexterous myoelectric prosthesis. Nat Mach Intell 1, 400–411 (2019). https://doi.org/10.1038/s42256-019-0093-5

Download citation

Further reading

Search

Sign up for the Nature Briefing newsletter for a daily update on COVID-19 science.
Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing