Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

The neural resource allocation problem when enhancing human bodies with extra robotic limbs

Abstract

The emergence of robotic body augmentation provides exciting innovations that will revolutionize the fields of robotics, human–machine interaction and wearable electronics. Although augmentative devices such as extra robotic arms and fingers are informed by restorative technologies in many ways, they also introduce unique challenges for bidirectional human–machine collaboration. Can humans adapt and learn to operate a new robotic limb collaboratively with their biological limbs, without restricting other physical abilities? To successfully achieve robotic body augmentation, we need to ensure that, by giving a user an additional (artificial) limb, we are not trading off the functionalities of an existing (biological) one. Here, we introduce the ‘neural resource allocation problem’ and discuss how to allow the effective voluntary control of augmentative devices without compromising control of the biological body. In reviewing the relevant literature on extra robotic fingers and arms, we critically assess the range of potential solutions available for this neural resource allocation problem. For this purpose, we combine multiple perspectives from engineering and neuroscience with considerations including human–machine interaction, sensory–motor integration, ethics and law. In summary, we aim to define common foundations and operating principles for the successful implementation of robotic body augmentation.

This is a preview of subscription content

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Body enhancement continuum with respect to bodily ability.
Fig. 2: Examples of use case scenarios of body augmentation with XRLs.
Fig. 3: Taxonomy for XRAs and XRFs and a comparison with established robotics paradigms.
Fig. 4: Possible interfaces for control and sensory feedback for an XRA.

References

  1. 1.

    Bergamasco, M. & Herr, H. in Springer Handbook of Robotics (eds Siciliano, B. & Khatib, O.) 1875–1906 (Springer International Publishing, 2016).

  2. 2.

    Windrich, M., Grimmer, M., Christ, O., Rinderknecht, S. & Beckerle, P. Active lower limb prosthetics: a systematic review of design issues and solutions. Biomed. Eng. Online 15, 140 (2016).

    Article  Google Scholar 

  3. 3.

    Mendez, V., Iberite, F., Shokur, S. & Micera, S. Current solutions and future trends of robotic prosthetic hands. Annu. Rev. Control Robot. Auton. Syst. 4, 595–627 (2021).

    Article  Google Scholar 

  4. 4.

    Dollar, A. M. & Herr, H. Lower extremity exoskeletons and active orthoses: challenges and state-of-the-art. IEEE Trans. Robot. 24, 144–158 (2008).

    Article  Google Scholar 

  5. 5.

    Guggenheim, J., Hoffman, R., Song, H. & Asada, H. H. Leveraging the human operator in the design and control of supernumerary robotic limbs. IEEE Robot. Autom. Lett. 5, 2177–2184 (2020).

    Article  Google Scholar 

  6. 6.

    Salvietti, G. et al. Compensating hand function in chronic stroke patients through the robotic sixth finger. IEEE Trans. Neural Syst. Rehabil. Eng. 25, 142–150 (2017).

    Article  Google Scholar 

  7. 7.

    Kieliba, P., Clode, D., Maimon-Mor, R. O. & Makin, T. R. Robotic hand augmentation drives changes in neural body representation. Sci. Robot. 6, eabd7935 (2021).

    Article  Google Scholar 

  8. 8.

    Xie, H., Mitsuhashi, K. & Torii, T. Augmenting human with a tail. In Proc. 10th Augmented Human International Conference 2019 Vol. 35, 1–7 (ACM, 2019); https://doi.org/10.1145/3311823.3311847

  9. 9.

    Parietti, F. & Asada, H. H. Independent, voluntary control of extra robotic limbs. In Proc. 2017 IEEE International Conference on Robotics and Automation ICRA 5954–5961 (IEEE, 2017).

  10. 10.

    Di Pino, G., Maravita, A., Zollo, L., Guglielmelli, E. & Di Lazzaro, V. Augmentation-related brain plasticity. Front. Syst. Neurosci. 8, 109 (2014).

    Google Scholar 

  11. 11.

    Makin, T., de Vigneromont, F. & Micera, S. Soft embodiment for engineering artificial limbs. Trends Cogn. Sci. 24, 965–968 (2020).

    Article  Google Scholar 

  12. 12.

    Mehring, C. et al. Augmented manipulation ability in humans with six-fingered hands. Nat. Commun. 10, 2401 (2019).

    Article  Google Scholar 

  13. 13.

    Penaloza, C. I. & Nishio, S. BMI control of a third arm for multitasking. Sci. Robot. 3, eaat1228 (2018).

    Article  Google Scholar 

  14. 14.

    Bernshtein, N. A. The Co-ordination and Regulation of Movements (Pergamon, 1967); http://books.google.com/books?id=F9dqAAAAMAAJ

  15. 15.

    Lisini Baldi, T. et al. Exploiting implicit kinematic kernel for controlling a wearable robotic extra-finger. Preprint at https://arxiv.org/pdf/2012.03600.pdf (2020).

  16. 16.

    Baldi, T. L., Farina, F., Garulli, A., Giannitrapani, A. & Prattichizzo, D. Upper body pose estimation using wearable inertial sensors and multiplicative Kalman filter. IEEE Sens. J. 20, 492–500 (2020).

    Article  Google Scholar 

  17. 17.

    Hussain, I., Meli, L., Pacchierotti, C., Salvietti, G. & Prattichizzo, D. Vibrotactile haptic feedback for intuitive control of robotic extra fingers. Proc. IEEE World Haptics Conference 394–399 (IEEE, 2015); https://doi.org/10.1109/WHC.2015.7177744

  18. 18.

    d’Avella, A., Saltiel, P. & Bizzi, E. Combinations of muscle synergies in the construction of a natural motor behavior. Nat. Neurosci. 6, 300–308 (2003).

    Article  Google Scholar 

  19. 19.

    Bräcklein, M., Ibáñez, J., Barsakcioglu, D. & Farina, D. Towards human motor augmentation by voluntary decoupling beta activity in the neural drive to muscle and force production. J. Neural Eng. 18, 016001 (2021).

    Article  Google Scholar 

  20. 20.

    Aoyama, T., Shikida, H., Schatz, R. & Hasegawa, Y. Operational learning with sensory feedback for controlling a robotic thumb using the posterior auricular muscle. Adv. Robot. 33, 243–253 (2019).

    Article  Google Scholar 

  21. 21.

    Guggenheim, J., Parietti, F., Flash, T. & Asada, H. Laying the groundwork for intra-robotic-natural limb coordination: is fully manual control viable?. ACM Trans. Hum. Robot. Interact. 9, 18 (2020).

    Article  Google Scholar 

  22. 22.

    Borzelli, D., Cesqui, B., Berger, D. J., Burdet, E. & d’Avella, A. Muscle patterns underlying voluntary modulation of co-contraction. PLoS ONE 13, e0205911 (2018).

    Article  Google Scholar 

  23. 23.

    Berger, D. J., Gentner, R., Edmunds, T., Pai, D. K. & d’Avella, A. Differences in adaptation rates after virtual surgeries provide direct evidence for modularity. J. Neurosci. 33, 12384–12394 (2013).

    Article  Google Scholar 

  24. 24.

    Gurgone, S. et al. Simultaneous control of natural and extra degrees-of-freedom by isometric force and EMG null space activation. In Converging Clinical and Engineering Research on Neurorehabilitation IV. Proceedings of the Fifth International Conference on Neurorehabilitation IV (INCR2020) (eds. Torricelli, D., Akay, M. & Pons, J. L.) 863–868 (2020); https://www.springer.com/gp/book/9783030703158

  25. 25.

    Lebedev, M. A. & Nicolelis, M. A. L. Brain–machine interfaces: from basic science to neuroprostheses and neurorehabilitation. Physiol. Rev. 97, 767–837 (2017).

    Article  Google Scholar 

  26. 26.

    Schwartz, A. B. Cortical neural prosthetics. Annu. Rev. Neurosci. 27, 487–507 (2004).

    Article  Google Scholar 

  27. 27.

    Omrani, M., Kaufman, M. T., Hatsopoulos, N. G. & Cheney, P. D. Perspectives on classical controversies about the motor cortex. J. Neurophysiol. 118, 1828–1848 (2017).

    Article  Google Scholar 

  28. 28.

    Fetz, E. E. Operant conditioning of cortical unit activity. Science 163, 955–958 (1969).

    Article  Google Scholar 

  29. 29.

    Carmena, J. M. et al. Learning to control a brain–machine interface for reaching and grasping by primates. PLoS Biol. 1, e42 (2003).

    Article  Google Scholar 

  30. 30.

    Ifft, P. J., Shokur, S., Li, Z., Lebedev, M. A. & Nicolelis, M. A. L. A brain-machine interface enables bimanual arm movements in monkeys. Sci. Transl. Med. 5, 210ra154 (2013).

    Article  Google Scholar 

  31. 31.

    Bashford, L. et al. Concurrent control of a brain–computer interface and natural overt movements. J. Neural Eng. 15, 066021 (2018).

    Article  Google Scholar 

  32. 32.

    Artoni, F., Delorme, A. & Makeig, S. Applying dimension reduction to EEG data by principal component analysis reduces the quality of its subsequent independent component decomposition. NeuroImage 175, 176–187 (2018).

    Article  Google Scholar 

  33. 33.

    Zhuang, K. Z. et al. Shared human–robot proportional control of a dexterous myoelectric prosthesis. Nat. Mach. Intell. 1, 400–411 (2019).

    Article  Google Scholar 

  34. 34.

    Johansson, R. S. & Flanagan, J. R. Coding and use of tactile signals from the fingertips in object manipulation tasks. Nat. Rev. Neurosci. 10, 345–359 (2009).

    Article  Google Scholar 

  35. 35.

    Bensmaia, S. J., Tyler, D. J. & Micera, S. Restoration of sensory information via bionic hands. Nat. Biomed. Eng. https://doi.org/10.1038/s41551-020-00630-8 (2020).

  36. 36.

    Clemente, F., D’Alonzo, M., Controzzi, M., Edin, B. B. & Cipriani, C. Non-invasive, temporally discrete feedback of object contact and release improves grasp control of closed-loop myoelectric transradial prostheses. IEEE Trans. Neural Syst. Rehabil. Eng. 24, 1314–1322 (2016).

    Article  Google Scholar 

  37. 37.

    Valle, G. et al. Hand control with invasive feedback is not impaired by increased cognitive load. Front. Bioeng. Biotechnol. 8, 287 (2020).

    Article  Google Scholar 

  38. 38.

    Guggenheim, J. W. & Asada, H. H. Inherent haptic feedback from supernumerary robotic limbs. IEEE Trans. Haptics 14, 123–131 (2020).

    Article  Google Scholar 

  39. 39.

    Amoruso, E. et al. Somatosensory signals from the controllers of an extra robotic finger support motor learning. Preprint at bioRxiv https://doi.org/10.1101/2021.05.18.444661 (2021).

  40. 40.

    Kim, J. H. & Lee, B.-H. Mirror therapy combined with biofeedback functional electrical stimulation for motor recovery of upper extremities after stroke: a pilot randomized controlled trial. Occup. Ther. Int. 22, 51–60 (2015).

    Article  Google Scholar 

  41. 41.

    Risi, N., Shah, V., Mrotek, L. A., Casadio, M. & Scheidt, R. A. Supplemental vibrotactile feedback of real-time limb position enhances precision of goal-directed reaching. J. Neurophysiol. 122, 22–38 (2019).

    Article  Google Scholar 

  42. 42.

    Vargas, L. et al. Object stiffness recognition using haptic feedback delivered through transcutaneous proximal nerve stimulation. J. Neural Eng. 17, 016002 (2019).

    Article  Google Scholar 

  43. 43.

    Wang, W. et al. Building multi-modal sensory feedback pathways for SRL with the aim of sensory enhancement via BCI. In Proc. 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO) 2439–2444 (IEEE, 2019); https://doi.org/10.1109/ROBIO49542.2019.8961383

  44. 44.

    Raspopovic, S. et al. Restoring natural sensory feedback in real-time bidirectional hand prostheses. Sci. Transl. Med. 6, 222ra19 (2014).

    Article  Google Scholar 

  45. 45.

    Ganzer, P. D. et al. Restoring the sense of touch using a sensorimotor demultiplexing neural interface. Cell 181, 763–773 (2020).

    Article  Google Scholar 

  46. 46.

    D’Anna, E. et al. A closed-loop hand prosthesis with simultaneous intraneural tactile and position feedback. Sci. Robot. 4, eaau8892 (2019).

    Article  Google Scholar 

  47. 47.

    Dadarlat, M. C., O’Doherty, J. E. & Sabes, P. N. A learning-based approach to artificial sensory feedback leads to optimal integration. Nat. Neurosci. 18, 138–144 (2015).

    Article  Google Scholar 

  48. 48.

    Ortiz-Catalan, M., Mastinu, E., Greenspon, C. M. & Bensmaia, S. J. Chronic use of a sensitized bionic hand does not remap the sense of touch. Cell Rep. 33, 108539 (2020).

    Article  Google Scholar 

  49. 49.

    Merzenich, M. M. et al. Topographic reorganization of somatosensory cortical areas 3b and 1 in adult monkeys following restricted deafferentation. Neuroscience 8, 33–55 (1983).

    MathSciNet  Article  Google Scholar 

  50. 50.

    Merzenich, M. M. et al. Somatosensory cortical map changes following digit amputation in adult monkeys. J. Comp. Neurol. 224, 591–605 (1984).

    Article  Google Scholar 

  51. 51.

    Jenkins, W. M., Merzenich, M. M., Ochs, M. T., Allard, T. & Guic-Robles, E. Functional reorganization of primary somatosensory cortex in adult owl monkeys after behaviorally controlled tactile stimulation. J. Neurophysiol. 63, 82–104 (1990).

    Article  Google Scholar 

  52. 52.

    Allard, T., Clark, S. A., Jenkins, W. M. & Merzenich, M. M. Reorganization of somatosensory area 3b representations in adult owl monkeys after digital syndactyly. J. Neurophysiol. 66, 1048–1058 (1991).

    Article  Google Scholar 

  53. 53.

    Wang, X., Merzenich, M. M., Sameshima, K. & Jenkins, W. M. Remodelling of hand representation in adult cortex determined by timing of tactile stimulation. Nature 378, 71–75 (1995).

    Article  Google Scholar 

  54. 54.

    Gindrat, A.-D., Chytiris, M., Balerna, M., Rouiller, E. & Ghosh, A. Use-dependent cortical processing from fingertips in touchscreen phone users. Curr. Biol. 25, 109–116 (2015).

    Article  Google Scholar 

  55. 55.

    Muret, D. & Makin, T. R. The homeostatic homunculus: rethinking deprivation-triggered reorganisation. Neurobiol. Learn. Plast. 67, 115–122 (2021).

    Google Scholar 

  56. 56.

    Peng, G., Wang, Y. & Han, G. Information technology and employment: the impact of job tasks and worker skills. J. Ind. Relat. 60, 201–223 (2018).

    Article  Google Scholar 

  57. 57.

    Oertelt, N. et al. Human by design: an ethical framework for human augmentation. IEEE Technol. Soc. Mag. 36, 32–36 (2017).

    Article  Google Scholar 

  58. 58.

    The Transhumanist Reader (eds More, M. & Vita-More, N.) Ch. 4 (John Wiley & Sons, 2013); https://doi.org/10.1002/9781118555927

  59. 59.

    Raisamo, R. et al. Human augmentation: past, present and future. Int. J. Hum. Comput. Stud. 131, 131–143 (2019).

    Article  Google Scholar 

  60. 60.

    Buckingham, G. et al. The impact of using an upper-limb prosthesis on the perception of real and illusory weight differences. Psychon. Bull. Rev. 25, 1507–1516 (2018).

    Article  Google Scholar 

  61. 61.

    Blanke, O. Multisensory brain mechanisms of bodily self-consciousness. Nat. Rev. Neurosci. 13, 556–571 (2012).

    Article  Google Scholar 

  62. 62.

    Simon-Martinez, C. et al. Age-related changes in upper limb motion during typical development. PLoS ONE 13, e0198524 (2018).

    Article  Google Scholar 

  63. 63.

    Ciullo, A. S. et al. A novel soft robotic supernumerary hand for severely affected stroke patients. IEEE Trans. Neural Syst. Rehabil. Eng. 28, 1168–1177 (2020).

    Article  Google Scholar 

  64. 64.

    Wesselink, D. B. et al. Obtaining and maintaining cortical hand representation as evidenced from acquired and congenital handlessness. eLife 8, e37227 (2019).

    Article  Google Scholar 

  65. 65.

    Makin, T. R. & Bensmaia, S. J. Stability of sensory topographies in adult cortex. Trends Cogn. Sci. 21, 195–204 (2017).

    Article  Google Scholar 

  66. 66.

    Wu, F. & Asada, H. Bio-artificial synergies for grasp posture control of supernumerary robotic fingers. In Robotics: Science and Systems X (2014); https://doi.org/10.15607/RSS.2014.X.027

  67. 67.

    Hussain, I., Salvietti, G., Spagnoletti, G. & Prattichizzo, D. The Soft-SixthFinger: a wearable EMG controlled robotic extra-finger for grasp compensation in chronic stroke patients. IEEE Robot. Autom. Lett. 1, 1000–1006 (2016).

    Article  Google Scholar 

  68. 68.

    Abdi, E., Burdet, E., Bouri, M. & Bleuler, H. Control of a supernumerary robotic hand by foot: an experimental study in virtual reality. PLoS ONE 10, e0134501 (2015).

    Article  Google Scholar 

  69. 69.

    Saraiji, M. Y., Sasaki, T., Kunze, K., Minamizawa, K. & Inami, M. MetaArms: body remapping using feet-controlled artificial arms. In Proc. 31st Annual ACM Symposium on User Interface Software and Technology 65–74 (ACM, 2018); https://doi.org/10.1145/3242587.3242665

Download references

Acknowledgements

G.D. was funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 813713). S.S. was funded by the CHRONOS project, the Wyss Center for Bio and Neuroengineering and the Bertarelli Foundation. F.D.V. was funded by ANR grants nos. ANR-17-EURE-0017 FrontCog and ANR-16-CE28-0015 Developmental tool. T.R.M. was funded by an ERC Starting Grant (715022 EmbodiedTech) and a Wellcome Trust Senior Research Fellowship (grant no. 215575/Z/19/Z). G.S. and D.P. were supported by Progetto Prin 2017 ‘TIGHT: Tactile InteGration for Humans and arTificial systems’, protocol 2017SB48FP. S.M. was funded by the Swiss National Science Foundation through the National Centre of Competence in Research (NCCR) Robotics, the ERC under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 813713) and the Bertarelli Foundation.

Author information

Affiliations

Authors

Contributions

G.D. and S.S. contributed equally to writing, editing and the overall vision of the paper, and also the figures. They and T.M. wrote the introductory section. G.S. and S.R. wrote the ‘Sensory feedback for XRAs and XRFs’ section. E.P. and F.D.V. wrote the ‘Regulatory, legal and ethical considerations’ section, and A.d’A. and D.P. wrote the ‘Motor control of XRAs and XRFs’ section. T.R.M. and S.M. wrote the final section. S.B. contributed to the editing of the paper, including reformulating of conceptual content. A.d’A., T.R.M., D.P. and S.M. contributed equally to the overall structure and the underlying idea of the review. All authors reviewed the manuscript.

Corresponding author

Correspondence to Silvestro Micera.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Machine Intelligence thanks the anonymous reviewers for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Dominijanni, G., Shokur, S., Salvietti, G. et al. The neural resource allocation problem when enhancing human bodies with extra robotic limbs. Nat Mach Intell 3, 850–860 (2021). https://doi.org/10.1038/s42256-021-00398-9

Download citation

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing