Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Active tactile exploration using a brain–machine–brain interface

Nature volume 479, pages 228–231 (2011)Cite this article

Subjects

Abstract

Brain–machine interfaces1,2 use neuronal activity recorded from the brain to establish direct communication with external actuators, such as prosthetic arms. It is hoped that brain–machine interfaces can be used to restore the normal sensorimotor functions of the limbs, but so far they have lacked tactile sensation. Here we report the operation of a brain–machine–brain interface (BMBI) that both controls the exploratory reaching movements of an actuator and allows signalling of artificial tactile feedback through intracortical microstimulation (ICMS) of the primary somatosensory cortex. Monkeys performed an active exploration task in which an actuator (a computer cursor or a virtual-reality arm) was moved using a BMBI that derived motor commands from neuronal ensemble activity recorded in the primary motor cortex. ICMS feedback occurred whenever the actuator touched virtual objects. Temporal patterns of ICMS encoded the artificial tactile properties of each object. Neuronal recordings and ICMS epochs were temporally multiplexed to avoid interference. Two monkeys operated this BMBI to search for and distinguish one of three visually identical objects, using the virtual-reality arm to identify the unique artificial texture associated with each. These results suggest that clinical motor neuroprostheses might benefit from the addition of ICMS feedback to generate artificial somatic perceptions associated with mechanical, robotic or even virtual prostheses.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Rent or buy this article

Get just this article for as long as you need it

$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Figure 1: The brain–machine–brain interface.
Figure 2: Learning to use ICMS feedback.
Figure 3: Statistics of object exploration.
Figure 4: M1 modulations during active control versus passive observation.

References

  1. Lebedev, M. A. & Nicolelis, M. A. Brain-machine interfaces: past, present and future. Trends Neurosci. 29, 536–546 (2006)

    Article  CAS  Google Scholar 

  2. Nicolelis, M. A. & Lebedev, M. A. Principles of neural ensemble physiology underlying the operation of brain-machine interfaces. Nature Rev. Neurosci. 10, 530–540 (2009)

    Article  CAS  Google Scholar 

  3. Chapin, J. K., Moxon, K. A., Markowitz, R. S. & Nicolelis, M. A. L. Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nature Neurosci. 2, 664–670 (1999)

    Article  CAS  Google Scholar 

  4. Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S. & Schwartz, A. B. Cortical control of a prosthetic arm for self-feeding. Nature 453, 1098–1101 (2008)

    Article  CAS  ADS  Google Scholar 

  5. Moritz, C. T., Perlmutter, S. I. & Fetz, E. E. Direct control of paralysed muscles by cortical neurons. Nature 456, 639–642 (2008)

    Article  CAS  ADS  Google Scholar 

  6. Wessberg, J. et al. Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408, 361–365 (2000)

    Article  CAS  ADS  Google Scholar 

  7. Taylor, D. M., Helms-Tillery, S. I. & Schwartz, A. B. Direct cortical control of 3D neuroprosthetic devices. Science 296, 1829–1832 (2002)

    Article  CAS  ADS  Google Scholar 

  8. Serruya, M. D., Hatsopoulos, N. G., Paninski, L., Fellows, M. R. & Donoghue, J. P. Instant neural control of a movement signal. Nature 416, 141–142 (2002)

    Article  CAS  ADS  Google Scholar 

  9. Carmena, J. M. et al. Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biol. 1, e42 (2003)

    Article  Google Scholar 

  10. Johansson, R. S. & Westling, G. Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects. Exp. Brain Res. 56, 550–564 (1984)

    Article  CAS  Google Scholar 

  11. Flanagan, J. R. & Wing, A. M. Modulation of grip force with load force during point-to-point arm movements. Exp. Brain Res. 95, 131–143 (1993)

    Article  CAS  Google Scholar 

  12. James, T. W., Kim, S. & Fisher, J. S. The neural basis of haptic object processing. Can. J. Exp. Psychol. 61, 219–229 (2007)

    Article  Google Scholar 

  13. Chatterjee, A. Aggarwal, V. Ramos, A., Acharya, S. & Thakor, N. V. A brain-computer interface with vibrotactile biofeedback for haptic information. J. Neuroeng. Rehabil. 4, 40 (2007)

    Article  Google Scholar 

  14. Kaczmarek, K., Webster, J., Bach-y-Rita, P. & Tompkins, W. Electrotactile and vibrotactile displays for sensory substitution systems. IEEE Trans. Biomed. Eng. 38, 1–16 (1991)

    Article  CAS  Google Scholar 

  15. Marasco, P. D., Schultz, A. E. & Kuiken, T. A. Sensory capacity of reinnervated skin after redirection of amputated upper limb nerves to the chest. Brain 132, 1441–1448 (2009)

    Article  Google Scholar 

  16. O’Doherty, J. E., Lebedev, M. A., Hanson, T. L., Fitzsimmons, N. A. & Nicolelis, M. A. A brain-machine interface instructed by direct intracortical microstimulation. Front. Integr. Neurosci. 3, 20 (2009)

    PubMed  PubMed Central  Google Scholar 

  17. Richer, F., Martinez, M., Robert, M., Bouvier, G. & Saint-Hilaire, J. M. Stimulation of human somatosensory cortex: tactile and body displacement perceptions in medial regions. Exp. Brain Res. 93, 173–176 (1993)

    Article  CAS  Google Scholar 

  18. London, B. M., Jordan, L. R., Jackson, C. R. & Miller, L. E. Electrical stimulation of the proprioceptive cortex (area 3a) used to instruct a behaving monkey. IEEE Trans. Neural Syst. Rehabil. Eng. 16, 32–36 (2008)

    Article  Google Scholar 

  19. Romo, R., Hernandez, A., Zainos, A. & Salinas, E. Somatosensory discrimination based on cortical microstimulation. Nature 392, 387–390 (1998)

    Article  CAS  ADS  Google Scholar 

  20. Fitzsimmons, N. A., Drake, W., Hanson, T. L., Lebedev, M. A. & Nicolelis, M. A. Primate reaching cued by multichannel spatiotemporal cortical microstimulation. J. Neurosci. 27, 5593–5602 (2007)

    Article  CAS  Google Scholar 

  21. Lederman, S. J. & Klatzky, R. L. Hand movements: a window into haptic object recognition. Cognit. Psychol. 19, 342–368 (1987)

    Article  CAS  Google Scholar 

  22. Lebedev, M. A. et al. Cortical ensemble adaptation to represent velocity of an artificial actuator controlled by a brain-machine interface. J. Neurosci. 25, 4681–4693 (2005)

    Article  CAS  Google Scholar 

  23. Li, Z. et al. Unscented Kalman filter for brain-machine interfaces. PLoS ONE 4, e6243 (2009)

    Article  ADS  Google Scholar 

  24. Lebedev, M. A., Denton, J. M. & Nelson, R. J. Vibration-entrained and premovement activity in monkey primary somatosensory cortex. J. Neurophysiol. 72, 1654–1673 (1994)

    Article  CAS  Google Scholar 

  25. Liu, Y., Denton, J. M. & Nelson, R. J. Neuronal activity in primary motor cortex differs when monkeys perform somatosensory and visually guided wrist movements. Exp. Brain Res. 167, 571–586 (2005)

    Article  Google Scholar 

  26. Cisek, P. & Kalaska, J. F. Neural correlates of mental rehearsal in dorsal premotor cortex. Nature 431, 993–996 (2004)

    Article  CAS  ADS  Google Scholar 

  27. Graziano, M. S., Cooke, D. F. & Taylor, C. S. Coding the location of the arm by sight. Science 290, 1782–1786 (2000)

    Article  CAS  ADS  Google Scholar 

  28. Maravita, A. & Iriki, A. Tools for the body (schema). Trends Cogn. Sci. 8, 79–86 (2004)

    Article  Google Scholar 

  29. Tkach, D., Reimer, J. & Hatsopoulos, N. G. Observation-based learning for brain-machine interfaces. Curr. Opin. Neurobiol. 18, 589–594 (2008)

    Article  CAS  Google Scholar 

  30. Dushanova, J. & Donoghue, J. Neurons in primary motor cortex engaged during action observation. Eur. J. Neurosci. 31, 386–398 (2010)

    Article  Google Scholar 

Download references

Acknowledgements

We thank D. Dimitrov for conducting the animal neurosurgeries; G. Lehew and J. Meloy for building brain implants; J. Fruh for rendering the virtual-reality monkey arm; T. Phillips, L. Oliveira and S. Halkiotis for technical support; and E. Thomson and Z. Li for comments. This research was supported by DARPA (award N66001-06-C-2019), TATRC (award W81XWH-08-2-0119), the NIH (award NS073125), NICHD/OD (award RC1HD063390) and NIH Director’s Pioneer Award DP1OD006798, to M.A.L.N. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Office of the NIH Director or the NIH.

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, Duke University, Durham, 27708, North Carolina, USA

    Joseph E. O’Doherty, Peter J. Ifft, Katie Z. Zhuang & Miguel A. L. Nicolelis

  2. Center for Neuroengineering, Duke University, Durham, 27710, North Carolina, USA

    Joseph E. O’Doherty, Mikhail A. Lebedev, Peter J. Ifft, Katie Z. Zhuang & Miguel A. L. Nicolelis

  3. Department of Neurobiology, Duke University, Durham, 27710, North Carolina, USA

    Mikhail A. Lebedev & Miguel A. L. Nicolelis

  4. STI IMT, École Polytechnique Fédérale de Lausanne, Lausanne CH1015, Switzerland

    Solaiman Shokur & Hannes Bleuler

  5. Department of Psychology and Neuroscience, Duke University, Durham, 27708, North Carolina, USA

    Miguel A. L. Nicolelis

  6. Edmond and Lily Safra International Institute of Neuroscience of Natal, Natal 59066-060, Brazil

    Miguel A. L. Nicolelis

Authors
  1. Joseph E. O’Doherty
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mikhail A. Lebedev
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Peter J. Ifft
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Katie Z. Zhuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Solaiman Shokur
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hannes Bleuler
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Miguel A. L. Nicolelis
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

J.E.O’D., M.A.L. and M.A.L.N. designed experiments, analysed data and wrote the paper; J.E.O’D., M.A.L., P.J.I. and K.Z.Z. conducted experiments; and S.S. and H.B. developed the virtual-reality monkey arm.

Corresponding author

Correspondence to Miguel A. L. Nicolelis.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Figures

This file contains Supplementary Figures 1-7 with legends. (PDF 3026 kb)

Supplementary Movie 1

This movie sequence depicts monkey M performing task IV in HC mode. Annotations are provided during the freeze frames. The audio track is the spiking activity of a single M1 neuron; ICMS pulses can be completion of a trial and the delivery of a juice reward. Please note that the monkeys could not hear the ICMS pulses or neural activity; these sounds are included for the benefit of the viewer. (MOV 11640 kb)

Supplementary Movie 2

This movie depicts the same sequence as Supplementary Movie 1 but for BC. Please note that the monkeys could not hear the ICMS pulses or neural activity; these sounds are included for the benefit of the viewer. (MOV 14860 kb)

Supplementary Movie 3

This movie sequence depicts monkey M performing task V in HC. Please note that the monkeys could not hear the ICMS pulses or neural activity; these sounds are included for the benefit of the viewer. (MOV 14763 kb)

Supplementary Movie 4

This movie depicts the same sequence as Supplementary Movie 3 but for BC. Please note that the monkeys could not hear the ICMS pulses or neural activity; these sounds are included for the benefit of the viewer. (MOV 16711 kb)

PowerPoint slides

Rights and permissions

Reprints and Permissions

About this article

Cite this article

O’Doherty, J., Lebedev, M., Ifft, P. et al. Active tactile exploration using a brain–machine–brain interface. Nature 479, 228–231 (2011). https://doi.org/10.1038/nature10489

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nature10489

This article is cited by

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing