Article | Published:

A learning-based approach to artificial sensory feedback leads to optimal integration

Nature Neuroscience volume 18, pages 138144 (2015) | Download Citation

Abstract

Proprioception—the sense of the body's position in space—is important to natural movement planning and execution and will likewise be necessary for successful motor prostheses and brain–machine interfaces (BMIs). Here we demonstrate that monkeys were able to learn to use an initially unfamiliar multichannel intracortical microstimulation signal, which provided continuous information about hand position relative to an unseen target, to complete accurate reaches. Furthermore, monkeys combined this artificial signal with vision to form an optimal, minimum-variance estimate of relative hand position. These results demonstrate that a learning-based approach can be used to provide a rich artificial sensory feedback signal, suggesting a new strategy for restoring proprioception to patients using BMIs, as well as a powerful new tool for studying the adaptive mechanisms of sensory integration.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    & Multisensory integration during motor planning. J. Neurosci. 23, 6982–6992 (2003).

  2. 2.

    & Flexible strategies for sensory integration during motor planning. Nat. Neurosci. 8, 490–497 (2005).

  3. 3.

    , & Integration of proprioceptive and visual position-information: an experimentally supported model. J. Neurophysiol. 81, 1355–1364 (1999).

  4. 4.

    & Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).

  5. 5.

    , & Multisensory integration in macaque visual cortex depends on cue reliability. Neuron 59, 662–673 (2008).

  6. 6.

    & Sensory transformations and the use of multiple reference frames for reach planning. Nat. Neurosci. 12, 1056–1061 (2009).

  7. 7.

    , & Loss of proprioception produces deficits in interjoint coordination. J. Neurophysiol. 70, 2136–2147 (1993).

  8. 8.

    , , & Control of limb dynamics in normal subjects and patients without proprioception. J. Neurophysiol. 73, 820–835 (1995).

  9. 9.

    , , & Incorporating feedback from multiple sensory modalities enhances brain–machine interface control. J. Neurosci. 30, 16777–16787 (2010).

  10. 10.

    et al. Biomimetic brain machine interfaces for the control of movement. J. Neurosci. 27, 11842–11846 (2007).

  11. 11.

    , , & An electric field model for prediction of somatosensory (S1) cortical field potentials induced by ventral posterior lateral (VPL) thalamic microstimulation. IEEE Trans. Neural Syst. Rehabil. Eng. 20, 161–169 (2012).

  12. 12.

    , , & Optimal space–time precoding of artificial sensory feedback through mutichannel microstimulation in bi-directional brain–machine interfaces. J. Neural Eng. 9, 065004 (2012).

  13. 13.

    , & Interfacing the somatosensory system to restore touch and proprioception: essential considerations. J. Mot. Behav. 44, 403–418 (2012).

  14. 14.

    et al. Restoring the sense of touch with a prosthetic hand through a brain interface. Proc. Natl. Acad. Sci. USA 110, 18279–18284 (2013).

  15. 15.

    & Movement-produced stimulation in the development of visually guided behavior. J. Comp. Physiol. Psychol. 56, 872–876 (1963).

  16. 16.

    , , , & Incorporating cross-modal statistics in the development and maintenance of multisensory integration. J. Neurosci. 32, 2287–2298 (2012).

  17. 17.

    , & The statistical determinants of adaptation rate in human reaching. J. Vis. 8, 1–19 (2008).

  18. 18.

    , & Visual-shift adaptation is composed of separable sensory and task-dependent effects. J. Neurophysiol. 98, 2827–2841 (2007).

  19. 19.

    , & Multisensory calibration is independent of cue reliability. J. Neurosci. 31, 13949–13962 (2011).

  20. 20.

    , & Learning multisensory integration and coordinate transformation via density estimation. PLoS Comput. Biol. 9, e1003035 (2013).

  21. 21.

    The representation of arm movements in postcentral and parietal cortex. Can. J. Physiol. Pharmacol. 66, 455–463 (1988).

  22. 22.

    Parietal cortex area 5 and visuomotor behavior. Can. J. Physiol. Pharmacol. 74, 483–498 (1996).

  23. 23.

    & Sensory reweighting in targeted reaching: effects of conscious effort, error history, and target salience. J. Neurophysiol. 103, 206–217 (2010).

  24. 24.

    & Calibration of visually-guided reaching is driven by error corrective learning and internal dynamics. J. Neurophysiol. 97, 3057–3069 (2007).

  25. 25.

    & Learning from sensory and reward prediction errors during motor adaptation. PLoS Comput. Biol. 7, e1002012 (2011).

  26. 26.

    , , & Parietal area 5 neuronal activity encodes movement kinematics, not movement dynamics. Exp. Brain Res. 80, 351–364 (1990).

  27. 27.

    , , & Reach plans in eye-centered coordinates. Science 285, 257–260 (1999).

  28. 28.

    et al. Early coding of reaching in the parietooccipital cortex. J. Neurophysiol. 83, 2374–2391 (2000).

  29. 29.

    , & Coding the location of the arm by sight. Science 290, 1782–1786 (2000).

  30. 30.

    & Coding of the reach vector in parietal area 5d. Neuron 75, 342–351 (2012).

  31. 31.

    , & Efficient computation and cue integration with noisy population codes. Nat. Neurosci. 4, 826–831 (2001).

  32. 32.

    , , & Bayesian inference with probabilistic population codes. Nat. Neurosci. 9, 1432–1438 (2006).

  33. 33.

    & Idiosyncratic and systematic aspects of spatial representations in the macaque parietal cortex. Proc. Natl. Acad. Sci. USA 107, 7951–7956 (2010).

  34. 34.

    , , & Reaching activity in parietal area V6A of macaque: eye influence on arm activity or retinocentric coding of reaching movements? Eur. J. Neurosci. 27, 775–789 (2008).

  35. 35.

    & Heterogeneous representations in the superior parietal lobule are common across reaches to visual and proprioceptive targets. J. Neurosci. 31, 6661–6673 (2011).

  36. 36.

    , , & Premotor and parietal cortex: corticocortical connectivity and combinatorial computations. Annu. Rev. Neurosci. 20, 25–42 (1997).

  37. 37.

    & Intentional maps in posterior parietal cortex. Annu. Rev. Neurosci. 25, 189–220 (2002).

  38. 38.

    , , & Multiple levels of representation of reaching in the parieto-frontal network. Cereb. Cortex 13, 1009–1022 (2003).

  39. 39.

    Sensory integration for reaching: models of optimality in the context of behavior and the underlying neural circuits. Prog. Brain Res. 191, 195–209 (2011).

  40. 40.

    , & Lesions of cortical area LIP affect reach onset only when the reach is accompanied by a saccade, revealing an active eye–hand coordination circuit. Proc. Natl. Acad. Sci. USA 110, 2371–2376 (2013).

  41. 41.

    , , & Specificity of human cortical areas for reaches and saccades. J. Neurosci. 27, 4687–4696 (2007).

  42. 42.

    & The cortico-cortical connections to area 5 of the parietal lobe from the primary somatic sensory cortex of the monkey. Proc. R. Soc. Lond. B Biol. Sci. 200, 103–108 (1978).

  43. 43.

    & Corticocortical connections of visual, sensorimotor, and multimodal processing areas in the parietal lobe of the macaque monkey. J. Comp. Neurol. 428, 112–137 (2000).

  44. 44.

    & Strategic calibration and spatial alignment: a model from prism adaptation. J. Mot. Behav. 34, 126–138 (2002).

  45. 45.

    , & How humans combine simultaneous proprioceptive and visual position information. Exp. Brain Res. 111, 253–261 (1996).

  46. 46.

    & Proprioceptive control of interjoint coordination. Can. J. Physiol. Pharmacol. 73, 273–284 (1995).

  47. 47.

    , & The use of overlapping submovements in the control of rapid hand movements. Exp. Brain Res. 144, 351–364 (2002).

  48. 48.

    & An Introduction to the Bootstrap (Chapman & Hall, 1993).

Download references

Acknowledgements

We thank M.R. Fellows for initial behavioral training; R.R. Torres for suggesting the 2AFC task for ICMS detection; A. Leggitt for help with data analysis; K.B. Andrews and K. MacLeod for animal-related support; and J.G. Makin, A. Yazdan-Shahmorad and T.L. Hanson for discussion and comments on the manuscript. This research was supported by the Defense Advanced Research Projects Agency (DARPA) Reorganization and Plasticity to Accelerate Injury Recovery (REPAIR; N66001-10-C-2010) and the US National Institutes of Health NEI (EY015679, EY007120).

Author information

Affiliations

  1. Department of Physiology, University of California, San Francisco, California, USA.

    • Maria C Dadarlat
    • , Joseph E O'Doherty
    •  & Philip N Sabes
  2. Center for Integrative Neuroscience, University of California, San Francisco, California, USA.

    • Maria C Dadarlat
    • , Joseph E O'Doherty
    •  & Philip N Sabes
  3. UC Berkeley–UCSF Center for Neural Engineering and Prosthetics, University of California, San Francisco, California, USA.

    • Maria C Dadarlat
    • , Joseph E O'Doherty
    •  & Philip N Sabes
  4. UC Berkeley–UCSF Graduate Program in Bioengineering, University of California, San Francisco, California, USA.

    • Maria C Dadarlat
    •  & Philip N Sabes

Authors

  1. Search for Maria C Dadarlat in:

  2. Search for Joseph E O'Doherty in:

  3. Search for Philip N Sabes in:

Contributions

M.C.D. and P.N.S. designed the experiments; M.C.D. and J.E.O. developed and tested multielectrode stimulation capabilities, including behavioral validation; M.C.D. performed the experiments; M.C.D. and P.N.S. analyzed the data; M.C.D., P.N.S. and J.E.O. wrote the manuscript.

Competing interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to Philip N Sabes.

Integrated supplementary information

Supplementary information

PDF files

  1. 1.

    Supplementary Text and Figures

    Supplementary Figures 1–5 and Supplementary Tables 1–4

  2. 2.

    Supplementary Methods Checklist

Videos

  1. 1.

    Sample trial of monkey F reaching with VIS+ICMS feedback.

    This video is generated from behavioral data collected on 29 Dec., 2013. The left panel shows a simulated version of a portion of the virtual reality environment that the monkey viewed during the trial: position of the fingertip (filled white circle), dot field (here displayed at 100% coherence for clarity; the coherence presented to the monkey for this trial was 50%), and start target (open green circle, radius 10 mm). The right panel shows the patterns of ICMS delivered during the trial, where each vertical line denotes a pulse of stimulation from an electrode with a preferred direction indicated by the corresponding red arrow at left. Stimulation rasters shown have been subsampled for clarity. In the video, the monkey is shown first acquiring the start target. After an instructed delay interval, during which VIS+ICMS information about the instructed movement vector become available, a go cue sounds (noted by text), and the monkey completes the reach to the unseen, 12 mm radius reach target (illustrated here with dashed white circle).

  2. 2.

    Sample trial of monkey F reaching with only ICMS feedback.

    This video is generated from behavioral data collected on 29 Dec., 2013. The left panel shows a simulated version of a portion of the virtual reality environment that the monkey viewed during the trial: position of the fingertip (filled white circle), dot field (here displayed at 100% coherence for clarity; the coherence presented to the monkey for this trial was 50%), and start target (open green circle, radius 10 mm). The right panel shows the patterns of ICMS delivered during the trial, where each vertical line denotes a pulse of stimulation from an electrode with a preferred direction indicated by the corresponding red arrow at left. Stimulation rasters shown have been subsampled for clarity. In the video, the monkey is shown first acquiring the start target. After an instructed delay interval, during which ICMS information about the instructed movement vector become available, a go cue sounds (noted by text), and the monkey completes the reach to the unseen, 12 mm radius reach target (illustrated here with dashed white circle).

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nn.3883

Further reading