Letter | Published:

Sensing with tools extends somatosensory processing beyond the body

Naturevolume 561pages239242 (2018) | Download Citation

Abstract

The ability to extend sensory information processing beyond the nervous system1 has been observed throughout the animal kingdom; for example, when rodents palpate objects using whiskers2 and spiders localize prey using webs3. We investigated whether the ability to sense objects with tools4,5,6,7,8,9 represents an analogous information processing scheme in humans. Here we provide evidence from behavioural psychophysics, structural mechanics and neuronal modelling, which shows that tools are treated by the nervous system as sensory extensions of the body rather than as simple distal links between the hand and the environment10,11. We first demonstrate that tool users can accurately sense where an object contacts a wooden rod, just as is possible on the skin. We next demonstrate that the impact location is encoded by the modal response of the tool upon impact, reflecting a pre-neuronal stage of mechanical information processing akin to sensing with whiskers2 and webs3. Lastly, we use a computational model of tactile afferents12 to demonstrate that impact location can be rapidly re-encoded into a temporally precise spiking code. This code predicts the behaviour of human participants, providing evidence that the information encoded in motifs shapes localization. Thus, we show that this sensory capability emerges from the functional coupling between the material, biomechanical and neural levels of information processing13,14.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. 1.

    Clark, A. & Chalmers, D. The extended mind. Analysis 58, 7–19 (1998).

  2. 2.

    Bagdasarian, K. et al. Pre-neuronal morphological processing of object location by individual whiskers. Nat. Neurosci. 16, 622–631 (2013).

  3. 3.

    Japyassú, H. F. & Laland, K. N. Extended spider cognition. Anim. Cogn. 20, 375–395 (2017).

  4. 4.

    Chan, T. C. & Turvey, M. T. Perceiving the vertical distances of surfaces by means of a hand-held probe. J. Exp. Psychol. Hum. Percept. Perform. 17, 347–358 (1991).

  5. 5.

    Brisben, A. J., Hsiao, S. S. & Johnson, K. O. Detection of vibration transmitted through an object grasped in the hand. J. Neurophysiol. 81, 1548–1558 (1999).

  6. 6.

    Klatzky, R. L. & Lederman, S. J. Tactile roughness perception with a rigid link interposed between skin and surface. Percept. Psychophys. 61, 591–607 (1999).

  7. 7.

    Yamamoto, S. & Kitazawa, S. Sensation at the tips of invisible tools. Nat. Neurosci. 4, 979–980 (2001).

  8. 8.

    Saig, A., Gordon, G., Assa, E., Arieli, A. & Ahissar, E. Motor-sensory confluence in tactile perception. J. Neurosci. 32, 14022–14032 (2012).

  9. 9.

    Kilteni, K. & Ehrsson, H. H. Sensorimotor predictions and tool use: Hand-held tools attenuate self-touch. Cognition 165, 1–9 (2017).

  10. 10.

    Katz, D. The World of Touch (Erlbaum, Hillsdale, 1989).

  11. 11.

    Holmes, N. P., Calvert, G. A. & Spence, C. Extending or projecting peripersonal space with tools? Multisensory interactions highlight only the distal and proximal ends of tools. Neurosci. Lett. 372, 62–67 (2004).

  12. 12.

    Saal, H. P., Delhaye, B. P., Rayhaun, B. C. & Bensmaia, S. J. Simulating tactile signals from the whole hand with millisecond precision. Proc. Natl Acad. Sci. USA 114, E5693–E5702 (2017).

  13. 13.

    Chiel, H. J. & Beer, R. D. The brain has a body: adaptive behavior emerges from interactions of nervous system, body and environment. Trends Neurosci. 20, 553–557 (1997).

  14. 14.

    Makin, T. R., De Vignemont, F. & Faisal, A. A. Neurocognitive barriers to the embodiment of technology. Nat. Biomed. Eng. 1, 0014 (2017).

  15. 15.

    Johnson-Frey, S. H. The neural bases of complex tool use in humans. Trends Cogn. Sci. 8, 71–78 (2004).

  16. 16.

    Martel, M., Cardinali, L., Roy, A. C. & Farnè, A. Tool-use: an open window into body representation and its plasticity. Cogn. Neuropsychol. 33, 82–101 (2016).

  17. 17.

    Miller, L. E., Longo, M. R. & Saygin, A. P. Tool morphology constrains the effects of tool use on body representations. J. Exp. Psychol. Hum. Percept. Perform. 40, 2143–2153 (2014).

  18. 18.

    Descartes, R. in Œuvres et lettres (Gallimard, Paris, 1937).

  19. 19.

    Schenkman, B. N. & Jansson, G. The detection and localization of objects by the blind with the aid of long-cane tapping sounds. Hum. Factors 28, 607–618 (1986).

  20. 20.

    Mancini, F., Longo, M. R., Iannetti, G. D. & Haggard, P. A supramodal representation of the body surface. Neuropsychologia 49, 1194–1201 (2011).

  21. 21.

    Kersten, D., Mamassian, P. & Yuille, A. Object perception as Bayesian inference. Annu. Rev. Psychol. 55, 271–304 (2004).

  22. 22.

    Johnson, K. O. The roles and functions of cutaneous mechanoreceptors. Curr. Opin. Neurobiol. 11, 455–461 (2001).

  23. 23.

    Shao, Y., Hayward, V. & Visell, Y. Spatial patterns of cutaneous vibration during whole-hand haptic interactions. Proc. Natl Acad. Sci. USA 113, 4188–4193 (2016).

  24. 24.

    Hayward, V. in Human Haptic Perception: Basics and Applications (ed. Grunwald, M.) 381–392 (Birkhäuser, Basel, 2008).

  25. 25.

    Bell, J., Bolanowski, S. & Holmes, M. H. The structure and function of Pacinian corpuscles: a review. Prog. Neurobiol. 42, 79–128 (1994).

  26. 26.

    Mackevicius, E. L., Best, M. D., Saal, H. P. & Bensmaia, S. J. Millisecond precision spike timing shapes tactile perception. J. Neurosci. 32, 15309–15317 (2012).

  27. 27.

    Talbot, W. H. & Mountcastle, B. The sense of flutter–vibration: comparison of the human capacity with response patterns of mechanoreceptive afferents from the monkey hand. J. Neurophysiol. 31, 301–334 (1968).

  28. 28.

    Johansson, R. S. & Birznieks, I. First spikes in ensembles of human tactile afferents code complex spatial fingertip events. Nat. Neurosci. 7, 170–177 (2004).

  29. 29.

    Imamizu, H. et al. Human cerebellar activity reflecting an acquired internal model of a new tool. Nature 403, 192–195 (2000).

  30. 30.

    Young, R. W. Evolution of the human hand: the role of throwing and clubbing. J. Anat. 202, 165–174 (2003).

  31. 31.

    Cortes, C. & Vapnik, V. Support-vector networks. Mach. Learn. 20, 273–297 (1995).

  32. 32.

    Chang, C.-C. & Lin, C.-J. LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2, 27 (2011).

  33. 33.

    Team, R. C. R. A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, Vienna, 2014).

  34. 34.

    Pruszynski, J. A. & Johansson, R. S. Edge-orientation processing in first-order tactile neurons. Nat. Neurosci. 17, 1404–1409 (2014).

  35. 35.

    Hayward, V. et al. Spatio-temporal skin strain distributions evoke low variability spike responses in cuneate neurons. J. R. Soc. Interface 11, 20131015 (2014).

  36. 36.

    Jörntell, H. et al. Segregation of tactile input features in neurons of the cuneate nucleus. Neuron 83, 1444–1452 (2014).

  37. 37.

    Sestelo, M., Villanueva, N. M., Meira-Machado, L. & Roca-Pardiñas, J. FWDselect: an R package for variable selection in regression models. The R Journal 8, 132–148 (2016)

Download references

Acknowledgements

We thank F. Volland for his help constructing the experimental setup; B. Miller and A. Schork for their statistical advice; and L. Quadt, A. Roy, E. Leonardis and K. Kilteni for their feedback on an earlier version of the manuscript. This work was supported by an FRM postdoctoral fellowship to L.E.M., ANR-16-CE28-0015 Developmental Tool Mastery to A.F. and V.H., a Leverhulme Trust Visiting Professorship Grant to V.H. and IHU CeSaMe ANR-10-IBHU-0003, Defi Auton Sublima and the James S. McDonnell Scholar Award to A.F. All work was performed within the framework of the LABEX CORTEX (ANR-11-LABX-0042) of Université de Lyon.

Reviewer information

Nature thanks S. Bensmaia and G. Stanley for their contribution to the peer review of this work.

Author information

Affiliations

  1. Integrative Multisensory Perception Action & Cognition Team—ImpAct, Lyon Neuroscience Research Center, INSERM U1028, CNRS U5292, Lyon, France

    • Luke E. Miller
    • , Luca Montroni
    • , Eric Koun
    • , Romeo Salemme
    •  & Alessandro Farnè
  2. University of Lyon 1, Lyon, France

    • Luke E. Miller
    • , Luca Montroni
    • , Eric Koun
    • , Romeo Salemme
    •  & Alessandro Farnè
  3. Hospices Civils de Lyon, Neuro-immersion, Lyon, France

    • Luke E. Miller
    • , Romeo Salemme
    •  & Alessandro Farnè
  4. Sorbonne Université, Institut des Systèmes Intelligents et de Robotique (ISIR), Paris, France

    • Vincent Hayward
  5. Centre for the Study of the Senses, School of Advanced Study, University of London, London, UK

    • Vincent Hayward
  6. Center for Mind/Brain Sciences, University of Trento, Rovereto, Italy

    • Alessandro Farnè

Authors

  1. Search for Luke E. Miller in:

  2. Search for Luca Montroni in:

  3. Search for Eric Koun in:

  4. Search for Romeo Salemme in:

  5. Search for Vincent Hayward in:

  6. Search for Alessandro Farnè in:

Contributions

L.E.M., V.H. and A.F. conceived the study and designed the experiments. L.E.M. and L.M. collected and analysed the behavioural experiments. E.K. and R.S. constructed equipment for the vibration experiment and helped process the data. L.E.M., V.H. and A.F. designed and analysed the neuronal modelling and vibration experiments. V.H. developed the theoretical framework presented in the Supplementary Data. L.E.M., V.H. and A.F. wrote the paper. All authors approved the final version of the paper.

Competing interests

The authors declare no competing interests.

Corresponding authors

Correspondence to Luke E. Miller or Alessandro Farnè.

Extended data figures and tables

  1. Extended Data Fig. 1 Setup for the behavioural and vibration experiments.

    a, The experimental setup for all behavioural experiments (see Methods for details). The object shown below the rod was used in the first three behavioural experiments. b, The hybrid tool used in experiment 6, which was half rigid (wood and insulation) and half non-rigid (insulation only). The foam is displayed as being open for presentation purposes only. The dimensions of the rod have also been altered for presentation purposes. See Methods for more details. c, Setup for the vibration experiment, where accelerometers were attached to the handle and the participant’s index finger.

  2. Extended Data Fig. 2 Results of behavioural experiments.

    a–f, Group-level affine regression for experiments 2–5. Experiments 1, 2, 3 and 5 (n = 10), experiment 4 (n = 20). Coloured dashed lines around the model fits correspond to its 95% confidence interval. The grey line corresponds to the equality line. c, Experiment 4: Pearson’s correlation between the regression slopes for when the drawing was displayed in parallel with the actual rod and rotated 90° counter-clockwise. g, Left, slope for every dataset from experiments 1–5 (n = 60). The prediction of the distalization model (that is, chance performance) is shown by the orange line. Right, average slope with 95% confidence intervals. h, Experiment 6, contact at identical locations on a wood (green) and foam (purple) tool leads to drastically different vibration patterns. In the case of the hybrid tool, participants could only feel the wooden portion of the rod with their hand, making the vibration pattern from the foam portion of the rod unexpected and therefore uninformative. i, Localization for each participant in the vibration experiment was within the range of behaviour observed in the other six experiments.

  3. Extended Data Fig. 3 Modal responses of a rod under different initial and boundary conditions.

    a, Independent of the material of a rod, the modal frequencies in the free case alias to the next highest modes in the clamped case. In the plot, this can be clearly seen for a simulated rod with a circular cross-section (length: 83 cm; cross-section radius: 0.8 cm) that was made of one of a diverse set of materials of varying elasticities and densities. The grey line corresponds to the equality line between the frequencies of each limit case. b, Mode shapes for the first four modes in the free case. c, Modes shapes for the first five modes in the clamped case. Modes with similar shapes as the free case are matched by colour. d, e, Simulated displacement, velocity and acceleration after impact on a free rod at l* = 0.33 (d) and 0.5 (e). The zero crossings of velocity and acceleration are presented as tick marks above the relevant curves. The weights are taken from the mode shapes. f, g, Same for the clamped case, but minus the ‘whipping’ first mode. This is justified as all other mode shapes and frequencies are shared between cases. Notably, there is a high degree of similarity between the modal responses in each case for impacts at identical locations. Furthermore, it can be noticed in all panels that the sequences of zero crossings tend to repeat themselves owing to the special distribution of modal frequencies reflected in the phase differences. After a few periods of the low frequency modes these sequences can generally be easily discriminated. Thus, an effective feature space could be simply a relatively small number of time intervals between extrema in a suitably filtered signal.

  4. Extended Data Fig. 4 Trial-by-trial vibrations for each participant were highly consistent.

    ac, When held in the hand, vibrations after impact at each location on the tool were highly consistent for each participant: LO (a), AY (b) and EA (c). The upper left plot in each panel corresponds to the histogram of each within-location Pearson’s correlation (0–100 ms after impact, corresponding to 250 data points per test). The shift in the distribution towards high correlations for each participant (LO: median r = 0.58, interquartile range (IQR) = 0.19; AY: median r = 0.73, IQR = 0.16; EA: median r = 0.79, IQR = 0.23) provides evidence for the emergence of vibratory motifs during passive and active sensing. The traces correspond to the motifs for each location (landmark 1 (blue) to landmark 7 (red), from left to right). The grey traces correspond to each individual impact and the colour traces correspond to the mean trace (0–100 ms after impact; colour coded by location).

  5. Extended Data Fig. 5 The dimensionality of motifs can be reduced to their zero crossings.

    a, b, The motif (black; acceleration), zero crossing in acceleration (green) and velocity (blue), and spikes (orange) of a representative trial from the datasets of participants (a) EA and (b) LO. We observe a precise temporal relationship between zero crossings in velocity (blue) and the spikes of a Pacinian mechanoreceptor simulated with TouchSim (orange). c, We observed high classification accuracy when decoding impact location from the pattern of zero crossings in acceleration of motifs for each dataset (chance = approximately 14%). d, We could accurately model each participant’s trial-by-trial behaviour given the temporal pattern of zero crossing in the acceleration of the corresponding motif (140 data points per test). We plot the goodness of fit as a function of the number of predictors used in the model. Solid lines correspond to the R2 and dashed lines correspond to the predictive R2.

  6. Extended Data Fig. 6 Results for skin recordings.

    a, b, We found similar spectral content between the mechanical (tool) and cutaneous (D2m) vibrations for both LO (a) and AY (b). Both participants showed similar peaks in the power spectrum for the vibrations (0–200 ms after impact) on the tool (black line) and skin (grey line). Furthermore, the trial-by-trial correlations between the spectral content of the wood and skin vibrations (histograms on the right) were high for both LO and AY. c, Histograms of all within-location comparisons for participants LO (left; median r = 0.65, IQR = 0.21) and AY (right; median r = 0.41, IQR = 0.17). Pearson’s correlation on the first 100 ms after impact, corresponding to 250 data points per test. d, The observed speed that location information accumulated within the vibrations on the skin was extremely rapid for each participant, mirroring what was observed when the tool was clamped with a bench vice (green line; see the experiment in the Supplementary Data section 3).

  7. Extended Data Fig. 7 The effect of temporal smoothing on population spiking.

    Population-level spiking (−15–215 ms after impact) of the simulated afferents for randomly chosen trials from four different locations: landmark 1 (blue; trial 12), landmark 2 (purple; trial 39), landmark 4 (green; trial 68), and landmark 6 (brown; trial 114). To reduce the temporal resolution of spiking, we smoothed the response with Gaussian kernels at six different widths. The resulting traces were then used to investigate whether impact location coding was dependent on millisecond-resolution spike-timing (see Fig. 3c). Only trials from the dataset of participant EA are shown in this figure, but nearly identical patterns of results were found for both LO and AY.

  8. Extended Data Fig. 8 Model fits for each participant.

    ac, The trial-by-trial population spike-timing of the putative afferents (left plots) precisely predicted the behaviour of each participant: LO (a), AY (b), and EA (c). When the population rate code (centre plots) was used as features, the model did not provide a precise fit to the behaviour (see Extended Data Table 2). Vibratory motifs (right plots) also predicted behaviour with high precision. The grey line in all plots represents the equality line between actual and predicted behaviour, not the actual regression line.

  9. Extended Table 1 Behavioural results
  10. Extended Table 2 Multivariate models with forty-two predictor variables

Supplementary information

  1. Supplementary Information

    This file contains Supplementary Data.

  2. Reporting Summary

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/s41586-018-0460-0

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.