Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Sensing with tools extends somatosensory processing beyond the body

Nature volume 561, pages 239–242 (2018)Cite this article

Subjects

Abstract

The ability to extend sensory information processing beyond the nervous system1 has been observed throughout the animal kingdom; for example, when rodents palpate objects using whiskers2 and spiders localize prey using webs3. We investigated whether the ability to sense objects with tools4,5,6,7,8,9 represents an analogous information processing scheme in humans. Here we provide evidence from behavioural psychophysics, structural mechanics and neuronal modelling, which shows that tools are treated by the nervous system as sensory extensions of the body rather than as simple distal links between the hand and the environment10,11. We first demonstrate that tool users can accurately sense where an object contacts a wooden rod, just as is possible on the skin. We next demonstrate that the impact location is encoded by the modal response of the tool upon impact, reflecting a pre-neuronal stage of mechanical information processing akin to sensing with whiskers2 and webs3. Lastly, we use a computational model of tactile afferents12 to demonstrate that impact location can be rapidly re-encoded into a temporally precise spiking code. This code predicts the behaviour of human participants, providing evidence that the information encoded in motifs shapes localization. Thus, we show that this sensory capability emerges from the functional coupling between the material, biomechanical and neural levels of information processing13,14.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Rent or buy this article

Get just this article for as long as you need it

$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Methods and behavioural results.
Fig. 2: Vibratory motifs emerge rapidly during extended location sensing.
Fig. 3: Impact location is encoded in the spike timing of tactile mechanoreceptors in the hand.

References

  1. Clark, A. & Chalmers, D. The extended mind. Analysis 58, 7–19 (1998).

    Google Scholar 

  2. Bagdasarian, K. et al. Pre-neuronal morphological processing of object location by individual whiskers. Nat. Neurosci. 16, 622–631 (2013).

    CAS  Google Scholar 

  3. Japyassú, H. F. & Laland, K. N. Extended spider cognition. Anim. Cogn. 20, 375–395 (2017).

    Google Scholar 

  4. Chan, T. C. & Turvey, M. T. Perceiving the vertical distances of surfaces by means of a hand-held probe. J. Exp. Psychol. Hum. Percept. Perform. 17, 347–358 (1991).

    CAS  Google Scholar 

  5. Brisben, A. J., Hsiao, S. S. & Johnson, K. O. Detection of vibration transmitted through an object grasped in the hand. J. Neurophysiol. 81, 1548–1558 (1999).

    CAS  Google Scholar 

  6. Klatzky, R. L. & Lederman, S. J. Tactile roughness perception with a rigid link interposed between skin and surface. Percept. Psychophys. 61, 591–607 (1999).

    CAS  Google Scholar 

  7. Yamamoto, S. & Kitazawa, S. Sensation at the tips of invisible tools. Nat. Neurosci. 4, 979–980 (2001).

    CAS  Google Scholar 

  8. Saig, A., Gordon, G., Assa, E., Arieli, A. & Ahissar, E. Motor-sensory confluence in tactile perception. J. Neurosci. 32, 14022–14032 (2012).

    CAS  Google Scholar 

  9. Kilteni, K. & Ehrsson, H. H. Sensorimotor predictions and tool use: Hand-held tools attenuate self-touch. Cognition 165, 1–9 (2017).

    Google Scholar 

  10. Katz, D. The World of Touch (Erlbaum, Hillsdale, 1989).

  11. Holmes, N. P., Calvert, G. A. & Spence, C. Extending or projecting peripersonal space with tools? Multisensory interactions highlight only the distal and proximal ends of tools. Neurosci. Lett. 372, 62–67 (2004).

    CAS  Google Scholar 

  12. Saal, H. P., Delhaye, B. P., Rayhaun, B. C. & Bensmaia, S. J. Simulating tactile signals from the whole hand with millisecond precision. Proc. Natl Acad. Sci. USA 114, E5693–E5702 (2017).

    CAS  Google Scholar 

  13. Chiel, H. J. & Beer, R. D. The brain has a body: adaptive behavior emerges from interactions of nervous system, body and environment. Trends Neurosci. 20, 553–557 (1997).

    CAS  Google Scholar 

  14. Makin, T. R., De Vignemont, F. & Faisal, A. A. Neurocognitive barriers to the embodiment of technology. Nat. Biomed. Eng. 1, 0014 (2017).

    Google Scholar 

  15. Johnson-Frey, S. H. The neural bases of complex tool use in humans. Trends Cogn. Sci. 8, 71–78 (2004).

    Google Scholar 

  16. Martel, M., Cardinali, L., Roy, A. C. & Farnè, A. Tool-use: an open window into body representation and its plasticity. Cogn. Neuropsychol. 33, 82–101 (2016).

    Google Scholar 

  17. Miller, L. E., Longo, M. R. & Saygin, A. P. Tool morphology constrains the effects of tool use on body representations. J. Exp. Psychol. Hum. Percept. Perform. 40, 2143–2153 (2014).

    Google Scholar 

  18. Descartes, R. in Œuvres et lettres (Gallimard, Paris, 1937).

  19. Schenkman, B. N. & Jansson, G. The detection and localization of objects by the blind with the aid of long-cane tapping sounds. Hum. Factors 28, 607–618 (1986).

    Google Scholar 

  20. Mancini, F., Longo, M. R., Iannetti, G. D. & Haggard, P. A supramodal representation of the body surface. Neuropsychologia 49, 1194–1201 (2011).

    Google Scholar 

  21. Kersten, D., Mamassian, P. & Yuille, A. Object perception as Bayesian inference. Annu. Rev. Psychol. 55, 271–304 (2004).

    Google Scholar 

  22. Johnson, K. O. The roles and functions of cutaneous mechanoreceptors. Curr. Opin. Neurobiol. 11, 455–461 (2001).

    CAS  Google Scholar 

  23. Shao, Y., Hayward, V. & Visell, Y. Spatial patterns of cutaneous vibration during whole-hand haptic interactions. Proc. Natl Acad. Sci. USA 113, 4188–4193 (2016).

    CAS  Google Scholar 

  24. Hayward, V. in Human Haptic Perception: Basics and Applications (ed. Grunwald, M.) 381–392 (Birkhäuser, Basel, 2008).

  25. Bell, J., Bolanowski, S. & Holmes, M. H. The structure and function of Pacinian corpuscles: a review. Prog. Neurobiol. 42, 79–128 (1994).

    CAS  Google Scholar 

  26. Mackevicius, E. L., Best, M. D., Saal, H. P. & Bensmaia, S. J. Millisecond precision spike timing shapes tactile perception. J. Neurosci. 32, 15309–15317 (2012).

    CAS  Google Scholar 

  27. Talbot, W. H. & Mountcastle, B. The sense of flutter–vibration: comparison of the human capacity with response patterns of mechanoreceptive afferents from the monkey hand. J. Neurophysiol. 31, 301–334 (1968).

    CAS  Google Scholar 

  28. Johansson, R. S. & Birznieks, I. First spikes in ensembles of human tactile afferents code complex spatial fingertip events. Nat. Neurosci. 7, 170–177 (2004).

    CAS  Google Scholar 

  29. Imamizu, H. et al. Human cerebellar activity reflecting an acquired internal model of a new tool. Nature 403, 192–195 (2000).

    CAS  Google Scholar 

  30. Young, R. W. Evolution of the human hand: the role of throwing and clubbing. J. Anat. 202, 165–174 (2003).

    Google Scholar 

  31. Cortes, C. & Vapnik, V. Support-vector networks. Mach. Learn. 20, 273–297 (1995).

    Google Scholar 

  32. Chang, C.-C. & Lin, C.-J. LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2, 27 (2011).

    Google Scholar 

  33. Team, R. C. R. A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, Vienna, 2014).

    Google Scholar 

  34. Pruszynski, J. A. & Johansson, R. S. Edge-orientation processing in first-order tactile neurons. Nat. Neurosci. 17, 1404–1409 (2014).

    CAS  Google Scholar 

  35. Hayward, V. et al. Spatio-temporal skin strain distributions evoke low variability spike responses in cuneate neurons. J. R. Soc. Interface 11, 20131015 (2014).

    Google Scholar 

  36. Jörntell, H. et al. Segregation of tactile input features in neurons of the cuneate nucleus. Neuron 83, 1444–1452 (2014).

    Google Scholar 

  37. Sestelo, M., Villanueva, N. M., Meira-Machado, L. & Roca-Pardiñas, J. FWDselect: an R package for variable selection in regression models. The R Journal 8, 132–148 (2016)

    Google Scholar 

Download references

Acknowledgements

We thank F. Volland for his help constructing the experimental setup; B. Miller and A. Schork for their statistical advice; and L. Quadt, A. Roy, E. Leonardis and K. Kilteni for their feedback on an earlier version of the manuscript. This work was supported by an FRM postdoctoral fellowship to L.E.M., ANR-16-CE28-0015 Developmental Tool Mastery to A.F. and V.H., a Leverhulme Trust Visiting Professorship Grant to V.H. and IHU CeSaMe ANR-10-IBHU-0003, Defi Auton Sublima and the James S. McDonnell Scholar Award to A.F. All work was performed within the framework of the LABEX CORTEX (ANR-11-LABX-0042) of Université de Lyon.

Reviewer information

Nature thanks S. Bensmaia and G. Stanley for their contribution to the peer review of this work.

Author information

Authors and Affiliations

  1. Integrative Multisensory Perception Action & Cognition Team—ImpAct, Lyon Neuroscience Research Center, INSERM U1028, CNRS U5292, Lyon, France

    Luke E. Miller, Luca Montroni, Eric Koun, Romeo Salemme & Alessandro Farnè

  2. University of Lyon 1, Lyon, France

    Luke E. Miller, Luca Montroni, Eric Koun, Romeo Salemme & Alessandro Farnè

  3. Hospices Civils de Lyon, Neuro-immersion, Lyon, France

    Luke E. Miller, Romeo Salemme & Alessandro Farnè

  4. Sorbonne Université, Institut des Systèmes Intelligents et de Robotique (ISIR), Paris, France

    Vincent Hayward

  5. Centre for the Study of the Senses, School of Advanced Study, University of London, London, UK

    Vincent Hayward

  6. Center for Mind/Brain Sciences, University of Trento, Rovereto, Italy

    Alessandro Farnè

Authors
  1. Luke E. Miller
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luca Montroni
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eric Koun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Romeo Salemme
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Vincent Hayward
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Alessandro Farnè
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

L.E.M., V.H. and A.F. conceived the study and designed the experiments. L.E.M. and L.M. collected and analysed the behavioural experiments. E.K. and R.S. constructed equipment for the vibration experiment and helped process the data. L.E.M., V.H. and A.F. designed and analysed the neuronal modelling and vibration experiments. V.H. developed the theoretical framework presented in the Supplementary Data. L.E.M., V.H. and A.F. wrote the paper. All authors approved the final version of the paper.

Corresponding authors

Correspondence to Luke E. Miller or Alessandro Farnè.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data figures and tables

Extended Data Fig. 1 Setup for the behavioural and vibration experiments.

a, The experimental setup for all behavioural experiments (see Methods for details). The object shown below the rod was used in the first three behavioural experiments. b, The hybrid tool used in experiment 6, which was half rigid (wood and insulation) and half non-rigid (insulation only). The foam is displayed as being open for presentation purposes only. The dimensions of the rod have also been altered for presentation purposes. See Methods for more details. c, Setup for the vibration experiment, where accelerometers were attached to the handle and the participant’s index finger.

Extended Data Fig. 2 Results of behavioural experiments.

a–f, Group-level affine regression for experiments 2–5. Experiments 1, 2, 3 and 5 (n = 10), experiment 4 (n = 20). Coloured dashed lines around the model fits correspond to its 95% confidence interval. The grey line corresponds to the equality line. c, Experiment 4: Pearson’s correlation between the regression slopes for when the drawing was displayed in parallel with the actual rod and rotated 90° counter-clockwise. g, Left, slope for every dataset from experiments 1–5 (n = 60). The prediction of the distalization model (that is, chance performance) is shown by the orange line. Right, average slope with 95% confidence intervals. h, Experiment 6, contact at identical locations on a wood (green) and foam (purple) tool leads to drastically different vibration patterns. In the case of the hybrid tool, participants could only feel the wooden portion of the rod with their hand, making the vibration pattern from the foam portion of the rod unexpected and therefore uninformative. i, Localization for each participant in the vibration experiment was within the range of behaviour observed in the other six experiments.

Extended Data Fig. 3 Modal responses of a rod under different initial and boundary conditions.

a, Independent of the material of a rod, the modal frequencies in the free case alias to the next highest modes in the clamped case. In the plot, this can be clearly seen for a simulated rod with a circular cross-section (length: 83 cm; cross-section radius: 0.8 cm) that was made of one of a diverse set of materials of varying elasticities and densities. The grey line corresponds to the equality line between the frequencies of each limit case. b, Mode shapes for the first four modes in the free case. c, Modes shapes for the first five modes in the clamped case. Modes with similar shapes as the free case are matched by colour. d, e, Simulated displacement, velocity and acceleration after impact on a free rod at l* = 0.33 (d) and 0.5 (e). The zero crossings of velocity and acceleration are presented as tick marks above the relevant curves. The weights are taken from the mode shapes. f, g, Same for the clamped case, but minus the ‘whipping’ first mode. This is justified as all other mode shapes and frequencies are shared between cases. Notably, there is a high degree of similarity between the modal responses in each case for impacts at identical locations. Furthermore, it can be noticed in all panels that the sequences of zero crossings tend to repeat themselves owing to the special distribution of modal frequencies reflected in the phase differences. After a few periods of the low frequency modes these sequences can generally be easily discriminated. Thus, an effective feature space could be simply a relatively small number of time intervals between extrema in a suitably filtered signal.

Extended Data Fig. 4 Trial-by-trial vibrations for each participant were highly consistent.

a–c, When held in the hand, vibrations after impact at each location on the tool were highly consistent for each participant: LO (a), AY (b) and EA (c). The upper left plot in each panel corresponds to the histogram of each within-location Pearson’s correlation (0–100 ms after impact, corresponding to 250 data points per test). The shift in the distribution towards high correlations for each participant (LO: median r = 0.58, interquartile range (IQR) = 0.19; AY: median r = 0.73, IQR = 0.16; EA: median r = 0.79, IQR = 0.23) provides evidence for the emergence of vibratory motifs during passive and active sensing. The traces correspond to the motifs for each location (landmark 1 (blue) to landmark 7 (red), from left to right). The grey traces correspond to each individual impact and the colour traces correspond to the mean trace (0–100 ms after impact; colour coded by location).

Extended Data Fig. 5 The dimensionality of motifs can be reduced to their zero crossings.

a, b, The motif (black; acceleration), zero crossing in acceleration (green) and velocity (blue), and spikes (orange) of a representative trial from the datasets of participants (a) EA and (b) LO. We observe a precise temporal relationship between zero crossings in velocity (blue) and the spikes of a Pacinian mechanoreceptor simulated with TouchSim (orange). c, We observed high classification accuracy when decoding impact location from the pattern of zero crossings in acceleration of motifs for each dataset (chance = approximately 14%). d, We could accurately model each participant’s trial-by-trial behaviour given the temporal pattern of zero crossing in the acceleration of the corresponding motif (140 data points per test). We plot the goodness of fit as a function of the number of predictors used in the model. Solid lines correspond to the R2 and dashed lines correspond to the predictive R2.

Extended Data Fig. 6 Results for skin recordings.

a, b, We found similar spectral content between the mechanical (tool) and cutaneous (D2m) vibrations for both LO (a) and AY (b). Both participants showed similar peaks in the power spectrum for the vibrations (0–200 ms after impact) on the tool (black line) and skin (grey line). Furthermore, the trial-by-trial correlations between the spectral content of the wood and skin vibrations (histograms on the right) were high for both LO and AY. c, Histograms of all within-location comparisons for participants LO (left; median r = 0.65, IQR = 0.21) and AY (right; median r = 0.41, IQR = 0.17). Pearson’s correlation on the first 100 ms after impact, corresponding to 250 data points per test. d, The observed speed that location information accumulated within the vibrations on the skin was extremely rapid for each participant, mirroring what was observed when the tool was clamped with a bench vice (green line; see the experiment in the Supplementary Data section 3).

Extended Data Fig. 7 The effect of temporal smoothing on population spiking.

Population-level spiking (−15–215 ms after impact) of the simulated afferents for randomly chosen trials from four different locations: landmark 1 (blue; trial 12), landmark 2 (purple; trial 39), landmark 4 (green; trial 68), and landmark 6 (brown; trial 114). To reduce the temporal resolution of spiking, we smoothed the response with Gaussian kernels at six different widths. The resulting traces were then used to investigate whether impact location coding was dependent on millisecond-resolution spike-timing (see Fig. 3c). Only trials from the dataset of participant EA are shown in this figure, but nearly identical patterns of results were found for both LO and AY.

Extended Data Fig. 8 Model fits for each participant.

a–c, The trial-by-trial population spike-timing of the putative afferents (left plots) precisely predicted the behaviour of each participant: LO (a), AY (b), and EA (c). When the population rate code (centre plots) was used as features, the model did not provide a precise fit to the behaviour (see Extended Data Table 2). Vibratory motifs (right plots) also predicted behaviour with high precision. The grey line in all plots represents the equality line between actual and predicted behaviour, not the actual regression line.

Extended Table 1 Behavioural results
Full size table
Extended Table 2 Multivariate models with forty-two predictor variables
Full size table

Supplementary information

Supplementary Information

This file contains Supplementary Data.

Reporting Summary

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Miller, L.E., Montroni, L., Koun, E. et al. Sensing with tools extends somatosensory processing beyond the body. Nature 561, 239–242 (2018). https://doi.org/10.1038/s41586-018-0460-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41586-018-0460-0

Keywords

  • Local Impact
  • Wooden Rod
  • Sensory Extensions
  • Object Contact
  • Sensory Embodiment

This article is cited by

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing