The ability to extend sensory information processing beyond the nervous system1 has been observed throughout the animal kingdom; for example, when rodents palpate objects using whiskers2 and spiders localize prey using webs3. We investigated whether the ability to sense objects with tools4,5,6,7,8,9 represents an analogous information processing scheme in humans. Here we provide evidence from behavioural psychophysics, structural mechanics and neuronal modelling, which shows that tools are treated by the nervous system as sensory extensions of the body rather than as simple distal links between the hand and the environment10,11. We first demonstrate that tool users can accurately sense where an object contacts a wooden rod, just as is possible on the skin. We next demonstrate that the impact location is encoded by the modal response of the tool upon impact, reflecting a pre-neuronal stage of mechanical information processing akin to sensing with whiskers2 and webs3. Lastly, we use a computational model of tactile afferents12 to demonstrate that impact location can be rapidly re-encoded into a temporally precise spiking code. This code predicts the behaviour of human participants, providing evidence that the information encoded in motifs shapes localization. Thus, we show that this sensory capability emerges from the functional coupling between the material, biomechanical and neural levels of information processing13,14.
Subscribe to Journal
Get full journal access for 1 year
only $3.90 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Clark, A. & Chalmers, D. The extended mind. Analysis 58, 7–19 (1998).
Bagdasarian, K. et al. Pre-neuronal morphological processing of object location by individual whiskers. Nat. Neurosci. 16, 622–631 (2013).
Japyassú, H. F. & Laland, K. N. Extended spider cognition. Anim. Cogn. 20, 375–395 (2017).
Chan, T. C. & Turvey, M. T. Perceiving the vertical distances of surfaces by means of a hand-held probe. J. Exp. Psychol. Hum. Percept. Perform. 17, 347–358 (1991).
Brisben, A. J., Hsiao, S. S. & Johnson, K. O. Detection of vibration transmitted through an object grasped in the hand. J. Neurophysiol. 81, 1548–1558 (1999).
Klatzky, R. L. & Lederman, S. J. Tactile roughness perception with a rigid link interposed between skin and surface. Percept. Psychophys. 61, 591–607 (1999).
Yamamoto, S. & Kitazawa, S. Sensation at the tips of invisible tools. Nat. Neurosci. 4, 979–980 (2001).
Saig, A., Gordon, G., Assa, E., Arieli, A. & Ahissar, E. Motor-sensory confluence in tactile perception. J. Neurosci. 32, 14022–14032 (2012).
Kilteni, K. & Ehrsson, H. H. Sensorimotor predictions and tool use: Hand-held tools attenuate self-touch. Cognition 165, 1–9 (2017).
Katz, D. The World of Touch (Erlbaum, Hillsdale, 1989).
Holmes, N. P., Calvert, G. A. & Spence, C. Extending or projecting peripersonal space with tools? Multisensory interactions highlight only the distal and proximal ends of tools. Neurosci. Lett. 372, 62–67 (2004).
Saal, H. P., Delhaye, B. P., Rayhaun, B. C. & Bensmaia, S. J. Simulating tactile signals from the whole hand with millisecond precision. Proc. Natl Acad. Sci. USA 114, E5693–E5702 (2017).
Chiel, H. J. & Beer, R. D. The brain has a body: adaptive behavior emerges from interactions of nervous system, body and environment. Trends Neurosci. 20, 553–557 (1997).
Makin, T. R., De Vignemont, F. & Faisal, A. A. Neurocognitive barriers to the embodiment of technology. Nat. Biomed. Eng. 1, 0014 (2017).
Johnson-Frey, S. H. The neural bases of complex tool use in humans. Trends Cogn. Sci. 8, 71–78 (2004).
Martel, M., Cardinali, L., Roy, A. C. & Farnè, A. Tool-use: an open window into body representation and its plasticity. Cogn. Neuropsychol. 33, 82–101 (2016).
Miller, L. E., Longo, M. R. & Saygin, A. P. Tool morphology constrains the effects of tool use on body representations. J. Exp. Psychol. Hum. Percept. Perform. 40, 2143–2153 (2014).
Descartes, R. in Œuvres et lettres (Gallimard, Paris, 1937).
Schenkman, B. N. & Jansson, G. The detection and localization of objects by the blind with the aid of long-cane tapping sounds. Hum. Factors 28, 607–618 (1986).
Mancini, F., Longo, M. R., Iannetti, G. D. & Haggard, P. A supramodal representation of the body surface. Neuropsychologia 49, 1194–1201 (2011).
Kersten, D., Mamassian, P. & Yuille, A. Object perception as Bayesian inference. Annu. Rev. Psychol. 55, 271–304 (2004).
Johnson, K. O. The roles and functions of cutaneous mechanoreceptors. Curr. Opin. Neurobiol. 11, 455–461 (2001).
Shao, Y., Hayward, V. & Visell, Y. Spatial patterns of cutaneous vibration during whole-hand haptic interactions. Proc. Natl Acad. Sci. USA 113, 4188–4193 (2016).
Hayward, V. in Human Haptic Perception: Basics and Applications (ed. Grunwald, M.) 381–392 (Birkhäuser, Basel, 2008).
Bell, J., Bolanowski, S. & Holmes, M. H. The structure and function of Pacinian corpuscles: a review. Prog. Neurobiol. 42, 79–128 (1994).
Mackevicius, E. L., Best, M. D., Saal, H. P. & Bensmaia, S. J. Millisecond precision spike timing shapes tactile perception. J. Neurosci. 32, 15309–15317 (2012).
Talbot, W. H. & Mountcastle, B. The sense of flutter–vibration: comparison of the human capacity with response patterns of mechanoreceptive afferents from the monkey hand. J. Neurophysiol. 31, 301–334 (1968).
Johansson, R. S. & Birznieks, I. First spikes in ensembles of human tactile afferents code complex spatial fingertip events. Nat. Neurosci. 7, 170–177 (2004).
Imamizu, H. et al. Human cerebellar activity reflecting an acquired internal model of a new tool. Nature 403, 192–195 (2000).
Young, R. W. Evolution of the human hand: the role of throwing and clubbing. J. Anat. 202, 165–174 (2003).
Cortes, C. & Vapnik, V. Support-vector networks. Mach. Learn. 20, 273–297 (1995).
Chang, C.-C. & Lin, C.-J. LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2, 27 (2011).
Team, R. C. R. A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, Vienna, 2014).
Pruszynski, J. A. & Johansson, R. S. Edge-orientation processing in first-order tactile neurons. Nat. Neurosci. 17, 1404–1409 (2014).
Hayward, V. et al. Spatio-temporal skin strain distributions evoke low variability spike responses in cuneate neurons. J. R. Soc. Interface 11, 20131015 (2014).
Jörntell, H. et al. Segregation of tactile input features in neurons of the cuneate nucleus. Neuron 83, 1444–1452 (2014).
Sestelo, M., Villanueva, N. M., Meira-Machado, L. & Roca-Pardiñas, J. FWDselect: an R package for variable selection in regression models. The R Journal 8, 132–148 (2016)
We thank F. Volland for his help constructing the experimental setup; B. Miller and A. Schork for their statistical advice; and L. Quadt, A. Roy, E. Leonardis and K. Kilteni for their feedback on an earlier version of the manuscript. This work was supported by an FRM postdoctoral fellowship to L.E.M., ANR-16-CE28-0015 Developmental Tool Mastery to A.F. and V.H., a Leverhulme Trust Visiting Professorship Grant to V.H. and IHU CeSaMe ANR-10-IBHU-0003, Defi Auton Sublima and the James S. McDonnell Scholar Award to A.F. All work was performed within the framework of the LABEX CORTEX (ANR-11-LABX-0042) of Université de Lyon.
Nature thanks S. Bensmaia and G. Stanley for their contribution to the peer review of this work.
The authors declare no competing interests.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data figures and tables
a, The experimental setup for all behavioural experiments (see Methods for details). The object shown below the rod was used in the first three behavioural experiments. b, The hybrid tool used in experiment 6, which was half rigid (wood and insulation) and half non-rigid (insulation only). The foam is displayed as being open for presentation purposes only. The dimensions of the rod have also been altered for presentation purposes. See Methods for more details. c, Setup for the vibration experiment, where accelerometers were attached to the handle and the participant’s index finger.
a–f, Group-level affine regression for experiments 2–5. Experiments 1, 2, 3 and 5 (n = 10), experiment 4 (n = 20). Coloured dashed lines around the model fits correspond to its 95% confidence interval. The grey line corresponds to the equality line. c, Experiment 4: Pearson’s correlation between the regression slopes for when the drawing was displayed in parallel with the actual rod and rotated 90° counter-clockwise. g, Left, slope for every dataset from experiments 1–5 (n = 60). The prediction of the distalization model (that is, chance performance) is shown by the orange line. Right, average slope with 95% confidence intervals. h, Experiment 6, contact at identical locations on a wood (green) and foam (purple) tool leads to drastically different vibration patterns. In the case of the hybrid tool, participants could only feel the wooden portion of the rod with their hand, making the vibration pattern from the foam portion of the rod unexpected and therefore uninformative. i, Localization for each participant in the vibration experiment was within the range of behaviour observed in the other six experiments.
a, Independent of the material of a rod, the modal frequencies in the free case alias to the next highest modes in the clamped case. In the plot, this can be clearly seen for a simulated rod with a circular cross-section (length: 83 cm; cross-section radius: 0.8 cm) that was made of one of a diverse set of materials of varying elasticities and densities. The grey line corresponds to the equality line between the frequencies of each limit case. b, Mode shapes for the first four modes in the free case. c, Modes shapes for the first five modes in the clamped case. Modes with similar shapes as the free case are matched by colour. d, e, Simulated displacement, velocity and acceleration after impact on a free rod at l* = 0.33 (d) and 0.5 (e). The zero crossings of velocity and acceleration are presented as tick marks above the relevant curves. The weights are taken from the mode shapes. f, g, Same for the clamped case, but minus the ‘whipping’ first mode. This is justified as all other mode shapes and frequencies are shared between cases. Notably, there is a high degree of similarity between the modal responses in each case for impacts at identical locations. Furthermore, it can be noticed in all panels that the sequences of zero crossings tend to repeat themselves owing to the special distribution of modal frequencies reflected in the phase differences. After a few periods of the low frequency modes these sequences can generally be easily discriminated. Thus, an effective feature space could be simply a relatively small number of time intervals between extrema in a suitably filtered signal.
a–c, When held in the hand, vibrations after impact at each location on the tool were highly consistent for each participant: LO (a), AY (b) and EA (c). The upper left plot in each panel corresponds to the histogram of each within-location Pearson’s correlation (0–100 ms after impact, corresponding to 250 data points per test). The shift in the distribution towards high correlations for each participant (LO: median r = 0.58, interquartile range (IQR) = 0.19; AY: median r = 0.73, IQR = 0.16; EA: median r = 0.79, IQR = 0.23) provides evidence for the emergence of vibratory motifs during passive and active sensing. The traces correspond to the motifs for each location (landmark 1 (blue) to landmark 7 (red), from left to right). The grey traces correspond to each individual impact and the colour traces correspond to the mean trace (0–100 ms after impact; colour coded by location).
a, b, The motif (black; acceleration), zero crossing in acceleration (green) and velocity (blue), and spikes (orange) of a representative trial from the datasets of participants (a) EA and (b) LO. We observe a precise temporal relationship between zero crossings in velocity (blue) and the spikes of a Pacinian mechanoreceptor simulated with TouchSim (orange). c, We observed high classification accuracy when decoding impact location from the pattern of zero crossings in acceleration of motifs for each dataset (chance = approximately 14%). d, We could accurately model each participant’s trial-by-trial behaviour given the temporal pattern of zero crossing in the acceleration of the corresponding motif (140 data points per test). We plot the goodness of fit as a function of the number of predictors used in the model. Solid lines correspond to the R2 and dashed lines correspond to the predictive R2.
a, b, We found similar spectral content between the mechanical (tool) and cutaneous (D2m) vibrations for both LO (a) and AY (b). Both participants showed similar peaks in the power spectrum for the vibrations (0–200 ms after impact) on the tool (black line) and skin (grey line). Furthermore, the trial-by-trial correlations between the spectral content of the wood and skin vibrations (histograms on the right) were high for both LO and AY. c, Histograms of all within-location comparisons for participants LO (left; median r = 0.65, IQR = 0.21) and AY (right; median r = 0.41, IQR = 0.17). Pearson’s correlation on the first 100 ms after impact, corresponding to 250 data points per test. d, The observed speed that location information accumulated within the vibrations on the skin was extremely rapid for each participant, mirroring what was observed when the tool was clamped with a bench vice (green line; see the experiment in the Supplementary Data section 3).
Population-level spiking (−15–215 ms after impact) of the simulated afferents for randomly chosen trials from four different locations: landmark 1 (blue; trial 12), landmark 2 (purple; trial 39), landmark 4 (green; trial 68), and landmark 6 (brown; trial 114). To reduce the temporal resolution of spiking, we smoothed the response with Gaussian kernels at six different widths. The resulting traces were then used to investigate whether impact location coding was dependent on millisecond-resolution spike-timing (see Fig. 3c). Only trials from the dataset of participant EA are shown in this figure, but nearly identical patterns of results were found for both LO and AY.
a–c, The trial-by-trial population spike-timing of the putative afferents (left plots) precisely predicted the behaviour of each participant: LO (a), AY (b), and EA (c). When the population rate code (centre plots) was used as features, the model did not provide a precise fit to the behaviour (see Extended Data Table 2). Vibratory motifs (right plots) also predicted behaviour with high precision. The grey line in all plots represents the equality line between actual and predicted behaviour, not the actual regression line.
About this article
Cite this article
Miller, L.E., Montroni, L., Koun, E. et al. Sensing with tools extends somatosensory processing beyond the body. Nature 561, 239–242 (2018). https://doi.org/10.1038/s41586-018-0460-0
- Local Impact
- Wooden Rod
- Sensory Extensions
- Object Contact
- Sensory Embodiment
Predictive perception of self-generated movements: Commonalities and differences in the neural processing of tool and hand actions
Acta Psychologica (2020)
American Journal of Human Biology (2020)
Gulliver’s virtual travels: active embodiment in extreme body sizes for modulating our body representations
Cognitive Processing (2020)
Altered updating of bodily and spatial representations after tool-use in complex regional pain syndrome