The effect of face patch microstimulation on perception of faces and objects

Article metrics

Abstract

What is the range of stimuli encoded by face-selective regions of the brain? We asked how electrical microstimulation of face patches in macaque inferotemporal cortex affects perception of faces and objects. We found that microstimulation strongly distorted face percepts and that this effect depended on precise targeting to the center of face patches. While microstimulation had no effect on the percept of many non-face objects, it did affect the percept of some, including non-face objects whose shape is consistent with a face (for example, apples) as well as somewhat facelike abstract images (for example, cartoon houses). Microstimulation even perturbed the percept of certain objects that did not activate the stimulated face patch at all. Overall, these results indicate that representation of facial identity is localized to face patches, but activity in these patches can also affect perception of face-compatible non-face objects, including objects normally represented in other parts of inferotemporal cortex.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Figure 1: Experimental scheme.
Figure 2: Experiment 1: face patch stimulation exerts a large effect on perception of facial identity.
Figure 3: The dependence of effect magnitude on proximity to the center of a face patch.
Figure 4: Experiments 2 and 3: effect of stimulation outside the face patches on face perception and effect of stimulation inside face patch AM on perception of both general and round non-face objects.
Figure 5: Experiment 4a: effect of face patch stimulation on perception of abstract faces.
Figure 6: fMRI and electrophysiological responses to abstract Mooney stimuli.
Figure 7: Experiment 4b: effect of stimulation inside face patch AM on perception of non-face objects, part II.
Figure 8: Summary of results.

References

  1. 1

    Gross, C.G., Rocha-Miranda, C.E. & Bender, D.B. Visual properties of neurons in inferotemporal cortex of the macaque. J. Neurophysiol. 35, 96–111 (1972).

  2. 2

    Tsao, D.Y., Freiwald, W.A., Tootell, R.B.H. & Livingstone, M.S. A cortical region consisting entirely of face-selective cells. Science 311, 670–674 (2006).

  3. 3

    Freiwald, W.A. & Tsao, D.Y. Functional compartmentalization and viewpoint generalization within the macaque face-processing system. Science 330, 845–851 (2010).

  4. 4

    Issa, E.B. & DiCarlo, J.J. Precedence of the eye region in neural processing of faces. J. Neurosci. 32, 16666–16682 (2012).

  5. 5

    McMahon, D.B., Jones, A.P., Bondar, I.V. & Leopold, D.A. Face-selective neurons maintain consistent visual responses across months. Proc. Natl. Acad. Sci. USA 111, 8251–8256 (2014).

  6. 6

    Haxby, J.V. et al. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425–2430 (2001).

  7. 7

    Hasson, U., Levy, I., Behrmann, M., Hendler, T. & Malach, R. Eccentricity bias as an organizing principle for human high-order object areas. Neuron 34, 479–490 (2002).

  8. 8

    Konkle, T. & Oliva, A. A real-world size organization of object responses in occipitotemporal cortex. Neuron 74, 1114–1124 (2012).

  9. 9

    Wiggett, A.J., Pritchard, I.C. & Downing, P.E. Animate and inanimate objects in human visual cortex: evidence for task-independent category effects. Neuropsychologia 47, 3111–3117 (2009).

  10. 10

    Haushofer, J., Baker, C.I., Livingstone, M.S. & Kanwisher, N. Privileged coding of convex shapes in human object-selective cortex. J. Neurophysiol. 100, 753–762 (2008).

  11. 11

    Yue, X., Pourladian, I.S., Tootell, R.B. & Ungerleider, L.G. Curvature-processing network in macaque visual cortex. Proc. Natl. Acad. Sci. USA 111, E3467–E3475 (2014).

  12. 12

    Srihasam, K., Vincent, J.L. & Livingstone, M.S. Novel domain formation reveals proto-architecture in inferotemporal cortex. Nat. Neurosci. 17, 1776–1783 (2014).

  13. 13

    Verhoef, B.-E., Vogels, R. & Janssen, P. Contribution of inferior temporal and posterior parietal activity to three-dimensional shape perception. Curr. Biol. 20, 909–913 (2010).

  14. 14

    Salzman, C.D., Britten, K.H. & Newsome, W.T. Cortical microstimulation influences perceptual judgements of motion direction. Nature 346, 174–177 (1990).

  15. 15

    Histed, M.H., Ni, A.M. & Maunsell, J.H. Insights into cortical mechanisms of behavior from microstimulation experiments. Prog. Neurobiol. 103, 115–130 (2013).

  16. 16

    Afraz, S.-R., Kiani, R. & Esteky, H. Microstimulation of inferotemporal cortex influences face categorization. Nature 442, 692–695 (2006).

  17. 17

    Penfield, W. & Perot, P. The brain's record of auditory and visual experience. A final summary and discussion. Brain 86, 595–696 (1963).

  18. 18

    Allison, T. et al. Face recognition in human extrastriate cortex. J. Neurophysiol. 71, 821–825 (1994).

  19. 19

    Puce, A., Allison, T. & McCarthy, G. Electrophysiological studies of human face perception. III: Effects of top-down processing on face-specific potentials. Cereb. Cortex 9, 445–458 (1999).

  20. 20

    Parvizi, J. et al. Electrical stimulation of human fusiform face-selective regions distorts face perception. J. Neurosci. 32, 14915–14920 (2012).

  21. 21

    Afraz, A., Boyden, E.S. & DiCarlo, J.J. Optogenetic and pharmacological suppression of spatial clusters of face neurons reveal their causal role in face gender discrimination. Proc. Natl. Acad. Sci. USA 112, 6730–6735 (2015).

  22. 22

    Fisher, R.A. On the interpretation of χ2 from contingency tables, and the calculation of P. J.R. Stat. Soc. 85, 87–94 (1922).

  23. 23

    Grimaldi, P., Saleem, K.S. & Tsao, D. Anatomical connections of the functionally defined “face patches” in the macaque monkey. Neuron 90, 1325–1342 (2016).

  24. 24

    Freiwald, W.A., Tsao, D.Y. & Livingstone, M.S. A face feature space in the macaque temporal lobe. Nat. Neurosci. 12, 1187–1196 (2009).

  25. 25

    Meyers, E.M., Borzello, M., Freiwald, W.A. & Tsao, D. Intelligent information loss: the coding of facial identity, head pose, and non-face information in the macaque face patch system. J. Neurosci. 35, 7069–7081 (2015).

  26. 26

    Mooney, C.M. Age in the development of closure ability in children. Can. J. Psychol. 11, 219–226 (1957).

  27. 27

    Taubert, J., Van Belle, G., Vanduffel, W., Rossion, B. & Vogels, R. Neural correlate of the thatcher face illusion in a monkey face-selective patch. J. Neurosci. 35, 9872–9878 (2015).

  28. 28

    Jonas, J. et al. Focal electrical intracerebral stimulation of a face-sensitive area causes transient prosopagnosia. Neuroscience 222, 281–288 (2012).

  29. 29

    Moeller, S., Freiwald, W.A. & Tsao, D.Y. Patches with links: a unified system for processing faces in the macaque temporal lobe. Science 320, 1355–1359 (2008).

  30. 30

    Knoll, M. & Kugler, J. Subjective light pattern spectroscopy in the encephalographic frequency range. Nature 184 (Suppl. 23), 1823–1824 (1959).

  31. 31

    Brindley, G.S. & Lewin, W.S. The sensations produced by electrical stimulation of the visual cortex. J. Physiol. (Lond.) 196, 479–493 (1968).

  32. 32

    Vignal, J.P., Chauvel, P. & Halgren, E. Localised face processing by the human prefrontal cortex: stimulation-evoked hallucinations of faces. Cogn. Neuropsychol. 17, 281–291 (2000).

  33. 33

    Fuster, J.M. & Jervey, J.P. Neuronal firing in the inferotemporal cortex of the monkey in a visual memory task. J. Neurosci. 2, 361–375 (1982).

  34. 34

    Baylis, G.C. & Rolls, E.T. Responses of neurons in the inferior temporal cortex in short term and serial recognition memory tasks. Exp. Brain Res. 65, 614–622 (1987).

  35. 35

    Vogels, R. & Orban, G.A. Activity of inferior temporal neurons during orientation discrimination with successively presented gratings. J. Neurophysiol. 71, 1428–1451 (1994).

  36. 36

    Mikami, A. & Kubota, K. Inferotemporal neuron activities and color discrimination with delay. Brain Res. 182, 65–78 (1980).

  37. 37

    Miller, E.K., Li, L. & Desimone, R. Activity of neurons in anterior inferior temporal cortex during a short-term memory task. J. Neurosci. 13, 1460–1478 (1993).

  38. 38

    Vogels, B. & Orban, G.A. Does practice in orientation discrimination lead to changes in the response properties of macaque inferior temporal neurons? Eur. J. Neurosci. 6, 1680–1690 (1994).

  39. 39

    Chelazzi, L., Miller, E.K., Duncan, J. & Desimone, R. A neural basis for visual search in inferior temporal cortex. Nature 363, 345–347 (1993).

  40. 40

    Ohayon, S., Freiwald, W.A. & Tsao, D.Y. What makes a cell face selective? The importance of contrast. Neuron 74, 567–581 (2012).

  41. 41

    Ohayon, S. & Tsao, D.Y. MR-guided stereotactic navigation. J. Neurosci. Methods 204, 389–397 (2012).

  42. 42

    Geusebroek, J.M., Burghouts, G.J. & Smeulders, A.W. The Amsterdam Library of Object Images. Int. J. Computer Vision 61, 103–112 (2005).

  43. 43

    Stanislaw, H. & Todorov, N. Calculation of signal detection theory measures. Behav. Res. Methods Instrum. Comput. 31, 137–149 (1999).

Download references

Acknowledgements

We thank N. Schweers for technical support, as well as N. Kanwisher, S. Kornblith, M. Livingstone, members of the Tsao lab and the three anonymous reviewers for critical comments. This work was supported by the Howard Hughes Medical Institute, the National Institutes of Health (1R01EY019702) and the Humboldt Foundation (to S.M.).

Author information

S.M. and D.Y.T. designed the experiments, interpreted the data and wrote the paper. S.M. and D.Y.T. conducted all experiments except those shown in Figure 6, which were done by T.C. and L.C., and Supplementary Figure 8, which was done by L.C. S.M. analyzed the data.

Correspondence to Doris Y Tsao.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Integrated supplementary information

Supplementary Figure 1 Mean change in performance caused by microstimulation of each face patch.

Dark gray bars for same identity trials, light gray bars for different identity trials, error bars show the standard error of the mean. The results of individual sessions are shown as circles, filled circles show session in which the effect size was significant with p <= 0.01. As shown in Figure 3, effect size per session correlates with the face selectivity of each stimulation site, explaining the sessions with smaller effect sizes.

Supplementary Figure 2 The dependence of effect magnitude for faces on stimulation current strength.

(a, b) Behavioral performance on same- and different-identity trials, with and without microstimulation, for stimulation strength magnitude of 100 μA (a) and 300 μA (b), with stimulation performed in the same session (AM, M1). (c, d) Another session, comparing effect on behavior of face patch stimulation at 100 μA and 300 μA (AM, M1). (e, f) Effect of face patch stimulation at 100 μA and 200 μA during the same session (AM, M2). (g, h) Effect of face patch stimulation at 100 μA and 200 μA during the same session (AM, M2). (i, j, k) Effect of face patch stimulation at 100 μA, 200 μA, and 300 μA during the same session (ML, M2). Gray bars denote trials without, red bars trials with electrical micro-stimulation; Darker bars show the performance for same identity trials, lighter bars for different identity trials. *: P < 0.05; **: P < 0.01; ***: P < 0.005; Fisher’s exact test (see Supplementary Table 3 for exact P-values). The lower plot in each panel shows the change in percentage points caused by electrical stimulation for same identity trials in dark gray and for the different identity trials in light gray.

Supplementary Figure 3 The dependence of effect magnitude for round objects on stimulation current strength and electrode position, for subject M1 for three different sessions.

(a) Low stimulation current (50 μA) inside face patch AM. (b) Intermediate stimulation current (100 μA) inside face patch AM. (c) Large stimulation current (200 μA) outside the face patches. Top row: Behavioral performance on same and different trials, with and without microstimulation, for stimulation in face patch AM of monkey M1, for faces, apples, citrus fruits, teapots, and clocks. 2nd row: Size of stimulation effect in percentage points for the same five categories. 3rd row: Behavioral performance on same and different identity trials of the face identification task of the same session. 4th row: Size of stimulation effect for the face identification task. Conventions as in Suppl. Fig. 2. *: P < 0.05; **: P < 0.01; ***: P < 0.005; Fisher’s exact test (see Supplementary Table 3 for exact P-values).

Supplementary Figure 4 The dependence of effect magnitude on proximity of the stimulation site to the center of the face patch, for the five different object categories from Experiment 3.

Same conventions as in Figure 3a. Pooled data for 14 sessions (M1: 9, M2: 5) showing how the magnitude of electrical microstimulation effect on same (left column) and different identity trials (right column) correlates with the face selectivity of the target location as measured by fMRI. Out of the five object categories (faces (a), apples (b), citrus fruit (c), pots (d) and clocks (e)), only for faces was there a significant correlation (same identity trials: p: 0.044706, correlation coefficient r = -0.54319, r2 = 0.2951; different identity trials: p: 0.0053008, correlation coefficient r = 0.70014, r2 = 0.4902).

Supplementary Figure 5 Experiment 2: effect of stimulation inside face patch AM on the perception of nonround objects.

Conventions as in Fig. 2. (a) Effect of AM stimulation on non-round object perception in monkey M1. Performance was significantly worse on microstimulation “same” trials (reduction by 7.77 percentage points). (c) Performance change in percentage points caused by electrical microstimulation for same- (dark gray bar) and different-identity trials (light gray bar) (b) Same as (a) and (d) same as (c), for stimulation in monkey M2. Stimulation current was 300 μA. *: P < 0.05; Fisher’s exact test (see Supplementary Table 3 for exact P-values).

Supplementary Figure 6 Experiment 4b: effect of stimulation inside face patch AM on perception of non-face objects II in M2.

(a) Effect of face patch stimulation on perception of house line drawings, house cartoons, house silhouettes, Mooney faces, and upside down Mooney faces. (b) Effect obtained in same experimental session, for face stimuli of Experiment 1. (c) Performance change in percentage points caused by electrical microstimulation for same- (dark gray bar) and different-identity trials (light gray bar) for each of the five categories of non-face objects II, and (d) for the faces of Experiment 1. Stimulation current was 200 μA. Conventions as in Fig. 4. *: P < 0.05; **: P < 0.01; ***: P < 0.005; Fisher’s exact test (see Supplementary Table 3 for exact P-values).

Supplementary Figure 7 Electrode position for M1 showing that, for the data presented in Figure 7, the electrode tip was located inside face patch AM.

Scale bar (top left) = 1 cm.

Supplementary Figure 8 Activation to faces, cartoon houses, real houses and real objects.

(a) Coronal slices showing fMRI activation to the contrast faces > objects. Face patches are indicated by arrows. (b, c) Slices from the same animal as (a), showing activation to the contrast cartoon houses > objects (b), and real houses > objects (c). (d) Flat maps of the left and right hemisphere visual cortex, showing the same data as (a-c). Face patches are indicated by green outlines. Notice overlap between face patches ML and PL and cartoon house activation (middle two panels), which is absent for real house activation (right two panels). Face patches indicated by green outlines in middle and right panels. (e) Beta values for faces, cartoon houses, real houses, and real objects, from PL/ML, showing strong activation to faces and to cartoon houses. Box-and-whisker plot indicates the median value (red line), the 25–75th percentiles (box) and +/- 2.7 sigma (whiskers).

Supplementary Figure 9 Experiment 5: dependence of effect magnitude on stimulation timing.

Behavioral performance for same- and different-identity trials, across four different stimulation conditions: no stimulation (gray bars), stimulation during cue 1 and cue 2 (yellow bars), stimulation during cue 1 (green bars), stimulation during cue 2 (red bars). Plots on the right show the change in percentage points caused by electrical stimulation (electrical stimulation during cue1: green; during cue2: red; during both cues: yellow). Darker bars denote same-identity trials, lighter bars different-identity trials. *: P < 0.05; **: P < 0.01; ***: P < 0.005; Fisher’s exact test (see Supplementary Table 3 for exact P-values). (a): ML, M1, (b): AM M1, (c): AM M2. (d): AM M1: Stimulation experiment in M1, with stimulation trains of 50 ms duration (images were presented for 200 ms each), instead of 200 ms (b). All trains were delayed by 75 ms relative to visual stimulation to account for the typical response latency of anterior temporal cortex.

Supplementary Figure 10 Analysis of data in Supplementary Figure 9, separating data for each of 32 different cue 2 identities.

(a, b) Analysis of data in panels (a), (b) of Supplementary Figure 9, respectively. Numbers below each bar indicate number of contributing trials, *: P < 0.05; **: P < 0.01; ***: P < 0.005; Fisher’s exact test (see Supplementary Table 5 for exact P-values). A 3-way ANOVA examining main effects and interactions of identity, trial type (same/different) and microstimulation (yes/no) showed significant main effects for trial (same/different) and stimulation condition, but not for identity, and a significant interaction between trial and stimulation. The results do not support the idea that effect of stimulation depended on the specific identity of the face: this would show up as a two-way interaction between stimulation and identity or three-way interaction between stimulation, trial type, and identity. M1 (a), trial type F(1, 92) = 401.58, p < 0.0001; stimulation F(3, 92) = 110.74, p < 0.0001; identity F(31, 92) = 1.51, p = 0.0666; trial*stimulation F(3, 92) = 39.75, p < 0.0001; trial*identity F(31, 92) = 1.51, p = 0.0665, stimulation*identity F(93, 92) = 0.94, p = 0.6137; M2 (b), trial type F(1, 92) = 150.51, p < 0.0001; stimulation F(3, 92) = 57.88, p < 0.0001; identity F(31, 92) = 1.15, p = 0.2937; trial*stimulation F(3, 92) = 10.22, p < 0.0001; trial*identity F(31, 92) = 1.16, p = 0.2846, stimulation*identity F(93, 92) = 0.83, p = 0.8079.

Supplementary Figure 11 Behavioral performance from training sessions and from experimental sessions from nonmicrostimulated trials.

This figure shows the task performance over the course of multiple sessions, always starting with the first session a stimulus set was introduced. Performance was averaged over same and different identity trials. (a) M1: The red line shows the initial task that consisted of performing same/different discrimination between five images of grape clusters and five images of single tomatoes; for the same condition we always presented the exact same image, for different we always selected one image from the five exemplars of the other category. The green line shows the initial performance on the initial face identity discrimination task. In this task we presented color photographs of 5 persons at 5 different views each, for the same condition we only selected the 2nd cue image from the 4 images of each identity not used as 1st cue, for the different condition we randomly selected one of the 20 images belonging to the remaining identities. Note how the performance for grapes and tomatoes ramps up over 4 sessions and how well the animal transferred to the face identification task (the animal had discriminated different face stimuli in past experiments). (b) Example stimuli. Left: all grape and tomato images used in initial training. Right: three exemplary identities at five views. (c) M1 and (d) M2 performance in >40 sessions with the 32 identities at 6 different facial expressions task (see Experiment 1). Note with M2 there was an initial ramping up from near chance performance, but both animals generalized quickly from the initially trained face set to the new set. (e & f) Performance in the different versions of the object identification task (as described in Experiment 2); both M1 and M2 immediately generalized to the different object sets. Note that we only added more stimuli, so the 16 objects set was a subset of both the 19 and 28 objects sets. (g & h) Performance for Experiment 3: both animals immediately generalized to the five new categories. Note that on the 8th session, M1 changed his strategy to guessing only to return to “normal” performance on the next session. (i & j) Performance on the abstract face and house stimuli from Experiment 4b; M1 immediately generalized to the new stimuli, while M2’s performance improved over the first few sessions before stabilizing.

Supplementary Figure 12 Neuronal response selectivity at the different stimulation sites, across animals.

The upper part of each panel shows the normalized mean response over 200 ms for each unit to each of the 96 consensus images, while the middle part shows the population mean response. Note that most units were recorded for the minimal time to see whether there was a difference between responses to face and non-face categories, so often, images were not repeated enough times to allow the normalization to work; such occurrences are colored black in the upper panels and ignored for the averaging shown in the middle panels. The lower part of each panel shows the distribution of the face selectivity indices for all units per patch and animal, distance from the midpoint at zero correlates with face selectivity. The category of each image is color-coded by the rainbow bar. The order of the presented population data mainly follows Fig. 2 to facilitate direct comparison: (a) AM of monkey M1 (b) AM of M2, (c) ML of M1, (d) ML of M2, (e) AL of M1, (f) MF of M2, (g) AF of M1 (h) control site outside of the face patches in M1.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–12 and Supplementary Tables 1–4 (PDF 4116 kb)

Supplementary Methods Checklist (PDF 472 kb)

Supplementary Table 5

Selected statistics for Supplementary Figure 10. (XLSX 20 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Moeller, S., Crapse, T., Chang, L. et al. The effect of face patch microstimulation on perception of faces and objects. Nat Neurosci 20, 743–752 (2017) doi:10.1038/nn.4527

Download citation

Further reading