On-line, voluntary control of human temporal lobe neurons

Journal name:
Nature
Volume:
467,
Pages:
1104–1108
Date published:
DOI:
doi:10.1038/nature09510
Received
Accepted
Published online

Daily life continually confronts us with an exuberance of external, sensory stimuli competing with a rich stream of internal deliberations, plans and ruminations. The brain must select one or more of these for further processing. How this competition is resolved across multiple sensory and cognitive regions is not known; nor is it clear how internal thoughts and attention regulate this competition1, 2, 3, 4. Recording from single neurons in patients implanted with intracranial electrodes for clinical reasons5, 6, 7, 8, 9, here we demonstrate that humans can regulate the activity of their neurons in the medial temporal lobe (MTL) to alter the outcome of the contest between external images and their internal representation. Subjects looked at a hybrid superposition of two images representing familiar individuals, landmarks, objects or animals and had to enhance one image at the expense of the other, competing one. Simultaneously, the spiking activity of their MTL neurons in different subregions and hemispheres was decoded in real time to control the content of the hybrid. Subjects reliably regulated, often on the first trial, the firing rate of their neurons, increasing the rate of some while simultaneously decreasing the rate of others. They did so by focusing onto one image, which gradually became clearer on the computer screen in front of their eyes, and thereby overriding sensory input. On the basis of the firing of these MTL neurons, the dynamics of the competition between visual images in the subject’s mind was visualized on an external display.

At a glance

Figures

  1. Experimental set-up.
    Figure 1: Experimental set-up.

    a, Continuous voltage traces are recorded by 64 microelectrodes from the subject’s medial temporal lobe. A four-dimensional vector, corresponding to the number of action potentials of four responsive units in the previous 100ms, is sent to a decoding algorithm determining the composition of the hybrid seen by the subject with a total delay of less than 100ms. b, The closest distance (weighted by the standard deviation) of this vector to the four clusters representing the four images is computed. If the ‘winning’ cluster represents the target or the distractor image, the visibility ratio of these two is adjusted accordingly.

  2. Task performance and neuronal spiking.
    Figure 2: Task performance and neuronal spiking.

    Two American actors, ‘Josh Brolin’ and ‘Marilyn Monroe’, constituted the preferred stimulus for two units. a, One multi-unit responded selectively to Monroe and was located in the left parahippocampal cortex. Below each illustration are the corresponding raster plots (twelve trials are ordered from top to bottom) and post-stimulus time histograms obtained during the control presentation. Vertical dashed lines indicate image onset (left) and offset (right), 1-s apart. Spike shapes are shown in blue, and the average spike shape in black. Below are the total number of spikes during the session. On the right is an illustration of the brain regions competing in these trials, and a fusion of the coronal CT and MRI scans taken after electrode implantation. Here, competing units were located in different hemispheres and regions. See Supplementary Video of the actual experiment. c, Time (running downwards for 10s) versus percentage visibility of eight trials in which the subject had to fade a 50%/50% hybrid image into a pure Monroe image. The subject was able to do so all eight times, even though these were her first trials ever. b, d, When Brolin was the target, she succeeded seven out of eight times. All subjects show similar trends of controlled fading (Fig. 3). The hybrid image was controlled in real time by the spiking of four units selective to the image of Brolin, Monroe, Michael Jackson or Venus Williams. e, f, Spiking activity of all four units for one successful Monroe (e) and Brolin (f) trial. The spike shapes and the four images each unit is selective to are shown on the right. Below are the images as seen by the subject during the trial at different times. For another example, see Supplementary Figs 4 and 7. For copyright reasons, some of the original images were replaced in this and all subsequent figures by very similar ones (same subject, similar pose, similar colour and so on). The image of Josh Brolin is copyright The Goonies, Warner Bros. Inc. RA, right amygdala; RH, right hippocampus; LH, left hippocampus; LP, left parahippocampal cortex.

  3. Successful fading.
    Figure 3: Successful fading.

    a, Percentage of trials in which subjects successfully controlled the activity of four units and faded to the target image within 10s. Yellow lines indicate chance performance—determined by bootstrapping 1,000 random trials for each subject (P<0.001; Wilcoxon rank-sum). The red bar is the performance averaged over all 12 subjects. Error bars show the standard deviation. b, Percentage of successful trials of the entire data set in which the competition between the two units was across hemispheres, within the same hemisphere but in different regions, or within the same region. Error bars show standard deviations. Note that in a, performance is analysed across subjects, whereas in b it is analysed across eight trial fading sessions; hence, the means differ.

  4. Voluntary control at the single unit level.
    Figure 4: Voluntary control at the single unit level.

    a, b, Normalized firing rates of the units in Fig. 2 as a function of visibility. We averaged the firing rates every 100ms for every level of visibility for all successful trials where the target either was the unit’s preferred (solid, black) or non-preferred stimulus (dashed, blue). Units fired significantly above baseline (grey dashed line) when the target was the preferred stimulus, and less than baseline when the target was the non-preferred stimulus. The TDC index is shown on the right. The shaded area reflects the bins used to calculate TDC. c, d, Averaging target and distractor trials across all subjects and all units for all successful fading trials reveals that the firing rate is significantly higher when the target is the preferred stimulus than in the competing situation, no matter what the visual input is. This is not true for failed trials (right). Red and dark grey vertical error bars are standard deviations. See Supplementary Fig. 8 for additional examples.

References

  1. Chalupa, L., Werner, J. & Barnstable, C. The Visual Neurosciences (MIT Press, 2004)
  2. Thorpe, S. Single units and sensation: still just as relevant today. Perception 38, 804807 (2009)
  3. Blake D. A. R., ed. Binocular Rivalry (MIT Press, 2005)
  4. Reynolds, J. & Chelazzi, L. Attentional modulation of visual processing. Annu. Rev. Neurosci. 27, 611648 (2004)
  5. Quian Quiroga, R. et al. Explicit encoding of multimodal percepts by single neurons in the human brain. Curr. Biol. 19, 1308–1313 (2009).
  6. Quian Quiroga, R. et al. Invariant visual representation by single neurons in the human brain. Nature 435, 11021107 (2005)
  7. Földiák, P. Neural coding: non-local but explicit and conceptual. Curr. Biol. 19, R904R906 (2009)
  8. Kreiman, G., Koch, C. & Fried, I. Imagery neurons in the human brain. Nature 408, 357361 (2000)
  9. Gelbard-Sagiv, H. et al. Internally generated reactivation of single neurons in human hippocampus during free recall. Science 322, 96101 (2008)
  10. Reddy, L., Kanwisher, N. & VanRullen, R. Attention and biased competition in multi-voxel object representations. Proc. Natl Acad. Sci. USA 106, 2144721452 (2009)
  11. Desimone, R. & Duncan, J. Neural mechanisms of selective visual attention. Annu. Rev. Neurosci. 18, 193222 (1995)
  12. Serences, J. et al. Control of object-based attention in human cortex. Cereb. Cortex 14, 13461357 (2004)
  13. Fried, I., MacDonald, K. & Wilson, C. Single neuron activity in human hippocampus and amygdala during recognition of faces and objects. Neuron 18, 753765 (1997)
  14. Quian Quiroga, R. et al. Decoding visual inputs from multiple neurons in the human temporal lobe. J. Neurophysiol. 98, 19972007 (2007)
  15. Musallam, S. et al. Cognitive control signals for neural prosthetics. Science 305, 258262 (2004)
  16. Wessberg, J. et al. Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408, 361365 (2000)
  17. Velliste, M. et al. Cortical control of a prosthetic arm for self-feeding. Nature 453, 10981101 (2008)
  18. Moritz, C., Perlmutter, S. & Fetz, E. Direct control of paralysed muscles by cortical neurons. Nature 456, 639642 (2008)
  19. Hochberg, L. et al. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164171 (2006)
  20. Kim, S. et al. Neural control of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia. J. Neural Eng. 5, 455476 (2008)
  21. Kennedy, P. et al. Direct control of a computer from the human central nervous system. IEEE Trans. Rehabil. Eng. 8, 198202 (2000)
  22. Guenther, F. et al. A wireless brain-machine interface for real-time speech synthesis. PLoS ONE 4, e8218 (2009)
  23. Waydo, S. et al. Sparse representation in the human medial temporal lobe. J. Neurosci. 26, 1023210234 (2006)

Download references

Author information

  1. These authors contributed equally to this work.

    • Christof Koch &
    • Itzhak Fried

Affiliations

  1. Computation and Neural Systems, California Institute of Technology, Pasadena, California 91125, USA

    • Moran Cerf,
    • Nikhil Thiruvengadam,
    • Florian Mormann,
    • Alexander Kraskov,
    • Rodrigo Quian Quiroga &
    • Christof Koch
  2. Department of Neurosurgery, University of California, Los Angeles, California 90095, USA

    • Moran Cerf &
    • Itzhak Fried
  3. Stern School of Business, New York University, New York, New York 10012, USA

    • Moran Cerf
  4. School of Computer Science, Carnegie Mellon University, Pittsburgh, Pennsylvania 15213, USA

    • Nikhil Thiruvengadam
  5. Department of Epileptology, University of Bonn, Bonn 53105, Germany

    • Florian Mormann
  6. Department of Engineering, University of Leicester, Leicester LE1 7RH, UK

    • Rodrigo Quian Quiroga
  7. Department of Brain and Cognitive Engineering, Korea University, Seoul, 136-713, Korea

    • Christof Koch
  8. Semel Institute for Neuroscience and Human Behavior, University of California, Los Angeles, California 90095, USA

    • Itzhak Fried
  9. Functional Neurosurgery Unit, Tel-Aviv Medical Center, Tel-Aviv 64239, Israel

    • Itzhak Fried
  10. Sackler Faculty of Medicine, Tel-Aviv University, Tel-Aviv 69978, Israel

    • Itzhak Fried

Contributions

M.C., F.M., R.Q.Q., C.K. and I.F. designed the experiment; M.C. performed the experiments; I.F. performed the surgeries; M.C. and N.T. analysed the data; M.C., C.K. and I.F. wrote the manuscript. All authors discussed the data and the analysis methods and contributed to the manuscript.

Competing financial interests

The authors declare no competing financial interests.

Corresponding authors

Correspondence to:

Author details

Supplementary information

PDF files

  1. Supplementary Information (6M)

    This file contains Supplementary Methods and Results, Supplementary Figures 1-9 with legends, legends for Supplementary Movie 1 and additional references.

Movies

  1. Supplementary Movie 1 (12.4M)

    An example of a feedback experiment, this movie has three parts. The first part shows the control presentation, part two shows a sequence of trials from the actual experiment and part three shows the 16 Monroe Brolin trials in the order they appeared in the experiment - see Supplementary Information file for full legend.

Additional data