Focus on neural computation and theory

Efficient codes and balanced networks

Journal name:
Nature Neuroscience
Volume:
19,
Pages:
375–382
Year published:
DOI:
doi:10.1038/nn.4243
Received
Accepted
Published online

Abstract

Recent years have seen a growing interest in inhibitory interneurons and their circuits. A striking property of cortical inhibition is how tightly it balances excitation. Inhibitory currents not only match excitatory currents on average, but track them on a millisecond time scale, whether they are caused by external stimuli or spontaneous fluctuations. We review, together with experimental evidence, recent theoretical approaches that investigate the advantages of such tight balance for coding and computation. These studies suggest a possible revision of the dominant view that neurons represent information with firing rates corrupted by Poisson noise. Instead, tight excitatory/inhibitory balance may be a signature of a highly cooperative code, orders of magnitude more precise than a Poisson rate code. Moreover, tight balance may provide a template that allows cortical neurons to construct high-dimensional population codes and learn complex functions of their inputs.

At a glance

Figures

  1. The conundrum of Poisson rate codes and E/I balance.
    Figure 1: The conundrum of Poisson rate codes and E/I balance.

    (a) When seeking to produce an average rate of 4 spikes per second, the number of spikes in each trial will fluctuate for a Poisson rate code (left), yet remain constant for a perfect, regular rate code (right). (b) As the average rate or number of spikes per second needed grows, the variability of the Poisson rate code (s.d.; black bars) grows as well. (c) When encoding an analog number, any rate code will cause coding errors due to the unavoidable discretization of the spike count. While these coding errors drop as 1/M for M spikes (green), the Poisson rate coding errors drop only as owing to the additional problems caused by unreliability (black). (d) In a neuron receiving both excitatory and inhibitory currents, even a small imbalance toward excitation will cause a spike train output that is far more regular than a Poisson spike train. (e) If excitatory and inhibitory currents are balanced on a slow time scale yet exhibit fast and uncorrelated (loose) fluctuations, then the net input current will depend on these faster fluctuations only, causing a random walk to threshold in the membrane potentials. (f) If the faster fluctuations are strongly correlated but shifted in time—for example, by inhibition trailing excitation (see inset)—then the system moves into a regime of tight E/I balance.

  2. Schematic illustration of key experimental findings.
    Figure 2: Schematic illustration of key experimental findings.

    (a) Co-tuning of excitatory (red) and inhibitory (blue) inputs into sensory neurons. Black represents the tuning curve of the neuron. We distinguish two types of observations. In one set of observations (left), excitatory and inhibitory inputs are balanced for all conditions or stimuli; firing occurs essentially because of temporal mismatch between excitation and inhibition (see b). In another set of observations (right), they are balanced only at the preferred stimuli of a neuron (right), and otherwise inhibition dominates, causing a sharpening of the tuning curve. (b) During stimulus presentations, inhibitory inputs match excitatory inputs in size and trail them by a few milliseconds only. (c,d) During spontaneous activity, inhibition still matches and trails excitation by a few milliseconds. Shown are the time course of inhibitory and excitatory currents and a magnification in time. (c) Switching between UP and DOWN states. (d) Gamma oscillations in the hippocampus. (e) Membrane and spike train correlations between nearby or similar pairs of neurons. Neurons with similar inputs usually exhibit strong co-fluctuations in the membrane voltage and much weaker or nonexistent correlations in their spike times (right). Panel e reproduced from ref. 54, Elsevier.

  3. Coding with tightly balanced networks.
    Figure 3: Coding with tightly balanced networks.

    (a) Using balance for coding. Here the network receives an excitatory input signal (red), which for technical reasons combines x(t) and its derivative (refs. 16, 17, 18). The excitatory input signal in turn is balanced by an inhibitory feedback loop (blue). The feedback loop subtracts the network's representation of the input from the actual input, so that only the momentary representation or decoding error is fed into the network. In the tightly balanced regime, the inhibitory loop cancels the excitatory input, and therefore . (b) This schema can be implemented in an architecture of excitatory (red) and inhibitory (blue) neurons. Here the signal x(t) is fed into the dendritic tree of the excitatory neurons. The spiking output of these neurons represents the readout , which is fed back through an inhibitory loop to cancel the input signals. (c) Toy example with four identical neurons. Shown are the input signal x(t), its estimate , and the voltages Vi, and spikes (black dots) of the four neurons. The neuron's voltages track the error . The neurons take turns firing whenever this error exceeds a threshold. The fast mutual inhibition prevents more than one neuron from firing in each integration cycle. In turn, the input signal is well approximated by the population of four neurons. (d) Error space plot. In larger networks, the voltages of the neurons track projections (small green dots) of the reconstruction error (large green dot). Every neuron (index i) represents one specific direction in this error space, given by its feedforward weights Di. Whenever the error becomes too large in this direction, the neuron's voltage Vi exceeds its threshold Ti and the neuron spikes, thereby updating the readout and minimizing the reconstruction error . (e) In larger networks—that is, ones with hundreds or thousands of neurons that represent many variables—the voltages and spike trains no longer look as cartoonish as in c. Shown here are the membrane potentials of two neurons with similar tuning. The membrane potentials are strongly correlated, while the spike trains are only weakly correlated.

  4. Computations with tightly balanced networks.
    Figure 4: Computations with tightly balanced networks.

    (a) Using balance for computation. Here the network balances an excitatory input signal c(t) and excitatory recurrent feedback with an inhibitory feedback loop (compare Fig. 3a). In the tightly balanced regime, the inhibitory and excitatory inputs cancel, and therefore , so that the network implements a particular differential equation. (b) This schema can likewise be implemented in a network of excitatory and inhibitory neurons. Just as in Figure 3b, some of the inhibitory neurons will mediate fast inhibition to construct the efficient population code. In addition to those found in Figure 3b, a second set of recurrent connections, which may include additional inhibitory or excitatory neurons (not shown), provides slower feedback and allows the system to perform computations. (c) A neural integrator as a specific implementation of this type of architecture. Here the balanced condition dictates that the recurrent input, –dx/dt, cancels the feedforward input c. The resulting computation, dx/dt = c(t), implements an integrator. Negative derivative feedback is implemented through a fast inhibitory and a slower excitatory loop. (d) Implementation of a harmonic oscillator in a network of N = 100 neurons, using the network architecture in b, adapted from ref. 16. Shown are the self-generated oscillations of the readout variables [] and []. The inset highlights the discrete nature of the representation. (e) Spike raster of all 100 neurons in the network during a single oscillation phase. (f) Trial-averaged, time-varying firing rates of selected neurons.

References

  1. Adrian, E.D. & Zotterman, Y. The impulses produced by sensory nerve-endings: Part II. The response of a single end-organ. J. Physiol. (Lond.) 61, 151171 (1926).
  2. Softky, W.R. & Koch, C. The highly irregular firing of cortical cells is inconsistent with temporal integration of random EPSPs. J. Neurosci. 13, 334350 (1993).
  3. Perkel, D.H. & Bullock, T.H. Neural coding. Neurosci. Res. Program Bull. 6, 220349 (1968).
  4. Abeles, M. Role of the cortical neuron: integrator or coincidence detector? Isr. J. Med. Sci. 18, 8392 (1982).
  5. Hopfield, J.J. Pattern recognition computation using action potential timing for stimulus representation. Nature 376, 3336 (1995).
  6. Rieke, F., Warland, D., de Ruyter van Steveninck, R. & Bialek, W. Spikes: Exploring the Neural Code (MIT Press, 1997).
  7. Fiser, J., Berkes, P., Orbán, G. & Lengyel, M. Statistically optimal perception and learning: from behavior to neural representations. Trends Cogn. Sci. 14, 119130 (2010).
  8. Shadlen, M.N. & Newsome, W.T. Noise, neural codes and cortical organization. Curr. Opin. Neurobiol. 4, 569579 (1994).
  9. van Vreeswijk, C. & Sompolinsky, H. Chaos in neuronal networks with balanced excitatory and inhibitory activity. Science 274, 17241726 (1996).
  10. Amit, D.J. & Brunel, N. Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex. Cereb. Cortex 7, 237252 (1997).
  11. Shadlen, M.N. & Newsome, W.T. The variable discharge of cortical neurons: implications for connectivity, computation, and information coding. J. Neurosci. 18, 38703896 (1998).
  12. Brunel, N. Dynamics of sparsely connected networks of excitatory and inhibitory spiking neurons. J. Comput. Neurosci. 8, 183208 (2000).
  13. Shu, Y., Hasenstaub, A. & McCormick, D.A. Turning on and off recurrent balanced cortical activity. Nature 423, 288293 (2003).
  14. Wehr, M. & Zador, A.M. Balanced inhibition underlies tuning and sharpens spike timing in auditory cortex. Nature 426, 442446 (2003).
  15. Isaacson, J.S. & Scanziani, M. How inhibition shapes cortical activity. Neuron 72, 231243 (2011).
  16. Boerlin, M., Machens, C.K. & Denève, S. Predictive coding of dynamical variables in balanced spiking networks. PLoS Comput. Biol. 9, e1003258 (2013).
  17. Boerlin, M. & Denève, S. Spike-based population coding and working memory. PLoS Comput. Biol. 7, e1001080 (2011).
  18. Bourdoukan, R., Barrett, D., Deneve, S. & Machens, C.K. Learning optimal spike-based representations. In Advances in Neural Information Processing Systems (eds. Pereira, F., Burges, C.J.C., Bottou, L. & Weinberger, K.Q.) 22852293 (MIT Press, 2012).
  19. Barrett, D.G., Deneve, S. & Machens, C.K. Firing rate predictions in optimal balanced networks. In Advances in Neural Information Processing Systems (eds. Pereira, F., Burges, C.J.C., Bottou, L. & Weinberger, K.Q.) 15381546 (MIT Press, 2013).
  20. Thalmeier, D., Uhlmann, M., Kappen, H.J. & Memmesheimer, R.-M. Learning universal computations with spikes. Preprint at http://arxiv.org/abs/1505.07866 (2015).
  21. Moreno-Bote, R. & Drugowitsch, J. Causal inference and explaining away in a spiking network. Sci. Rep. 5, 17531 (2015).
  22. van Vreeswijk, C. & Sompolinsky, H. Chaotic balanced state in a model of cortical circuits. Neural Comput. 10, 13211371 (1998).
  23. Monteforte, M. & Wolf, F. Dynamical entropy production in spiking neuron networks in the balanced state. Phys. Rev. Lett. 105, 268104 (2010).
  24. London, M., Roth, A., Beeren, L., Häusser, M. & Latham, P.E. Sensitivity to perturbations in vivo implies high noise and suggests rate coding in cortex. Nature 466, 123127 (2010).
  25. Monier, C., Fournier, J. & Frégnac, Y. In vitro and in vivo measures of evoked excitatory and inhibitory conductance dynamics in sensory cortices. J. Neurosci. Methods 169, 323365 (2008).
  26. Haider, B., Duque, A., Hasenstaub, A.R. & McCormick, D.A. Neocortical network activity in vivo is generated through a dynamic balance of excitation and inhibition. J. Neurosci. 26, 45354545 (2006).
  27. Xue, M., Atallah, B.V. & Scanziani, M. Equalizing excitation-inhibition ratios across visual cortical neurons. Nature 511, 596600 (2014).
  28. Destexhe, A., Rudolph, M. & Paré, D. The high-conductance state of neocortical neurons in vivo. Nat. Rev. Neurosci. 4, 739751 (2003).
  29. Anderson, J.S., Carandini, M. & Ferster, D. Orientation tuning of input conductance, excitation, and inhibition in cat primary visual cortex. J. Neurophysiol. 84, 909926 (2000).
  30. Martinez, L.M., Alonso, J.M., Reid, R.C. & Hirsch, J.A. Laminar processing of stimulus orientation in cat visual cortex. J. Physiol. (Lond.) 540, 321333 (2002).
  31. Tan, A.Y., Zhang, L.I., Merzenich, M.M. & Schreiner, C.E. Tone-evoked excitatory and inhibitory synaptic conductances of primary auditory cortex neurons. J. Neurophysiol. 92, 630643 (2004).
  32. Tan, A.Y., Brown, B.D., Scholl, B., Mohanty, D. & Priebe, N.J. Orientation selectivity of synaptic input to neurons in mouse and cat primary visual cortex. J. Neurosci. 31, 1233912350 (2011).
  33. Tan, A.Y. & Wehr, M. Balanced tone-evoked synaptic excitation and inhibition in mouse auditory cortex. Neuroscience 163, 13021315 (2009).
  34. Wilent, W.B. & Contreras, D. Dynamics of excitation and inhibition underlying stimulus selectivity in rat somatosensory cortex. Nat. Neurosci. 8, 13641370 (2005).
  35. Liu, B.H. et al. Broad inhibition sharpens orientation selectivity by expanding input dynamic range in mouse simple cells. Neuron 71, 542554 (2011).
  36. Wu, G.K., Arbuckle, R., Liu, B.H., Tao, H.W. & Zhang, L.I. Lateral sharpening of cortical frequency tuning by approximately balanced inhibition. Neuron 58, 132143 (2008).
  37. Cardin, J.A., Palmer, L.A. & Contreras, D. Stimulus feature selectivity in excitatory and inhibitory neurons in primary visual cortex. J. Neurosci. 27, 1033310344 (2007).
  38. Runyan, C.A. et al. Response features of parvalbumin-expressing interneurons suggest precise roles for subtypes of inhibition in visual cortex. Neuron 67, 847857 (2010).
  39. Liu, B.H. et al. Visual receptive field structure of cortical inhibitory neurons revealed by two-photon imaging guided recording. J. Neurosci. 29, 1052010532 (2009).
  40. Kerlin, A.M., Andermann, M.L., Berezovskii, V.K. & Reid, R.C. Broadly tuned response properties of diverse inhibitory neuron subtypes in mouse visual cortex. Neuron 67, 858871 (2010).
  41. Niell, C.M. & Stryker, M.P. Modulation of visual responses by behavioral state in mouse visual cortex. Neuron 65, 472479 (2010).
  42. Poo, C. & Isaacson, J.S. Odor representations in olfactory cortex: “sparse” coding, global inhibition, and oscillations. Neuron 62, 850861 (2009).
  43. D'amour, J.A. & Froemke, R.C. Inhibitory and excitatory spike-timing-dependent plasticity in the auditory cortex. Neuron 86, 514528 (2015).
  44. Dorrn, A.L., Yuan, K., Barker, A.J., Schreiner, C.E. & Froemke, R.C. Developmental sensory experience balances cortical excitation and inhibition. Nature 465, 932936 (2010).
  45. Froemke, R.C., Merzenich, M.M. & Schreiner, C.E. A synaptic memory trace for cortical receptive field plasticity. Nature 450, 425429 (2007).
  46. Marlin, B.J., Mitre, M., D'amour, J.A., Chao, M.V. & Froemke, R.C. Oxytocin enables maternal behaviour by balancing cortical inhibition. Nature 520, 499504 (2015).
  47. Froemke, R.C. et al. Long-term modification of cortical synapses improves sensory perception. Nat. Neurosci. 16, 7988 (2013).
  48. Graupner, M. & Reyes, A.D. Synaptic input correlations leading to membrane potential decorrelation of spontaneous activity in cortex. J. Neurosci. 33, 1507515085 (2013).
  49. Okun, M. & Lampl, I. Instantaneous correlation of excitation and inhibition during ongoing and sensory-evoked activities. Nat. Neurosci. 11, 535537 (2008).
  50. Tan, A.Y.Y., Andoni, S. & Priebe, N.J. A spontaneous state of weakly correlated synaptic excitation and inhibition in visual cortex. Neuroscience 247, 364375 (2013).
  51. Salkoff, D.B., Zagha, E., Yüzgeç, Ö. & McCormick, D.A. Synaptic mechanisms of tight spike synchrony at gamma frequency in cerebral cortex. J. Neurosci. 35, 1023610251 (2015).
  52. Atallah, B.V. & Scanziani, M. Instantaneous modulation of gamma oscillation frequency by balancing excitation with inhibition. Neuron 62, 566577 (2009).
  53. Cohen, M.R. & Kohn, A. Measuring and interpreting neuronal correlations. Nat. Neurosci. 14, 811819 (2011).
  54. Yu, J. & Ferster, D. Membrane potential synchrony in primary visual cortex during sensory stimulation. Neuron 68, 11871201 (2010).
  55. Gentet, L.J., Avermann, M., Matyas, F., Staiger, J.F. & Petersen, C.C. Membrane potential dynamics of GABAergic neurons in the barrel cortex of behaving mice. Neuron 65, 422435 (2010).
  56. Renart, A. et al. The asynchronous state in cortical circuits. Science 327, 587590 (2010).
  57. Vogels, T.P., Sprekeler, H., Zenke, F., Clopath, C. & Gerstner, W. Inhibitory plasticity balances excitation and inhibition in sensory pathways and memory networks. Science 334, 15691573 (2011).
  58. Barrett, D.G.T., Deneve, S. & Machens, C.K. Optimal compensation for neuron death. Preprint at http://biorxiv.org/content/early/2015/10/20/029512 (2015).
  59. Schwemmer, M.A., Fairhall, A.L., Denéve, S. & Shea-Brown, E.T. Constructing precisely computing networks with biophysical spiking neurons. J. Neurosci. 35, 1011210134 (2015).
  60. Chalk, M., Gutkin, B. & Deneve, S. Neural oscillations as a signature of efficient coding in the presence of synaptic delays. Preprint at doi:10.1101/034736 (2015).
  61. Dayan, P. & Abbott, L.F. Theoretical Neuroscience vol. 806 (MIT Press, 2001).
  62. Jaeger, H. The echo state approach to analysing and training recurrent neural networks–with an erratum note. GMD Technical Report 148:34 (German National Research Center for Information Technology, Bonn, 2001).
  63. Maass, W., Natschläger, T. & Markram, H. Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput. 14, 25312560 (2002).
  64. Compte, A., Brunel, N., Goldman-Rakic, P.S. & Wang, X.J. Synaptic mechanisms and network dynamics underlying spatial working memory in a cortical network model. Cereb. Cortex 10, 910923 (2000).
  65. Hansel, D. & Mato, G. Short-term plasticity explains irregular persistent activity in working memory tasks. J. Neurosci. 33, 133149 (2013).
  66. Murphy, B.K. & Miller, K.D. Balanced amplification: a new mechanism of selective amplification of neural activity patterns. Neuron 61, 635648 (2009).
  67. Hennequin, G., Vogels, T.P. & Gerstner, W. Optimal control of transient dynamics in balanced networks supports generation of complex movements. Neuron 82, 13941406 (2014).
  68. Churchland, M.M. et al. Neural population dynamics during reaching. Nature 487, 5156 (2012).
  69. Bourdoukan, R. & Deneve, S. Enforcing balance allows local supervised learning in spiking recurrent networks. In Advances in Neural Information Processing Systems (in the press).
  70. Abbott, L.F., DePasquale, B. & Memmesheimer, R.-M. Building functional networks of spiking model. Nat. Neurosci. 19, 350355 (2016).
  71. Poirazi, P., Brannon, T. & Mel, B.W. Pyramidal neuron as two-layer neural network. Neuron 37, 989999 (2003).
  72. Lim, S. & Goldman, M.S. Balanced cortical microcircuitry for spatial working memory based on corrective feedback control. J. Neurosci. 34, 67906806 (2014).
  73. Lim, S. & Goldman, M.S. Balanced cortical microcircuitry for maintaining information in working memory. Nat. Neurosci. 16, 13061314 (2013).
  74. Mello, G.B., Soares, S. & Paton, J.J. A scalable population code for time in the striatum. Curr. Biol. 25, 11131122 (2015).
  75. Harvey, C.D., Coen, P. & Tank, D.W. Choice-specific sequences in parietal cortex during a virtual-navigation decision task. Nature 484, 6268 (2012).
  76. Stettler, D.D. & Axel, R. Representations of odor in the piriform cortex. Neuron 63, 854864 (2009).
  77. Monier, C., Chavane, F., Baudot, P., Graham, L.J. & Frégnac, Y. Orientation and direction selectivity of synaptic inputs in visual cortical neurons: a diversity of combinations produces spike tuning. Neuron 37, 663680 (2003).
  78. Haider, B., Häusser, M. & Carandini, M. Inhibition dominates sensory responses in the awake cortex. Nature 493, 97100 (2013).
  79. Zhou, M. et al. Scaling down of balanced excitation and inhibition by active behavioral states in auditory cortex. Nat. Neurosci. 17, 841850 (2014).
  80. Hubel, D.H. & Wiesel, T.N. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol. (Lond.) 160, 106154 (1962).
  81. Ohki, K., Chung, S., Ch'ng, Y.H., Kara, P. & Reid, R.C. Functional imaging with cellular resolution reveals precise micro-architecture in visual cortex. Nature 433, 597603 (2005).
  82. Fino, E. & Yuste, R. Dense inhibitory connectivity in neocortex. Neuron 69, 11881203 (2011).
  83. Packer, A.M., McConnell, D.J., Fino, E. & Yuste, R. Axo-dendritic overlap and laminar projection can explain interneuron connectivity to pyramidal cells. Cereb. Cortex 23, 27902802 (2013).
  84. Ren, M., Yoshimura, Y., Takada, N., Horibe, S. & Komatsu, Y. Specialized inhibitory synaptic actions between nearby neocortical pyramidal neurons. Science 316, 758761 (2007).
  85. Bock, D.D. et al. Network anatomy and in vivo physiology of visual cortical neurons. Nature 471, 177182 (2011).

Download references

Author information

Affiliations

  1. Laboratoire de Neurosciences Cognitives, École Normale Supérieure, Paris, France.

    • Sophie Denève
  2. Champalimaud Centre for the Unknown, Lisbon, Portugal.

    • Christian K Machens

Competing financial interests

The authors declare no competing financial interests.

Corresponding authors

Correspondence to:

Author details

Additional data