Learning, especially rapid learning, is critical for survival. However, learning is hard; a large number of synaptic weights must be set based on noisy, often ambiguous, sensory information. In such a high-noise regime, keeping track of probability distributions over weights is the optimal strategy. Here we hypothesize that synapses take that strategy; in essence, when they estimate weights, they include error bars. They then use that uncertainty to adjust their learning rates, with more uncertain weights having higher learning rates. We also make a second, independent, hypothesis: synapses communicate their uncertainty by linking it to variability in postsynaptic potential size, with more uncertainty leading to more variability. These two hypotheses cast synaptic plasticity as a problem of Bayesian inference, and thus provide a normative view of learning. They generalize known learning rules, offer an explanation for the large variability in the size of postsynaptic potentials and make falsifiable experimental predictions.
Subscribe to Journal
Get full journal access for 1 year
only $4.92 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
Data are available for download at: https://github.com/Jegmi/the-bayesian-synapse/releases/tag/v2/.
Code is available for download at: https://github.com/Jegmi/the-bayesian-synapse/releases/tag/v2/.
Poggio, T. A theory of how the brain might work. Cold Spring Harb. Symp. Quant. Biol. 55, 899–910 (1990).
Knill, D. C. & Richards, W. Perception as Bayesian Inference (Cambridge University Press, 1996).
Pouget, A., Beck, J. M., Ma, W. J. & Latham, P. E. Probabilistic brains: knowns and unknowns. Nat. Neurosci. 16, 1170–1178 (2013).
Aitchison, L. Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods. Adv. Neural Inf. Process. Syst. https://proceedings.neurips.cc/paper/2020/file/d33174c464c877fb03e77efdab4ae804-Paper.pdf (2020).
Tripathy, S. J., Burton, S. D., Geramita, M., Gerkin, R. C. & Urban, N. N. Brain-wide analysis of electrophysiological diversity yields novel categorization of mammalian neuron types. J. Neurophysiol. 113, 3474–3489 (2015).
Schiess, M., Urbanczik, R. & Senn, W. Somato-dendritic synaptic plasticity and error-backpropagation in active dendrites. PLoS Comput. Biol. 12, e1004638 (2016).
Bono, J. & Clopath, C. Modeling somatic and dendritic spike mediated plasticity at the single neuron and network level. Nat. Commun. 8, 706 (2017).
Sacramento, J., Ponte Costa, R., Bengio, Y. & Senn, W. Dendritic cortical microcircuits approximate the backpropagation algorithm. Adv. Neural Inf. Process. Syst. 31, 8711 (2018).
Illing, B., Gerstner, W. & Brea, J. Biologically plausible deep learning—but how far can we go with shallow networks? Neural Netw. 118, 90–101 (2019).
Akrout, M., Wilson, C., Humphreys, P. C., Lillicrap, T. & Tweed, D. Deep learning without weight transport. Adv. Neural Inf. Process. Syst. 32, 976 (2019).
Ito, M., Sakurai, M. & Tongroach, P. Climbing fibre induced depression of both mossy fibre responsiveness and glutamate sensitivity of cerebellar Purkinje cells. J. Physiol. 324, 113–134 (1982).
Eccles, J., Llinas, R. & Sasaki, K. The excitatory synaptic action of climbing fibres on the purkinje cells of the cerebellum. J. Physiol. 182, 268–296 (1966).
Widrow, B. & Hoff, M. E. Adaptive switching circuits. Technical Report no. 1553-1. https://apps.dtic.mil/dtic/tr/fulltext/u2/241531.pdf (Office of Naval Research, 1960).
Dayan, P. & Abbott, L. F. Theoretical Neuroscience (MIT Press, 2001).
Ko, H. et al. The emergence of functional microcircuits in visual cortex. Nature 496, 96–100 (2013).
Thomson, A. M. Presynaptic frequency- and pattern-dependent filtering. J. Comput. Neurosci. 15, 159–202 (2003).
Tsodyks, M. V. & Markram, H. The neural code between neocortical pyramidal neurons depends on neurotransmitter release probability. Proc. Natl Acad. Sci. USA 94, 719–723 (1997).
Maffei, A. & Turrigiano, G. G. Multiple modes of network homeostasis in visual cortical layer 2/3. J. Neurosci. 28, 4377–4384 (2008).
Hoyer, P. O. & Hyvarinen, A. Interpreting neural response variability as Monte Carlo sampling of the posterior. Adv. Neural Inf. Process. Syst. 15, 293–300 (2002).
Fiser, J., Berkes, P., Orbán, G. & Lengyel, M. Statistically optimal perception and learning: from behavior to neural representations. Trends Cogn. Sci. 14, 119–130 (2010).
Berkes, P., Fiser, J., Orbán, G. & Lengyel, M. Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment. Science 331, 83–87 (2011).
Orbán, G., Berkes, P., Fiser, J. & Lengyel, M. Neural variability and sampling-based probabilistic representations in the visual cortex. Neuron 92, 530–543 (2016).
Haefner, R. M., Berkes, P. & Fiser, J. Perceptual decision-making as probabilistic inference by neural sampling. Neuron 90, 649–660 (2016).
Aitchison, L. & Lengyel, M. The hamiltonian brain: efficient probabilistic inference with excitatory–inhibitory neural circuit dynamics. PLoS Comput. Biol. 12, e1005186 (2016).
Lange, R. D. & Haefner, R. M. Task-induced neural covariability as a signature of approximate bayesian learning and inference. Preprint at bioRxiv https://doi.org/10.1101/081661 (2020).
Ma, W. J., Beck, J. M., Latham, P. E. & Pouget, A. Bayesian inference with probabilistic population codes. Nat. Neurosci. 9, 1432–1438 (2006).
Buntine, W. L. & Weigend, A. S. Bayesian backpropagation. Complex Syst. 5, 603–643 (1991).
MacKay, D. J. A practical bayesian framework for backpropagation networks. Neural Comput. 4, 448–472 (1992).
Blundell, C., Cornebise, J., Kavukcuoglu, K. & Dean, W. Weight uncertainty in neural networks. Proc. Mach. Learn. Res. 37, 1613–1622 (2015).
Kirkpatrick, J. et al. Overcoming catastrophic forgetting in neural networks. Proc. Natl Acad. Sci. USA 106, 10296–10301 (2016).
Dayan, P. & Kakade, S. Explaining away in weight space. Adv. Neural Inf. Process. Syst. 13, 451–457 (2001).
Kappel, D., Habenschuss, S., Legenstein, R. & Maass, W. Network plasticity as bayesian inference. PLoS Comput. Biol. 11, e1004485 (2015).
Hiratani, N. & Fukai, T. Redundancy in synaptic connections enables neurons to learn optimally. Proc. Natl Acad. Sci. USA 115, E6871–E6879 (2018).
Drugowitsch, J., Mendonça, A. G., Mainen, Z. F. & Pouget, A. Learning optimal decisions with confidence. Proc. Natl Acad. Sci. USA 116, 24872–24880 (2019).
Pfister, J.-P., Dayan, P. & Lengyel, M. Synapses with short-term plasticity are optimal estimators of presynaptic membrane potentials. Nat. Neurosci. 13, 1271–1275 (2010).
Kasai, H., Takahashi, N. & Tokumaru, H. Distinct initial SNARE configurations underlying the diversity of exocytosis. Physiol. Rev. 92, 1915–1964 (2012).
Südhof, T. C. The presynaptic active zone. Neuron 75, 11–25 (2012).
Michel, K., Müller, J. A., Oprisoreanu, A.-M. & Schoch, S. The presynaptic active zone: a dynamic scaffold that regulates synaptic efficacy. Exp. Cell Res. 335, 157–164 (2015).
Frey, U. & Morris, R. G. Synaptic tagging and long-term potentiation. Nature 385, 533–536 (1997).
Redondo, R. L. & Morris, R. G. M. Making memories last: the synaptic tagging and capture hypothesis. Nat. Rev. Neurosci. 12, 17–30 (2011).
Rogerson, T. et al. Synaptic tagging during memory allocation. Nat. Rev. Neurosci. 15, 157–169 (2014).
Abraham, W. C. & Bear, M. F. Metaplasticity: the plasticity of synaptic plasticity. Trends Neurosci. 19, 126–130 (1996).
Abraham, W. C. Metaplasticity: tuning synapses and networks for plasticity. Nat. Rev. Neurosci. 9, 387 (2008).
Hulme, S. R., Jones, O. D., Raymond, C. R., Sah, P. & Abraham, W. C. Mechanisms of heterosynaptic metaplasticity. Philos. Trans. R. Soc. Lond. B Biol. Sci. 369, 20130148 (2014).
Vogelstein, J. T. et al. Fast nonnegative deconvolution for spike train inference from population calcium imaging. J. Neurophysiol. 104, 3691–3704 (2010).
Packer, A. M., Russell, L. E., Dalgleish, H. W. P. & Häusser, M. Simultaneous all-optical manipulation and recording of neural circuit activity with cellular resolution in vivo. Nat. Methods 12, 140–146 (2015).
Loewenstein, Y., Kuras, A. & Rumpel, S. Multiplicative dynamics underlie the emergence of the log-normal distribution of spine sizes in the neocortex in vivo. J. Neurosci. 31, 9481–9488 (2011).
Matsuzaki, M., Honkura, N., Ellis-Davies, G. C. & Kasai, H. Structural basis of long-term potentiation in single dendritic spines. Nature 429, 761–766 (2004).
Song, S., Sjöström, P. J., Reigl, M., Nelson, S. & Chklovskii, D. B. Highly nonrandom features of synaptic connectivity in local cortical circuits. PLoS Biol. 3, e68 (2005).
O’Connor, D. H., Peron, S. P., Huber, D. & Svoboda, K. Neural activity in barrel cortex underlying vibrissa-based object localization in mice. Neuron 67, 1048–1061 (2010).
Mizuseki, K. & Buzsáki, G. Preconfigured, skewed distribution of firing rates in the hippocampus and entorhinal cortex. Cell Rep. 4, 1010–1021 (2013).
Minka, T. P. A family of algorithms for approximate Bayesian inference. Dissertation, Massachusetts Institute of Technology (2001).
L.A. and P.E.L. were supported by the Gatsby Charitable Foundation. P.E.L. was also supported by the Wellcome Trust (110114/Z/15/Z). J.J. and J.-P.P. were supported by the Swiss National Science Foundation (PP00P3 150637 and 31003A 175644). J.A.M. was supported by University College London (UCL) Graduate Research and UCL Overseas Research Scholarships. A.P. was supported by a grant from the Simons Collaboration for the Global Brain and the Swiss National Foundation (31003A 165831).
The authors declare no competing interests.
Peer review information Nature Neuroscience thanks the anonymous reviewers for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Aitchison, L., Jegminat, J., Menendez, J.A. et al. Synaptic plasticity as Bayesian inference. Nat Neurosci 24, 565–571 (2021). https://doi.org/10.1038/s41593-021-00809-5