Simulations are an important tool for investigating brain function but large models are needed to faithfully reproduce the statistics and dynamics of brain activity. Simulating large spiking neural network models has, until now, needed so much memory for storing synaptic connections that it required high performance computer systems. Here, we present an alternative simulation method we call ‘procedural connectivity’ where connectivity and synaptic weights are generated ‘on the fly’ instead of stored and retrieved from memory. This method is particularly well suited for use on graphical processing units (GPUs)—which are a common fixture in many workstations. Using procedural connectivity and an additional GPU code generation optimization, we can simulate a recent model of the macaque visual cortex with 4.13 × 106 neurons and 24.2 × 109 synapses on a single GPU—a significant step forward in making large-scale brain modeling accessible to more researchers.
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
The raw data40 used to produce Figs. 1 and 2; and the pre-processed data used to produce Fig. 3 are available at https://doi.org/10.5281/zenodo.4277749. Furthermore, the raw data used to produce Figs. 1 and 2 are also available with this manuscript. The raw spiking data from the GeNN simulations of the multi-area model41 are available at https://doi.org/10.25377/sussex.12912699. The authors of ref. 15 hold the copyright for the raw spiking data from their NEST simulations and the data is available from them upon request.
All experiments were carried out using GeNN 4.3.3 (ref. 42), available at https://doi.org/10.5281/zenodo.4022384. A GeNN port of the multi-area model43 is available at https://doi.org/10.5281/zenodo.4271816. The models used to produce Figs. 1 and 2 as well as the code to produce all figures are available at https://doi.org/10.5281/zenodo.4277749. The latest version of the code used for this paper is also availalble at https://github.com/BrainsOnBoard/procedural_paper/.
Herculano-Houzel, S., Mota, B. & Lent, R. Cellular scaling rules for rodent brains. Proc. Natl Acad. Sci. USA 103, 12138–12143 (2006).
Gewaltig, M.-O. & Diesmann, M. NEST (NEural Simulation Tool). Scholarpedia 2, 1430 (2007).
Carnevale, N. T. & Hines, M. L. The NEURON book (Cambridge Univ. Press, 2006).
Jordan, J. et al. Extremely scalable spiking neuronal network simulation code: from laptops to exascale computers. Front. Neuroinf. 12, 2 (2018).
Frenkel, C., Lefebvre, M., Legat, J.-D. & Bol, D. A 0.086-mm2 12.7-pJ/SOP 64k-synapse 256-neuron online-learning digital spiking neuromorphic processor in 28-nm CMOS. IEEE Trans. Biomed. Circuits Syst. 13, 145–158 (2019).
Furber, S. B., Galluppi, F., Temple, S. & Plana, L. A. The SpiNNaker Project. Proc. IEEE 102, 652–665 (2014).
Merolla, P. A. et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345, 668–673 (2014).
Qiao, N. et al. A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128K synapses. Front. Neurosci. 9, 141 (2015).
Schemmel, J., Kriener, L., Müller, P. & Meier, K. An accelerated analog neuromorphic hardware system emulating NMDA- and calcium-based non-linear dendrites. In Proc. International Joint Conference on Neural Networks 2217–2226 (2017).
van Albada, S. J., Helias, M. & Diesmann, M. Scalability of asynchronous networks is limited by one-to-one mapping between effective connectivity and correlations. PLoS Comput. Biol. 11, e1004490 (2015).
Rhodes, O. et al. Real-time cortical simulation on neuromorphic hardware. Philos. Trans. R. Soc. A 378, 20190160 (2020).
Knight, J. C. & Nowotny, T. GPUs outperform current HPC and neuromorphic solutions in terms of speed and energy when simulating a highly-connected cortical model. Front. Neurosci. 12, 941 (2018).
Li, A. et al. Evaluating modern GPU interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect. IEEE Trans. Parallel Distrib. Syst. 31, 94–110 (2020).
Izhikevich, E. M. Large-Scale Simulation of the Human Brain. The Neurosciences Institute http://www.izhikevich.org/human_brain_simulation/Blue_Brain.htm (2005).
Schmidt, M. et al. A multi-scale layer-resolved spiking network model of resting-state dynamics in macaque visual cortical areas. PLoS Comput. Biol. 14, e1006359 (2018).
Brette, R. et al. Simulation of networks of spiking neurons: a review of tools and strategies. J. Comput. Neurosci. 23, 349–398 (2007).
Yavuz, E., Turner, J. & Nowotny, T. GeNN: a code generation framework for accelerated brain simulations. Sci. Rep. 6, 18854 (2016).
Blundell, I. et al. Code generation in computational neuroscience: a review of tools and techniques. Front. Neuroinf. 12, 68 (2018).
Plotnikov, D. et al. NESTML: a modeling language for spiking neurons. In Lecture Notes in Informatics (LNI) Vol. P-254, 93–108 (2016); https://juser.fz-juelich.de/record/826510
Wang, G., Lin, Y. S. & Yi, W. Kernel fusion: an effective method for better power efficiency on multithreaded GPU. In Proc. 2010 IEEE/ACM International Conference on Green Computing and Communications (2010).
Stimberg, M., Brette, R. & Goodman, D. F. Brian 2, an intuitive and efficient neural simulator. eLife 8, e47314 (2019).
Izhikevich, E. M. & Edelman, G. M. Large-scale model of mammalian thalamocortical systems. Proc. Natl Acad. Sci. USA 105, 3593–3598 (2008).
Potjans, T. C. & Diesmann, M. The cell-type specific cortical microcircuit: relating structure and activity in a full-scale spiking network model. Cereb. Cortex 24, 785–806 (2014).
Cabral, J., Kringelbach, M. L. & Deco, G. Exploring the network dynamics underlying brain activity during rest. Prog. Neurobiol. 114, 102–131 (2014).
Belitski, A. et al. Low-frequency local field potentials and spikes in primary visual cortex convey independent visual information. J. Neurosci. 28, 5696–5709 (2008).
Schmidt, M., Bakker, R., Hilgetag, C. C., Diesmann, M. & van Albada, S. J. Multi-scale account of the network structure of macaque visual cortex. Brain Struct. Funct. 223, 1409–1435 (2018).
Bakker, R., Wachtler, T. & Diesmann, M. CoCoMac 2.0 and the future of tract-tracing databases. Front. Neuroinf. 6, 30 (2012).
Markov, N. T. et al. A weighted and directed interareal connectivity matrix for macaque cerebral cortex. Cereb. Cortex 24, 17–36 (2014).
Ercsey-Ravasz, M. et al. A predictive network model of cerebral cortical connectivity based on a distance rule. Neuron 80, 184–197 (2013).
Markov, N. T. et al. Anatomy of hierarchy: feedforward and feedback pathways in macaque visual cortex. J. Comp. Neurol. 522, 225–259 (2014).
Binzegger, T., Douglas, R. J. & Martin, K. A. A quantitative map of the circuit of cat primary visual cortex. J. Neurosci. 24, 8441–8453 (2004).
van Albada, S. J., Pronold, J., van Meegen, A. & Diesmann, M. in Brain-Inspired Computing (eds. Amunts, K., Grandinetti, L., Lippert, T. & Petkov, N.) (Springer, in press).
Shinomoto, S. et al. Relating neuronal firing patterns to functional differentiation of cerebral cortex. PLoS Comput. Biol. 5, e1000433 (2009).
Freedman, D. & Diaconis, P. On the histogram as a density estimator: L2 theory. Z. Wahrscheinlichkeitstheorie Verwandte Gebiete 57, 453–476 (1981).
Brader, J. M., Senn, W. & Fusi, S. Learning real-world stimuli in a neural network with spike-driven synaptic dynamics. Neural Comput. 19, 2881–2912 (2007).
Clopath, C., Büsing, L., Vasilaki, E. & Gerstner, W. Connectivity reflects coding: a model of voltage-based STDP with homeostasis. Nat. Neurosci. 13, 344–352 (2010).
Devroye, L. in Non-uniform Random Variate Generation 2nd edn, Ch. X.2, 498–500 (Springer, 2013).
Salmon, J. K., Moraes, M. A., Dror, R. O. & Shaw, D. E. Parallel random numbers: As easy as 1, 2, 3. In Proc. 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (ACM Press, 2011); http://dl.acm.org/citation.cfm?doid=2063384.2063405
Vogels, T. P. & Abbott, L. F. Signal propagation and logic gating in networks of integrate-and-fire neurons. J. Neurosci. 25, 10786–10795 (2005).
Knight, J. C. & Nowotny, T. BrainsOnBoard/procedural_paper (2020); https://doi.org/10.5281/zenodo.4277749
Knight, J. C. & Nowotny, T. Dataset for paper ‘Larger GPU-accelerated brain simulations with procedural connectivity’ figshare https://doi.org/10.25377/sussex.12912699.v1 (2020).
Knight, J. C. et al. GeNN 4.3.3 (2020); https://doi.org/10.5281/zenodo.4022384
van Albada, S. J., van Meegen, A., Knight, J. C., Schuecker, J. & Pronold, J. neworderofjamie/multi-area-model: PyGeNN multiarea model 1.0.0 (2020); https://doi.org/10.5281/zenodo.4271816
We would like to thank J. Pronold, S. van Albada, A. Korcsak-Gorzo and M. Schmidt for their help with the multi-area model data; and D. Goodman and M. Mikaitis for their feedback on the manuscript. J.K. and T.N. were funded by the EPSRC (Brains on Board project, grant number EP/P006094/1). T.N. was also funded by the the European Union’s Horizon 2020 research and innovation program under grant agreements 785907 (HBP SGA2) and 945539 (HBP SGA3).
The authors declare no competing interests.
Peer review information Nature Computational Science thanks Susanne Kunkel, Oliver Rhodes and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Yann Sweeney was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Knight, J.C., Nowotny, T. Larger GPU-accelerated brain simulations with procedural connectivity. Nat Comput Sci 1, 136–142 (2021). https://doi.org/10.1038/s43588-020-00022-7