Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Larger GPU-accelerated brain simulations with procedural connectivity

Nature Computational Science volume 1, pages 136–142 (2021)Cite this article

Subjects

A preprint version of the article is available at bioRxiv.

Abstract

Simulations are an important tool for investigating brain function but large models are needed to faithfully reproduce the statistics and dynamics of brain activity. Simulating large spiking neural network models has, until now, needed so much memory for storing synaptic connections that it required high performance computer systems. Here, we present an alternative simulation method we call ‘procedural connectivity’ where connectivity and synaptic weights are generated ‘on the fly’ instead of stored and retrieved from memory. This method is particularly well suited for use on graphical processing units (GPUs)—which are a common fixture in many workstations. Using procedural connectivity and an additional GPU code generation optimization, we can simulate a recent model of the macaque visual cortex with 4.13 × 106 neurons and 24.2 × 109 synapses on a single GPU—a significant step forward in making large-scale brain modeling accessible to more researchers.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Scaling of 1 s balanced random network simulation using different algorithms on a range of modern GPUs.
Fig. 2: Performance of a 1 s simulation of 1 × 106 LIF neurons driven by a Gaussian input current, partitioned into varying numbers (Npop) of populations and running on a workstation equipped with a Titan RTX GPU.
Fig. 3: Results of full-scale multi-area model simulation.
Fig. 4: Comparison of full-scale multi-area model spike statistics between three GeNN simulations with different seeds; and between the three GeNN simulations and a NEST simulation.

Similar content being viewed by others

Data availability

The raw data40 used to produce Figs. 1 and 2; and the pre-processed data used to produce Fig. 3 are available at https://doi.org/10.5281/zenodo.4277749. Furthermore, the raw data used to produce Figs. 1 and 2 are also available with this manuscript. The raw spiking data from the GeNN simulations of the multi-area model41 are available at https://doi.org/10.25377/sussex.12912699. The authors of ref. 15 hold the copyright for the raw spiking data from their NEST simulations and the data is available from them upon request.

Code availability

All experiments were carried out using GeNN 4.3.3 (ref. 42), available at https://doi.org/10.5281/zenodo.4022384. A GeNN port of the multi-area model43 is available at https://doi.org/10.5281/zenodo.4271816. The models used to produce Figs. 1 and 2 as well as the code to produce all figures are available at https://doi.org/10.5281/zenodo.4277749. The latest version of the code used for this paper is also availalble at https://github.com/BrainsOnBoard/procedural_paper/.

References

  1. Herculano-Houzel, S., Mota, B. & Lent, R. Cellular scaling rules for rodent brains. Proc. Natl Acad. Sci. USA 103, 12138–12143 (2006).

    Article  Google Scholar 

  2. Gewaltig, M.-O. & Diesmann, M. NEST (NEural Simulation Tool). Scholarpedia 2, 1430 (2007).

    Article  Google Scholar 

  3. Carnevale, N. T. & Hines, M. L. The NEURON book (Cambridge Univ. Press, 2006).

  4. Jordan, J. et al. Extremely scalable spiking neuronal network simulation code: from laptops to exascale computers. Front. Neuroinf. 12, 2 (2018).

    Article  Google Scholar 

  5. Frenkel, C., Lefebvre, M., Legat, J.-D. & Bol, D. A 0.086-mm2 12.7-pJ/SOP 64k-synapse 256-neuron online-learning digital spiking neuromorphic processor in 28-nm CMOS. IEEE Trans. Biomed. Circuits Syst. 13, 145–158 (2019).

    Article  Google Scholar 

  6. Furber, S. B., Galluppi, F., Temple, S. & Plana, L. A. The SpiNNaker Project. Proc. IEEE 102, 652–665 (2014).

    Article  Google Scholar 

  7. Merolla, P. A. et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345, 668–673 (2014).

    Article  Google Scholar 

  8. Qiao, N. et al. A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128K synapses. Front. Neurosci. 9, 141 (2015).

    Article  Google Scholar 

  9. Schemmel, J., Kriener, L., Müller, P. & Meier, K. An accelerated analog neuromorphic hardware system emulating NMDA- and calcium-based non-linear dendrites. In Proc. International Joint Conference on Neural Networks 2217–2226 (2017).

  10. van Albada, S. J., Helias, M. & Diesmann, M. Scalability of asynchronous networks is limited by one-to-one mapping between effective connectivity and correlations. PLoS Comput. Biol. 11, e1004490 (2015).

    Article  Google Scholar 

  11. Rhodes, O. et al. Real-time cortical simulation on neuromorphic hardware. Philos. Trans. R. Soc. A 378, 20190160 (2020).

    Article  MathSciNet  Google Scholar 

  12. Knight, J. C. & Nowotny, T. GPUs outperform current HPC and neuromorphic solutions in terms of speed and energy when simulating a highly-connected cortical model. Front. Neurosci. 12, 941 (2018).

    Article  Google Scholar 

  13. Li, A. et al. Evaluating modern GPU interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect. IEEE Trans. Parallel Distrib. Syst. 31, 94–110 (2020).

    Article  Google Scholar 

  14. Izhikevich, E. M. Large-Scale Simulation of the Human Brain. The Neurosciences Institute http://www.izhikevich.org/human_brain_simulation/Blue_Brain.htm (2005).

  15. Schmidt, M. et al. A multi-scale layer-resolved spiking network model of resting-state dynamics in macaque visual cortical areas. PLoS Comput. Biol. 14, e1006359 (2018).

    Article  Google Scholar 

  16. Brette, R. et al. Simulation of networks of spiking neurons: a review of tools and strategies. J. Comput. Neurosci. 23, 349–398 (2007).

    Article  MathSciNet  Google Scholar 

  17. Yavuz, E., Turner, J. & Nowotny, T. GeNN: a code generation framework for accelerated brain simulations. Sci. Rep. 6, 18854 (2016).

    Article  Google Scholar 

  18. Blundell, I. et al. Code generation in computational neuroscience: a review of tools and techniques. Front. Neuroinf. 12, 68 (2018).

  19. Plotnikov, D. et al. NESTML: a modeling language for spiking neurons. In Lecture Notes in Informatics (LNI) Vol. P-254, 93–108 (2016); https://juser.fz-juelich.de/record/826510

  20. Wang, G., Lin, Y. S. & Yi, W. Kernel fusion: an effective method for better power efficiency on multithreaded GPU. In Proc. 2010 IEEE/ACM International Conference on Green Computing and Communications (2010).

  21. Stimberg, M., Brette, R. & Goodman, D. F. Brian 2, an intuitive and efficient neural simulator. eLife 8, e47314 (2019).

    Article  Google Scholar 

  22. Izhikevich, E. M. & Edelman, G. M. Large-scale model of mammalian thalamocortical systems. Proc. Natl Acad. Sci. USA 105, 3593–3598 (2008).

    Article  Google Scholar 

  23. Potjans, T. C. & Diesmann, M. The cell-type specific cortical microcircuit: relating structure and activity in a full-scale spiking network model. Cereb. Cortex 24, 785–806 (2014).

    Article  Google Scholar 

  24. Cabral, J., Kringelbach, M. L. & Deco, G. Exploring the network dynamics underlying brain activity during rest. Prog. Neurobiol. 114, 102–131 (2014).

    Article  Google Scholar 

  25. Belitski, A. et al. Low-frequency local field potentials and spikes in primary visual cortex convey independent visual information. J. Neurosci. 28, 5696–5709 (2008).

    Article  Google Scholar 

  26. Schmidt, M., Bakker, R., Hilgetag, C. C., Diesmann, M. & van Albada, S. J. Multi-scale account of the network structure of macaque visual cortex. Brain Struct. Funct. 223, 1409–1435 (2018).

    Article  Google Scholar 

  27. Bakker, R., Wachtler, T. & Diesmann, M. CoCoMac 2.0 and the future of tract-tracing databases. Front. Neuroinf. 6, 30 (2012).

    Article  Google Scholar 

  28. Markov, N. T. et al. A weighted and directed interareal connectivity matrix for macaque cerebral cortex. Cereb. Cortex 24, 17–36 (2014).

    Article  Google Scholar 

  29. Ercsey-Ravasz, M. et al. A predictive network model of cerebral cortical connectivity based on a distance rule. Neuron 80, 184–197 (2013).

    Article  Google Scholar 

  30. Markov, N. T. et al. Anatomy of hierarchy: feedforward and feedback pathways in macaque visual cortex. J. Comp. Neurol. 522, 225–259 (2014).

    Article  Google Scholar 

  31. Binzegger, T., Douglas, R. J. & Martin, K. A. A quantitative map of the circuit of cat primary visual cortex. J. Neurosci. 24, 8441–8453 (2004).

    Article  Google Scholar 

  32. van Albada, S. J., Pronold, J., van Meegen, A. & Diesmann, M. in Brain-Inspired Computing (eds. Amunts, K., Grandinetti, L., Lippert, T. & Petkov, N.) (Springer, in press).

  33. Shinomoto, S. et al. Relating neuronal firing patterns to functional differentiation of cerebral cortex. PLoS Comput. Biol. 5, e1000433 (2009).

  34. Freedman, D. & Diaconis, P. On the histogram as a density estimator: L2 theory. Z. Wahrscheinlichkeitstheorie Verwandte Gebiete 57, 453–476 (1981).

    Article  MathSciNet  Google Scholar 

  35. Brader, J. M., Senn, W. & Fusi, S. Learning real-world stimuli in a neural network with spike-driven synaptic dynamics. Neural Comput. 19, 2881–2912 (2007).

    Article  MathSciNet  Google Scholar 

  36. Clopath, C., Büsing, L., Vasilaki, E. & Gerstner, W. Connectivity reflects coding: a model of voltage-based STDP with homeostasis. Nat. Neurosci. 13, 344–352 (2010).

    Article  Google Scholar 

  37. Devroye, L. in Non-uniform Random Variate Generation 2nd edn, Ch. X.2, 498–500 (Springer, 2013).

  38. Salmon, J. K., Moraes, M. A., Dror, R. O. & Shaw, D. E. Parallel random numbers: As easy as 1, 2, 3. In Proc. 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (ACM Press, 2011); http://dl.acm.org/citation.cfm?doid=2063384.2063405

  39. Vogels, T. P. & Abbott, L. F. Signal propagation and logic gating in networks of integrate-and-fire neurons. J. Neurosci. 25, 10786–10795 (2005).

    Article  Google Scholar 

  40. Knight, J. C. & Nowotny, T. BrainsOnBoard/procedural_paper (2020); https://doi.org/10.5281/zenodo.4277749

  41. Knight, J. C. & Nowotny, T. Dataset for paper ‘Larger GPU-accelerated brain simulations with procedural connectivity’ figshare https://doi.org/10.25377/sussex.12912699.v1 (2020).

  42. Knight, J. C. et al. GeNN 4.3.3 (2020); https://doi.org/10.5281/zenodo.4022384

  43. van Albada, S. J., van Meegen, A., Knight, J. C., Schuecker, J. & Pronold, J. neworderofjamie/multi-area-model: PyGeNN multiarea model 1.0.0 (2020); https://doi.org/10.5281/zenodo.4271816

Download references

Acknowledgements

We would like to thank J. Pronold, S. van Albada, A. Korcsak-Gorzo and M. Schmidt for their help with the multi-area model data; and D. Goodman and M. Mikaitis for their feedback on the manuscript. J.K. and T.N. were funded by the EPSRC (Brains on Board project, grant number EP/P006094/1). T.N. was also funded by the the European Union’s Horizon 2020 research and innovation program under grant agreements 785907 (HBP SGA2) and 945539 (HBP SGA3).

Author information

Authors and Affiliations

  1. Centre for Computational Neuroscience and Robotics, School of Engineering and Informatics, University of Sussex, Brighton, UK

    James C. Knight & Thomas Nowotny

Authors
  1. James C. Knight
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas Nowotny
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

J.K. and T.N. wrote the paper. T.N. is the original developer of GeNN. J.K. is currently the primary GeNN developer and was responsible for extending the code generation approach to the procedural simulation of synaptic connectivity. J.K. performed the experiments and the analysis of the results that are presented in this work.

Corresponding author

Correspondence to James C. Knight.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Computational Science thanks Susanne Kunkel, Oliver Rhodes and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Yann Sweeney was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Source data

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Knight, J.C., Nowotny, T. Larger GPU-accelerated brain simulations with procedural connectivity. Nat Comput Sci 1, 136–142 (2021). https://doi.org/10.1038/s43588-020-00022-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s43588-020-00022-7

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics