Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Imaging whole-brain activity to understand behaviour

Nature Reviews Physics volume 4, pages 292–305 (2022)Cite this article

Subjects

Abstract

Until now, most brain studies have focused on small numbers of neurons that interact in limited circuits, allowing analysis of individual computations or steps of neural processing. During behaviour, however, brain activity must integrate multiple circuits in different brain regions. Whole-brain recording with cellular resolution provides a new opportunity to dissect the neural basis of behaviour, but whole-brain activity is mutually contingent on behaviour itself, especially for natural behaviours such as navigation, mating or hunting, which require dynamic interaction between the animal, its environment and other animals. Many of the signalling and feedback pathways that animals use to guide behaviour only occur in freely moving animals. Recent technological advances have enabled whole-brain recording in small behaving animals including the nematode Caenorhabditis elegans, the fruit fly Drosophila melanogaster and the larval zebrafish Danio rerio. These whole-brain experiments capture neural activity with cellular resolution spanning sensory, decision-making and motor circuits, and thereby demand new theoretical approaches that integrate brain dynamics with behavioural dynamics. We review the experimental and theoretical methods used to understand animal behaviour and whole-brain activity, and the opportunities for physics to contribute to this emerging field of systems neuroscience.

Key points

  • Advances in optical microscopy allow brain-wide imaging with cellular resolution throughout the sensory, decision-making and motor circuits of behaving animals.

  • A complete understanding brain-wide dynamics requires requires the context provided by behavioural dynamics: ongoing behaviour emerges from brain activity, and brain activity itself is contingent on past behaviours and experiences.

  • Brain activity is organized by structural, functional and physiological mechanisms. The wiring diagram of the brain (the connectome) represents pathways of synaptic information flow. The molecular properties of synapses and cells determine the neuronal responses to sensory and synaptic inputs. Non-synaptic mechanisms organize brain-wide activities corresponding to different behavioural states.

  • Small animals like nematodes, insects and larval fish are tractable models for comprehensively exploring and modelling the mechanisms of brain-wide activity and behaviour.

  • Modelling brain-wide activity is a multiscale problem from synapses to cells to circuits, across brain areas and across behaviours.

  • Both top-to-bottom modelling — posing a theory of neural computation and modelling biological mechanisms that might carry it out — and bottom-to-top modelling — looking for structure in high-dimensional activity patterns that might explain correlated behavioural patterns — are important strategies for building towards an understanding of brain-wide dynamics and animal behaviour.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Recording from the brains of behaving animals.
Fig. 2: Samples of pan-neuronal recordings in behaving animals.
Fig. 3: Computational methods for neural and behavioural analysis.

Similar content being viewed by others

References

  1. Dupre, C. & Yuste, R. Non-overlapping neural networks in Hydra vulgaris. Curr. Biol. 27, 1085–1097 (2017).

    Article  Google Scholar 

  2. Chen, J. L., Andermann, M. L., Keck, T., Xu, N. L. & Ziv, Y. Imaging neuronal populations in behaving rodents: paradigms for studying neural circuits underlying behavior in the mammalian cortex. J. Neurosci. 33, 17631–17640 (2013).

    Article  Google Scholar 

  3. Hong, G. & Lieber, C. M. Novel electrode technologies for neural recordings. Nat. Rev. Neurosci. 20, 330–345 (2019).

    Article  Google Scholar 

  4. Allen, W. E. Thirst regulates motivated behavior through modulation of brainwide neural population dynamics. Science 364, 253 (2019).

    Article  ADS  Google Scholar 

  5. Allen, W. E. et al. Global representations of goal-directed behavior in distinct cell types of mouse neocortex. Neuron 94, 891–907 (2017).

    Article  Google Scholar 

  6. Stringer, C. et al. Spontaneous behaviors drive multidimensional, brainwide activity. Science 364, 255 (2019).

    Article  ADS  Google Scholar 

  7. Steinmetz, N. A. et al. Neuropixels 2.0: a miniaturized high-density probe for stable, long-term brain recordings. Science 372, eabf4588 (2021).

    Article  Google Scholar 

  8. Steinmetz, N. A., Koch, C., Harris, K. D. & Carandini, M. Challenges and opportunities for large-scale electrophysiology with neuropixels probes. Curr. Opin. Neurobiol. 50, 92–100 (2018).

    Article  Google Scholar 

  9. DeYoe, E. A., Bandettini, P., Neitz, J., Miller, D. & Winans, P. Functional magnetic resonance imaging (FMRI) of the human brain. J. Neurosci. Methods 54, 171–187 (1994).

    Article  Google Scholar 

  10. Lee, J. H. et al. Global and local FMRI signals driven by neurons defined optogenetically by type and wiring. Nature 465, 788–792 (2010).

    Article  ADS  Google Scholar 

  11. Calhoun, A. J. & Murthy, M. Quantifying behavior to solve sensorimotor transformations: advances from worms and flies. Curr. Opin. Neurobiol. 46, 90–98 (2017).

    Article  Google Scholar 

  12. Kandel, E. R., Schwartz, J. H., Jessell, T. M., Siegelbaum, S. A. & Hudspeth, A. J. Principles of Neural Science 5th edn (McGraw-Hill, 2013).

  13. Chen, Z., Truong, T. M. & Ai, H.-w. Illuminating brain activities with fluorescent protein-based biosensors. Chemosensors 5, 32 (2017).

    Article  ADS  Google Scholar 

  14. Deo, C. & Lavis, L. D. Synthetic and genetically encoded fluorescent neural activity indicators. Curr. Opin. Neurobiol. 50, 101–108 (2018).

    Article  Google Scholar 

  15. Pal, A. & Tian, L. Imaging voltage and brain chemistry with genetically encoded sensors and modulators. Curr. Opin. Chem. Biol. 57, 166–176 (2020).

    Article  Google Scholar 

  16. Laviv, T. & Yasuda, R. Imaging neuronal protein signaling dynamics in vivo. Curr. Opin. Neurobiol. 69, 68–75 (2021).

    Article  Google Scholar 

  17. Stephens, D. & Allan, V. Light microscopy techniques for live cell imaging. Science 300, 82–86 (2003).

    Article  ADS  Google Scholar 

  18. Huisken, J. & Stainier, D. Y. R. Selective plane illumination microscopy techniques in developmental biology. Development 136, 1963–1975 (2009).

    Article  Google Scholar 

  19. Svoboda, K. & Yasuda, R. Principles of two-photon excitation microscopy and its applications to neuroscience. Neuron 50, 823–839 (2006).

    Article  Google Scholar 

  20. Bouchard, M. B. et al. Swept confocally-aligned planar excitation (SCAPE) microscopy for high-speed volumetric imaging of behaving organisms. Nat. Photonics 9, 113–119 (2015).

    Article  ADS  Google Scholar 

  21. Voleti, V. et al. Real-time volumetric microscopy of in vivo dynamics and large-scale samples with SCAPE 2.0. Nat. Methods 16, 1054–1062 (2019).

    Article  Google Scholar 

  22. Karagyozov, D., Mihovilovic Skanata, M., Lesar, A. & Gershow, M. Recording neural activity in unrestrained animals with three-dimensional tracking two-photon microscopy. Cell Reports 25, 1371–1383 (2018).

    Article  Google Scholar 

  23. Griffiths, V. A. et al. Real-time 3D movement correction for two-photon imaging in behaving animals. Nat. Methods 17, 741–748 (2020).

    Article  Google Scholar 

  24. Power, R. M. & Huisken, J. A guide to light-sheet fluorescence microscopy for multiscale imaging. Nat. Methods 14, 360–373 (2017).

    Article  Google Scholar 

  25. Prabhat, P., Ram, S., Ward, E. S. & Ober, R. J. Simultaneous imaging of different focal planes in fluorescence microscopy for the study of cellular dynamics in three dimensions. IEEE Trans. Nanobiosci. 3, 237–242 (2004).

    Article  Google Scholar 

  26. Abrahamsson, S. et al. Multifocus microscopy with precise color multi-phase diffractive optics applied in functional neuronal imaging.Biomed. Opt. Express 7, 855–869 (2016).

    Article  Google Scholar 

  27. Bimber, O. & Schedl, D. C. Light-field microscopy: a review. J. Neurol. Neuromed. 4, 1–6 (2019).

    Article  Google Scholar 

  28. Aimon, S. et al. Fast near-whole-brain imaging in adult Drosophila during responses to stimuli and behavior. PLoS Biol. 17, e2006732 (2019).

    Article  Google Scholar 

  29. Cong, L. et al. Rapid whole brain imaging of neural activity in freely behaving larval zebrafish (Danio rerio). eLife 6, e28158 (2017).

    Article  Google Scholar 

  30. Zhang, Z. et al. Imaging volumetric dynamics at high speed in mouse and zebrafish brain with confocal light field microscopy. Nat. Biotechnol. 39, 74–83 (2021).

    Article  Google Scholar 

  31. Wang, Z. et al. Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning. Nat. Methods 18, 551–556 (2021).

    Article  Google Scholar 

  32. Aragon, M. J. et al. Multiphoton imaging of neural structure and activity in Drosophila through the intact cuticle. eLife 11, e69094 (2022).

    Article  Google Scholar 

  33. Mahou, P., Vermot, J., Beaurepaire, E. & Supatto, W. Multicolor two-photon light-sheet microscopy. Nat. Methods 11, 600–601 (2014).

    Article  Google Scholar 

  34. Venkatachalam, V. et al. Pan-neuronal imaging in roaming Caenorhabditis elegans. Proc. Natl Acad. Sci. USA 113, 1082–8 (2016).

    Article  Google Scholar 

  35. Nguyen, J. P., Linder, A. N., Plummer, G. S., Shaevitz, J. W. & Leifer, A. M. Automatically tracking neurons in a moving and deforming brain.PLoS Comput. Biol. 13, e1005517 (2017).

    Article  ADS  Google Scholar 

  36. Schaffer, E. S. et al. Flygenvectors: the spatial and temporal structure of neural activity across the fly brain. Preprint at bioRxiv https://doi.org/10.1101/2021.09.25.461804 (2021).

    Article  Google Scholar 

  37. Brown, A. E. & De Bivort, B. Ethology as a physical science. Nat. Phys. 14, 653–657 (2018).

    Article  Google Scholar 

  38. Kim, D. H. et al. Pan-neuronal calcium imaging with cellular resolution in freely swimming zebrafish. Nat. Methods 14, 1107–1114 (2017).

    Article  Google Scholar 

  39. Wu, Y. et al. Rapid detection and recognition of whole brain activity in a freely behaving Caenorhabditis elegans. Preprint at https://arxiv.org/abs/2109.10474 (2021).

  40. Yu, X. et al. Fast deep neural correspondence for tracking and identifying neurons in C. elegans using semi-synthetic training. eLife 10, e66410 (2021).

    Article  Google Scholar 

  41. Randlett, O. et al. Whole-brain activity mapping onto a zebrafish brain atlas. Nat. Methods 12, 1039–1046 (2015).

    Article  Google Scholar 

  42. Pereira, T. D. et al. Fast animal pose estimation using deep neural networks. Nat. Methods 16, 117–125 (2019).

    Article  Google Scholar 

  43. Pereira, T. D. et al. SLEAP: multi-animal pose tracking. Preprint at bioRxiv https://doi.org/10.1101/2020.08.31.276246 (2020).

    Article  Google Scholar 

  44. Graving, J. M. et al. DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning. eLife 8, e47994 (2019).

    Article  Google Scholar 

  45. Berman, G. J., Choi, D. M., Bialek, W. & Shaevitz, J. W. Mapping the stereotyped behaviour of freely moving fruit flies. J. R. Soc. Interface 11, 20140672 (2014).

    Article  Google Scholar 

  46. Datta, S. R., Anderson, D. J., Branson, K., Perona, P. & Leifer, A. Computational neuroethology: a call to action. Neuron 104, 11–24 (2019).

    Article  Google Scholar 

  47. White, J., Southgate, E., Thomson, J. & Brenner, S. The structure of the nervous system of the nematode Caenorhabditis elegans. Phil. Trans. R Soc. Lond. B 314, 1–340 (1986).

    Article  ADS  Google Scholar 

  48. Witvliet, D., Mulcahy, B., Mitchell, J.K. et al. Connectomes across development reveal principles of brain maturation. Nature 596, 257–261 (2021).

    Article  ADS  Google Scholar 

  49. Barr, M. M. & Garcia, L. R. Male mating behavior. WormBook https://doi.org/10.1895/wormbook.1.78.1 (2006).

    Article  Google Scholar 

  50. Schrödel, T., Prevedel, R., Aumayr, K., Zimmer, M. & Vaziri, A. Brain-wide 3D imaging of neuronal activity in Caenorhabditis elegans with sculpted light. Nat. Methods 10, 1013–1020 (2013).

    Article  Google Scholar 

  51. Kato, S. et al. Global brain dynamics embed the motor command sequence of Caenorhabditis elegans. Cell 163, 656–669 (2015).

    Article  Google Scholar 

  52. Zhen, M. & Samuel, A. D. C. elegans locomotion: small circuits, complex functions. Curr. Opin. Neurobiol. 33, 117–126 (2015).

    Article  Google Scholar 

  53. Kaplan, H. S., Salazar Thula, O., Khoss, N. & Zimmer, M. Nested neuronal dynamics orchestrate a behavioral hierarchy across timescales. Neuron 105, 562–576 (2020).

    Article  Google Scholar 

  54. Nguyen, J. P. et al. Whole-brain calcium imaging with cellular resolution in freely behaving Caenorhabditis elegans. Proc. Natl Acad. Sci. USA 113, 1074–81 (2016).

    Article  Google Scholar 

  55. Hallinen, K. M. et al. Decoding locomotion from population neural activity in moving C. elegans. eLife 10, e66135 (2021).

    Article  Google Scholar 

  56. Yemini, E. et al. NeuroPAL: a multicolor atlas for whole-brain neuronal identification in C. elegans. Cell 184, 272–288 (2021).

    Article  Google Scholar 

  57. Sulston, J. E., Albertson, D. G. & Thomson, J. N. The Caenorhabditis elegans male: postembryonic development of nongonadal structures. Dev. Biol. 78, 542–576 (1980).

    Article  Google Scholar 

  58. Susoy, V. et al. Natural sensory context drives diverse brain-wide activity during C. elegans mating. Cell 184, 5122–5137 (2021).

    Article  Google Scholar 

  59. Nejatbakhsh, A. et al. Demixing calcium imaging data in C. elegans via deformable non-negative matrix factorization. Med. Image Comput. Comput. Assist. Interv. 12265, 14–24 (2020).

    Google Scholar 

  60. Chaudhary, S., Lee, S. A., Li, Y., Patel, D. S. & Lu, H. Automated annotation of cell identities in dense cellular images. eLife 10, e60321 (2021).

    Article  Google Scholar 

  61. Yu, X. et al. Fast deep learning correspondence for neuron tracking and identification in C. elegans using synthetic training. eLife 10, e66410 (2021).

    Article  Google Scholar 

  62. Hendricks, M., Ha, H., Maffey, N. & Zhang, Y. Compartmentalized calcium dynamics in a C. elegans interneuron encode head movement. Nature 487, 99–103 (2012).

    Article  ADS  Google Scholar 

  63. Ouellette, M. H., Desrochers, M. J., Gheta, I., Ramos, R. & Hendricks, M. A gate-and-switch model for head orientation behaviors in C. elegans. eNeuro 5, ENEURO.0121-18.2018 (2018).

    Article  Google Scholar 

  64. Cuentas-Condori, A. et al. C. elegans neurons have functional dendritic spines. eLife 8, e47918 (2019).

    Article  Google Scholar 

  65. Mullins, O. J., Hackett, J. T., Buchanan, J. T. & Friesen, W. O. Neuronal control of swimming behavior: comparison of vertebrate and invertebrate model systems. Prog. Neurobiol. 93, 244–269 (2011).

    Article  Google Scholar 

  66. Kiehn, O. Development and functional organization of spinal locomotor circuits. Curr. Opin. Neurobiol. 21, 100–109 (2011).

    Article  Google Scholar 

  67. Rybak, I. A., Dougherty, K. J. & Shevtsova, N. A. Organization of the mammalian locomotor CPG: review of computational model and circuit architectures based on genetically identified spinal interneurons. eNeuro 2, ENEURO.0069-15.2015 (2015).

  68. Falgairolle, M., Puhl, J. G., Pujala, A., Liu, W. & O’Donovan, M. J. Motoneurons regulate the central pattern generator during drug-induced locomotor-like activity in the neonatal mouse. eLife 6, e26622 (2017).

    Article  Google Scholar 

  69. Cohen, N. & Sanders, T. Nematode locomotion: nissecting the neuronal-environmental loop. Curr. Opin. Neurobiol. 25, 99–106 (2014).

    Article  Google Scholar 

  70. Scheffer, L. K. et al. A connectome and analysis of the adult Drosophila central brain. eLife 9, e57443 (2020).

    Article  Google Scholar 

  71. Seelig, J. D. et al. Two-photon calcium imaging from head-fixed Drosophila during optomotor walking behavior. Nat. Methods 7, 535–540 (2010).

    Article  Google Scholar 

  72. Ito, K. et al. A systematic nomenclature for the insect brain. Neuron 81, 755–765 (2014).

    Article  Google Scholar 

  73. Mann, K., Gallen, C. L. & Clandinin, T. R. Whole-brain calcium imaging reveals an intrinsic functional network in Drosophila. Curr. Biol. 27, 2389–2396 (2017).

    Article  Google Scholar 

  74. Mann, K., Deny, S., Ganguli, S. et al. Coupling of activity, metabolism and behaviour across the Drosophila brain. Nature 593, 244–248 (2021).

    Article  Google Scholar 

  75. Isaacman-Beck, J. et al. SPARC enables genetic manipulation of precise proportions of cells. Nat. Neurosci. 23, 1168–1175 (2020).

    Article  Google Scholar 

  76. Pacheco, D. A., Thiberge, S. Y., Pnevmatikakis, E. & Murthy, M. Auditory activity is diverse and widespread throughout the central brain of Drosophila. Nat. Neurosci 24, 93–104 (2021).

    Article  Google Scholar 

  77. Deutsch, D. et al. The neural basis for a persistent internal state in Drosophila females. eLife 9, e59502 (2020).

    Article  Google Scholar 

  78. Lemon, W. C. et al. Whole-central nervous system functional imaging in larval Drosophila. Nat. Commun. 6, 7924 (2015).

    Article  ADS  Google Scholar 

  79. Sun, X. & Heckscher, E. S. Using linear agarose channels to study Drosophila larval crawling behavior. J. Vis. Exp. 2016, 1–7 (2016).

    Google Scholar 

  80. Hinsch, K. & Zupanc, G. Generation and long-term persistence of new neurons in the adult zebrafish brain: a quantitative analysis. Neuroscience 146, 679–696 (2007).

    Article  Google Scholar 

  81. Orger, M. B. & De Polavieja, G. G. Zebrafish behavior: opportunities and challenges. Ann. Rev. Neurosci. 40, 125–147 (2017).

    Article  Google Scholar 

  82. Mu, Y., Narayan, S., Mensh, B. D. & Ahrens, M. B. Brain-wide, scale-wide physiology underlying behavioral flexibility in zebrafish. Curr. Opin. Neurobiol. 64, 151–160 (2020).

    Article  Google Scholar 

  83. Kunst, M. et al. A cellular-resolution atlas of the larval zebrafish brain. Neuron 103, 21–38.e5 (2019).

    Article  ADS  Google Scholar 

  84. Ahrens, M. B., Orger, M. B., Robson, D. N., Li, J. M. & Keller, P. J. Whole-brain functional imaging at cellular resolution using light-sheet microscopy. Nat. Methods 10, 413–420 (2013).

    Article  Google Scholar 

  85. Vanwalleghem, G. C., Ahrens, M. B. & Scott, E. K. Integrative whole-brain neuroscience in larval zebrafish. Curr. Opin. Neurobiol. 50, 136–145 (2018).

    Article  Google Scholar 

  86. Ahrens, M. B., Huang, K. H., Narayan, S., Mensh, B. D. & Engert, F. Two-photon calcium imaging during fictive navigation in virtual environments. Front. Neural Circuits 7, 104 (2013).

    Article  Google Scholar 

  87. Dunn, T. W. et al. Brain-wide mapping of neural activity controlling zebrafish exploratory locomotion. eLife 5, e12741 (2016).

    Article  Google Scholar 

  88. Haesemeyer, M., Robson, D. N., Li, J. M., Schier, A. F. & Engert, F. A brain-wide circuit model of heat-evoked swimming behavior in larval zebrafish. Neuron 98, 817–831 (2018).

    Article  Google Scholar 

  89. Lovett-Barron, M. et al. Multiple convergent hypothalamus-brainstem circuits drive defensive behavior. Nat. Neurosci. 23, 959–967 (2020).

    Article  Google Scholar 

  90. Bahl, A. & Engert, F. Neural circuits for evidence accumulation and decision making in larval zebrafish. Nat. Neurosci. 23, 94–102 (2020).

    Article  Google Scholar 

  91. Dragomir, E. I., Štih, V. & Portugues, R. Evidence accumulation during a sensorimotor decision task revealed by whole-brain imaging. Nat. Neurosci. 23, 85–93 (2020).

    Article  Google Scholar 

  92. Kawashima, T., Zwart, M. F., Yang, C. T., Mensh, B. D. & Ahrens, M. B. The serotonergic system tracks the outcomes of actions to mediate short-term motor learning. Cell 167, 933–946.e20 (2016).

    Article  Google Scholar 

  93. Lin, Q. et al. Cerebellar neurodynamics predict decision timing and outcome on the single-trial level. Cell 180, 536–551.e17 (2020).

    Article  Google Scholar 

  94. Mu, Y. et al. Glia accumulate evidence that actions are futile and suppress unsuccessful behavior. Cell 178, 27–43 (2019).

    Article  Google Scholar 

  95. Wolf, S. et al. Sensorimotor computation underlying phototaxis in zebrafish. Nat. Commun. 8, 651 (2017).

    Article  ADS  Google Scholar 

  96. Chen, X. et al. Brain-wide organization of neuronal activity and convergent sensorimotor transformations in larval zebrafish. Neuron 100, 876–890 (2018).

    Article  Google Scholar 

  97. Oikonomou, G. et al. The serotonergic raphe promote sleep in zebrafish and mice. Neuron 103, 686–701.e8 (2019).

    Article  Google Scholar 

  98. Andalman, A. S. et al. Neuronal dynamics regulating brain and behavioral state transitions. Cell 177, 970–985.e20 (2019).

    Article  Google Scholar 

  99. Lovett-Barron, M. et al. Ancestral circuits for the coordinated modulation of brain state. Cell 171, 1411–1423.e17 (2017).

    Article  Google Scholar 

  100. Migault, G. et al. Whole-brain calcium imaging during physiological vestibular stimulation in larval zebrafish. Curr. Biol. 28, 3723–3735.e6 (2018).

    Article  Google Scholar 

  101. Marques, J. C., Li, M., Schaak, D., Robson, D. N. & Li, J. M. Internal state dynamics shape brainwide activity and foraging behaviour. Nature 577, 239–243 (2020).

    Article  ADS  Google Scholar 

  102. Chalfie, M. et al. The neural circuit for touch sensitivity in Caenorhabditis elegans. J. Neurosci. 5, 956–964 (1985).

    Article  Google Scholar 

  103. Berck, M. E. et al. The wiring diagram of a glomerular olfactory system. eLife 5, e14859 (2016).

    Article  Google Scholar 

  104. Gerhard, S., Andrade, I., Fetter, R. D., Cardona, A. & Schneider-Mizell, C. M. Conserved neural circuit structure across Drosophila larval development revealed by comparative connectomics. eLife 6, e29089 (2017).

    Article  Google Scholar 

  105. Zheng, Z. et al. A complete electron microscopy volume of the brain of adult Drosophila melanogaster. Cell 174, 730–743.e22 (2018).

    Article  Google Scholar 

  106. Dorkenwald, S. et al. FlyWire: online community for whole-brain connectomics. Nat. Methods 19, 119–128 (2022).

    Article  Google Scholar 

  107. Turner, M. H., Mann, K. & Clandinin, T. R. The connectome predicts resting-state functional connectivity across the Drosophila brain. Curr. Biol. 31, 2386–2394 (2021).

    Article  Google Scholar 

  108. Hildebrand, D. G. C. et al. Whole-brain serial-section electron microscopy in larval zebrafish. Nature 545, 345–349 (2017).

    Article  ADS  Google Scholar 

  109. Lichtman, J. W. & Denk, W. The big and the small: challenges of imaging the brain’s circuits. Science 334, 618–623 (2011).

    Article  ADS  Google Scholar 

  110. Swanson, L. W. & Lichtman, J. W. From Cajal to connectome and beyond. Annu. Rev. Neurosci. 39, 197–216 (2016).

    Article  Google Scholar 

  111. Pnevmatikakis, E. A. et al. Simultaneous denoising, deconvolution, and demixing of calcium imaging data. Neuron 89, 285–299 (2016).

    Article  Google Scholar 

  112. Zhou, P. et al. Efficient and accurate extraction of in vivo calcium signals from microendoscopic video data. eLife 7, e28728 (2018).

    Article  Google Scholar 

  113. Pachitariu, M., Stringer, C. & Harris, K. D. Robustness of spike deconvolution for neuronal calcium imaging. J. Neurosci. 38, 7976–7985 (2018).

    Article  Google Scholar 

  114. Giovannucci, A. et al. CaImAn an open source tool for scalable calcium imaging data analysis. eLife 8, e38173 (2019).

    Article  Google Scholar 

  115. Mathis, A. et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 21, 1281–1289 (2018).

    Article  Google Scholar 

  116. Wu, A. et al. Deep graph pose: a semi-supervised deep graphical model for improved animal pose tracking. In Adv. Neural Inf. Proc. Syst. 33 (NeurIPS, 2020).

  117. Bala, P. C. et al. Automated markerless pose estimation in freely moving macaques with openmonkeystudio. Nat. Commun. 11, 4560 (2020).

    Article  ADS  Google Scholar 

  118. Zhang, L., Dunn, T., Marshall, J., Olveczky, B. & Linderman, S. Animal pose estimation from video data with a hierarchical von Mises–Fisher–Gaussian model. Proc. Machine Learning Res. 130, 2800–2808 (2021).

  119. Abbott, L. F. et al. The mind of a mouse. Cell 182, 1372–1376 (2020).

    Article  Google Scholar 

  120. Linderman, S. W. & Gershman, S. J. Using computational theory to constrain statistical models of neural data. Curr. Opin. Neurobiol. 46, 14–24 (2017).

    Article  Google Scholar 

  121. Tukey, J. W. Exploratory Data Analysis Vol. 2 (Addison-Wesley, 1977).

  122. Cunningham, J. P. & Yu, B. M. Dimensionality reduction for large-scale neural recordings. Nat. Neurosci. 17, 1500–1509 (2014).

    Article  Google Scholar 

  123. York, R. A., Carreira-Rosario, A., Giocomo, L. M. & Clandinin, T. R. Flexible analysis of animal behavior via time-resolved manifold embedding. Preprint at bioRxiv https://doi.org/10.1101/2020.09.30.321406 (2021).

    Article  Google Scholar 

  124. Wiltschko, A. B. et al. Mapping sub-second structure in mouse behavior. Neuron 88, 1121–1135 (2015).

    Article  Google Scholar 

  125. Calhoun, A. J., Pillow, J. W. & Murthy, M. Unsupervised identification of the internal states that shape natural behavior. Nat. Neurosci. 22, 2040–2049 (2019).

    Article  Google Scholar 

  126. Markowitz, J. E. et al. The striatum organizes 3D behavior via moment-to-moment action selection. Cell 174, 44–58.e17 (2018).

    Article  Google Scholar 

  127. Linderman, S. W., Nichols, A. L. A., Blei, D. M., Zimmer, M. & Paninski, L. Hierarchical recurrent state space models reveal discrete and continuous dynamics of neural activity in C. elegans. Preprint at bioRxiv https://doi.org/10.1101/621540 (2019).

    Article  Google Scholar 

  128. Paninski, L. et al. A new look at state-space models for neural data. J. Comput. Neurosci. 29, 107–126 (2010).

    Article  MathSciNet  MATH  Google Scholar 

  129. Murphy, K. P. Machine Learning: A Probabilistic Perspective Ch. 17–18 (MIT Press, 2012).

  130. Eyjolfsdottir, E., Branson, K., Yue, Y. & Perona, P. Learning recurrent representations for hierarchical behavior modeling. Preprint at https://arxiv.org/abs/1611.00094 (2016).

  131. Zhao, Y. & Park, I. M. Variational latent Gaussian process for recovering single-trial dynamics from population spike trains. Neural Comput. 29, 1293–1316 (2017).

    Article  MathSciNet  MATH  Google Scholar 

  132. Linderman, S. W. et al. Bayesian learning and inference in recurrent switching linear dynamical systems. PMLR 54, 914–922 (2017).

    Google Scholar 

  133. Pandarinath, C. et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nat. Methods 15, 805–815 (2018).

    Article  Google Scholar 

  134. Hernandez, D. et al. A novel variational family for hidden nonlinear Markov models. Preprint at https://arxiv.org/abs/1811.02459 (2018).

  135. Duncker, L., Bohner, G., Boussard, J. & Sahani, M. Learning interpretable continuous-time models of latent stochastic dynamical systems. Preprint at https://arxiv.org/abs/1902.04420 (2019).

  136. Batty, E. et al. BehaveNet: nonlinear embedding and Bayesian neural decoding of behavioral videos. In Adv. Neural Inf. Process. Syst. 32 (NeurIPS, 2019).

  137. Nassar, J., Linderman, S., Bugallo, M. & Park, I. Tree-structured recurrent switching linear dynamical systems for multi-scale modeling. ICLR https://openreview.net/forum?id=HkzRQhR9YX (2019).

  138. Glaser, J. I., Whiteway, M., Cunningham, J. P., Paninski, L. & Linderman, S. W. Recurrent switching dynamical systems models for multiple interacting neural populations. In Adv. Neural Inf. Proc. Syst. 33 (NeurIPS, 2020).

  139. Zoltowski, D., Pillow, J. & Linderman, S. A general recurrent state space framework for modeling neural dynamics during decision-making. PMLR 119, 11680–11691 (2020).

    Google Scholar 

  140. Sussillo, D. & Barak, O. Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural Comput. 25, 626–649 (2013).

    Article  MathSciNet  MATH  Google Scholar 

  141. Vyas, S., Golub, M. D., Sussillo, D. & Shenoy, K. V. Computation through neural population dynamics. Annu. Rev. Neurosci. 43, 249–275 (2020).

    Article  Google Scholar 

  142. Ben-Yishai, R., Bar-Or, R. L. & Sompolinsky, H. Theory of orientation tuning in visual cortex. Proc. Natl Acad. Sci. USA 92, 3844–3848 (1995).

    Article  ADS  Google Scholar 

  143. Skaggs, W. E., Knierim, J. J., Kudrimoti, H. S. & McNaughton, B. L. A model of the neural basis of the rat’s sense of direction. Adv. Neural Inf. Process. Syst. 7, 173–180 (1995).

    Google Scholar 

  144. Kim, S. S., Rouault, H., Druckmann, S. & Jayaraman, V. Ring attractor dynamics in the Drosophila central brain. Science 356, 849–853 (2017).

    Article  ADS  Google Scholar 

  145. Yamins, D. L. K. & DiCarlo, J. J. Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci. 19, 356–365 (2016).

    Article  Google Scholar 

  146. Nayebi, A. et al. Task-driven convolutional recurrent models of the visual system. Adv. Neural Inf. Proc. Syst. 31, 5290–5301 (2018).

  147. Sussillo, D., Churchland, M. M., Kaufman, M. T. & Shenoy, K. V. A neural network that finds a naturalistic solution for the production of muscle activity. Nat. Neurosci. 18, 1025–1033 (2015).

    Article  Google Scholar 

  148. Kriegeskorte, N., Mur, M. & Bandettini, P. Representational similarity analysis — connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008).

    Google Scholar 

  149. Wadhams, G. H. & Armitage, J. P. Making sense of it all: bacterial chemotaxis. Nat. Rev. Mol. Cell Biol. 5, 1024–1037 (2004).

    Article  Google Scholar 

  150. Frankel, N. W. et al. Adaptability of non-genetic diversity in bacterial chemotaxis. eLife 3, e03526 (2014).

    Article  Google Scholar 

  151. Waite, A. J. et al. Non-genetic diversity modulates population performance. Mol. Syst Biol. 12, 895 (2016).

    Article  Google Scholar 

  152. Vogels, T. P., Rajan, K. & Abbott, L. F. Neural network dynamics. Annu. Rev. Neurosci. 28, 357–376 (2005).

    Article  Google Scholar 

  153. Chen, X., Randi, F., Leifer, A. M. & Bialek, W. Searching for collective behavior in a small brain. Phys. Rev. E 99, 052418 (2019).

    Article  ADS  Google Scholar 

  154. Williams, A. H., Degleris, A., Wang, Y. & Linderman, S. W. Point process models for sequence detection in high-dimensional neural spike trains. Adv. Neural Inf. Proc. Syst. 33, 14350–14361 (NeurIPS, 2020).

Download references

Acknowledgements

The authors thank J. Kanwal, D. Zimmerman and V. Susoy for discussions. This work was partially supported by funding from the Simons Foundation SCGB 697092 and US National Institutes of Health (NIH) Brain Initiatives U19NS113201 and R01NS11311 awarded to S.W.L., US National Science Foundation (IOS-1452593) and NIH (R01 NS082525, R01 GM130842-01 and U01-NS111697) grants to A.D.T.S., and a Burroughs Wellcome Career Award and American Federation for Aging Research Junior Faculty Award to V.V.

Author information

Authors and Affiliations

  1. Department of Physics, Harvard University, Cambridge, MA, USA

    Albert Lin, Daniel Witvliet & Aravinthan D. T. Samuel

  2. Center for Brain Science, Harvard University, Cambridge, MA, USA

    Albert Lin, Daniel Witvliet, Luis Hernandez-Nunez, Aravinthan D. T. Samuel & Vivek Venkatachalam

  3. Center for the Physics of Biological Function, Princeton University, Princeton, NJ, USA

    Albert Lin

  4. Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA

    Albert Lin

  5. Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA

    Luis Hernandez-Nunez

  6. Department of Statistics, Stanford University, Stanford, CA, USA

    Scott W. Linderman

  7. Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA

    Scott W. Linderman

  8. Department of Physics, Northeastern University, Boston, MA, USA

    Vivek Venkatachalam

Authors
  1. Albert Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Witvliet
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luis Hernandez-Nunez
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Scott W. Linderman
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aravinthan D. T. Samuel
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Vivek Venkatachalam
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to all aspects of the Review.

Corresponding author

Correspondence to Aravinthan D. T. Samuel.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Reviews Physics thanks Joshua Shaevitz, Ralph Greenspan and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, A., Witvliet, D., Hernandez-Nunez, L. et al. Imaging whole-brain activity to understand behaviour. Nat Rev Phys 4, 292–305 (2022). https://doi.org/10.1038/s42254-022-00430-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42254-022-00430-w

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing