Inference of action potentials (‘spikes’) from neuronal calcium signals is complicated by the scarcity of simultaneous measurements of action potentials and calcium signals (‘ground truth’). In this study, we compiled a large, diverse ground truth database from publicly available and newly performed recordings in zebrafish and mice covering a broad range of calcium indicators, cell types and signal-to-noise ratios, comprising a total of more than 35 recording hours from 298 neurons. We developed an algorithm for spike inference (termed CASCADE) that is based on supervised deep networks, takes advantage of the ground truth database, infers absolute spike rates and outperforms existing model-based algorithms. To optimize performance for unseen imaging data, CASCADE retrains itself by resampling ground truth data to match the respective sampling rate and noise level; therefore, no parameters need to be adjusted by the user. In addition, we developed systematic performance assessments for unseen data, openly released a resource toolbox and provide a user-friendly cloud-based implementation.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
International Journal of Computer Vision Open Access 19 December 2022
Machine Intelligence Research Open Access 14 July 2022
Calcium Imaging and Electrophysiology of hippocampal Activity under Anesthesia and natural Sleep in Mice
Scientific Data Open Access 29 March 2022
Subscribe to Nature+
Get immediate online access to Nature and 55 other Nature journal
Subscribe to Journal
Get full journal access for 1 year
only $6.58 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
Ground truth data, including extracted spike times and calcium traces, are deposited in the GitHub repository together with demo scripts (https://github.com/HelmchenLabSoftware/Cascade). We provide a cloud-based Colaboratory Notebook that allows for interactive browsing through all datasets (https://colab.research.google.com/github/HelmchenLabSoftware/Cascade/blob/master/Demo%20scripts/Explore_ground_truth_datasets.ipynb). Raw data were recorded in different formats, and all newly recorded raw datasets are also available upon reasonable request in their original formats. Publicly available datasets are described in detail in the Methods (‘Extraction of ground truth from publicly available datasets’).
Additional information on experimental design and reagents is available in the Research Life Sciences Reporting Summary linked to this paper.
A cloud-based version of CASCADE is available as a Colaboratory Notebook (https://colab.research.google.com/github/HelmchenLabSoftware/Cascade/blob/master/Demo%20scripts/Calibrated_spike_inference_with_Cascade.ipynb). The code is also available as a GitHub repository together with demo scripts, installation instructions and FAQs (https://github.com/HelmchenLabSoftware/Cascade). Pre-trained models for CASCADE are archived in an online server (https://www.switch.ch/drive/) and retrieved automatically by the CASCADE code.
Göbel, W. & Helmchen, F. In vivo calcium imaging of neural network function. Physiology 22, 358–365 (2007).
Harris, K. D., Quiroga, R. Q., Freeman, J. & Smith, S. L. Improving data quality in neuronal population recordings. Nat. Neurosci. 19, 1165–1174 (2016).
Rose, T., Goltstein, P. M., Portugues, R. & Griesbeck, O. Putting a finishing touch on GECIs. Front. Mol. Neurosci. 7, 88 (2014).
Sabatini, B. L. The impact of reporter kinetics on the interpretation of data gathered with fluorescent reporters. Preprint at https://www.biorxiv.org/content/10.1101/834895v1 (2019).
Wei, Z. et al. A comparison of neuronal population dynamics measured with calcium imaging and electrophysiology. PLoS Comput. Biol. 16, e1008198 (2020).
Ali, F. & Kwan, A. C. Interpreting in vivo calcium signals from neuronal cell bodies, axons, and dendrites: a review. Neurophotonics 7, 011402 (2020).
Yaksi, E. & Friedrich, R. W. Reconstruction of firing rate changes across neuronal populations by temporally deconvolved Ca2+ imaging. Nat. Methods 3, 377–383 (2006).
Greenberg, D. S., Houweling, A. R. & Kerr, J. N. D. Population imaging of ongoing neuronal activity in the visual cortex of awake rats. Nat. Neurosci. 11, 749–751 (2008).
Vogelstein, J. T. et al. Spike inference from calcium imaging using sequential Monte Carlo methods. Biophys. J. 97, 636–655 (2009).
Vogelstein, J. T. et al. Fast nonnegative deconvolution for spike train inference from population calcium imaging. J. Neurophysiol. 104, 3691–3704 (2010).
Lütcke, H., Gerhard, F., Zenke, F., Gerstner, W. & Helmchen, F. Inference of neuronal network spike dynamics and topology from calcium imaging data. Front. Neural Circuits 7, 201 (2013).
Deneux, T. et al. Accurate spike estimation from noisy calcium signals for ultrafast three-dimensional imaging of large neuronal populations in vivo. Nat. Commun. 7, 1–17 (2016).
Greenberg, D. S. et al. Accurate action potential inference from a calcium sensor protein through biophysical modeling. Preprint at https://www.biorxiv.org/content/10.1101/479055v1 (2018).
Pachitariu, M., Stringer, C. & Harris, K. D. Robustness of spike deconvolution for neuronal calcium imaging. J. Neurosci. 38, 7976–7985 (2018).
Friedrich, J., Zhou, P. & Paninski, L. Fast online deconvolution of calcium imaging data. PLoS Comput. Biol. 13, e1005423 (2017).
Berens, P. et al. Community-based benchmarking improves spike rate inference from two-photon calcium imaging data. PLoS Comput. Biol. 14, e1006157 (2018).
Jewell, S. & Witten, D. Exact spike inference via l0 optimization. Ann. Appl. Stat. 12, 2457–2482 (2018).
Sasaki, T., Takahashi, N., Matsuki, N. & Ikegaya, Y. Fast and accurate detection of action potentials from somatic calcium fluctuations. J. Neurophysiol. 100, 1668–1676 (2008).
Theis, L. et al. Benchmarking spike rate inference in population calcium imaging. Neuron 90, 471–482 (2016).
Sebastian, J., Sur, M., Murthy, H. A. & Magimai-Doss, M. Signal-to-signal neural networks for improved spike estimation from calcium imaging data. PLoS Comput. Biol. 17, e1007921 (2021).
Hoang, H. et al. Improved hyperacuity estimation of spike timing from calcium imaging. Sci. Rep. 10, 17844 (2020).
Éltes, T., Szoboszlay, M., Kerti-Szigeti, K. & Nusser, Z. Improved spike inference accuracy by estimating the peak amplitude of unitary [Ca2+] transients in weakly GCaMP6f-expressing hippocampal pyramidal cells. J. Physiol. 597, 2925–2947 (2019).
Evans, M. H., Petersen, R. S. & Humphries, M. D. On the use of calcium deconvolution algorithms in practical contexts. Preprint at https://www.biorxiv.org/content/10.1101/871137v1 (2019).
Zhu, P., Fajardo, O., Shum, J., Zhang Schärer, Y.-P. & Friedrich, R. W. High-resolution optical control of spatiotemporal neuronal activity patterns in zebrafish using a digital micromirror device. Nat. Protoc. 7, 1410–1425 (2012).
Schoenfeld, G., Carta, S., Rupprecht, P., Ayaz, A. & Helmchen, F. In vivo calcium imaging of CA3 pyramidal neuron populations in adult mouse hippocampus. Preprint at https://www.biorxiv.org/content/10.1101/2021.01.21.427642v1 (2021).
Bethge, P. et al. An R-CaMP1.07 reporter mouse for cell-type-specific expression of a sensitive red fluorescent calcium indicator. PLoS ONE 12, e0179460 (2017).
Tada, M., Takeuchi, A., Hashizume, M., Kitamura, K. & Kano, M. A highly sensitive fluorescent indicator dye for calcium imaging of neural activity in vitro and in vivo. Eur. J. Neurosci. 39, 1720–1728 (2014).
Khan, A. G. et al. Distinct learning-induced changes in stimulus selectivity and interactions of GABAergic interneuron classes in visual cortex. Nat. Neurosci. 21, 851–859 (2018).
Kwan, A. C. & Dan, Y. Dissection of cortical microcircuits by single-neuron stimulation in vivo. Curr. Biol. CB 22, 1459–1467 (2012).
Huang, L. et al. Relationship between simultaneously recorded spiking activity and fluorescence signal in GCaMP6 transgenic mice. Elife 10, e51675 (2021).
Ledochowitsch, P. et al. On the correspondence of electrical and optical physiology in in vivo population-scale two-photon calcium imaging. Preprint at https://www.biorxiv.org/content/10.1101/800102v1 (2019).
Dana, H. et al. Sensitive red protein calcium indicators for imaging neural activity. eLife 5, e12727 (2016).
Akerboom, J. et al. Optimization of a GCaMP calcium indicator for neural activity imaging. J. Neurosci. 32, 13819–13840 (2012).
Chen, T.-W. et al. Ultrasensitive fluorescent proteins for imaging neuronal activity. Nature 499, 295–300 (2013).
Russakovsky, O. et al. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2015).
Mathis, A. et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 21, 1281–1289 (2018).
Deng, J. et al. ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition https://ieeexplore.ieee.org/document/5206848 (2009).
Giovannucci, A. et al. CaImAn an open source tool for scalable calcium imaging data analysis. eLife 8, e38173 (2019).
Keemink, S. W. et al. FISSA: a neuropil decontamination toolbox for calcium imaging signals. Sci. Rep. 8, 3493 (2018).
Charles, A. S., Song, A., Gauthier, J. L., Pillow, J. W. & Tank, D. W. Neural anatomy and optical microscopy (NAOMi) simulation for evaluating calcium imaging methods. J. Neurosci. Methods 358, 109173 (2019).
Pachitariu, M. et al. Suite2p: beyond 10,000 neurons with standard two-photon microscopy. Preprint at https://www.biorxiv.org/content/10.1101/061507v2 (2017).
Jewell, S., Hocking, T. D., Fearnhead, P. & Witten, D. Fast nonconvex deconvolution of calcium imaging data. Biostatistics 21, 709–726 (2019).
Rupprecht, P., Prendergast, A., Wyart, C. & Friedrich, R. W. Remote z-scanning with a macroscopic voice coil motor for fast 3D multiphoton laser scanning microscopy. Biomed. Opt. Express 7, 1656–1671 (2016).
Blumhagen, F. et al. Neuronal filtering of multiplexed odour representations. Nature 479, 493–498 (2011).
Rupprecht, P. & Friedrich, R. W. Precise synaptic balance in the zebrafish homolog of olfactory cortex. Neuron 100, 669–683.e5 (2018).
Mackevicius, E. L. et al. Unsupervised discovery of temporal sequences in high-dimensional datasets, with applications to neuroscience. eLife 8, e38471 (2019).
de Vries, S. E. J. et al. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nat. Neurosci. 23, 138–151 (2020).
Lin, I.-C., Okun, M., Carandini, M. & Harris, K. D. The nature of shared cortical variability. Neuron 87, 644–656 (2015).
Kaifosh, P., Zaremba, J. D., Danielson, N. B. & Losonczy, A. SIMA: Python software for analysis of dynamic fluorescence imaging data. Front. Neuroinformatics 8, 80 (2014).
Siegle, J. H. et al. Reconciling functional differences in populations of neurons recorded with two-photon imaging and electrophysiology. Preprint at https://www.biorxiv.org/content/10.1101/2020.08.10.244723v1.full (2020).
Vanwalleghem, G., Constantin, L. & Scott, E. K. Calcium imaging and the curse of negativity. Front. Neural Circuits 14, 607391 (2021).
Kay, K. et al. Constant sub-second cycling between representations of possible futures in the hippocampus. Cell 180, 552–567 (2020).
van der, Bourg,A. et al. Temporal refinement of sensory-evoked activity across layers in developing mouse barrel cortex. Eur. J. Neurosci. 50, 2955–2969 (2019).
Pégard, N. C. et al. Three-dimensional scanless holographic optogenetics with temporal focusing (3D-SHOT). Nat. Commun. 8, 1228 (2017).
Packer, A. M., Russell, L. E., Dalgleish, H. W. P. & Häusser, M. Simultaneous all-optical manipulation and recording of neural circuit activity with cellular resolution in vivo. Nat. Methods 12, 140–146 (2015).
Griffiths, V. A. et al. Real-time 3D movement correction for two-photon imaging in behaving animals. Nat. Methods 17, 741–748 (2020).
Inoue, M. et al. Rational engineering of XCaMPs, a multicolor GECI suite for in vivo imaging of complex brain circuit dynamics. Cell 177, 1346–1360 (2019).
Frank, T., Mönig, N. R., Satou, C., Higashijima, S. & Friedrich, R. W. Associative conditioning remaps odor representations and modifies inhibition in a higher olfactory brain area. Nat. Neurosci. 22, 1844–1856 (2019).
Kitamura, K., Judkewitz, B., Kano, M., Denk, W. & Häusser, M. Targeted patch-clamp recordings and single-cell electroporation of unlabeled neurons in vivo. Nat. Methods 5, 61–67 (2008).
Perkins, K. L. Cell-attached voltage-clamp and current-clamp recording and stimulation techniques in brain slices. J. Neurosci. Methods 154, 1–18 (2006).
Pologruto, T. A., Sabatini, B. L. & Svoboda, K. ScanImage: flexible software for operating laser scanning microscopes. Biomed. Eng. Online 2, 13 (2003).
Suter, B. A. et al. Ephus: multipurpose data acquisition software for neuroscience experiments. Front. Neural Circuits 4, 100 (2010).
Huang, K.-H. et al. A virtual reality system to analyze neural activity and behavior in adult zebrafish. Nat. Methods 17, 343–351 (2020).
Langer, D. et al. HelioScan: a software framework for controlling in vivo microscopy setups with high hardware flexibility, functional diversity and extendibility. J. Neurosci. Methods 215, 38–52 (2013).
Pecka, M., Han, Y., Sader, E. & Mrsic-Flogel, T. D. Experience-dependent specialization of receptive field surround for selective coding of natural scenes. Neuron 84, 457–469 (2014).
Pernía-Andrade, A. J. et al. A deconvolution-based method with high sensitivity and temporal resolution for detection of spontaneous synaptic currents in vitro and in vivo. Biophys. J. 103, 1429–1439 (2012).
Guzman, S. J., Schlögl, A. & Schmidt-Hieber, C. Stimfit: quantifying electrophysiological data with Python. Front. Neuroinformatics 8, 16 (2014).
GENIE project, Janelia Farm Campus, HHMI & Svoboda, K. Simultaneous imaging and loose-seal cell-attached electrical recordings from neurons expressing a variety of genetically encoded calcium indicators. https://crcns.org/data-sets/methods/cai-1/about-cai-1 (2015).
Boaz, M., Dana, H., Kim, D. S., Svoboda, K. & GENIE project, Janelia Farm Campus, HHMI. jRGECO1a and jRCaMP1a characterization in the intact mouse visual cortex, using AAV-based gene transfer, 2-photon imaging and loose-seal cell attached recordings. https://crcns.org/data-sets/methods/cai-2/about-cai-2 (2016).
Reynolds, S., Abrahamsson, T., Sjöström, P. J., Schultz, S. R. & Dragotti, P. L. CosMIC: a consistent metric for spike inference from calcium imaging. Neural Comput. 30, 2726–2756 (2018).
Ioffe, S. & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. 448-456 (PMLR, 2015).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).
Gers, F. A., Schmidhuber, J. & Cummins, F. Learning to forget: continual prediction with LSTM. Neural Comput. 12, 2451–2471 (1999).
Schuster, M. & Paliwal, K. Bidirectional recurrent neural networks. Signal Process. IEEE Trans. 45, 2673–2681 (1997).
Graves A., Fernández S., Schmidhuber J. Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition. Duch W., Kacprzyk J., Oja E., Zadrożny S. (eds) Artificial Neural Networks: Formal Models and Their Applications – ICANN 2005. Lecture Notes in Computer Science, vol 3697. (Springer, Berlin, Heidelberg, 2005).
Eden, U. T. & Kramer, M. A. Drawing inferences from Fano factor calculations. J. Neurosci. Methods 190, 149–152 (2010).
We thank the members of the GENIE project, the Allen Institute and the Spikefinder project for publicly providing existing ground truth datasets together with excellent documentation. We thank P. Berens and E. Froudarakis for providing additional information on the Spikefinder datasets. We thank G. Schoenfeld for helpful discussions on DS #18 and H. Heiser, N. Temiz, C. Satou, G. Schoenfeld and H. Luetcke for testing earlier versions of the toolbox. This work was supported by grants to F.H. from the Swiss National Science Foundation (project grant no. 310030-127091 and Sinergia grant no. CRSII5-18O316) and the European Research Council (ERC Advanced Grant BRAINCOMPATH, grant agreement no. 670757); by grants to K.K. from MEXT, Japan (Scientific Research for Innovative Areas, no. 17H06313); by grants to R.W.F. from the Swiss National Science Foundation (project grant no. 310030B-152833/1) and the European Research Council (ERC Advanced Grant MCircuits, grant agreement no. 742576); by the Novartis Research Foundation; by a UZH Forschungskredit and a fellowship from the Boehringer Ingelheim Fonds to P.R.
The authors declare no competing financial interests.
Peer review information Nature Neuroscience thanks the anonymous reviewers for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The kernels are optimized such that when the ground truth spike times are linearly convolved with the kernel, the experimentally recorded ΔF/F trace is ideally approximated. In practice, this is achieved using regularized linear deconvolution of calcium traces based on spike times (Methods). Kernels vary both in amplitude and shape across datasets and within datasets. For single neurons, the kernel area (right panels) is only shown if the kernel could be reliably determined, as tested with the variability of the kernel across the recording (Methods). The red arrow in panel (r) indicates an outlier case that is discussed in Extended Data Fig. 4a. m: Mouse, zf: Zebrafish.
ΔF/F ground truth traces were resampled with added noise to reach the target noise level ν. a-d, Noise level illustration from ν = 15 (very high noise level) to ν = 1 (very low noise level). Standardized noise ν is given in units of %·Hz−1/2.
Same as Fig. 2e–g, but with each column (testing level) normalized in order to highlight that the optimal training level for each testing noise level lies close to the diagonal. The correlation (a) was normalized by the maximum of each column, while error and bias metrics have been normalized by the minimum of each column. ν in units of standardized noise, %·Hz−1/2.
The deep network was trained on all neurons of a specific dataset except one, and then tested with the remaining neuron. This analysis shows how the network is able to generalize to new neurons recorded under the same conditions, as a function of the standardized noise level ν in %·Hz−1/2. a-d, Performance of the predictions for 4 selected ground truth datasets in terms of correlation, error and bias as a function of the standardized noise level. Error values were cropped at a value of 5 for display purposes. Single neurons in grey, median across neurons in blue. Grey lines highlighted by arrows indicate outlier neurons with particularly low spike rates (black and green arrows) and particularly distinct calcium response kernel (red arrow, see main text for discussion). e, Correlation, error and biases as a distribution across neurons within each dataset (number of neurons for each dataset as indicated in Table 1). For box plots, the median is indicated by the central line, 25th and 75th percentiles by the box, and maximum/minimum values excluding outliers (points) by the whiskers. All datasets were re-sampled at a frame rate of 7.5 Hz.
Calcium trace (ΔF/F), true action potentials (APs), inferred spiking activity (SR) and true ground truth spiking activity (GT). a, The baseline of this recording is unstable, exhibiting irregular bumps (arrowheads). The supervised deep network can learn to ignore these movement artifacts if their dynamics is dissimilar from the sharp onset of calcium transients. Predictions of the deep network are shown in black, ground truth in grey. Green arrowheads indicate movement artifacts that are not associated with high spiking acitivity (correct rejections of artifacts), while black arrowheads indicate movement artifacts that are not recognized as artifacts by the network (false positives). The zoom-in on the right shows an example where a movement artifact is associated with a negligeable spike rate (correct rejection). b, Fluorescence transients without corresponding action potentials are clearly visible (red arrowheads). These are induced by contamination through bright neuropil. The deep network is unable to distinguish this artifact from true calcium transients. c, Negative transients (arrowheads) are generated by standard neuropil decontamination (subtraction of the neuropil surround). The deep network can learn to partially ignore these events (correct rejections). d, Trace showing periodic movement artifacts that do not correspond to action potentials. e, A power spectral density of the recording in (d) exhibits a peak at ca. 1.5 Hz, suggesting breathing of the anaesthetized animal underlying the movement artifact.
The global EXC model (see Fig. 3) was trained as before, but using only a subset of the ground truth data points (x-axis). The performance (correlation) across each dataset was normalized to the performance with 5 million data points (horizontal dashed line). The performance approaches an asymptote at approximately 100,000 data points. A typical single ground truth dataset contains ca. 400,000 data points (median across all datasets; vertical dashed line). This result also indicates that a diverse but smaller training dataset sampled from all ground truth datasets results in better generalization than a larger training dataset from a single ground truth dataset.
Example predictions from the deep-learning based method (CASCADE) and five model-based algorithms (MLSpike, CaImAn, Peeling, Suite2p, Jewell&Witten) of a ΔF/F recording. Inferred spike rates are in black, ground truth spike rates in orange. r indicates correlation of predictions with ground truth. Events that are not detected across all algorithms (false negatives) are labeled with red arrowheads. Compared to the example in Fig. 4a, the calcium recording here is rather noisy due to the insensitivity of GCaMP to single action potentials in this neuron.
Comparison of the six algorithms when optimized for a single dataset, showing relative error and relative bias for all neurons, grouped by ground truth dataset.
All algorithms were optimized via the mean squared error to infer spike rates at a specific temporal precision defined by the smoothing of the ground truth (default: Gaussian smoothing with kernel of σ = 200 ms). For all model-based algorithms, the inferred spike traces were shifted in time to optimize the mean squared error. a, Predictions from an example ΔF/F trace (top; dataset #09). Ground truth spike rates are shown in orange, inferred spike rates as black overlay. Correlation values are indicated at the right. The scale bars for ΔF/F and time are the same as in Fig. 4a. b, Highlighted excerpt from (a). Due to the high temporal precisions of the inferred spike rates, small time shifts lead to low performance (clearly visible for the Peeling algorithm in this example). The CaImAn and Suite2p algorithms deconvolve less aggressively, therefore making less dramatic errors. CASCADE and MLSpike perform best for this example neuron, with CASCADE detecting more events than MLSpike. c, Overall performance (correlation) change with temporal precision of predictions (smoothing kernels shown below) on a subset of datasets (datasets #4, #6, #9, #11-14 and #18). As expected, correlation with ground truth decreased with higher temporal resolution of the desired temporal resolution. This decrease was especially prominent for algorithms that, by design, aim at the inference of precise (discrete) spike rates (Peeling, Jewell&Witten). The decrease was less pronounced for CASCADE compared to for example MLSpike. Shaded corridors indicate SEM across n = 8 datasets. All recordings resampled at a noise level of 2 with a frame rate of 7.5 Hz.
Extended Data Fig. 10 Predictions of spiking probabilities and discrete spikes from the Allen Brain Institute Visual Coding dataset.
Predictions were produced with the global EXC model trained at 30 Hz. From dataset ID ‘552195520’, plotting a total of 40 neurons out of 74, approximately 1 minute out of 63.2 minutes of recording for this dataset. Discrete spikes are the most likely fit, generated with an algorithm using Metropolis-Monte Carlo sampling as starting point (see Methods).
About this article
Cite this article
Rupprecht, P., Carta, S., Hoffmann, A. et al. A database and deep learning toolbox for noise-optimized, generalized spike inference from calcium imaging. Nat Neurosci 24, 1324–1337 (2021). https://doi.org/10.1038/s41593-021-00895-5
This article is cited by
Calcium Imaging and Electrophysiology of hippocampal Activity under Anesthesia and natural Sleep in Mice
Scientific Data (2022)
A deep learning framework for inference of single-trial neural population dynamics from calcium imaging with subframe temporal resolution
Nature Neuroscience (2022)
Nature Reviews Methods Primers (2022)
Nature Reviews Neuroscience (2022)
International Journal of Computer Vision (2022)