Real-time denoising enables high-sensitivity fluorescence time-lapse imaging beyond the shot-noise limit

A fundamental challenge in fluorescence microscopy is the photon shot noise arising from the inevitable stochasticity of photon detection. Noise increases measurement uncertainty and limits imaging resolution, speed and sensitivity. To achieve high-sensitivity fluorescence imaging beyond the shot-noise limit, we present DeepCAD-RT, a self-supervised deep learning method for real-time noise suppression. Based on our previous framework DeepCAD, we reduced the number of network parameters by 94%, memory consumption by 27-fold and processing time by a factor of 20, allowing real-time processing on a two-photon microscope. A high imaging signal-to-noise ratio can be acquired with tenfold fewer photons than in standard imaging approaches. We demonstrate the utility of DeepCAD-RT in a series of photon-limited experiments, including in vivo calcium imaging of mice, zebrafish larva and fruit flies, recording of three-dimensional (3D) migration of neutrophils after acute brain injury and imaging of 3D dynamics of cortical ATP release. DeepCAD-RT will facilitate the morphological and functional interrogation of biological dynamics with a minimal photon budget.


CONTENT I. Supplementary Figures
Supplementary Figure 1 In silico simulation of calcium imaging data.

Supplementary Figure 2
Evaluating the performance of different model complexity.

Supplementary Figure 3
Evaluating the data dependency of DeepCAD-RT.

Supplementary Figure 4
Comparing DeepCAD-RT with other methods at different noise levels.

Supplementary Figure 5
Representative images for method comparison.

Supplementary Figure 6
Performance comparison between DeepCAD-RT and DeepInterpolation on neutrophil imaging data.

Supplementary Figure 7
System point spread function (PSF).

Supplementary Figure 8
ATP annotation pipeline. Tables   Supplementary Table 1 Comparison of different model complexity.

Supplementary Table 2
Parameters for the simulation of calcium imaging data.

III. Supplementary Videos
Supplementary Video 1 Demonstrating real-time denoising on a two-photon microscope using DeepCAD-RT.

Supplementary Video 2
DeepCAD-RT enhances the in vivo recording of calcium transients in dendritic spines.

Supplementary Video 3
DeepCAD-RT massively improves the imaging SNR of neuronal population recordings in the zebrafish brain.

Supplementary Video 4
DeepCAD-RT massively improves the imaging SNR of neuronal population recordings across multiple brain regions in the zebrafish brain.

Supplementary Video 5
DeepCAD-RT enhances the neuronal population imaging of Drosophila mushroom body.

Supplementary Video 6
Denoising performance of DeepCAD-RT on two-photon imaging of neutrophils in the mouse brain.

Supplementary Video 7
DeepCAD-RT facilitates high-SNR observations of retraction fiber dynamics during neutrophil migration.

Supplementary Video 8
DeepCAD-RT reveals the 3D migration of neutrophils in vivo after acute brain injury.

Supplementary Video 9
Denoising performance of DeepCAD-RT on a recently developed genetically encoded ATP (adenosine 5'-triphosphate) sensor.

Supplementary Figure 1
In silico simulation of calcium imaging data.
Noise-free two-photon calcium imaging videos were simulated with in silico Neural Anatomy and Optical Microscopy (NAOMi) 1 . Different levels of Mixed Poisson-Gaussian (MPG) noise 2, 3 were added subsequently. Noise-free images were used as the ground truth for quantitative evaluations of the denoising performance. a, The relation between imaging SNR and relative photon number (Q). b, Representative images of different imaging SNRs and corresponding ground truth (GT). Scale bar, 50 μm.

Supplementary Figure 2
Evaluating the performance of different model complexity.
Simulated calcium imaging data (6000 frames, 30 Hz frame rate, SNR=-2.51 dB) were used in this experiment for quantitative evaluation. The best training epoch was selected by validation. a, The relation between denoising performance (output SNR) and the number of model parameters. The line shows mean values and error bars represent the minimum and maximum values. b, Example ground truth (GT) images, raw data before denoising, and images after denoising. Scale bar, 100 μm.

Supplementary Figure 3
Evaluating the data dependency of DeepCAD-RT.
Simulated calcium imaging data (30 Hz frame rate, SNR=-2.51 dB) were used in this experiment for quantitative evaluation. The network architecture has been simplified (~1.0 million trainable parameters) and data augmentation was applied. Each model was trained for 20 epochs and the last epoch was used for comparison. a, The relation between denoising performance (output SNR) and the number of training frames (N). The line shows mean values and error bars represent the minimum and maximum values. b, Example ground truth (GT) images, raw data before denoising, and images after denoising. Scale bar, 100 μm.

Supplementary Figure 4
Comparing DeepCAD-RT with other methods at different noise levels.
The relationship between the input SNR and output SNR of different denoising methods. Simulated calcium imaging data (6000 frames, 30 Hz frame rate) were used for the training of all methods. DeepCAD-RT (simplified network architecture with ~1.0 million trainable parameters) was trained from scratch for 20 epochs with data augmentation. DeepInterpolation was implemented with the companion code of relevant papers 4 and two kinds of DeepInterpolation models were trained. The first one was trained from scratch. The other model was fine-tuned based on a pre-trained model by presenting the training data only once according to the DeepInterpolation paper. The supervised baseline was obtained with a 3D-Unet (4.1 M trainable parameters) trained in a supervised manner. Noise2Void 5 and Hierarchical DivNoising (HDN) 6 are 2D methods and were implemented with the companion code of relevant papers. Each Noise2Void model was trained for 50 epochs. Each HDN model was trained for 150 epochs and the minimum mean square error (MMSE) estimate of each frame was obtained by averaging 100 denoised samples. For each method, a specified model was trained for each SNR level. Lines represent mean values and error bars represent the minimum and maximum values. The black dashed line represents the shot-noise limit (i.e., the input SNR is equal to the output SNR).

Supplementary Figure 5
Representative images for method comparison. First, low-SNR recordings (4D, xyz-t) were denoised with DeepCAD-RT. Denoised data were visualization in Imaris software. Second, all ATP-release events during the whole imaging session were manually annotated to obtain their position and time. Then, intensity profiles along all three dimensions of each event were extracted from the denoised data. Finally, Gaussian fitting was performed and fitted curves were deconvoluted with the Richardson-Lucy algorithm to eliminate the influence of limited and anisotropic spatial resolution. The full width at half maximum (FWHM) of each deconvoluted curve was extracted as the size of the event.

Comparison of different model complexity.
We compared the memory cost, time consumption, and denoising performance of different model complexity using simulated calcium imaging data with clean images for better quantification. All hyper-parameters were kept unchanged except the total