Image restoration of degraded time-lapse microscopy data mediated by near-infrared imaging

Time-lapse fluorescence microscopy is key to unraveling biological development and function; however, living systems, by their nature, permit only limited interrogation and contain untapped information that can only be captured by more invasive methods. Deep-tissue live imaging presents a particular challenge owing to the spectral range of live-cell imaging probes/fluorescent proteins, which offer only modest optical penetration into scattering tissues. Herein, we employ convolutional neural networks to augment live-imaging data with deep-tissue images taken on fixed samples. We demonstrate that convolutional neural networks may be used to restore deep-tissue contrast in GFP-based time-lapse imaging using paired final-state datasets acquired using near-infrared dyes, an approach termed InfraRed-mediated Image Restoration (IR2). Notably, the networks are remarkably robust over a wide range of developmental times. We employ IR2 to enhance the information content of green fluorescent protein time-lapse images of zebrafish and Drosophila embryo/larval development and demonstrate its quantitative potential in increasing the fidelity of cell tracking/lineaging in developing pescoids. Thus, IR2 is poised to extend live imaging to depths otherwise inaccessible.

and may vary depending on the depth in tissue or the depth inside the agarose bead column for which the refocus calibration is made.
Moreover, the refocusing does not correct for lateral chromatic aberrations.Since the IR 2 training is reliant on the 1:1 correspondence between different color channels, residual misalignments must be corrected for.Any residual axial/lateral chromatic aberrations are accounted for via a registration step (see "Deep Learning" in Methods).As a demonstration, a two-color fluorescent bead sample is shown in Supplementary Figure 1.As shown in A-C, there is a noticeable misalignment between the two color channels.The misalignment is smaller in z, highlighting that the refocus scheme broadly corrects for axial color.Following registration, the beads are well aligned (D-E).
Supplementary Figure 1 The fluorescent beads used are sub-diffraction-sized, providing a route to explore the resolving performance of the IR-SPIM via the PSF.The scaling of lateral spatial resolution with wavelength would suggest that the performance is highly dependent on the specific emission band of the fluorophore and, hence, that the IR dyes would provide about half the resolving capability of GFP.However, SPIM often employs undersampling in the imaging path (with respect to NA), to allow for a larger field of view while maintaining the light collection and axial resolution benefits of high NA.The data presented are captured at ca. 11.1⨉ and 22.2⨉.The magnification can be switched between these two values by exchanging the tube lens (effl = 200 or 400 mm for 11.1⨉, 22.2⨉, respectively, when paired with an Olympus 10⨉ objective, effl = 18 mm).In the former case, given a pixel size of 6.5 µm, the Nyquist-Shannon criterion provides a sampling-determined resolution limit of ca.1.2 µm.Using the Rayleigh criterion for resolution r: And taking  0 in each case as the pass-band center (525, 845 nm respectively for GFP, AF800/CF800), the expected resolution assuming diffraction-limited performance at NA = 0.6 is ca.534, 859 nm, respectively, both of which are below the Nyquist-Shannon criterion.Consequently, it is sampling rather than wavelength that determines the spatial resolution.However, in the latter case, the Nyquist-Shannon criterion provides a sampling-determined resolution limit of ca.0.6 µm, which could allow diffraction-limited resolution for wavelengths > 590 nm and a substantial difference in resolution for GFP, AF800/CF800, respectively.The assumption made is that the imaging system performs in a diffraction-limited manner over the full wavelength range.The illumination system may be safely disregarded for consideration of lateral resolution; however, the imaging system formed by the imaging objective lens (Olympus XLPLNS10XSSVMP 10x/0.6, 8 mm WD) and ultra-broadband tube lens (Thorlabs, TTL200MP/AC508-400-AB-ML) may not provide diffraction-limited resolution, particularly the objective, which is optimized for two-photon imaging (most commonly ca.920 nm).Given that axial chromatic aberrations are substantial for wavelengths < 700 nm, it is reasonable to also expect a decrease in the effective NA for shorter visible wavelengths and a convergence in the resolution for GFP and CF800/AF800.To explore whether this was apparent, bead stacks were analyzed using PSFj 1 .The lower bound for the lateral PSF FWHM for excitation/emission centers of 488/525 nm, 640/697 nm, 808/845 nm was found to be 741 ± 14, 797 ± 15, and 974 ± 18 nm, which are equivalent to maximum NAs of 0.44, 0.54, 0.54 respectively (from the Rayleigh criterion), demonstrating that the optical performance is best in the farred -NIR as expected and approaches the theoretical value of 0.6.Although the lateral PSF for the GFP equivalent is slightly narrower, we note that this will be apparent only in extremely superficial regions.In any case, the difference is minor, and cell nuclei (ca.5-10 µm), which are the smallest structural details that we sought to resolve, were easily resolvable for GFP, AF647, and CF800 in both sparsely labeled samples such as pescoids and Drosophila embryos, where the spacing between nuclei is substantially smaller (ca.2-5 µm) as well as densely packed tissues, including the brain in the zebrafish larvae.The axial PSF FWHM is also similar for the three imaging bands: 5.46 ± 0.22, 5.36 ± 0.25 and 6.38 ± 0.41 µm for 488/525 nm, 640/697 nm, 808/845 nm.In the future, IR 2 could be combined with deconvolution strategies to improve the axial resolving power of multi-view light sheet microscopy as necessary 2 .

Fixation, permeabilization and staining strategies.
The deep learning-based restoration requires as close to 1:1 correspondence between i) the live and fixed state of the animal and ii) the distributions of GFP and the near-infrared dye therein post-staining.Several key challenges are apparent.Firstly, the fixation should maintain GFP fluorescence and not enhance autofluorescence.Secondly, the permeabilization step should allow antibody penetration but not distort or degrade the sample morphology.Thirdly, the staining step should allow permeation of the antibodies throughout the tissue to evenly stain without non-specific binding.
To ensure these requirements were met, various approaches for fixation, permeabilization, and tissue staining were tested and optimized for zebrafish samples.More generally, PFA fixation was found to be suitable for maintaining GFP fluorescence in all cases presented.However, the quenching of aldehydes via glycine washing reduced fixation-induced autofluorescence substantially.Various permeabilization steps were explored, including organic solvents (methanol/acetone), nonionic surfactants (Tween/Triton/DMSO), and proteinases (trypsin/proteinase K).Specifically, we tested reported protocols for zebrafish staining with slight modifications [3][4][5] .In some cases, antibody staining was lengthened to 7 days primary, 7 days secondary in an unsuccessful attempt to improve penetration.We found the standard protocol discussed in the methods section was sufficient for penetration of the vasculature label (Tg(kdrl:GFP)), best preserved the structure, and gave good labeling fidelity for a number of dyes.Nevertheless, the results from some dyes/affinity-tags were better than others.A large number of IR dye candidates were tested, and protocols utilizing primary antibodies only, primary and secondary antibodies, as well as nanobodies were assessed.Note that there are many important characteristics of a dye and affinity tag, such as brightness, photostability, excitation/emission maxima, solubility, specificity (IR dyes are typically large lipophilic molecules bearing several charged functional groups that offer watersolubility at a potential cost to specificity), staining time required, and completeness of staining.It was not possible to complete a full combinatorial assessment of all dyes and affinity tags independently, rather, we were limited by the commercially available options and cost constraints.The dyes, antibodies, and nanobodies tested that result in inferior (dimmer/less specific) staining than the case presented for the transgenic vasculature line Tg(kdrl:GFP) in Figure 1 are given in Supplementary Tables 1 and 2. We note, that other lines, permeabilization, and staining strategies or model organisms may provide better results with these products.None of the protocols attempted provided anything more than superficial penetration in the nuclear label (Tg(h2b:GFP) when used with antibody labeling.The custom-labeled CF800 nanobody (see Methods section), however, provided excellent staining in zebrafish and drosophila alike. of zebrafish and drosophila larva/embryos, respectively.For approximately equivalent light-sheet imaging schemes, the photon burden experienced by the live sample can be described by the laser power summed for the total number of exposures over the duration of the experiment.For the zebrafish (ca.72 hrs, dt = 300 s, one color channel) and drosophila (ca.24 hrs, dt = 300 s, one color channel) data presented, we calculate 1,848 and 275 mW equivalent exposures, respectively.For comparison, Schmid et al. used 17,280 mW equivalent exposures for early imaging of zebrafish embryogenesis (6 mW, ca. 12 hrs, dt = 30 s, one color channel) 8 .Shah et al. used 13,824 mW for conceptually similar imaging (8 mW, ca. 12 hrs, dt = 150 s, three color channels) 9 .Weber et al. used 2 mW at the illumination objective back aperture (ca.1.6 mW at the object assuming 80% transmission through the objective lens), which was lower than the 5 mW (4 mW at the object) threshold for a measurable increase in heart rate in the zebrafish embryo/larva 10 .Chhetri et al. performed developmental imaging of drosophila using 2,160 mW equivalent exposures (0.1 mW, ca. 3 hrs, dt = 4 s, two color channels).In this case since the light sheet was produced temporally by line scanning, the instantaneous or peak intensity is far higher than the other examples, or indeed the studies presented herein, making direct comparison difficult 11 .Despite the challenges in directly comparing the different studies, we note that the <2.1 mW laser power used at 488 nm for zebrafish is below the threshold shown by Weber et al. which is below the threshold shown by Weber et al. for perturbation of the heart beat period.The mW equivalent exposures being substantially lower than the and substantially below the laser power usage of Schmid and Shah et al.For live imaging of zebrafish/drosophila, we used a maximum laser power at source of 17.2/8.6mW resulting in a maximum power at the sample of 2.2/1.1 mW, for single-color live imaging from a single illumination view (presented data).For the zebrafish (ca.72 hrs, every 300 s) and drosophila (ca.24 hrs every 300 s) data presented, = 1,848 and 275 mW equivalent exposures respectively, lower than all other examples considered other examples considered demonstrates that the photon burden associated with live imaging presented is well within a normal range for live imaging of developing embryos and well within ranges of exposure considered safe for long term live imaging.
: Multi-color patch registration.A maximum intensity projection from a stack of fluorescent beads: cyan: 488 nm excitation, 525/50 nm emission (band center/full-width at half maximum bandwidth), magenta: 640 nm excitation, 697/60 nm emission.A: The center 200 µm ⨉ 200 µm of the full imaging field of view (scale bar = 50 µm).B-E one patch extracted from the full volume: 18.7 µm ⨉ 18.7 µm ⨉ 39.7 µm shown as maximum intensity projections over the excluded axis (scale bars: 5 µm).B/D xy-views C/E xz-views.B/C pre registration.D/E post registration.