Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Evaluation and development of deep neural networks for image super-resolution in optical microscopy

Nature Methods volume 18, pages 194–202 (2021)Cite this article

Subjects

Abstract

Deep neural networks have enabled astonishing transformations from low-resolution (LR) to super-resolved images. However, whether, and under what imaging conditions, such deep-learning models outperform super-resolution (SR) microscopy is poorly explored. Here, using multimodality structured illumination microscopy (SIM), we first provide an extensive dataset of LR–SR image pairs and evaluate the deep-learning SR models in terms of structural complexity, signal-to-noise ratio and upscaling factor. Second, we devise the deep Fourier channel attention network (DFCAN), which leverages the frequency content difference across distinct features to learn precise hierarchical representations of high-frequency information about diverse biological structures. Third, we show that DFCAN’s Fourier domain focalization enables robust reconstruction of SIM images under low signal-to-noise ratio conditions. We demonstrate that DFCAN achieves comparable image quality to SIM over a tenfold longer duration in multicolor live-cell imaging experiments, which reveal the detailed structures of mitochondrial cristae and nucleoids and the interaction dynamics of organelles and cytoskeleton.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Evaluation of the four DLSR models.
Fig. 2: FCA mechanism and comparison of SISR images via DFCAN and RCAN.
Fig. 3: Characterization of DFCAN and DFGAN.
Fig. 4: Live-cell SISR imaging via DFCAN and DFGAN.
Fig. 5: Comparison of SIM reconstruction via DFCAN, DFGAN and scU-net.
Fig. 6: Two-color SR live-cell imaging by DFCAN- and DFGAN-SIM.

Similar content being viewed by others

Data availability

The BioSR dataset including more than 2,200 pairs of LR–SR images covering four biology structures, nine SNR levels and two upscaling factors are publicly accessible at figshare repository (https://doi.org/10.6084/m9.figshare.13264793). Source data are provided with this paper.

Code availability

The tensorflow codes of DFCAN and DFGAN, several representative trained models, as well as some example images for testing are publicly available at https://github.com/qc17-THU/DL-SR.

References

  1. Dong, C., Loy, C. C., He, K. & Tang, X. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 295–307 (2016).

    Article  Google Scholar 

  2. Lim, B., Son, S., Kim, H., Nah, S. & Mu Lee, K. Enhanced deep residual networks for single image super-resolution. In Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops (eds. Chellappa, R. et al.) 1132–1140 (IEEE, 2017).

  3. Lai, W.-S., Huang, J.-B., Ahuja, N. & Yang, M.-H. Deep Laplacian pyramid networks for fast and accurate super-resolution. in Proc. IEEE Conference on Computer Vision and Pattern Recognition (eds. Chellappa, R. et al.) 624–632 (IEEE, 2017).

  4. Ledig, C. et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (eds. Chellappa, R. et al.) 4681–4690 (IEEE, 2017).

  5. Zhang, Y. et al. Image super-resolution using very deep residual channel attention networks. In Proc. European Conference on Computer Vision (ECCV) (eds. Ferrari, V. et al.) 286–301 (Springer, 2018).

  6. Yang, W. et al. Deep learning for single image super-resolution: a brief review. IEEE Trans. Multimed. 21, 3106–3121 (2019).

    Article  Google Scholar 

  7. Cai, J., Zeng, H., Yong, H., Cao, Z. & Zhang, L. Toward real-world single image super-resolution: a new benchmark and a new model. In Proc. IEEE International Conference on Computer Vision (eds. Davis, L. S. et al.) 3086–3095 (IEEE, 2019).

  8. Cai, J., Gu, S., Timofte, R. & Zhang, L. Ntire 2019 challenge on real image super-resolution: methods and results. In Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops (eds. Brown, M. S. et al.) 2211–2223 (IEEE, 2019).

  9. Lugmayr, A., Danelljan, M. & Timofte, R. NTIRE 2020 challenge on real-world image super-resolution: methods and results. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (Boult, T. T. et al.) 494–495 (IEEE, 2020).

  10. Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).

    Article  CAS  Google Scholar 

  11. Wang, H. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat. Methods 16, 103–110 (2019).

    Article  CAS  Google Scholar 

  12. Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 16, 1215–1225 (2019).

    Article  CAS  Google Scholar 

  13. Guo, Y. et al. Visualizing intracellular organelle and cytoskeletal interactions at nanoscale resolution on millisecond timescales. Cell 175, 1430–1442 e1417 (2018).

    Article  CAS  Google Scholar 

  14. Li, D. et al. Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics. Science 349, aab3500 (2015).

    Article  Google Scholar 

  15. Hu, J., Shen, L. & Sun, G. Squeeze-and-excitation networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (eds. Brown, M. S. et al.) 7132–7141 (IEEE, 2018).

  16. Gustafsson, M. G. et al. Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophys. J. 94, 4957–4970 (2008).

    Article  CAS  Google Scholar 

  17. Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A. Image-to-image translation with conditional adversarial networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (eds. Chellappa, R. et al.) 1125–1134 (IEEE, 2017).

  18. Ronneberger, O., Fischer, P. & Brox, T. U-Net: Convolutional networks for biomedical image segmentation. In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention (eds Navab, N. et al.) 234–241 (Springer, 2015).

  19. Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K. Q. Densely connected convolutional networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (eds. Chellappa, R. et al.) 4700–4708 (IEEE, 2017).

  20. Westermann, B. Mitochondrial fusion and fission in cell life and death. Nat. Rev. Mol. Cell Biol. 11, 872–884 (2010).

    Article  CAS  Google Scholar 

  21. Stephan, T., Roesch, A., Riedel, D. & Jakobs, S. Live-cell STED nanoscopy of mitochondrial cristae. Sci. Rep. 9, 12419 (2019).

    Article  Google Scholar 

  22. Shim, S. H. et al. Super-resolution fluorescence imaging of organelles in live cells with photoswitchable membrane probes. Proc. Natl Acad. Sci. USA 109, 13978–13983 (2012).

    Article  CAS  Google Scholar 

  23. Wang, C. et al. A photostable fluorescent marker for the superresolution live imaging of the dynamic structure of the mitochondrial cristae. Proc. Natl Acad. Sci. USA 116, 15817–15822 (2019).

    Article  CAS  Google Scholar 

  24. Goldstein, R. E., Tuval, I. & van de Meent, J.-W. Microfluidics of cytoplasmic streaming and its implications for intracellular transport. Proc. Natl Acad. Sci. USA 105, 3663–3667 (2008).

    Article  CAS  Google Scholar 

  25. Wu, Y. & Shroff, H. Faster, sharper, and deeper: structured illumination microscopy for biological imaging. Nat. Methods 15, 1011–1019 (2018).

    Article  CAS  Google Scholar 

  26. Li, D. & Betzig, E. Response to Comment on “Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics”. Science 352, 527–527 (2016).

    CAS  PubMed  Google Scholar 

  27. Huang, X. et al. Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy. Nat. Biotechnol. 36, 451–459 (2018).

    Article  CAS  Google Scholar 

  28. Jin, L. et al. Deep learning enables structured illumination microscopy with low light levels and enhanced speed. Nat. Commun. 11, 1934 (2020).

    Article  CAS  Google Scholar 

  29. Collins, A., Warrington, A., Taylor, K. A. & Svitkina, T. Structural organization of the actin cytoskeleton at sites of clathrin-mediated endocytosis. Curr. Biol. 21, 1167–1175 (2011).

    Article  CAS  Google Scholar 

  30. Burke, L. A. & Hutchins, H. M. Training transfer: an integrative literature review. Hum. Resour. Dev. Rev. 6, 263–296 (2007).

    Article  Google Scholar 

  31. Christensen, C. N., Ward, E. N., Lio, P. & Kaminski, C. F. ML-SIM: a deep neural network for reconstruction of structured illumination microscopy images. Preprint at https://arxiv.org/abs/2003.11064 (2020).

  32. Hendrycks, D. & Gimpel, K. Gaussian error linear units (GELUs). Preprint at https://arxiv.org/abs/1606.08415 (2016).

  33. Goodfellow, I. et al. Generative adversarial nets. In Proc. Advances in Neural Information Processing Systems 27 (NIPS) (eds. Ghahramani, Z. et al.) 2672–2680 (MIT Press, 2014).

  34. Descloux, A., Grussmayer, K. S. & Radenovic, A. Parameter-free image resolution estimation based on decorrelation analysis. Nat. Methods 16, 918–924 (2019).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

This work was supported by grants from the National Natural Science Foundation of China (NSFC; nos. 91754202, 31827802, 31800717, and 62088102); Ministry of Science and Technology (MOST; nos. 2017YFA0505301 and 2016YFA0500203); Chinese Academy of Sciences (CAS; nos. XDB19040101 and ZDBS-LY-SM004); The Youth Innovation Promotion Association of Chinese Academy of Sciences (no. 2020094); China Postdoctoral Science Foundation (no. BX20190355); and the Tencent Foundation through the XPLORER PRIZE.

Author information

Author notes

  1. These authors contributed equally: Chang Qiao, Di Li, Yuting Guo, Chong Liu.

Authors and Affiliations

  1. Department of Automation, Tsinghua University, Beijing, China

    Chang Qiao & Qionghai Dai

  2. National Laboratory of Biomacromolecules, CAS Center for Excellence in Biomacromolecules, Institute of Biophysics, Chinese Academy of Sciences, Beijing, China

    Di Li, Yuting Guo, Chong Liu, Tao Jiang & Dong Li

  3. College of Life Sciences, University of Chinese Academy of Sciences, Beijing, China

    Chong Liu, Tao Jiang & Dong Li

  4. Bioland Laboratory, Guangzhou Regenerative Medicine and Health Guangdong Laboratory, Guangzhou, China

    Dong Li

Authors
  1. Chang Qiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Di Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuting Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tao Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Qionghai Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Dong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Dong Li conceived the project. Dong Li and Q.D. supervised the research. Dong Li, C.Q. and Di Li designed the experiments. Di Li, Y.G., C.L. and T.J. prepared samples and performed experiments. C.Q., Di Li, Y.G. and C.L. analyzed the data with conceptual advice from Dong Li. C.Q. and Di Li composed the figures and videos under the supervision of Dong Li. Dong Li, C.Q. and Q.D. wrote the manuscript with input from all authors. All authors discussed the results and commented on the manuscript.

Corresponding authors

Correspondence to Qionghai Dai or Dong Li.

Ethics declarations

Competing interests

Dong Li, C.Q., Di Li and Q.D. have a pending patent application on the presented framework.

Additional information

Peer review information Rita Strack was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Representative GT-SIM images of CCPs, ER, MTs, F-actin (linear SIM) and F-actin (nonlinear SIM).

Scale bar: 5 μm, and 1 μm (zoom-in regions). Gamma values: 0.8 for F-actin images of both linear and nonlinear SIM. For each type of specimens, experiments were repeated independently for more than 100 test images with similar characteristics and performance.

Extended Data Fig. 2 Representative super-resolution images inferred from the wide-field diffraction-limited images of medium fluorescence intensity by the four DLSR models.

The difference images with respect to ground truth (GT) are below the corresponding DLSR images. The mean absolute error (MAE) of each difference image is labeled in its top left corner. Scale bar: 0.75 μm for CCPs; 1.5 μm for other specimens. Gamma values: 0.6 for the F-actin images of upscaling 3×.

Extended Data Fig. 3 Network architecture of DFCAN and DFGAN.

a, The architecture of DFCAN or the generator of DFGAN. b, The architecture of the discriminator of DFGAN. c, Schematic of the training process of DFCAN. d, Schematic of the training process of the generator of DFGAN. The generator and the discriminator were trained alternatively. The discriminator’s parameters remained unchanged during each training step of the generator.

Extended Data Fig. 4 Comparison of Fourier channel attention (FCA) and spatial channel attention (SCA) mechanism.

a, b, The architecture of FCA (a) and SCA (b). c, d, The schematic illustrating the positions where the FCA or SCA module was added into U-net (c) and DenseNet (d). e, f, g, Comparisons of the validation NRMSE achieved over the training process of (e) DFCAN, a modified ResNet from DFCAN without using FCA-modules, and deep spatial channel attention network (DSCAN), whose architecture is the same as DFCAN except for replacing the FCA- with SCA modules. (f) conventional U-net, U-net with SCA (named U-net-SCA), and U-net with FCA (named U-net-FCA); (g) conventional DenseNet, DenseNet with SCA (named DenseNet-SCA), and DenseNet with FCA (named DenseNet-FCA). h, Representative SISR images of simulated tubular structure inferred by the aforementioned DLSR models. In each magnified image, the green arrows indicate the features correctly inferred, whereas magenta arrows indicate the features wrongly inferred by specific DLSR model. Yellow arrows indicate the same features in the corresponding GT-SIM and wide-field images for reference. Scale bar: 1 μm, and 0.3 μm for both magnified regions. Experiments were repeated independently for 30 test images, achieving similar results.

Source data

Extended Data Fig. 5 MS-SSIM and resolution comparisons of SISR images generated by different DLSR models.

MS-SSIM and resolution comparisons of the seven DLSR models that are classified into two categories: non-GAN models (a) and GAN-based models (b) in the cases of CCPs, ER, MTs, and F-actin (upscaling 2×), and F-actin (upscaling 3×).

Source data

Extended Data Fig. 6 Live-cell GI-SIM images confirming the phenomena observed in Fig. 4.

a, Time-lapse GI-SIM images of a nucleoid fission event concomitant with the formation of a new cristae in between. b, Time-lapse GI-SIM images of a nucleoids fusion event concomitant with the recession of the intermediate cristae. c, Time-lapse GI-SIM images illustrating the bidirectional rotation of circular mitochondrial tubes. Lower row shows that nucleoids inside mitochondria change their orientation accordingly. Scale bar: 1 µm (a-c). Experiments were repeated independently for n = 10 COS-7 cells, all showing similar characteristics and performance.

Extended Data Fig. 7 SR imaging comparisons of conventional non-deep-learning methods and DFCAN or DFGAN.

a, Single phase raw SIM image acquired at relatively low-fluorescence (average photon count = 120). b, Deconvolved image with Richardson-Lucy algorithm (NRMSE = 0.0755, MS-SSIM = 0.7512, Resolution=243 nm). c, Conventional SIM image (NRMSE = 0.0781, MS-SSIM = 0.7832, Resolution=96 nm). d, Hessian algorithm27 denoised image (µ=150, σ = 0) after Conventional SIM reconstruction (NRMSE = 0.0622, MS-SSIM = 0.8584, Resolution=101 nm). e, f, SIM image reconstructed by (e) DFCAN (NRMSE = 0.0593, MS-SSIM = 0.8665, Resolution=139 nm) and (f) DFGAN (NRMSE = 0.0586, MS-SSIM = 0.8680, Resolution=97 nm). g, GT-SIM image acquired at high SNR. h-k, magnified images of (h) Hessian-SIM, (i) DFCAN, (j) DFGAN, and (k) GT-SIM from the boxed regions in corresponding images. Scale bar: 3 μm; 1 μm in magnified image. Experiments were repeated with more than 100 test images, achieving similar results.

Extended Data Fig. 8 The robustness of DFCAN and DFGAN algorithms to reconstruct the raw SIM images of high out-of-focus background.

a, DFCAN and DFGAN can reconstruct the simulated raw images of background-to-signal ratio as high as 20, although the Moiré fringes in the raw images are so obscure that many reconstruction artifacts are riddled in the conventional SIM image. b, DFCAN and DFGAN also largely prevent the SR images from reconstruction artifacts, although the raw images are contaminated by large out-of-focus background because the used excitation NA (Exc. NA) is significantly lower than the critical NA for TIRF or grazing incidence (GI). Experiments in (a) and (b) were repeated with 20 images, achieving similar results.

Extended Data Fig. 9 NRMSE, MS-SSIM and resolution comparisons of SIM image reconstruction by different DLSR models.

NRMSE (a), MS-SSIM (b), and resolution (c) comparisons of DLSR images generaed by scU-net input of N×M raw SIM images (yellow), DFCAN input of single WF image (dark green), DFCAN input of N×M raw SIM images (light green), DFGAN input of single WF image (dark red), DFGAN input of N×M raw SIM images (red) in the cases of CCPs (first row), ER (second row), MTs (third row), and F-actin (linear SIM, 2× upscaling) (fourth row), and F-actin (nonlinear SIM, 3× upscaling) (fifth row).

Source data

Extended Data Fig. 10 Assessment of transfer learning capability of DFGAN.

a, Representative raw SIM images of F-actin, CCPs, MTs, endoplasmic reticulum (ER), and mitochondria. b, The super-resolution images inferred by the same DFGAN network, which is trained with only F-actin images. c, SIM images reconstructed by the DFGAN networks after transfer learning for each type of specimen, which are initialized by the DFGAN network trained with only F-actin data (Supplementary Note 7). d, SIM images reconstructed by the DFGAN networks trained with corresponding datasets. e, The ground truth SIM images shown for reference. Scale bar: 3 μm; 1 μm for the zoom-in regions. Experiments were repeated with more than 50 test images for each type of specimens, achieving similar results.

Supplementary information

Supplementary Information

Supplementary Notes 1–8, Figs. 1–15 and Tables 1–5.

Reporting Summary

Supplementary Video 1

The evolution in super-resolution imaging quality of clathrin-coated pits for 2× upscaling factor over low-to-high fluorescence levels by the DLSR models evaluated in this work.

Supplementary Video 2

The evolution in super-resolution imaging quality of endoplasmic reticulum for 2× upscaling factor over low-to-high fluorescence levels by the DLSR models evaluated in this work.

Supplementary Video 3

The evolution in super-resolution imaging quality of microtubules for 2× upscaling factor over low-to-high fluorescence levels by the DLSR models evaluated in this work.

Supplementary Video 4

The evolution in super-resolution imaging quality of F-actin for 2× upscaling factor over low-to-high fluorescence levels by the DLSR models evaluated in this work.

Supplementary Video 5

The evolution in super-resolution imaging quality of F-actin for 3× upscaling factor over low-to-high fluorescence levels by the DLSR models evaluated in this work. Gamma value: 0.8 for all F-actin images.

Supplementary Video 6

DFCAN/DFGAN-SISR live-cell imaging records the continuous reorganizations of nucleoids during the mitochondrial fission, fusion, and cristae deformation over 1230 frames at 2-s intervals from COS-7 cells expressing PHB2-mEmerald (green) and TFAM-mCherry (magenta)

Supplementary Video 7

Circular mitochondrial tubes undergo bidirectional rotation captured by DFCAN/DFGAN-SISR live-cell imaging over 931 frames at 2-s intervals from COS-7 cells expressing PHB2-mEmerald (green) and TFAM-mCherry (magenta).

Supplementary Video 8

Two-color DFCAN/DFGAN-SIM live-cell imaging reveals the dynamic interactions of F-actin (red) and clathrin coated pits (green) over 402 frames at 5-s intervals from COS-7 cells expressing Lifeact-mEmerald and clathrin-mCherry.

Supplementary Video 9

Two-color DFCAN/DFGAN-SIM live-cell imaging reveals the dynamic interactions of endoplasmic reticulum (magenta) and mitochondrial cristae (green) over 421 frames at 2-s intervals from COS-7 cells expressing PHB2-mEmerald and KDEL-mCherry.

Supplementary Video 10

Mitochondrial fission and fusion events at ER-Mito contact sites revealed by DFCAN/DFGAN-SIM live-cell imaging over 311 frames at 2-s intervals from COS-7 cells expressing PHB2-mEmerald (green) and KDEL-mCherry (magenta). White arrows indicate the association of dividing or coalescing site with ER tubules.

Source data

Source Data Fig. 1

Statistical source data.

Source Data Fig. 2

Statistical Source data.

Source Data Fig. 3

Statistical source data.

Source Data Fig. 5

Statistical source data.

Source Data Extended Data Fig. 4

Statistical source data.

Source Data Extended Data Fig. 5

Statistical source data.

Source Data Extended Data Fig. 9

Statistical source data.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qiao, C., Li, D., Guo, Y. et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat Methods 18, 194–202 (2021). https://doi.org/10.1038/s41592-020-01048-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41592-020-01048-5

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing