Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Pretraining a foundation model for generalizable fluorescence microscopy-based image restoration

Nature Methods (2024)Cite this article

Subjects

Abstract

Fluorescence microscopy-based image restoration has received widespread attention in the life sciences and has led to significant progress, benefiting from deep learning technology. However, most current task-specific methods have limited generalizability to different fluorescence microscopy-based image restoration problems. Here, we seek to improve generalizability and explore the potential of applying a pretrained foundation model to fluorescence microscopy-based image restoration. We provide a universal fluorescence microscopy-based image restoration (UniFMIR) model to address different restoration problems, and show that UniFMIR offers higher image restoration precision, better generalization and increased versatility. Demonstrations on five tasks and 14 datasets covering a wide range of microscopy imaging modalities and biological samples demonstrate that the pretrained UniFMIR can effectively transfer knowledge to a specific situation via fine-tuning, uncover clear nanoscale biomolecular structures and facilitate high-quality imaging. This work has the potential to inspire and trigger new research highlights for fluorescence microscopy-based image restoration.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Applying the proposed UniFMIR approach to reconstruct SR SIM images from diffraction-limited WF images.
Fig. 2: Applying UniFMIR to the isotropic reconstruction of 3D volumes.
Fig. 3: Applying UniFMIR to content-aware 3D image denoising.
Fig. 4: Applying UniFMIR to joint surface projection.
Fig. 5: Applying UniFMIR to volumetric reconstruction.
Fig. 6: Generalization ability analysis conducted on unseen datasets41.

Similar content being viewed by others

Data availability

All training and testing data involved in the experiments come from existing literature and can be downloaded from the corresponding links provided in Supplementary Table 2 or via Zenodo at https://doi.org/10.5281/zenodo.8401470 (ref. 55).

Code availability

The PyTorch code of our UniFMIR, together with trained models, as well as some example images for inference are publicly available at https://github.com/cxm12/UNiFMIR (https://doi.org/10.5281/zenodo.10117581)56. Furthermore, We also provide a live demo for UniFMIR at http://unifmir.fdudml.cn/. Users can also access the colab at https://colab.research.google.com/github/cxm12/UNiFMIR/blob/main/UniFMIR.ipynb or use the steps in our GitHub documentation to run the demo locally. This newly built interactive software platform facilitates users to freely and easily use the pretrained foundation model. It also makes it easy for us to continuously train the foundation model with new data and share it with the community. Finally, we shared all models on BioImage.IO at https://bioimage.io/#/. Data are available via Zenodo at https://doi.org/10.5281/zenodo.10577218, https://doi.org/10.5281/zenodo.10579778, https://doi.org/10.5281/zenodo.10579822, https://doi.org/10.5281/zenodo.10595428, https://doi.org/10.5281/zenodo.10595460, https://doi.org/10.5281/zenodo.8420081 and https://doi.org/10.5281/zenodo.8420100 (refs. 57,58,59,60,61,62,63). We used the Pycharm software for code development.

References

  1. Preibisch, S. et al. Efficient bayesian-based multiview deconvolution. Nat. Methods 11, 645–648 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Gustafsson, N. et al. Fast live-cell conventional fluorophore nanoscopy with ImageJ through super-resolution radial fluctuations. Nat. Commun. 7, 12471 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Arigovindan, M. et al. High-resolution restoration of 3D structures from widefield images with extreme low signal-to-noise-ratio. Proc. Natl Acad. Sci. USA 110, 17344–17349 (2013).

  4. Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).

    Article  CAS  PubMed  Google Scholar 

  5. Qiao, C. et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat. Methods 18, 194–202 (2021).

  6. Chen, J. et al. Three-dimensional residual channel attention networks denoise and sharpen fluorescence microscopy image volumes. Nat. Methods 18, 678–687 (2021).

  7. Wang, Z., Xie, Y. & Ji, S. Global voxel transformer networks for augmented microscopy. Nat. Mach. Intell. 3, 161–171 (2021).

    Article  Google Scholar 

  8. Wang, Z. et al. Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning. Nat. Methods 18, 551–556 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. Li, X. et al. Reinforcing neuron extraction and spike inference in calcium imaging using deep self-supervised denoising. Nat. Methods 18, 1395–1400 (2021).

    Article  CAS  PubMed  Google Scholar 

  10. Qiao, C. et al. Rationalized deep neural network for sustained super-resolution live imaging of rapid subcellular processes. Nat. Biotechol. 41, 367–377 (2022).

  11. Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 16, 1215–1225 (2019).

  12. Wu, Y. & Shroff, H. Faster, sharper, and deeper: structured illumination microscopy for biological imaging. Nat. Methods 15, 1011–1019 (2018).

    Article  CAS  PubMed  Google Scholar 

  13. Wu, Y. et al. Multiview confocal super-resolution microscopy. Nature 600, 279–284 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Chen, R. et al. Single-frame deep-learning super-resolution microscopy for intracellular dynamics imaging. Nat. Commun. 14, 2854 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Xu, Y. K. T. et al. Cross-modality supervised image restoration enables nanoscale tracking of synaptic plasticity in living mice. Nat. Methods 20, 935–944 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  16. Arigovindan, M. et al. High-resolution restoration of 3D structures from widefield images with extreme low signal-to-noise-ratio. Proc. Natl Acad. Sci. USA 110, 17344–17349 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Bommasani, R. et al. On the opportunities and risks of foundation models. Preprint at https://arxiv.org/abs/2108.07258 (2021).

  18. Fei, N. et al. Towards artificial general intelligence via a multimodal foundation model. Nat. Commun. 13, 3094 (2022).

  19. Zhang, Y. et al. DialoGPT: large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. 270–278 (2020).

  20. Yang, Z. et al. Xlnet: generalized autoregressive pretraining for language understanding. In Conference on Neural Information Processing Systems (NeurIPS) (2019).

  21. Dai, Z. et al. Coatnet: marrying convolution and attention for all data sizes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021).

  22. Kirillov, A. et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4015–4026 (2023).

  23. Achiam, J. et al. Gpt-4 technical report. Preprint at https://arxiv.org/abs/2303.08774 (2023).

  24. Bao, F. et al. One transformer fits all distributions in multi-modal diffusion at scale. In International Conference on Machine Learning (ICML) (2023).

  25. Bi, K. et al. Accurate medium-range global weather forecasting with 3D neural networks. Nature 619, 533–538 (2023).

  26. Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023).

  27. Jiang, L. Y. et al. Health system-scale language models are all-purpose prediction engines. Nature 619, 357–362 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  28. Huang, Z. et al. A visual-language foundation model for pathology image analysis using medical twitter. Nat. Methods 29, 2307–2316 (2023).

    CAS  Google Scholar 

  29. Zhou, Y. et al. A foundation model for generalizable disease detection from retinal images. Nature 622, 156–163 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Moor, M. et al. Foundation models for generalist medical artificial intelligence. Nature 616, 259–265 (2023).

    Article  CAS  PubMed  Google Scholar 

  31. Madani, A. et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnol. 41, 1099–1106 (2023).

    Article  CAS  Google Scholar 

  32. Theodoris, C. V. et al. Transfer learning enables predictions in network biology. Nature 618, 616–624 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. Henighan, T. et al. Scaling laws for autoregressive generative modeling. Preprint at https://arxiv.org/abs/2010.14701 (2020).

  34. Zamir, A. et al. Taskonomy: disentangling task transfer learning. In Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI), 3712–3722 (2019).

  35. Liu, Z. et al. Swin transformer: hierarchical vision transformer using shifted windows. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021).

  36. Xia, B. et al. Efficient non-local contrastive attention for image super-resolution. In Association for the Advancement of Artificial Intelligence (AAAI) (2022).

  37. Descloux, A., Grubmayer, K. S. & Radenovic, A. Parameter-free image resolution estimation based on decorrelation analysis. Nat. Methods 16, 918–924 (2019).

    Article  CAS  PubMed  Google Scholar 

  38. Nieuwenhuizen, R. et al. Measuring image resolution in optical nanoscopy. Nat. Methods 10, 557–562 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. Culley, S. et al. Quantitative mapping and minimization of super-resolution optical imaging artifacts. Nat. Methods 15, 263–266 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  40. Li, X. et al. Three-dimensional structured illumination microscopy with enhanced axial resolution. Nat. Biotechnol. 41, 1307–1319 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Spahn, C. et al. DeepBacs for multi-task bacterial image analysis using open-source deep learning approaches. Commun. Biol. 5, 688 (2022).

  42. Ouyang, W. et al. ShareLoc—an open platform for sharing localization microscopy data. Nat. Methods 19, 1331–1333 (2022).

    Article  CAS  PubMed  Google Scholar 

  43. Zhang, X. C. et al. Zoom to learn, learn to zoom. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019).

  44. Nehme, E. et al. Deep-storm: super-resolution single-molecule microscopy by deep learning. Optica 5, 458–464 (2018).

    Article  CAS  Google Scholar 

  45. Guo, L. L. et al. EHR foundation models improve robustness in the presence of temporal distribution shift. Sci. Rep. 13, 3767 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  46. Liang, J. et al. Swinir: image restoration using swin transformer. In IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 1833–1844 (2021).

  47. Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. In International Conference on Machine Learning (ICLR) (2015).

  48. Kingma, D. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).

  49. Wang, Z. et al. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004).

    Article  PubMed  Google Scholar 

  50. Abbe, E. Beiträge zur theorie des mikroskops und der mikroskopischen wahrnehmung. Archiv. f. Mikrosk. Anatomie 9, 413–418 (1873).

    Article  Google Scholar 

  51. Koho, S. et al. Fourier ring correlation simplifies image restoration in fluorescence microscopy. Nat. Commun. 10, 3103 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  52. Baskin, C. et al. UNIQ: uniform noise injection for non-uniform quantization of neural networks. ACM Transactions on Computer Systems (TOCS), 37 (1–4), 1–15 (2021).

  53. Arganda, C. et al. Trainable weka segmentation: a machine learning tool for microscopy pixel classification. Bioinformatics 33, 2424–2426 (2017).

    Article  Google Scholar 

  54. Jacob, B. et al. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2704–2713 (2018).

  55. Ma, C., Tan, W., He, R. & Yan, B. UniFMIR: pre-training a foundation model for universal fluorescence microscopy image restoration (2023.10.03). Zenodo https://doi.org/10.5281/zenodo.8401470 (2023).

    Article  Google Scholar 

  56. Ma, C., Tan, W., He, R., & Yan, B. UniFMIR: pre-training a foundation model for universal fluorescence microscopy image restoration (version 2023.11.13). Zenodo https://doi.org/10.5281/zenodo.10117581 (2023).

  57. Ma, C., Tan, W., He, R. & Yan, B. UniFMIRProjectionOnFlyWing. Zenodo https://doi.org/10.5281/zenodo.10577218 (2024).

    Article  Google Scholar 

  58. Ma, C., Tan, W., He, R. & Yan, B. UniFMIRDenoiseOnPlanaria. Zenodo https://doi.org/10.5281/zenodo.10579778 (2024).

    Article  Google Scholar 

  59. Ma, C., Tan, W., He, R. & Yan, B. UniFMIRDenoiseOnTribolium. Zenodo https://doi.org/10.5281/zenodo.10579822 (2024).

    Article  Google Scholar 

  60. Ma, C., Tan, W., He, R. & Yan, B. UniFMIRVolumetricReconstructionOnVCD. Zenodo https://doi.org/10.5281/zenodo.10595428 (2024).

    Article  Google Scholar 

  61. Ma, C., Tan, W., He, R. & Yan, B. UniFMIRIsotropicReconstructionOnLiver. Zenodo https://doi.org/10.5281/zenodo.10595460 (2024) .

  62. Ma, C., Tan, W., He, R. & Yan, B. UniFMIRSuperResolutionOnMicrotubules. Zenodo https://doi.org/10.5281/zenodo.8420081 (2023).

  63. Ma, C., Tan, W., He, R. & Yan, B. UniFMIRSuperResolutionOnFactin. Zenodo https://doi.org/10.5281/zenodo.8420100 (2023).

Download references

Acknowledgements

We gratefully acknowledge support for this work provided by the National Natural Science Foundation of China (NSFC) (grant nos. U2001209 to B.Y. and 62372117 to W.T.) and the Natural Science Foundation of Shanghai (grant no. 21ZR1406600 to W.T.).

Author information

Author notes

  1. These authors contributed equally: Chenxi Ma, Weimin Tan.

Authors and Affiliations

  1. School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University, Shanghai, China

    Chenxi Ma, Weimin Tan, Ruian He & Bo Yan

Authors
  1. Chenxi Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weimin Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ruian He
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bo Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

B.Y. and W.T. supervised the research. C.M. and W.T. conceived of the technique. C.M. implemented the algorithm. C.M. and W.T. designed the validation experiments. C.M. trained the network and performed the validation experiments. R.H. implemented the interactive software platform and organized the codes and models. All authors had access to the study and wrote the paper.

Corresponding author

Correspondence to Bo Yan.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Methods thanks Ricardo Henriques and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Rita Strack, in collaboration with the Nature Methods team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Overall architecture of the UniFMIR.

The proposed UniFMIR approach is composed of three submodules: a multihead module, a Swin transformer-based feature enhancement module, and a multitail module. The numbers of parameters (M) and calculations (GFLOPs) required for the head, feature enhancement and tail modules for different tasks are marked below the structures of the respective modules. The input sizes and output sizes of training batches for different tasks are also marked below the images.

Extended Data Fig. 2 Network architecture of the Swin transformer-based feature enhancement module46.

The feature enhancement module consists of convolutional layers and a series of Swin transformer blocks (STB), each of which includes several Swin transformer layers (STL), a convolutional layer and a residual connection. The STL is composed of layer normalization operations, a multihead self-attention (MSA) mechanism and a multilayer perceptron (MLP). In the MSA mechanism, the input features are first divided into multiple small patches with a moving window operation, and then the self-attention in each patch is calculated to output features fout. The MLP is composed of two fully connected layers (FCs) and Gaussian-error linear unit (GELU) activation.

Extended Data Fig. 3 Generalization ability analysis of super-resolution on unseen modality of single-molecule localization microscopy data from the Shareloc platform52.

a, SR results obtained by the SOTA model (DeepSTORM54), the pretrained UniFMIR model without fine-tuning, Baseline (same network structure as UniFMIR trained from scratch), and our fine-tuned UniFMIR model. The GT dSTORM images of microtubules stained with Alexa 647 in U2OS cells incubated with nocodazole and the input synthesized LR images are also shown. The PSNR/NRMSE results of the SR outputs obtained on n = 16 synthetic inputs are shown on the right. b, SR results obtained on the real-world wide-field images. The NRMSE values are depicted on the residual images under different SR results and the raw input images. The PSNR/NRMSE results on n = 9 real-world inputs are shown on the right. Box-plot elements are defined as follows: center line (median); box limits (upper and lower quartiles); whiskers (1.5x interquartile range). The line plots show the pixel intensities along the dashed lines in the corresponding images. Scale bar: 6.5 μm.

Extended Data Table 1 Comparison between the training costs of the SOTA models and that of our model on different tasks
Full size table
Extended Data Table 2 Comparison between the prediction costs of the SOTA models and that of our model on different tasks
Full size table
Extended Data Table 3 Quantitative comparison among different models for the x2 SR task
Full size table
Extended Data Table 4 Quantitative comparison among different models for the isotropic reconstruction task
Full size table
Extended Data Table 5 Quantitative comparison among different models for the 3D image denoising task
Full size table
Extended Data Table 6 Quantitative comparison among different models for the 3D-to-2D projection task
Full size table

Supplementary information

Supplementary Information

Supplementary Notes 1–5, Figs. 1–17 and Tables 1 and 2.

Reporting Summary

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ma, C., Tan, W., He, R. et al. Pretraining a foundation model for generalizable fluorescence microscopy-based image restoration. Nat Methods (2024). https://doi.org/10.1038/s41592-024-02244-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s41592-024-02244-3

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing