A method for the reconstruction of multifocal structured illumination microscopy data with high efficiency

Article metrics

Abstract

We present and demonstrate an efficient method for the reconstruction of profiles acquired by multifocal structured illumination microscopy (MSIM) utilizing few raw images. Firstly, we propose a method to produce nine raw multifocal images with enhanced modulation depth to accomplish the uniform illumination of the sample. Then, combing with the parameter of the arrays, we perform the standard construct reconstruction procedure of structured illumination microscopy (SIM) row by row and column by column. Finally, we combine these restored images together to obtain the final image with enhanced resolution and good contrast. Based on theoretical analysis and numerical simulations, this method shows great potential in the field of the image reconstruction of MSIM data.

Introduction

The wave nature of visible light has hampered the development of optical microscopy in spatial resolution. The Abbe diffraction limit imposes a critical limit on the spatial resolution of optical microscopy, meaning that the maximum available resolution under a wide-field microscope is approximately equal to one-half of the fluorophores emission wavelength which remains two orders of magnitude above the size regime relevant to many molecular reactions and subcellular structures1. Therefore, it severely hinders the observation of the fine structure of biological samples. To circumvent the Abbe limit and achieve super-resolution, the past two decades have witnessed the impressive development in super resolution microscopy filed, and a variety of related techniques have come into being2,3,4,5 to enable measurements within spatial regimes that were previously inaccessible, such as photo-activated localization microscopy (PALM)6,7,8, stochastic optical reconstruction microscopy (STORM)9,10,11, stimulated emission depletion microscopy (STED)12,13,14, and structured illumination microscopy (SIM)15,16,17,18.

As one of the mainstream super resolution microscopies, SIM applies spatially patterned illumination to elicit sample fluorescence. By frequency-mixing between the excitation pattern and fluorophore density, it visualizes high-resolution information which is inaccessible in the conventional microscope. Linear SIM can double the resolution limit of the conventional imaging system while nonlinear structured light technology can improve the performance of lateral resolution further17,19,20. In addition, conventional SIM with sinusoidal illumination patterns offers the advantage which does not require additional complex sample preparation effort. However, under the circumstances of high scattering media and high light intensity, the advantages of SIM based on wide-field imaging are not obvious enough due to the optical aberrations and scattering of the sample. Utilizing point scanning to acquire images, the contrast and quality of the data can be proper maintained while it may be far too slow for monitoring many live cellular processes.

MSIM, with a sparse lattice of excitation foci would dramatically increase acquisition speed and show excellent performance. MSIM could not be well achieved without proper reconstruction methods for the data profiles. The number of original images used for image reconstruction is closely related to the efficiency of the method. The method reported in ref.21 is a joint Richardson-Lucy deconvolution algorithm to achieve the reconstruction of MSIM data along with enhanced resolution and good quality. However, the method needs too many images (143 images) and iterations (between 5–30 times) to accomplish the reconstruction, which is complicated and blocks its extensive application. Meanwhile, the method reported in ref.22 deconvolves the multifocal-excited, pinholed, scaled and summed (MPSS) images to achieve resolution doubling in live and multicellular organisms. In this method, to fully illuminate the sample, 120 raw multifocal images are acquired and processed to obtain a super resolution image. Although the final images show excellent quality, procedure of data process should been well taken into consideration and might consume lots of effort and time. In this paper, for the sake of image reconstruction of MSIM data, in first step we proposed a method to generate nine raw multifocal images with enhanced modulation depth, then combing with the parameter of the arrays we performed the standard construct reconstruction procedure of SIM row by row and column by column in the these images, respectively. Finally, we combined these restored images to obtain the final image with enhanced resolution and good contrast. Our method holds the merits of high efficiency and flexibility, which has great potential in the field of the reconstruction of MSIM data.

Result

As a validation of principle, we created artificial MSIM data based on the standard wide-field model and processed it with the proposed reconstruction algorithm which we implemented in MATLAB. The simulations were operated on the resolution test target and the Leaf image, respectively.

Generation of the multifocal arrays

To generate the multifocal array, we superimpose two sinusoidal stripes but orthogonal to each other as shown in Fig. 1(A), of which all have the same parameters including amplitude, frequency. When imaging thick biological tissue, light scattering and depth-induced spherical aberration can significantly weaken the modulation contrast of the structured pattern, which in turn contributes to the restored images with poor signal-to-ratio (SNR)23. To maintain the modulation depth of the multifocal array, we set a threshold based on the maximum intensity of the array and only the parts with the amplitude above the threshold of the array is retained. As illustrated in Fig. 1(E), we can see that the modulation depth of multifocal array with threshold is higher than that without the threshold. So far, a preliminary model shown in Fig. 1(B) of the multifocal array has been established. By changing the phase of the two stripes in the X and Y directions, the movement of the multi-focus array can be realized to achieve the entire illumination of the sample.

Figure 1
figure1

The generation of the multifocal array. (A) Result of superimposing the two orthogonal sinusoidal stripes. (B) The multifocal array parts with the intensity above the set threshold. (C) The wide-field image when A restricted by the OTF of the microscopy system. (D) The wide-field image when B restricted by the OTF of the microscopy system. (E) Normalized intensity curves along vertical or lateral direction in (C,D) respectively. As shown in (E) the modulation depth of (D) may be slighter higher than that of (C).

The improvement of spatial resolution provided by the proposed method

To confirm the improvement of spatial resolution provided by the proposed method, we analyze and compare the simulation results, where the sample is the resolution test target (USAF 1951 1X, Edmund). Figure 2(C,D) are the magnification of respective regions boxed in Fig. 2(A,B), while Fig. 2(A,B) show the wide-field and MSIM images, respectively. As shown in Fig. 2(E), the resolution of the wide-field image is confined to the element 4 of group −1 (around 10 pixels). From Fig. 2(F,G), we find that the reconstructed image can clearly distinguish the element 4 of group 0 (around 5 pixels), achieving a totally twofold resolution enhancement.

Figure 2
figure2

Simulated reconstruction results of the resolution target. (A) The wide-field image of the resolution target. (B) The reconstructed super-resolved MSIM image of the resolution target. (C,D) Are the magnifications of the dotted areas in the (A,B) respectively. (E) Normalized intensity curves of the profiles along the dotted vertical lines in (A) and the real image. (F) Normalized intensity curves of the profiles along the dotted vertical lines in (C,D) respectively. (G) Normalized intensity curves of the profiles along the dotted lateral lines in (C,D) respectively.

The necessity of setting thresholds to generate the multifocal arrays

To illustrate the necessity of setting thresholds to generate the multifocal arrays, we analyzed the results restored by our methods where the raw images are all contaminated by a combination of Poisson noise24 and Gaussian noise, which may be more close to the noise in practical experiments. Figure 3(C,D) are the leaf image reconstructed using the multifocal images without threshold and with threshold, respectively, while Fig. 3(B) is the wide-field image. As presented in Fig. 3(E), there is no significant difference between Fig. 3(C,D) in terms of resolution improvement, and the reconstructed image with the multifocal images with the threshold as shown in Fig. 3(D) is slightly better than that image without the threshold shown in Fig. 3(C). The threshold setting to the multifocal arrays can enhance the modulation contrast so as to contribute to the restored image with better contrast and SNR.

Figure 3
figure3

Simulated reconstruction results of the Leaf image under noise circumstance. (A) the Leaf photograph, where the area inside the dotted box is the image for subsequent simulations; (B) the wide-field image of the area inside the dotted box in (A,C) the reconstructed image with the nine raw multifocal images corrupted by noise, where the multifocal arrays are generated by superimposing two orthogonal stripes directly; (D) the reconstructed image with the nine raw multifocal images corrupted by noise, where the multifocal arrays are generated with our proposed approach (superimposing two orthogonal stripes and retaining the parts above a certain threshold). (E) Plots of normalized intensity along the corresponding colored lines in (BD).

Discussion and Conclusion

With our method, a super resolution image can be obtained just with nine original multifocal images. Compared with methods related to reconstruction of MSIM data21,22, the number of raw multifocal images for reconstruction is rather fewer and procedure of the reconstruction is relatively convenient to implement and operate, which is vital and suitable for the live cell imaging and monitoring along with less phototoxicity.

Based on wide-field microscopy, the field of view in conventional SIM is nearly equivalent to that in wide-field microscopy. Spot-scanning SIM can achieve same spatial resolution improvement as the traditional SIM, while its field of view is smaller than that of traditional SIM owning to the limitation of scanning angle of scanning device. If the scanning angle of scanning device is quite large, there would be large aberrations and distortions at the edges of the field of view resulting in degradation of resolution. Parallelization of spot-scanning SIM, MSIM could dramatically improve the efficiency of the data acquisition and realize the same resolution improvement as conventional SIM. Precise knowledge of the excitation pattern is quite imperative to MSIM in both applying digital pinholes and in image reconstruction. These effects are particularly apparent when it comes to imaging highly scattering or thick tissue samples, where the aberrations and noise produce reconstruction artefacts in the final image25. Besides, to achieve the fullest resolution improvement, the frequency of the multifocal arrays should be close to the cutoff frequency of the microscopy system.

To achieve the entire illumination of the sample, successive application of nine different patterns can be realized via a single programmable spatial light modulator (SLM) with the merits of program flexibility and the moveable parts is not required. Owing to the characteristics of multifocal arrays, every single excitation focus may have a relatively high intensity and the obtained multifocal arrays might physically reject out-of-focus light and permit imaging the sample even with high scatter property. It should also be taken into consideration that the proposed approach could be applied in two-photon excitation and adaptive optics (AO) for imaging thicker samples with the proper improvement of implements.

In conclusion, we proposed and demonstrated a strategy for the reconstruction of MSIM data. Meanwhile, we presented the feasible approach for the generation of multifocal arrays with considerable modulation depth which is used to maintain the final image with enhanced image contrast. With nine raw multifocal images, the proposed strategy to restore super-resolution images is of high efficiency and has great potential in live cell imaging.

Methods

Conventional SIM

In fluorescence microscopy, the illumination pattern Iem(x) multiplies the sample fluorescence S(x), consequently eliciting fluorescence emission, which is blurred by diffraction, yielding the image Iex(x):

$${I}_{em}(x)=({I}_{ex}(x)S(x))\otimes PS{F}_{em}(x)$$
(1)

where PSFem(x) is the emission point spread function (blurred image of a point source), and denotes the convolution operation. Fourier transformation of Eq. (1) yields

$${\tilde{I}}_{em}(k)=({\tilde{I}}_{ex}(k)\tilde{S}(k))\cdot OT{F}_{em}(k)$$
(2)

where the tilde represents the Fourier transform of the function and optical transfer function OTFem(k) is the Fourier transform of the PSFem(x).

As for conventional SIM, the illumination can be approximately described as:

$${I}_{ex}(x)={I}_{0}\cdot [1+m\cdot \,\cos \,(2\pi {k}_{0}x+{\phi })]$$
(3)

where k0 and φ are the spatial frequency and the phase of the illumination patterns, and I0 is the peak illumination intensity. By Fourier transforming Eq. (3) and substituting into Eq. (2), the expression for the observed image can be obtained as:

$${\tilde{I}}_{em}(k)={I}_{0}\cdot [\tilde{S}(k)+\frac{m}{2}\tilde{S}(k+{k}_{0}){e}^{-i{\phi }}+\frac{m}{2}\tilde{S}(k-{k}_{0}){e}^{i{\phi }}]\cdot OT{F}_{em}(k)$$
(4)

The first term in Eq. (4) denotes all the spatial frequencies normally observed by the microscope while the two remaining terms gives rise to resolution enhancement as they modulate high frequency components into the passband of the OTFem(k). By obtaining three images with three different phases (φx,y) of the illumination pattern we can obtain a system of three independent linear equations, allowing us to extract the three terms in Eq. (4). After that, we untangle and restore these frequency components to their proper location in Fourier space to reconstruct a super-resolved image. The maximum frequency of the illumination pattern is subject to the OTFem(k), as same as in the conventional wide-field microscopy. Consequently, it is possible to increase the resolution of the optical microscope along the pattern’s direction by a factor of approximately two in this case.

Method for the construction of MSIM

As for MSIM, the illumination shown in Fig. 4(A) can be approximately described as:

$${I}_{ex}(x,y)={I}_{0}\cdot [1+m\cdot \,\cos \,(2\pi {k}_{x}x+{{\phi }}_{x})+m\cdot \,\cos \,(2\pi {k}_{y}y+{{\phi }}_{y})]$$
(5)

where kx,y and φx,y are the spatial frequency and the phase of the two illumination patterns along the X or Y direction, respectively. Substituting the Fourier transformation of Eq. (5) into Eq. (2) yields

$${\mathop{I}\limits^{ \sim }}_{em}(k)={I}_{0}\cdot [\begin{array}{c}\mathop{S}\limits^{ \sim }(k)+\frac{m}{2}\mathop{S}\limits^{ \sim }(k+{k}_{x})\cdot {e}^{-i{\phi }_{x}}+\frac{m}{2}\mathop{S}\limits^{ \sim }(k-{k}_{x})\cdot {e}^{i{\phi }_{x}}\\ +\frac{m}{2}\mathop{S}\limits^{ \sim }(k+{k}_{y})\cdot {e}^{-i{\phi }_{y}}+\frac{m}{2}\mathop{S}\limits^{ \sim }(k-{k}_{y})\cdot {e}^{i{\phi }_{y}}\end{array}]\cdot OT{F}_{em}(k)$$
(6)
Figure 4
figure4

Multifocal array and its Fourier space. (A) The multifocal array. (B) Corresponding Fourier space of the array.

Compared with the Eq. (4), the Eq. (6) has five terms indicating that it has frequency shift along X and Y directions. In this case, the MSIM can enhance the spatial resolution of the sample in two directions simultaneously.

To construct the super-resolved images, we make it by performing the reconstruction from two directions separately based on standard reconstruction procedure of SIM. The multifocal arrays we generated here have frequency distributions in the X and Y direction, for the reason that we superimposed two sinusoidal stripes that are orthogonal to each other. After performing reconstruction with the raw multifocal images, we can obtain the final images with isotropic resolution improvement along the two directions simultaneously. To extract the three frequency terms in the X direction, we should shift the horizontal stripes three times to obtain a system of three independent equations. Meanwhile, to extract the three frequency terms in the Y direction, we should shift the vertical stripes three times to obtain a system of three independent equations. Therefore, nine raw multifocal images are required, which can accomplish the uniform illumination of the sample as well.

To obtain the super-resolved images, the restore procedure is performed in in the two direction respectively, after that the reconstruction results are superimposed. Nine images are arranged artificially in three rows and three columns, according to their phase relationship of each of the multifocal images in two directions: three multifocal array images in the same column have the same phase in the X direction and a phase difference of 2π/3 in the X direction, while three multifocal array images in the same column have the same phase in the X direction and a phase difference of 2π/3 in the Y direction, as shown in Fig. 5(a1–a9), respectively.

Figure 5
figure5

The procedure of the reconstruction for MSIM data. (a1a9) The nine raw images. The bracket on their side shows the phases of the image in X and Y directions, respectively. (a10a12) Are the reconstructed images by column while (a14a16) are the reconstructed images by row. (a13) Is the result by superimposing (a10a12) while (a17) is the result by superimposing (a14a16). (a18) The final reconstructed image with spatial resolution enhanced in the X and Y directions.

Figure 6 illustrates the schematic of the reconstruction procedure for MSIM data. We first estimate the parameters of the illumination patterns and then reconstruct the images row by row and column by column, finally we add up these restored images together to obtain the final image with enhancement of spatial resolution in the X and Y directions. The specific reconstruction procedures are described in detail as follows:

  1. (1)

    Applying three-phase standard reconstruction procedure column by column, we get three restored images, as shown in Fig. 5(a10–a12). Compared with the wide-field images, the resolution of these three images is enhanced in the Y direction.

    Since the reconstruction process in the X direction is not involved yet, the three obtained images still have a strip with a same frequency and have a phase difference of 2π/3 between each other. As Fig. 7 illustrated, we superimpose them together to eliminate influence of the stripes on the quality of the image and acquire an image, as demonstrated in Fig. 5(a13), with enhanced resolution in the X direction.

  2. (2)

    Repeat the step (1) by row by row, and obtain the image Fig. 5(a17) with enhanced resolution in the Y direction.

  3. (3)

    Add up the obtained images, Fig. 5(a13,a17) in the two previous steps, and the final image Fig. 5(a18) with improved resolution in both directions can be obtained.

Figure 6
figure6

Schematic of the reconstruction procedure for MSIM data.

Figure 7
figure7

Superimposing the three obtained images to eliminate influence of the stripes on the quality of the image. The three dotted curves in the figure correspond to three terms in the formula respectively, where kar represents the arbitrary spatial frequency. As the figure and formula illustrated, only the direct term irrelative with the frequency as the red line shown is remaining after superimposing the three obtained images, leading to no strips in the final image.

The spatial frequency of the multifocal array is limited by the OTFem(k) of the optical system as well. Compared with the traditional fluorescence microscopy, the proposed method could improve the spatial resolution by a factor of two as the traditional SIM under the linear situation.

Data Availability

The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. 1.

    Kaniecki, K., Tullio, L. D. & Greene, E. C. A change of view: homologous recombination at single-molecule resolution. Nature Reviews Genetics 19(4), 191 (2018).

  2. 2.

    Thompson, M. A., Lew, M. D. & Moerner, W. E. Extending Microscopic Resolution with Single-Molecule Imaging and Active Control. Annual Review of Biophysics 41(1), 321–342 (2012).

  3. 3.

    Sengupta, P., van Engelenburg, S. B. & Lippincottschwartz, J. Superresolution imaging of biological systems using photoactivated localization microscopy. Chemical Reviews 114(6), 3189–202 (2014).

  4. 4.

    Lambert, T. J. & Waters, J. C. Navigating challenges in the application of superresolution microscopy. The Journal of Cell Biology 216(1), 53–63 (2017).

  5. 5.

    Betzig, E. Single Molecules, Cells, and Super-Resolution Optics (Nobel Lecture). Angewandte Chemie 54(28), 8034–8053 (2015).

  6. 6.

    Betzig E., Patterson G. H. & Sougrat, R. et al. Imaging intracellular fluorescent proteins at nanometer resolution[J]. Science, 313(5793), 1642–1645 (2006).

  7. 7.

    Manley, S. et al. High-density mapping of single-molecule trajectories with photoactivated localization microscopy. Nature Methods 5(2), 155 (2008).

  8. 8.

    Sengupta, P., Engelenburg, S. V. & Lippincott-Schwartz, J. Visualizing cell structure and function with point-localization superresolution imaging. Developmental Cell 23(6), 1092–1102 (2012).

  9. 9.

    Bates, M. et al. Multicolor super-resolution imaging with photo-switchable fluorescent probes. Science 317(5845), 1749–1753 (2007).

  10. 10.

    Huang, B. et al. Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy. Science 319(5864), 810–813 (2008).

  11. 11.

    Rust, M. J., Bates, M. & Zhuang, X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nature Methods 3(10), 793 (2010).

  12. 12.

    Hell, S. W. & Wichmann, J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy. Optics Letters 19(11), 780–782 (1994).

  13. 13.

    Klar, T. A. & Hell, S. W. Subdiffraction Resolution in Far-Field Fluorescence Microscopy. Optics Letters 24(14), 954–956 (1999).

  14. 14.

    Westphal, V. et al. Video-rate far-field optical nanoscopy dissects synaptic vesicle movement. Science 320(5873), 246–249 (2008).

  15. 15.

    Gustafsson, M. G. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. Journal of Microscopy 198(2), 82–87 (2010).

  16. 16.

    Chen, J. et al. Super-resolution differential interference contrast microscopy by structured illumination. Optics Express 21(1), 112 (2013).

  17. 17.

    Gustafsson, G. L. M. Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution. Proceedings of the National Academy of Sciences 102(37), 13081–13086 (2005).

  18. 18.

    Mandula, O. et al. Line scan - structured illumination microscopy super-resolution imaging in thick fluorescent samples. Optics Express 20(22), 24167 (2012).

  19. 19.

    Rego, E. H. et al. Nonlinear structured-illumination microscopy with a photoswitchable protein reveals cellular structures at 50-nm resolution. Proceedings of the National Academy of Sciences 109(3), E135–43 (2011).

  20. 20.

    Zhang, H., Zhao, M. & Peng, L. Nonlinear structured illumination microscopy by surface plasmon enhanced stimulated emission depletion. Optics Express 19(24), 24783–24794 (2011).

  21. 21.

    Ströhl, F. & Kaminski, C. F. A joint Richardson—Lucy deconvolution algorithm for the reconstruction of multifocal structured illumination microscopy data. Methods and Applications in Fluorescence 3(1), 014002 (2015).

  22. 22.

    York, A. G. et al. Resolution doubling in live, multicellular organisms via multifocal structured illumination microscopy [J]. Nature Methods 9(7), 749–754 (2012).

  23. 23.

    Patwary, N., Doblas, A. & Preza, C. Image restoration approach to address reduced modulation contrast in structured illumination microscopy. Biomedical Optics Express 9(4), 1630 (2018).

  24. 24.

    Clerk, A. A. et al. Introduction to quantum noise, measurement, and amplification[J]. Reviews of Modern Physics 82(2), 1155–1208 (2010).

  25. 25.

    Ward, E. N. & Pal, R. Image scanning microscopy: an overview. Journal of Microscopy 266(2), 221–228 (2017).

Download references

Acknowledgements

This work is financially supported by the National Key Research and Development Program (2018YFB0504400), the National Natural Science Foundation of China (11574160, 61875093), and the Fundamental Research Funds for the Central Universities, Special Support Plan for Talent Development in Tianjin and 111 Project (B16027). W.L. acknowledges the support of the open research funds of the State Key Laboratory of High Field Laser Physics, Shanghai Institute of Optics and Fine Mechanics (SIOM).

Author information

X.W. and W.L. conceived the original idea. L.F., L.Z. and W.L. implemented the simulations. L.F., L.Z., X.S., Q.X., P.C., X.W. and W.L. analyzed all simulation data and wrote the manuscript. All authors contributed to technical discussion regarding this work.

Correspondence to Xiaolei Wang.

Ethics declarations

Competing Interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.