Main

Structured illumination microscopy (SIM) is the most universally implemented super-resolution modality in the life sciences because it offers fast longitudinal imaging with low phototoxicity and is highly compatible with fluorescent labeling1,2,3. With the flourishing of SIM, a variety of open-source reconstruction algorithms have been developed, such as OpenSIM4, fairSIM5, SIMtoolbox6, HiFi-SIM7 and so on. The availability of open-source software also boosts custom-built SIM hardware platforms, such as SLM-SIM8, DMD-SIM9, galvanometer-SIM10, Hessian-SIM11 and so on. Combining software and hardware has created an open and productive community for SIM researchers.

Compared with 2DSIM, 3DSIM doubles resolution along the z axis1,12,13,14 as well as in the xoy plane. 3DSIM reconstruction algorithms can be found in commercial systems such as GE OMX and Nikon N-SIM, or open-source software such as Cudasirecon1, AO-3DSIM14, SIMnoise15 and 4BSIM16. However, the commercial solutions are limited to specific microscopy platforms. The open-source solutions are all target-specific tools to solve certain imaging problems and are unsuitable for generic 3DSIM reconstruction. They may also lead to serious artifacts or offer poor user-friendliness. On the contrary, in the field of 2DSIM or single-layer 3DSIM, OpenSIM4 explains the principle of SIM reconstruction systematically; fairSIM5 integrates the algorithm into Fiji to facilitate use by biological researchers; HiFi-SIM7 notably optimizes reconstruction results and has a user-friendly graphical user interface. The lack of convenient multilayer 3DSIM software impedes users from accessing and using it, and serious artifacts challenge the fidelity and reliability of 3DSIM. Therefore, a well-established and user-friendly 3DSIM reconstruction tool is urgently needed in the 3DSIM field to ensure its further development.

To address this need, here we report Open-3DSIM, which can provide superior and robust multilayer 3DSIM reconstruction. We prepare the Fiji version to make it easily accessible for biological users; provide intermediate results to help hardware specialists to check their home-built 3DSIM data and open modular source codes for software developers to boost its future developments. Through comparisons with different algorithms on various specimens and signal-to-noise (SNR) levels, we demonstrate that Open-3DSIM offers superior performance due to the optimization of parameter estimation and spectral filtering, resulting in high-fidelity reconstructions with minimal-artifacts and preserved weak information. Furthermore, Open-3DSIM can extract the inherent dipole orientation information, unlocking the full potential of 3DSIM in multilayer, multicolor, polarization and time-lapse super-resolution reconstruction.

The principle of Open-3DSIM is shown in Supplementary Fig. 1 and Supplementary Note 1. The pattern of three-dimensional (3D) structured illumination17 is generated by the interference of three beams through grating diffraction. Then, 3D stack data are taken layer by layer, converted to the frequency domain and separated by a phase-separation matrix. The separated ±first frequency components are shifted to fill the leaky cone of zero frequency component, and the ±second frequency components are shifted to expand the spectrum range of the xoy plane. So, 3DSIM doubles the spectral range compared with wide field by filling the ‘missing cone’ in an optical conversion function (OTF) (Extended Data Fig. 1 and Supplementary Notes 1 and 2), and obtaining 3D super-resolution results.

We use the cross-correlation method5,7 to estimate the frequency, angle, phase and modulation depth of the structured illumination patterns to avoid the involvement of initial parameters that other 3DSIM algorithms require to input1,14,15,16. To obtain a correct frequency parameter, which is the first step of parameter estimation, we use both +first and +second frequency components. Although traditionally estimating the peak of the +second frequency is more accurate than +first, the +first frequency’s peak carries a higher contrast than the +first frequency in low SNR as shown in Fig. 1a. Therefore, we set up a criterion to determine whether estimating through the +second frequency’s peak is reliable. If not, we will use the +first spectrum to estimate instead to guarantee the validity of frequency estimation. Then the corresponding parameter such as phase and angle can be accurately resolved. This method can greatly improve the correctness of parameter estimation when reconstructing images under low SNR, thus reducing various artifacts caused by parameter estimation errors17,18.

Fig. 1: Principle and reconstruction results of Open-3DSIM.
figure 1

a, Separated frequency domain and image under low SNR, demonstrating the first frequency peak is higher than the second and is easier to recognize. The top left, top right and bottom left are the separated zeroth, first and second frequency domains. The bottom right is the raw image of actin filaments in U2OS under extremely low SNR, but the +first pattern (visible because of relatively low frequency) is easy to recognize. The color bars in the upper-right corner of a are in the format ‘Intensity (a.u.)’. b, Two-step filter designed by OTF, Notch, OTFnotch and Apo to optimize the frequency spectrum of 3DSIM, achieving the effect of reducing artifacts and improving resolution. The white lines below represent their corresponding profiles along the estimated frequency. c, Reconstruction results of actin filaments in U2OS, including the comparison of the single-layer algorithm (HiFi-SIM), the multilayer algorithm (Open-3DSIM), wide-field (WF) image and polarized Open-3DSIM image (pOpen). The color bars in ‘pOpen’ of c are the format ‘Angle (rad)’. Scale bar, 2 μm. Scale on the z axis, 12 layers, 0.125 μm per layer. The experiments were repeated three times independently with similar results.

Full size image

Next, inspired by Hifi-SIM7, we designed a two-step filter in the frequency domain based on the notch function (Notch), apodization function (Apo), optical conversion function (OTF) and OTFnotch according to the estimated frequency vector in the xoy and yoz plane. First, we designed a notchfilter corresponding to the estimated frequency and input image size to improve the compatibility of different images. We demonstrate that the cooperation of Filter 1 and Filter 2 will make the 3D spectrum approach the ideal spectrum, being smooth and even18,19, which can suppress the spectrum’s peak, reduce the noise and compensate for high-frequency blurring (Fig. 1b, Extended Data Fig. 1 and Supplementary Note 1 and 2). This method to correct an abnormal spectrum is important for controlling the artifacts of reconstructed images and further improving the 3D resolution, making it perform well under low SNR. We carefully illustrate the function and superiority of the spectrum filters (Extended Data Fig. 2 and Supplementary Note 3) and provide a guide for adjusting the parameters (Extended Data Fig. 3 and Supplementary Note 4).

Figure 1c shows the 3DSIM imaging of actin filaments in U2OS. Open-3DSIM can improve xyz resolution compared to the wide-field images. What is more, Open-3DSIM interpreting whole actin structure can greatly eliminate defocus and artifacts with z axis improvement compared with single-layer reconstruction just like HiFi-SIM (Extended Data Fig. 4, Supplementary Note 5). We also introduce polarization dimension on the xoy plane to realize dipole orientation imaging in Open-3DSIM20,21. The dipole orientation information of the actin filament analyzed is shown in Fig. 1c. The parallelism between dipole orientation and actin filament direction validates the correctness of the polarization information.

To further test the performance of Open-3DSIM, we first conduct the simulation using a petal-shaped structure and a resolution test 3D structure, gaining high-fidelity results with no observable artifacts (Extended Data Fig. 5 and Supplementary Note 6). We also compare Open-3DSIM with traditional Wenier-based 3DSIM reconstruction algorithms such as AO-3DSIM and 4BSIM, finding Open-3DSIM outperforms them compared with their well-adjusted reconstructed results (Extended Data Fig. 6 and Supplementary Note 7). Because SIMnoise optimizes the Wenier-based filter, we compare the reconstruction results of Open-3DSIM, OMX and SIMnoise under gradient SNR for actin filaments in U2OS in Fig. 2a. Here, Open-3DSIM outperforms OMX and SIMnoise in different levels of SNR with fewer artifacts and backgrounds. Also, the mean absolute error (MAE) and SNR of Open-3DSIM are almost the same as those of other algorithms with higher illumination conditions in Fig. 2b, proving the excellent performance of Open-3DSIM under low SNR. Although SIMnoise has optimized the reconstruction under low SNR, Open-3DSIM shows better edge information retention and a denoising effect compared with the state-of-the-art result of SIMnoise in Fig. 2c with less reconstruction time. Similarly, compared with OMX, Open-3DSIM has an excellent ability to remove artifacts and retain weak information in Fig. 2d. More comparisons of reconstruction on the fluorescent quantitative standard sheet (Argolight) and various biological samples can be seen in Extended Data Figs. 7 and 8 and Supplementary Notes 8 and 9 to further prove the excellent performance of Open-3DSIM.

Fig. 2: Reconstruction performance of Open-3DSIM.
figure 2

a, Reconstruction for actin filaments under different illumination conditions of extremely low (EL, intensity of 2%, exposure time of 5 ms), low (L, 5%, 5 ms), moderate (M, 10%, 20 ms) and high (H, 10%, 50 ms) with the comparison among SIMnoise, OMX and Open-3DSIM. b, MAE under different SNRs and algorithms, taking the corresponding highest SNR results as ground-truth images, with the relative SNR of the reconstruction results under different SNRs and algorithms to demonstrate the algorithm’s ability to suppress a defocused background. c, State-of-the-art reconstruction results of tubulin structure from SIMnoise under the lowest SNR with the corresponding comparison. d, Reconstruction for actin filaments in U2OS cell from OMX under a low SNR (5%, 20 ms) with the corresponding comparison. e, The multi-color reconstruction by Open-3DSIM of a COS cell labeled for actin filaments (yellow), clathrin (cyan), and tubulin (red), excited at 488, 561 and 640 nm in wavelength, respectively. f, Open-3DSIM analyzed the 3D mitochondrial ridge structure in live COS7 cell and showed the process of mitochondrial fusion, separation and apoptosis under a low SNR (10%, 3 ms) for 15 frames (10 s and six slices per frame). g, Reconstruction and polarization analysis of actin filaments in mouse kidney sections, including whole 3D sight (on the top) and the maximum-intensity projection on the xoy and xoz planes, with the comparison between pOMX, pSIMnoise and pOpen. The color bars in g are in the format ‘Angle (rad)’. Scale bar is 2 μm. Scales on the z axis are: a, 13 layers; c, 41 layers; d, ten layers; e, 13 layers; f, six layers; and g, 41 layers. a, c, d, f, g 0.125 μm per layer; e, 0.12 μm per layer.

Full size image

Furthermore, we demonstrate that Open-3DSIM can accurately reconstruct multicolor samples in Fig. 2e with N-SIM system and Extended Data Fig. 9, and Supplementary Note 10 with the OMX system, which shows excellent optical sectioning ability and compatibility with different microscope workstations. Open-3DSIM is also applied to analyze the 3D mitochondrial ridge structure in time-lapse imaging. We observe a clear ridge structure with the process of mitochondrial fusion, separation and apoptosis under low SNR as shown in Fig. 2f. Last, we use Open-3DSIM to reconstruct actin filament structure in a mouse kidney with dipole orientation imaging shown in Fig. 2g; it can be seen that polarized Open-3DSIM image (pOpen) outperforms polarized OMX image (pOMX) and polarized SIMnoise image (pSIMnoise) to a great extent in artifact suppression and defocus elimination, and more reconstructions of fluorescent dipole orientation are shown in Extended Data Fig. 10 (Supplementary Note 11). It is noteworthy that, with intensity calibration, the dipole orientation will be more accurately resolved by Open-3DSIM than with default assumptions. To help users to use Open-3DSIM, we also provide the parameters of reconstruction in Supplementary Table 1 and typical 3DSIM data in Supplementary Table 2 with results, parameters and comparisons.

With excellent performance on high-fidelity, low-artifact, defocus-removal and weak information retaining, Open-3DSIM can truly unleash the potential of 3DSIM in lateral and axial super-resolution, through multicolor, multilayer, time-lapse and dipole orientation imaging. Open-3DSIM is multiplatform, not limited by specific microscope workstations and can be customized and adjusted according to user needs. It is also extensible, as it is fully compatible with other algorithm optimization methods based on regular terms or machine learning. We expect Open-3DSIM to be the open-source standard for the 3DSIM multi-dimensional reconstruction and believe that open-source efforts will make a great difference to the community.

Methods

Sample preparation

Human osteosarcoma U2OS cell lines (HTB-96, ATCC) were cultured in Dulbecco’s modified Eagle’s medium (DMEM, GIBCO) containing 10% heat-inactivated fetal bovine serum (GIBCO) and 100 U ml−1 penicillin and 100 µg ml−1 streptomycin solution (PS, GIBCO) at 37 °C in an incubator with 95% humidity and 5% CO2. Fixed cells were plated on no. 1.5H cover glasses (CG15XH, Thorlabs) before they were grown to a suitable density (24 h) and fixed with 4% formaldehyde (R37814, Invitrogen) for 15 min. Then, the cells were washed three times with PBS to remove the formaldehyde. Alexa Fluor 488 Phalloidin (A12379, Invitrogen) was used to stain the actin filaments for 1 h at room temperature. Then, the coverslip was sealed on the slide with the prolong antifade mountant (P36941, Invitrogen).

The COS7 cells were cultured in DMEM (GIBCO) containing 10% (V/V) fetal bovine serum (GIBCO) and 100 U ml−1 penicillin and 100 µg ml−1 streptomycin solution (PS, GIBCO) at 37 °C in an incubator with 95% humidity atmosphere and 5% CO2. The selected cell lines were seeded on no. 1.5H cover glasses (CG15XH, Thorlabs) before they were grown to a suitable density (24 h) via incubation in a humid atmosphere containing 5% (V) CO2 at 37 °C. The cells were stained in DMEM containing 500 nM MitoTracker Orange (M7510, ThermoFisher) and 0.5% dimethyl sulfoxide for 20 min in a CO2 incubator. The cells were washed with PBS and fixed in 4% formaldehyde (R37814, Invitrogen) for 15 min at room temperature. The coverslip was sealed onto the cavity of the slide. The coverslip was sealed on the slide with the prolong antifade mountant (P36941, Invitrogen). For live COS7 cell imaging, the COS7 cells were seeded on µ-Slide 8 Wellhigh (80806, ibidi). The COS7 cells were stained in DMEM containing 500 nM IMMBright660 and 0.5% dimethyl sulfoxide for 20 min in a CO2 incubator. Then, the cells were kept in DMEM for live-cell imaging without washing. For multicolor COS7 cells imaging, COS7 cells were cultured, fixed with a glutaraldehyde-based extraction-fixation procedure, and stained for actin (phalloidin-Atto 488, Sigma 49409), clathrin heavy chain (abcam ab21679; donkey anti-rabbit Alexa Fluor 555, ThermoFisher Scientific A31572) and alpha-tubulin (Sigma T5168 and T6199; donkey anti-mouse Alexa Fluor 647, ThermoFisher Scientific A31571) according to a previously published protocol22.

The mouse kidney sections used to reconstruct the actin filaments were purchased from ThermoFisher. Actin filaments were labeled by Alexa Fluor 568 using Prolong Diamond to embed.

The COS7 cells used to reconstruct the nuclear pore complex were purchased from GATTA Quant. Anti-Nup was labeled by Alexa Fluor 555 using Prolong Diamond to embed.

The bovine large artery endothelial cells used to reconstruct the multicolor sample were purchased from ThermoFisher. Mitochondria were labeled with red-fluorescent MitoTracker Red CMXRos, F-actin was stained using green-fluorescent Alexa Fluor 488 phalloidin and blue-fluorescent 4,6-diamidino-2-phenylindole was used to label the nuclei using Prolong Diamond to embed.

Data acquisition

We obtain data based on the commercial OMX-SIM system (DeltaVision OMX SR, GE) using an oil immersion objective (Olympus, ×60 1.4 numerical aperture (NA)) and a commercial N-SIM system (Nikon) using an oil immersion objective (CFI Apochromat, ×100 1.49 NA).

For the OMX system, 3DSIM sequences were performed with a pixel size of 80 and 125 nm in the xoy and xoz plane (five phases, three angles and 15 raw images per plane). And for the N-SIM system, 3DSIM sequences were performed with a pixel size of 65 and 120 nm in the xoy and xoz plane (five phases, three angles and 15 raw images per plane).

Samples in Fig. 2c and Extended Data Fig. 8a were obtained from open-source data in SIMnoise (v.1.0)15,23 (https://data.4tu.nl/articles/_/12942932). COS7 cells in Fig. 2e were obtained from C. Leterrier (Aix Marseille University) using the Nikon N-SIM system equipped with a 100X, NA 1.49 objective and a Hamamatsu Fusion BT camera. Samples in Extended Data Figs. 2a, 3a and 4b were obtained from open-source data in fairSIM (v.1.5.0)5 (http://www.fairsim.org/). The simulated structure in Extended Data Fig. 5a was obtained from open-source 3D structure data (v.1.2)24 (https://github.com/Biomedical-Imaging-Group/GlobalBioIm). The simulated resolution test image in Extended Data Fig. 5b is ISO12233:2000 (Imatest). Extended Data Fig. 6a was obtained from AO-3DSIM (v.1.0.0)14 (https://www.ebi.ac.uk/biostudies/studies/S-BSST629). Extended Data Fig. 6b was obtained from 4BSIM (v.1.0)16 (https://zenodo.org/record/6727773).

Raw image format

Open-3DSIM supports three data formats including OMX (*.dv/*.tif/*.tiff), N-SIM (*.nd2/*.tif/*.tiff) and some home-built systems (*.tif/*.tiff). The picture sequence for OMX is phase, depth, channel, time, then angle. The picture sequence for N-SIM is image (angle in rows and phase in columns), depth, channel, then time. The picture sequence for home-built systems is phase, angle, depth, channel, then time. We suggest using samples of six or more layers for reconstruction to obtain optimal results.

Image processing

The frequency domain and SNR of images were generated by ImageJ. HiFi-SIM (v.1.01)7, SIMnoise (v.1.0)15, OMX, AO-SIM (v.1.0.0)14 and 4BSIM (v.1.0)16 are used for comparison. The reconstructions of Open-3DSIM were processed with MATLAB (v.2018a, 2021a and 2021b) and ImageJ. Figures were plotted with Imaris (v.9.0.1), Visio (v.2016), Origin (v.2021b) and Adobe Illustrator (v.2020).

Calibration of the illumination nonuniformity

Fluorescent beads (ThermoFisher Scientific) of 200 nm in diameter were prepared in a fixed slide20. The beads were imaged by OMX for three angles with five phases in each angle. By averaging images of the five phases, wide-field images of each angle can be obtained. To correct the light intensity at each point, the beads should be thick on the focal plane. Then interpolation is used to compensate the gap of beads to form a light intensity map of calibang1, calibang2 and calibang3. The calibration ratio of calib1 and calib2 can be expressed as:

$${\mathrm{calib}}_{1}={\mathrm{calib}}_{\mathrm{ang}2}/{\mathrm{calib}}_{\mathrm{ang}1},{\mathrm{calib}}_{2}={\mathrm{calib}}_{\mathrm{ang}3}/{\mathrm{calib}}_{\mathrm{ang}1}$$

Calibration of MAE and SNR

MAE and SNR in Fig. 2b are used to evaluate the reconstruction of Open-3DSIM, SIMnoise and OMX-SIM. After image calibration to compensate for vibrating during shooting, the corresponding results of high SNR under different algorithms are used as ground truth y(i), so the images of low SNR y′(i) are calculated using MAE:

$${\mathrm{MAE}}=\frac{1}{m}\mathop{\sum }\limits_{i=1}^{m}\left|\,y(i)-y{\prime} (i)\right|$$

where i denotes the ith pixel of whole m pixels, and SNR is used to evaluate the ability to remove the defocused background of different algorithms. We choose the part with signal and the part without signal to solve its mean value and variance, and use the following formula to solve SNR (dB):

$${\mathrm{SNR(dB)}}=10{\log }_{10}\frac{{\mathrm{mean}}\left({\mathrm{signal}}\right)-{\mathrm{mean}}({\mathrm{noise}})}{{\mathrm{s.d.}}({\mathrm{noise}})}$$

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.