Introduction

Three-dimensional (3D) imaging of live cells and their subcellular structures is crucial for the study of cell biology and provides invaluable information about related diseases1. Various 3D imaging approaches that were developed to investigate subcellular structures and constituents of live cells have several advantages, including high spatial and temporal resolution, molecular specificity, and noninvasiveness2,3,4,5. Fluorescence microscopy, a widely used microscopic method, can obtain molecule-specific information by labeling specific molecules inside cells with fluorescent proteins or dyes2. In particular, several super-resolution microscopic techniques that allow fluorescence imaging beyond the diffraction limit have been reported6,7,8,9,10. Among them, 3D structured illumination microscopy (SIM) provides 3D fluorescence images beyond both the lateral and the axial diffraction limit of conventional fluorescence microscopy by exploiting spatially structured excitation7,11,12,13. However, despite outstanding spatial resolution, 3D SIM has poor temporal resolution due to the time-consuming projection of the structured illumination sequence, mechanical axial scanning, and long acquisition time for dim fluorescence signals. Furthermore, 3D SIM may induce phototoxicity that could affect the cells, an inevitable problem of fluorescence microscopy2,14.

In parallel, optical diffraction tomography (ODT) or 3D quantitative phase imaging (QPI) techniques have emerged as methods for label-free quantitative imaging of 3D refractive index (RI) distributions of biological samples15,16,17,18,19,20. To reconstruct 3D RI distribution, multiple two-dimensional (2D) holograms of a sample are measured using various incident angles. From the holograms, a 3D RI tomogram can be reconstructed via the principle of inverse light scattering21. Because the imaging contrast of ODT is a function of the RI, ODT is label-free and noninvasive, requires no preparation, has rapid image acquisition, and provides quantitative imaging22. The RI distribution of live cells has provided structural and biochemical information about biological samples in studies in the fields of cell biology23,24, microbiology25, hematology26, infectious diseases27, plant biology28, and biophysics29,30. Nonetheless, the RI generally provides limited molecular specificity31, except for certain materials with distinct RI values such as lipids32,33 and metallic particles34,35.

Because both fluorescence imaging and QPI provide complementary information about cell pathophysiology, efforts have been made to combine and utilize both modalities for correlative bioimaging36,37,38,39,40,41,42,43,44,45. Several techniques for performing both 3D wide-field fluorescence imaging and 3D RI tomography of cells have been reported33,39,40. More importantly, for super-resolution 3D fluorescence imaging, Chowdhury et al. performed 3D SIM and ODT simultaneously utilizing a liquid crystal spatial light modulator (SLM) for structured illuminations41, and Descloux et al. combined 3D phase imaging with super-resolution optical fluctuation imaging by spatial filtering of stacked images measured at various axial positions42. However, use of an SLM prevents fast imaging, one of the advantages of ODT. Furthermore, photobleaching and phototoxicity inevitably follow the use of an SLM.

In this paper, we propose and demonstrate with experiments a multimodal approach for measuring both the 3D RI and fluorescence distributions of live cells that combines ODT and 3D SIM using a digital micromirror device (DMD). Time-multiplexed structured illumination patterns displayed by the DMD are used to perform 3D super-resolution fluorescence imaging and obtain 3D RI tomograms of samples simultaneously and effectively. The fast dynamic response of the DMD enables imaging of live cells with high spatiotemporal resolution, molecular specificity, and reduced damage from phototoxicity. Because the same optical imaging system with the DMD is used for both modes, the correlation between 3D fluorescence and 3D RI data can be easily analyzed. The present method will open a new avenue into the study of cell biology, medical research, and diagnosis.

Results and Discussion

Optical setup

Figure 1 shows the schematic of the optical setup, equipped with a DMD (DLP6500EVM, Texas Instruments, USA), used in our experiments. This setup was able to operate in both the ODT mode and the 3D SIM mode [Fig. 1(a–c)] by modifying a commercial ODT setup (HT-1H, Tomocube Inc., Republic of Korea) to incorporate fluorescence excitation.

Figure 1
figure 1

(a) Experimental setup: I1, I2, irises; 1×2 SMFC, 1×2 single-mode fiber optic coupler; BS1, BS2, beam splitters; L1-L4, lenses (f1 = 250 mm, f2 = 180 mm, f3 = f4 = 150 mm); P, polarizer; Cam1, Cam2, cameras. (b) Setup for ODT based on a Mach-Zehnder interferometer equipped with a DMD. (c) Setup for 3D SIM based on epifluorescence microscopy.

Full size image

To minimize photobleaching and phototoxicity that occur in fluorescence imaging of live cells, the two operating modes are separate utilizing dispersion property of a DMD and two lasers with different wavelengths (λODT = 473 nm, MSL-FN-473, CNI Laser, P.R. China) and λSIM = 532 nm (LSS-0532, Laserglow Technologies, Canada)] for coherent imaging and fluorescence excitation, respectively. To minimize phototoxicity46, the laser power for ODT was attenuated to 0.11 mW and the exposure time was shorter than 0.1 s. To simplify the setup, the two laser beams were coupled into a single-mode fiber using an RGB combiner (RGB30HF, Thorlabs Inc., USA). By selectively blocking one of the laser beams, the present setup alternated between the ODT mode and the 3D SIM mode. In both modes, the illumination part that included the DMD, lens L1 (f1 = 250 mm), and objective lens 1 [UPLSAPO 60XW, numerical aperture (NA) 1.2, Olympus Inc., Japan] was the same, but the modes used different measurement schemes [Fig. 1(b,c)].

ODT mode

In the ODT mode, a sample is illuminated with various illumination patterns and the optical fields diffracted from the sample are measured using Mach-Zehnder interferometry22 [Fig. 1(b)]. A 1 × 2 single-mode fiber optic coupler (SMFC) (TW560R3F1, Thorlabs) splits the laser beam (λODT = 473 nm) into a reference beam and a sample beam. To control the angle of the beam impinging on a sample, the DMD displays time-multiplexed patterns that are projected onto the sample47. Using objective lens 2 (UPLSAPO 60XW, NA 1.2) and lens L2, (f2 = 180 mm), the beam diffracted from the sample is collected and projected onto the image plane, where it interferes with the slightly tilted reference beam to generate an off-axis hologram. Multiple 2D holograms with various illumination patterns are recorded using an image sensor (Cam1; FL3-U3-13Y3M-C, FLIR Systems, USA).

The DMD displays a time-multiplexed pattern that systematically controls the illumination angles. The superposition of the two plane waves from the DMD results in the formation of structured illumination that impinges on the sample [Fig. 2(a,b)]. The two plane waves diffracted by the displayed pattern in the DMD are mutually conjugated optical fields. Thus, the phase shift Δϕ between the plane waves and the incident angles of the plane waves on the sample can be precisely controlled by the pattern displayed by the DMD. In addition, the time-multiplexing method significantly reduces unwanted diffraction noise that usually occurs when binary control is used in a DMD. By exploiting temporal averaging unwanted diffraction from a DMD, the time-multiplexing method enables to reduce unwanted diffraction from the DMD47. For efficiency, the limited number of time-multiplexed patterns stored on the DMD control board consists of a three-binary-pattern sequence.

Figure 2
figure 2

Reconstruction of the 3D RI tomogram of a sample. (a) Structured illumination generated by a DMD is the superposition of two plane waves with a controlled wave vector and phase shift Δϕ. (b) Time-multiplexed patterns displayed on the DMD and (c) the corresponding holograms of a sample. Insets: magnified image (upper) and corresponding Fourier spectrum in the spatial frequency k space (lower). Except for sample illumination at a normal angle, a pair of patterns projects two plane waves with different phase shifts onto a sample to decompose diffracted optical fields with respect to individual plane wave components. (d) Phase images of the retrieved optical fields corresponding to the plane wave components with various incident angles. Inset: corresponding Fourier spectrum. (e) A mapped object function in 3D Fourier space using the principle of ODT.

Full size image

In addition to the normal angle, we used 28 time-multiplexed patterns (14 pairs) to generate 28 plane waves at different azimuthal angles so that the illumination beams scanned a circular pattern within the NA of the condenser lens [Fig. 2(b)]. To decompose the time-multiplex pattern into two plane waves, a pair of structured patterns with different relative phase shifts, 0 and π, respectively, was needed. Therefore, 28 time-multiplexed patterns were constructed and displayed to produce the 28 plane waves used for circular scanning48.

The optical fields diffracted from a sample are retrieved from the measured holograms corresponding to the time-multiplexed structured illuminations [Fig. 2(c)] using a field retrieval algorithm based on Fourier transform [insets in Fig. 2(c)]49. Then, using the known phase shifts in the structured illuminations, the retrieved optical fields under the structured illuminations are decomposed into the scattered fields from the sample with respect to plane-wave illumination components as follows:

$$[\begin{array}{c}{S}_{1}\\ {S}_{2}\end{array}]={[\begin{array}{c}1\\ 1\end{array}\begin{array}{c}{e}^{i{\rm{\Delta }}{\varphi }_{1}}\\ {e}^{i{\rm{\Delta }}{\varphi }_{2}}\end{array}]}^{-1}\,[\begin{array}{c}{E}_{1}\\ {E}_{2}\end{array}],$$
(1)

where E1,2, Δϕ1,2, and S1,2 are the retrieved fields, the known phase shifts, and the decomposed scattered fields for each plane-wave component [Fig. 2(d)], respectively. Random fluctuations in an interferometer cause a varying global phase on the retrieved field. To compensate the random fluctuations, the global phase for each of the retrieved fields is normalized to let the coefficient of S1 become 1 before the decomposition. From the obtained scattered fields, an object function in 3D Fourier space is mapped using the principle of ODT to reconstruct the 3D RI distribution of the sample [Fig. 2(e)]15,21,50. To compensate for the uncollected side scattering signals that result from the limited NA of the objective lenses, the iterative regularization algorithm based on the assumption of non-negativity was used51,52.

3D SIM mode

In the 3D SIM mode, the same DMD generates structured illumination patterns that excite fluorescent molecules in the sample plane [Fig. 1(c)]. An excitation laser beam (λSIM = 532 nm) is reflected from the DMD displaying a time-multiplexed pattern, and then the time-multiplex pattern is projected onto a sample through lens L1 and objective lens 1. This excitation configuration enables fast and stable generation of structured illumination patterns. Based on an epifluorescence detection scheme, fluorescence images of the sample are obtained in sequence for each structured illumination pattern at each axial position using the same objective lens. An sCMOS camera (Cam2; C11440-22C, Hamamatsu Photonics, Japan) records the fluorescence images via objective lens 1 and relay lenses L1, L3, and L4 (f3 = f4 = 150 mm). A beam splitter (BS1) and a band-stop filter (NF01-532U-25) are used to block excitation beams in the path of the fluorescence beam.

To obtain a 3D fluorescence image with subdiffraction resolution, we used the principle of SIM (Fig. 3)7,13. SIM images were reconstructed from fluorescence images obtained using structured excitation patterns. To project a structured excitation pattern onto a sample, the DMD is used to rapidly generate a time-multiplexed pattern comprising three plane waves [Fig. 3(a,b)]. The two oblique plane waves are mutually conjugated fields that are diffracted by the displayed pattern, while the third plane wave has a normal incident angle. The various patterns displayed on the DMD systematically control the phase shift and the incident angles of the two oblique plane waves relative to the residual plane wave. To reconstruct the SIM images, 15 time-multiplexed patterns are displayed as a combination of three azimuthal angles and five phase shifts [Fig. 3(b)]7.

Figure 3
figure 3

Reconstruction of the 3D fluorescence image of a sample. (a) Epifluorescence microscopy geometry for structured illumination excitation of a sample and recording its fluorescence emission. A DMD controls both the phase shift and the illumination angles of three plane waves that result in the structured excitation pattern on a sample. (b) Time-multiplexed patterns displayed on the DMD (inset: corresponding Fourier spectrum) and (c) corresponding fluorescence images of a sample (inset: magnified images). (d) and (e) Illustrations in Fourier space for describing the SIM reconstruction from (d) measured fluorescence images to (e) reconstructed SIM images. (f) Reconstructed SIM and wide-field fluorescence (WF) images of a cluster of red fluorescent microspheres.

Full size image

A 2D SIM image is reconstructed from the recorded fluorescence image7,53. Fluorescence emission results from an excitation intensity pattern. A recorded fluorescence image is the sum of component fluorescence images with the five different phase shifts, \({F}_{n}={\sum }_{m\,=\,-2}^{2}{C}_{m}\,{e}^{mi{\rm{\Delta }}{\varphi }_{n}}\), where n is the phase-shift index, F n is a recorded fluorescence image, C m is an indexed component image, and Δϕ n is the relative phase shift of the three plane waves that result in structured illumination. Using the known phase shifts controlled by the DMD, the component images can be decomposed as follows:

$$[\begin{array}{c}{C}_{-2}\\ {C}_{-1}\\ {C}_{0}\\ {C}_{1}\\ {C}_{2}\end{array}]={[\begin{array}{c}{e}^{-2i{\rm{\Delta }}{\varphi }_{1}}\\ {e}^{-2i{\rm{\Delta }}{\varphi }_{2}}\\ {e}^{-2i{\rm{\Delta }}{\varphi }_{3}}\\ {e}^{-2i{\rm{\Delta }}{\varphi }_{4}}\\ {e}^{-2i{\rm{\Delta }}{\varphi }_{5}}\end{array}\begin{array}{c}{e}^{-i{\rm{\Delta }}{\varphi }_{1}}\\ {e}^{-i{\rm{\Delta }}{\varphi }_{2}}\\ {e}^{-i{\rm{\Delta }}{\varphi }_{3}}\\ {e}^{-i{\rm{\Delta }}{\varphi }_{4}}\\ {e}^{-i{\rm{\Delta }}{\varphi }_{5}}\end{array}\begin{array}{c}1\\ 1\\ 1\\ 1\\ 1\end{array}\begin{array}{c}{e}^{i{\rm{\Delta }}{\varphi }_{1}}\\ {e}^{i{\rm{\Delta }}{\varphi }_{2}}\\ {e}^{i{\rm{\Delta }}{\varphi }_{3}}\\ {e}^{i{\rm{\Delta }}{\varphi }_{4}}\\ {e}^{i{\rm{\Delta }}{\varphi }_{5}}\end{array}\begin{array}{c}{e}^{2i{\rm{\Delta }}{\varphi }_{1}}\\ {e}^{2i{\rm{\Delta }}{\varphi }_{2}}\\ {e}^{2i{\rm{\Delta }}{\varphi }_{3}}\\ {e}^{2i{\rm{\Delta }}{\varphi }_{4}}\\ {e}^{2i{\rm{\Delta }}{\varphi }_{5}}\end{array}]}^{-1}\,[\begin{array}{c}{F}_{1}\\ {F}_{2}\\ {F}_{3}\\ {F}_{4}\\ {F}_{5}\end{array}].$$
(2)

Next, the Fourier spectra of the decomposed component images are shifted in the Fourier space to reconstruct a SIM image using the principle of SIM [Fig. 3(d,e)]. Furthermore, a 3D SIM image can be obtained by stacking 100 reconstructed SIM images using a 150-nm interval along the axial direction [Fig. 3(f)]13,53. To compare the spatial resolution of the 3D SIM images with the 3D wide-field fluorescence (WF) images, 3D WF images are reconstructed by stacking the 2D WF images along the axial direction.

Measurements of a monolayer of fluorescent beads

To demonstrate the validity of the proposed method, we measured the 3D RI distribution and 3D fluorescence image of a cluster of red fluorescent 500-nm-diameter PS beads (Fig. 4). In Fig. 4(a,c,e) for the whole field of view (FOV) and in Fig. 4(b,d,f) for the magnified FOV, a monolayer of fluorescent beads is clearly seen in both the 3D RI distribution and the 3D fluorescence image. In addition, the reconstructed 3D RI distribution matches the reconstructed 3D SIM and WF images well. This clearly shows the feasibility of our method.

Figure 4
figure 4

3D RI distribution and 3D fluorescence image of a cluster of red fluorescent PS beads. (a) and (b) Cross-sectional slices of reconstructed 3D RI and (c–f) 3D fluorescence distributions in x-y, x-z, and y-z planes of a cluster of red fluorescent 500-nm-diameter PS beads. The slices in each column were reconstructed using (a,b) ODT, (c) and (d) SIM, and (e,f) WF. The slices in (b,d,f) are magnified images of the area in the dashed boxes in (a,c), and (e,g) and (h) Cross-sectional slices of the 3D Fourier spectrum of 3D fluorescence distributions reconstructed using WF and SIM, respectively. Illustrations on the right-hand side of (g,h) are representations of the theoretical transfer function of WF and SIM.

Full size image

The lateral and axial resolutions of ODT in the proposed method were 172 nm and 665 nm, respectively, as calculated from the spatial bandwidth of the reconstructed 3D RI along the lateral and axial directions50. Because the axial resolution of ODT in the proposed method was larger than the diameter of the PS beads, the reconstructed RI distribution of the beads showed elongated images in the axial direction as well as RI values smaller than the known RI value of PS (n = 1.608 at λ = 473 nm)15,17.

The cross-sectional slices of the reconstructed 3D SIM images [Fig. 4(c,d)] show the ability to perform optical sectioning and enhanced lateral and axial resolutions, 127 nm and 408 nm, respectively, of 3D SIM compared to 3D WF, 254 nm and 816 nm, respectively [Fig. 4(e,f)]. The lateral and axial resolutions are calculated from the spatial bandwidth of the reconstructed 3D image along the lateral and axial directions50. Moreover, the extended lateral and axial bandwidths are clearly seen in the 3D Fourier spectrum of the 3D SIM image in Fig. 4(h) compared to those seen in the spectrum of the 3D WF image [Fig. 4(g)]. In particular, the cross-sectional axial slice of the 3D Fourier spectrum of the 3D SIM does not exhibit the missing cone, which causes severe deterioration of the 3D WF image along the axial direction7.

4D Time-lapse measurements of fluorescent beads inside a HeLa cell

The proposed multimodal method combines the advantages of ODT and 3D SIM by providing label-free 3D RI distribution with superior spatiotemporal resolution along with high molecular specificity, which can be effectively demonstrated by measuring movements of a small object inside a complex 3D RI distribution54. To demonstrate the synergistic advantages and the applicability of the proposed method, we measured the dynamics of the 3D RI distribution of 500-nm-diameter fluorescent PS beads ingested by a HeLa cell (Fig. 5).

Figure 5
figure 5

Demonstration of the applicability of the proposed method. Cross-sectional slices of (a) the reconstructed 3D RI and (b) fluorescence distributions in the x-y, x-z, and y-z planes of a HeLa cell containing fluorescent PS beads. (c) 3D rendered isosurfaces of the HeLa cell from the reconstructed 3D RI distributions in (a). (d) 3D rendered isosurfaces of the HeLa cell after labeling the fluorescent beads by combining the reconstructed 3D RI and fluorescence distributions in (a) and (b). (e) Time-lapse 3D RI distributions of the HeLa cell. The labeled fluorescent beads are tracked using spatiotemporal correlations (see Visualization 1 and Visualization 2).

Full size image

First we measured the 3D RI distribution and the 3D fluorescence image of PS beads inside a HeLa cell using the ODT and 3D SIM modes, respectively [Fig. 5(a,b)]. The 3D isosurface of the cell [Fig. 5(c)] rendered from the reconstructed 3D RI distribution [Fig. 5(a)] shows that the fluorescent beads inside the cell are difficult to distinguish from the surrounding cytoplasm because the axial resolution of ODT is larger than the diameter of the beads (see Section 3.1). In contrast, the correlation of the 3D RI distribution with the measured 3D SIM image clearly shows the fluorescent beads [Fig. 5(d)], where the reconstructed RI values of the identified fluorescent beads replaced the known RI value of PS (n = 1.608 at λ = 473 nm).

After identifying the fluorescent beads in the reconstructed 3D RI distribution, we obtained time-lapse 3D RI distribution measurements of the HeLa cell every 30 s over 1 h using ODT [Fig. 5(e)]. Because the subcellular dynamics of the HeLa cell was slow compared to the tomographic acquisition speed, the reconstructed 3D RI distributions of the beads between consecutive measurements were highly correlated, which enabled the 3D dynamics of the fluorescent beads inside the HeLa cell to be tracked without additional 3D SIM measurements [Fig. 5(e)].

Conclusion

In this paper, we proposed multimodal approach to combine ODT and 3D SIM using a DMD and experimentally demonstrated its feasibility and applicability. Each mode can be operated separately by using dispersion property of a DMD and different illumination wavelengths. As a result, the combination of the respective advantages of ODT and 3D SIM yields the synergistic effects of our method. To demonstrate the feasibility of the proposed method, we reconstructed 3D RI and fluorescence distributions of a planar cluster of fluorescent PS beads using the ODT and 3D SIM modes, respectively. Furthermore, to demonstrate the applicability of our method, we reconstructed 3D fluorescence and time-lapse 3D RI distributions of beads inside a HeLa cell. By exploiting the spatiotemporal correlations of 3D fluorescence and time-lapse 3D RI distributions, the beads were tracked as the cell changed morphologically.

Our proposed method provides 3D structural and biochemical information on fluorescent-labeled live cells with high molecular specificity. The advantages of our method, such as its high spatiotemporal resolution and minimization of the photobleaching problem, provide it the potential for use in a wide range of applications, including the study of subcellular dynamics over a long time.

Materials and Methods

Microsphere preparation

The red fluorescent PS beads (L3280, Sigma-Aldrich, St. Louis, MO, USA) were diluted in distilled water, placed on a coverslip, and dried overnight so that the beads adhered to the surface of the coverslip. Before measurements were taken, phosphate-buffered saline (PBS) solution (pH 7.4, 50 mM, Welgene, Gyeongsangbuk-do, South Korea) was gently dropped on the coverslip, which was then covered by another coverslip.

HeLa cell preparation

HeLa cells, a human cervical cancer cell line, were cultured in Dulbecco’s modified Eagle’s medium (DMEM) (Welgene) with 10% fetal bovine serum (FBS) (Welgene) and 1% penicillin streptomycin [100× (v/v), Welgene] inside a humidified incubator (37 °C, 5% CO2, 95% air). After 12 h of incubation, trypsin EDTA (Welgene) was applied to detach the cells from the culture flask. The detached cells were then collected via centrifugation (1000 rpm for 3 min) and incubated on a coverslip using the same previous culture conditions for 3 h. The diluted red fluorescent PS beads (100× diluted from stock solution) were added to the culture medium and became inserted as fluorescent probes inside the HeLa cells. Cells on the coverslip were maintained with the bead solution for 12 h and then washed three times with PBS solution.