We explain how to concentrate light simultaneously at multiple selected volumetric positions by means of a 4D illumination light field. First, to select target objects, a 4D imaging light field is captured. A light field mask is then computed automatically for this selection to avoid illumination of the remaining areas. With one-photon illumination, simultaneous generation of complex volumetric light patterns becomes possible. As a full light-field can be captured and projected simultaneously at the desired exposure and excitation times, short readout and lighting durations are supported.
Fast optical volumetric recording, as required in numerous applications, still remains a challenge in microscopy. While regular scanning methods (e.g., confocal microscopy and its variants) can be too slow, specialized light-sheet scanning1, 3D imaging with spatial light modulators (SLMs)2, multi-focus microscopy3, and light-field microscopy (LFM)4 promise faster scanning rates.
LFM5 captures the 4D incident light by means of a microlens array (MLA) placed at the intermediate image plane of the imaging path, and thus supports the computation of a focal stack with a single sensor recording. The limited spatial resolution that is caused by simultaneously multiplexing 2D location and 2D direction information at the same image sensor can be increased computationally by 3D deconvolution6 and by using additional phase masks in the optical path7.
Besides fast readouts, precise controllable excitation is another requirement in recent microscopy techniques. In fluorescence microscopy, for instance, such methods are used to accurately illuminate only certain parts of a probe and to avoid out-of-focus excitation or even damaging the probe by photobleaching or phototoxicity. Spatially controllable excitation can be achieved by sequentially illuminating the sample with a focused beam while either moving the sample or scanning the beam with a mirror8. Multiple simultaneous beams can be generated with spatial light modulators (SLMs), such as digital micromirror devices9, or phase-modulators (PMs), such as liquid crystal PMs used for holographic projection10. The latter require coherent light produced by a laser light source.
One-photon excitation as discussed above, however, also excites the probe in the light cone above and below the focal plane, as illustrated in Fig. 1a, which makes controlled excitation within a volumetric probe impossible. Out-of-focus excitation can be avoided by two-photon techniques11,12,13,14, as shown in Fig. 1b. This allows deeper penetration in scattering media than simple one-photon methods, but limits simultaneous excitation to a single plane.
Various light-modulation approaches exist that strive for controlled, simultaneous excitation of multiple selected sample points within a volume. Combining two micromirror arrays in the illumination path of a microscope (one in the intermediate image plane and one in the back focal plane15,16), for instance, enables the generation of spatial-angular controlled light patterns to avoid out-of-focus excitation. However, the interplay of two 2D SLMs does not support arbitrary illumination patterns within a volume. Either angular sampling is constant for a spatial location, or spatial sampling is constant for a direction. Thus, an illumination pattern as shown in Fig. 1c cannot be achieved simultaneously, but requires sequential excitation.
Holographic projection with PMs2,17,18 overcomes this primary limitation. However, spatially varying diffraction efficiency and the presence of zero-order diffraction spots, ghosting, and intensity fluctuations (speckles) limit the excitation area and the spatial and temporal resolution of holographic projections19,20. Furthermore, coherent computer-generated holograms limit the types of illumination patterns that can be generated21.
We explain how to concentrate light simultaneously at multiple selected volumetric positions by means of a 4D illumination light field22. First, a 4D imaging light field is captured to select target objects. A light field mask is then computed automatically for this selection to avoid illumination of the remaining areas. For one-photon illumination, the simultaneous generation of complex volumetric illumination patterns, like the ones shown in Figs 1c and 7, becomes possible. In contrast to holographic projection approaches, light-field projection supports arbitrary patterns and non-coherent light sources, and retains the same sampling quality across the whole field of view. Since a full light field can be captured and projected simultaneously at the desired exposure and excitation times, short readout and illumination durations are supported.
Masked Light-Field Illumination
Figure 2 illustrates the principle of volumetric excitation by means of masked light-field illumination using 10–20 μm fluorescent microspheres. With one sensor recording, we capture a 4D light field of the probe under full illumination and apply synthetic aperture rendering23,24 to compute a 3D focal stack or individual perspective images from it. The focal stack undergoes 3D deconvolution25 to obtain a defocus-free z-stack. Within the z-stack, we select parts of the probe (i.e., 3D-segmented microspheres, shown in red in our example) that are to be excited. From the selection in the z-stack, we then determine a 4D light-field mask that is projected simultaneously into the probe such that as much light as possible is concentrated at the volumetric positions of the selection while minimizing the illumination of other regions. Rerecording a light field using volumetric excitation shows selected regions in perspective images and z-stack (Fig. 2 orange).
Computing the masked light field that leads to the desired volumetric excitation pattern is the goal of the following optimization: Let vi be the illumination reaching volumetric point i, αi the transmission of light at i, and rj the light intensity of light-field ray j, then βi,j is the total transmission of ray j to volumetric point through all volumetric points on its path through the probe to i: βi,j = α1 · α2 · … · αn, where volumetric points 1, 2, …, n are on the path of ray j to volumetric point i. The light transport for a given light field through a given volume can then be described as:
where r = [r1, r2, …, rJ] is the vector of all light-field ray intensities, v = [v1, v2, …, vI] the vector of all light reaching each volumetric point, and B the transport matrix of all ray-to-point transmission coefficients βi,j through the probe. With known transport matrix B, the vector v can be defined to contain the coefficient 1 for all volumetric sample points to be excited, and the coefficient 0 for all sample points not to be excited. Note that we assume 1 and 0 to correspond to the maximum and minimum brightness of the light source, respectively. Volumetric sample points whose illumination is irrelevant, such as carrier material (e.g., index matching gel), must not be included in v. Solving Eq. 1 for r results in an illumination light-field mask that maximizes irradiation in the selected points to be excited while minimizing the light in points that must not be excited. We apply the simultaneous algebraic reconstruction technique (SART)26 to solve Eq. 1.
For illumination light fields of J rays and volumes of I points, B is large and contains I × J coefficients. Despite being sparse, Eq. 1 is expensive to solve. For practical real-time applications, we approximate the illumination mask as follows:
We distinguish between three types of volumetric points: Those that should be excited (E, e.g., microspheres to be stimulated), those that should not be exited (N, e.g., microspheres not to be stimulated), and those for which illumination is irrelevant (O, e.g., carrier material such as an index-matching gel).
We then determine all light-field rays R that pass through points in E by raycasting the z-stack, and consider three general application-specific illumination strategies: (S1) All rays in R equal 1 (which corresponds to a full illumination along the rays). (S2) Rays in R equal 1 only if they do not pass through points in N. (S3) Rays in R equal 1 only if they do not pass through points in N before they pass through a point in E. All other rays of the illumination light field are set to 0 (no light).
These three strategies are illustrated in Fig. 3 using the example of two occluding microspheres i and ii (Fig. 3a,b), and an optical axial-slice simulation of eight excitation regions and eight non-excitation regions (Fig. 3g–i). In Fig. 3c,d, we selected the foreground sphere i for excitation (i.e., to be part of E) and background sphere ii to remain unexcited (i.e., to be part of N), and vice versa in Fig. 3e,f. By choosing strategy S1, E receives a maximum of light while—due to partial occlusion and transparency of the microspheres/excitation regions—N is also fractionally illuminated (Fig. 3c,e,g). Strategy S2 avoids illumination of N entirely, but—depending on the degree of occlusion—it may result in lower excitation levels in E (Fig. 3d,f,h). Both strategies assume that the probes to be exited are not fully opaque but transmit a certain amount of excitation light. If probes can be considered to block light, then strategy S3 is appropriate (Fig. 3i). For our example in Fig. 3b, illuminating the nonoccluded microsphere i would result in the same outcome as shown in Fig. 3c (S1), while exciting the occluded microshpere ii results in Fig. 3f (S2). While computing an expensive optimization based on Eq. 1 (results are also shown in Fig. 3 for comparison) requires up to several hours (including generation of B), our raycasting approximation is achieved easily at interactive rates (see Method section).
According to Sparrow’s criterion, the smallest resolvable spot for wavelength λ, numerical aperture NA, and magnification M is . For a light-field microscope, the theoretical number of resolvable directions D equals the microlens pitch m divided by o. However, if the pixel pitch p of the SLM or camera used exceeds o, then D is limited to m/p. The lateral resolution in the field plane corresponds to the resolution of the MLA. The axial depth of field is S ≈ ((2 + D)λn)/(2NA2), which corresponds to the smallest focus shift within a focusable range of F ≈ ((2 + D2)λn)/(2NA2), where n is the refractive index of the imaging medium between the probe and the objective front lens (e.g., air n = 1, water n = 1.33, or oil n = 1.52)5. The numerical aperture of the microlenses should match the numerical aperture of the objective (F-number is M/(2NA)5).
Figure 4 illustrates the sampling nature of our approach based on an optical simulation of an empty volume (i.e., without occlusions) within a lateral field of view of 263 μm and an axially focusable range F = 22.2 μm, achieved with a 40×/0.95 NA objective, a m = 125 μm microlens pitch, and a λ = 470 nm excitation wavelength. The point-spread function (PSF) of the illumination focused on the field plane is shown in Fig. 4a. The average full width at half maximum (FWHM) over all axial distances is approximately L = 3.75 μm (lateral) × A = 5.1 μm (axial), and corresponds to the smallest focusable point size (i.e., highest sampling resolution) in this example. The smallest axial focus shift is S = 2.9 μm. Since the light-field ray space (principal ray directions are illustrated in Fig. 4a) is discrete with a limited directional and spatial resolution, the ray-sampling density varies across the focusable range. The dimension of a focus point depends on the number of ray intersections at its position. Thus, the axial and lateral resolutions vary with the distance to the field plane, as shown in Fig. 4b,c. They range from 3.3 μm to 4.2 μm laterally and 3.5 μm to 7.2 μm axially for the example shown in Fig. 4. We are considering mean resolutions (dotted lines in Fig. 4c) in the remaining discussion.
While the lateral resolution of a light-field microscope can be increased mainly by downscaling the microlens pitch of the MLA, the axial resolution is enhanced by reducing the pixel pitch of the DMD (down to the diffraction limit set by o), increasing the NA of the objective, and reducing the excitation wavelength.
Figure 5 illustrates the impact of different microlens pitches (m) for three different objectives. While decreasing m increases the mean lateral and axial resolutions (L, A), the focusable range (F) is strongly reduced. The simulations in Fig. 4 uses a 40×/0.95 NA objective with a m = 125 μm microlens pitch. Thus, the following sampling is achieved: o = 9.3 μm, p = 13.7 μm, D = 9.12, S = 2.9 μm, F = 22.20 μm. The simulated PSF size for this configuration is lateral L = 3.75 μm and axial A = 5.1 μm, which results in a sampling density of 13.94 × 10−3 PSFs per μm3 within the excitable volume. The same 40×/0.95 NA objective with a m = 150 μm microlens pitch results in the following sampling parameters: o = 9.3 μm, p = 13.7 μm, D = 10.9, S = 3.37 μm, F = 31.74 μm. The PSF size for this configuration is lateral L = 4.5 μm (4.0–5.2 μm) and axial A = 7.5 μm (6.4–10.4 μm), which results in a sampling density of 6.63 × 10−3 μm−3.
Figure 6 illustrates the excitation contrast for the three masking strategies (S1, S2 and S3) and a varying number of excitation points (E) and non-excitation points (N). The points are randomly distributed based on a Poisson disk sampling27. Figure 6a shows an optical simulation with a density of 0.5 × 10−3 μm−3, while Fig. 6b simulates densities of 1.45 × 10−3 μm−3. For all three strategies the contrast decreases as the number of excitation positions E increases (consequently the number of N decreases). A high contrast indicates low light pollution at N and high excitation levels at E. Within transparent non-scattering volumes, strategy S2 generates the highest and S1 the lowest contrast. The simulation volumes and the resulting volumetric illumination patterns for strategy S2 are illustrated in Fig. 7a,b.
Our approach is currently limited to transparent and non-scattering probes, due to the usage of single-photon illumination. In the future we would like to investigate the combination of our method with two-photon excitation to support applications in dense scattering volumes. However, single shot two-photon light field acquisition is not possible. Therefore, alternative methods for focal stack recording—preliminary to volumetric two-photon light-field excitation—have to be employed. Furthermore, for two-photon excitation light efficient optics and high-power lasers are required.
Our experimental (Fig. 2) and simulated results (Figs 6 and 7) indicate that our technique might have potential applications in the field of optogenetics28,29. Typical neuron sizes and cell densities for model organisms—i.e., C. elegans and zebrafish larvae—are in the range of – μm and 0.15–6.44 × 10−3 μm−3, respectively. However, currently we neglect some effects occurring in organic probes, such as scattering, natural fluctuations in cell size and density, and the neuron’s activation sensitivity. Future experiments on actual organic tissue will provide further insights.
Our proof-of concept prototype is a replica of the light-field microscope from Levoy et al.22 and is based on a Nikon Eclipse 80i fluorescence microscope. Our prototype equips a 20×/0.75 NA and a water-immersion 60×/1.2 NA objective (n = 1.33). For controlled and repeatable experiments, we applied commercially available fluorescence microspheres (100 μm and 10–20 μm) and bedded them in a silicone elastomer carrier. The application to other scales requires different MLAs or objectives and is discussed in the optical simulations above.
The prototype and the optical layout are shown in Fig. 8. The illumination system employs a square-sided, planoconvex microlens array (MLA2) at the intermediate image plane and a DMD-SLM (Texas Instruments Digital Light Processor driven by a Discovery 1100 Controller from Digital Light Innovations, pixel size pil = 13.7 μm) with a resolution of 1024 × 768. As a light source we use a 120 W metal halide lamp (X-Cite 120, EXFO), with corresponding filters. For capturing light fields, the imaging system uses an MLA (MLA1) at the intermediate image plane. A monochrome CCD camera (Retiga 4000R, pixel size pil = 7.4 μm) with 2048 × 2048 resolution is used for recording. The non-square illumination area, caused by the DMD, crops the usable field of view. For our experiments in Fig. 2 we used the 60 /1.2 NA objective and for the results in Fig. 3 we use the 20×/0.75 NA objective. The achieved sampling parameters with these configurations are shown in Table 1. The illumination loss of our prototype setup from light source to probe is approx 90%.
Eq. 1 assumes optical symmetry between the imaging and illumination light fields. In practice, however, slight misalignments in the optical paths and different microlenses cause misalignments and multi-mappings between illumination and imaging rays. To consider this, we calibrate the mapping between both by one-time measurement of a transport matrix T between the illumination light field r′ and the imaging light field r using a front surface mirror (with a 0.17 mm cover slip on top, placed perpendicular underneath the objective) using structured-light calibration30. The matrix T contains in each column the transport coefficients for one illumination ray r′s,t,u,v to all imaging rays rs,t,u,v. Thus, the size of T equals the number of illumination rays (number of columns) times the number of imaging rays (number of rows). Note that, in order to compensate for the calibration with a front surface mirror, the angular coordinates of the illumination rays are inverted (i.e., r′s,t,u,v becomes r′s,t,−u,−v) before their coefficients are stored in T. Eq. 1 then extends to v = BTr′, assuming that the z-stack from which B is determined is defined in the imaging ray space of r. The same applies to our approximated masks: Light fields in imaging ray space can be mapped to illumination ray space (or vice versa) by multiplying with T (or with TT). Note that T is constant for a constant optical configuration and is invariant to focus changes of the microscope. In the masking experiments with our 20× and 60× prototype B is large and contains 1.4 1012 and 4.31 1011 coefficients, respectively.
Our prototype implementation uses custom software5 to drive the microscope, Matlab for processing tasks (i.e., deconvolution, solving Eq. 1) and ImageJ for volume segmentation (i.e., selecting E). In Fig. 2 the z-stack size is 55 × 45 × 61 and sparse (i.e., a high number of volumetric points i is zero). For this setup our approximate solution of Eq. 1 was calculated in 7–10 seconds on an Intel Core i7 CPU 2.7 GHz and 24 GB of RAM. Computation times are independent of the number of non-zero volumetric points. In our approximation the illumination light field is constructed by raycasting its rays j in the volume (z-stack) and applying the rules of the masking strategies (S1, S2 and S3) as discussed above. Note, that the resulting light field is binary—0 for no light and 1 for light. We estimate that a GPU implementation of our approximation can lead to significant speed-up (at least ×10). When solving Eq. 1 with SART the number of non-zero volumetric points affects the performance. Runtimes ranged from 4 minutes for 0.16% non-zero points (E and N) up to 8 hours for 12.25% non-zero points. Note that most of the computation time is spend on construction of the transport matrix B (e.g., 8 hours for construction and 1 minute for solving with SART). The matrix B is constructed by raycasting each light-field ray j in the volume (z-stack) and setting the corresponding total transmission values βi,j in the matrix. The computed illumination light field consists of continues values for . In our experiments we used 500 SART iterations and a single transmission value α for all non-empty volumetric points (E or N).
The samples in Fig. 2 were Fluorescent Green Polyethylene 10–20 μm microspheres (material density 0.99–1.01 g/cm3; peak excitation 470 nm; peak emission 505 nm; distributor cospheric) mixed with polydimethylsiloxane (Elastosil RT 604 from Wacker) of material density 0.97 g/cm3 (mixing ratio was 6.5% of spheres). In Fig. 3 we used Rhodamine B Polyethylene 100 μm microspheres (material density 0.98 g/cm3; peak excitation 540 nm; peak emission 630 nm; distributor cospheric) mixed with the same silicone elastomer (as above) at a mixing ratio of 5% spheres.
How to cite this article: Schedl, D. C. and Bimber, O. Volumetric Light-Field Excitation. Sci. Rep. 6, 29193; doi: 10.1038/srep29193 (2016).
We thank Marc Levoy and Gordon Wetzstein of Stanford University, Enrico Geissler of Carl Zeiss AG, and Bettina Heise and Franz Josef Hiptmair of Johannes Kepler University Linz for their support and for providing optical equipment.
This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/