The word ‘holography’ means a drawing that contains all of the information for light—both amplitude and wavefront. However, because of the insufficient bandwidth of current electronics, the direct measurement of the wavefront of light has not yet been achieved. Though reference-field-assisted interferometric methods have been utilized in numerous applications, introducing a reference field raises several fundamental and practical issues. Here we demonstrate a reference-free holographic image sensor. To achieve this, we propose a speckle-correlation scattering matrix approach; light-field information passing through a thin disordered layer is recorded and retrieved from a single-shot recording of speckle intensity patterns. Self-interference via diffusive scattering enables access to impinging light-field information, when light transport in the diffusive layer is precisely calibrated. As a proof-of-concept, we demonstrate direct holographic measurements of three-dimensional optical fields using a compact device consisting of a regular image sensor and a diffusor.
Photography does not present volumetric and depth perceptions because wavefront (or phase) information of light cannot be recorded on a film or an image sensor (Fig. 1a). The loss of wavefront information, the so-called phase problem1, has been a classical issue in numerous fields, including holography2,3, X-ray crystallography4 and three-dimensional imaging5.
To remedy the issue, Gabor proposed a holographic method that exploited the interferometric nature of waves2,3. By recording interference patterns between an unknown incident field and a known reference field, it is possible to deduce the incident field information (Fig. 1b). Due to its powerful ability, reference-assisted holography has been utilized in numerous disciplines6,7,8. Unfortunately, despite the significant efforts steadily continued for decades, the realizations and applications of holography are limited.
One prime reason is the practical issues of introducing a reference field. In an X-ray regime, for example, introducing a reference field is impractical due to the limited capability of X-ray optics9,10. Even in a visible regime, introducing a reference arm results in various limitations because it makes the system complex, incompatible and vulnerable to ambient noise11. However, although several reference-free holographic methods have been proposed, including Shack-Hartmann-type sensors12,13, transport-of-intensity equation14, ptycographic scanning methods10,15,16 and iterative algorithms17,18,19,20, most of them have to sacrifice the generality of a holographic system by introducing specific assumptions about a sample or an incident beam.
Ideal holography would be a method that directly captures the incident field rather than intensity without any additional constraints (Fig. 1c). One obvious solution is to enhance the bandwidth of a camera above the frequency of light, which is, however, still impractical because the optical frequency is several orders of magnitude faster than the current state-of-the-art electronic techniques.
Here we propose the ideal holographic image sensor using a regular image sensor and a disordered layer and experimentally demonstrate direct holographic measurements of three-dimensional optical fields. To achieve this, we propose a new method called the speckle-correlation scattering matrix (SSM) approach.
Hologram retrieval from an intensity speckle pattern
The basic principle of our method is based on the linear relationship between incident light and scattered light from the diffusive layer, and the randomly distributed (or complex Gaussian distributed) property of the diffused light. The light scattered from a disordered medium is linearly related to the incident light. This linear relationship between incident and scattered light fields is described using a scattering matrix21. Previously, measuring the scattering matrix has enabled the imaging or delivering of the designated optical field through a disordered layer22,23,24,25. However, since the scattering matrix is written by field-field correlation, previous studies had to include additional holographic methods to measure the field. We overcame this limitation by using the random nature of diffused light inspired by the Siegert relation of temporally random (chaotic) light26 and achieved reference-free holographic imaging using a commercial diffuser as a holographic imaging system (Fig. 1d).
For simplicity without losing generality, we focus on transmission geometry, providing an explanation using a transmission matrix (TM), which is a subset of a scattering matrix. For N orthonormal fields k1, k2, …, kN as the preset input basis, the p-th column of TM, tp, is the corresponding diffused field for the p-th input field kp (Fig. 2c–e). Then, for a given arbitrary incident field, , the corresponding diffused field can be expressed as , where α1, α2, …, αN are complex-valued coefficients that contain the incident field information, and the camera captures the diffused intensity, y*y (Fig. 2g). Note that y*y loses the wavefront information during the capturing process. Because TM describes the field-field correlation between the input and output light, the field information of diffused light y is required in order to retrieve an incident field x using the TM. Therefore, additional holographic methods have to be employed in conventional studies using TMs22,23,24,25.
Here, exploiting the random property of diffused fields, we find that calibrated TM, t1, t2, …, tN, and the intensity snapshot, y*y, are sufficient for the incident field retrieval. We propose a speckle-correlation scattering matrix Z,
where indicates an space average, and is a normalization constant. The random property of diffused light field enables an intriguing mathematical relation between fourth-order and second-order moments27,28,29,
where E1, E2, E3 and E4 are any randomly diffused light fields, and indicates the ensemble average. Applying equation (2) to the first term of equation (1), we obtain
where from the general orthogonality between uncorrelated diffused fields, . Although the assumption of orthogonality has generally been accepted for transport through a diffusive layer23,30,31, correlation exists for highly scattering media22,32,33,34,35, according to the random matrix theory36,37. Nonetheless, we note that this correlation can be ignored for a diffusive layer, which is used in this study. If it exists, these correlated elements can be numerically filtered out once a TM is calibrated and its eigenbasis is analysed (See Supplementary Note 1). Because the second term in equation (3) comprises the spatial average of other random fields and tqy, it vanishes as the ratio (γ) between the number of optical sampling modes (M) and the number of preset input modes (N) increases (see Supplementary Note 2). Therefore, for a sufficiently large γ, the second term of equation (2) vanishes, and the rank of Z matrix becomes unity; , whose sole eigenvector is the incident field.
To demonstrate the proposed idea, we experimentally construct the set-up (Fig. 2a,b). In addition to a diffuser, we insert an aperture to block ambient light, and a polarizer to maximize the speckle visibility. To avoid duplicated optical information, we set the speckle grain size at the camera plane to be comparable to a single camera pixel by adjusting the distance between the diffuser and the camera. Using a He-Ne laser (wavelength, 633 nm), the TM is measured by a spatial light modulator (SLM) (see Methods).
Because the retrieved hologram is mapped onto the preset input basis, the number of input modes N determine the field of view (FOV) and the diffraction-limited spot size (that is, the resolution) of the holographic image sensor. The number of optical sampling modes M relates to the intrinsic signal-to-noise ratio (SNR) originating from the second term of equation (3). In our system, we used 66 × 60 (N=3,960) equally spaced spatial frequency modes as an input basis. This corresponds to the FOV of 5.28 × 4.00 mm and a diffraction-limited spot size of 80 × 67 μm. Additionally, we set 512 × 512 central camera pixels (M=262,144) as the output modes, which corresponds to γ=66.2.
First, we test the feasibility of our idea by imaging incident fields, which are modulated by a SLM (wavefront modulation) or a binary mask (amplitude modulation). We use a 4-f telescopic imaging system with a demagnification factor of 3 to conjugate the focal position of the holographic image sensor to the position of the SLM or the binary mask, and also to increase the effective FOV. The Z matrix is calculated by substituting a calibrated TM and a measured y*y (Fig. 3a) into equation (1). The amplitude and wavefront results of the retrieved hologram are shown in Fig. 3c,f, respectively. Despite the presence of noise mainly originating from the second term of equation (3), the expected field (Fig. 3b,e) is successfully reconstructed.
To suppress speckle noise, we additionally present a modified Gerchberg-Saxton algorithm (see Methods, Supplementary Movie 1). We use a measured TM as an operator, instead of Fourier transformation, and adopt as a constraint. Starting from , the iteration process stops when it converges into . Note that the proposed iteration algorithm does not require any additional assumptions, free variables or regularizations. The iteration result is shown in Fig. 3d,g. Since only a limited portion of the Fourier modes is recorded in the TM, the effects of low-pass filtering are observed in the retrieved amplitude (Fig. 3d). In addition, several wavefront- and amplitude-modulated fields are tested (Fig. 3h–m).
The noise tolerance
To analyse the noise tolerance of the proposed method, we performed numerical simulations (Fig. 4). We added Gaussian (white) noise to both TMs, and measured the speckle intensity maps by adjusting γ=M/N (or the intrinsic noise level) and the SNR of Gaussian noise. In order to systematically investigate the effects of noise, the correlation between the incident field (x, ground truth) and the retrieved field ( or ) is quantified for both the SSM and the SSM with the iteration method. As quantitatively analysed in Fig. 4d,e, the validity of the present method is satisfied when the scattering correlation matrix is appropriately sampled and measured; that is, γ is set to be large. Importantly, the present method provides the successful retrieval of holographic images even in extremely noisy conditions (for example, SNR=1) when γ is large enough. Moreover, for a small γ (for example, γ=4), the modified GS algorithm significantly enhances the quality of the retrieved hologram when the SNR is sufficiently large (for example, SNR>10). For all numerical simulations, we used a fixed N=1,024, while M varied depending on γ. The detailed procedures for the numerical implementations and the used MATLAB code can be found in Supplementary Methods.
The hologram of real objects
To demonstrate the versatility of our method as a stand-alone 3D holographic image sensor, we prepare a real target consisting of two diffusive dices with a size of 5 × 5 × 5 mm, which are separated by 40 cm from each other (Fig. 5a). We impinge a laser beam onto the target for illumination and measure the reflected light field (Fig. 5b–d). To confirm the quality of the measured hologram, we perform numerical refocusing from −35 to +35 cm, as shown in Fig. 5e–k. As shown, the results exhibit the expected silhouette of targets at each refocusing position. Note that the speckled intensity and randomized phase (Fig. 5c,d) are a result of the diffusive reflection of an object38. For clear visualization, the speckle is suppressed by measuring 25 more holograms with slightly tilted laser illumination. For a given refocusing position, all 25 retrieved holograms are numerically propagated to the position, and their amplitude parts are compounded. The compounding results shown in Fig. 5l–r show clearer images of the target for every numerically refocused position. These results prove the capability of our holographic method, even for the complex form of the incident field. Once the TM of the system is calibrated, the system can be functional as long as the physical structures are not altered. We observe no significant degradation in the quality of holographic imaging, even 1 month after the calibration without any additional temperature or humidity controls.
In conclusion, we theoretically propose and experimentally demonstrate a reference-free holographic imaging sensor. The present method enables the direct retrieval of the phase without using reference-beam-assisted interferometry. Exploiting the random property, light-field information can be directly obtained from the measurement of a diffused intensity image. The principle is that light transport through an optical diffusor is described with a random matrix, and the intensity speckle patterns from two different input wavefronts are effectively not correlated with each other. To demonstrate the proposed idea, we present a compact and reference-free holographic image sensor, with which direct measurements of various light-field images are shown.
We should emphasize that the present approach is fundamentally different from previously reported methods based on speckle correlation or ghost imaging. For example, conventional speckle correlation methods such as ghost imaging39,40,41,42, single-pixel imaging43,44,45,46, memory effect-based imaging31,32 and Siegert-relation-based imaging47,48 have been able to retrieve only amplitude images, because their speckle correlation is based on intensity correlation, ; therefore, the wavefront information is inherently lost. In addition, unlike existing TM-based methods22,23,24,25, the current work does not require the use of a reference beam, which significantly expands the applicability of our proposed method. Our method also does not require any type of prior knowledge of incident light, including reference field, which enables direct hologram capture as photography does. From a technical point of view, the present method can be extended to image polarization- and wavelength-dependent imaging, as well as subwavelengths near-field imaging, by exploiting the large degree of freedom that multiple light scattering offers49,50,51,52. Although the present method requires a precise and time-consuming calibration step of a turbid layer prior to use, faster techniques for measuring TMs will further alleviate this technical issue.
Therefore, we expect that our method could be a solution for applications with which conventional holographic approaches have had difficulties. Furthermore, the assumption-free and reference-free capability of the present method would be advantageous and could also be expanded to direct holographic measurement in an X-ray regime. The compactness and single-shot property would be beneficial for more practical applications such as a handheld holographic camera using smartphone optics53 or quantitative-phase imaging8 for point-of-care biomedical diagnostic applications.
Holographic image sensor
We use a lens tube (SM1L10, Thorlabs Inc.) as the body of the holographic lens. From the front, a rectangular-shape aperture (4 × 5.3 mm, custom-made), an optical diffuser (ED1-C20, Thorlabs Inc.) and a polarizer (LPVISE100-A, Thorlabs Inc.) are sequentially stacked inside the tube. The direction of the polarizer is set to be parallel to the polarization of the incident laser. We directly assemble the lens with a camera (4,242 × 2,830 pixels; 3.1 μm pitch; MD120MU-SY, XIMEA GmbH) using an adapter (SM1A9, Thorlabs Inc.). The distance between the diffuser and the detector array is estimated to be 43 mm, including the additional space inside of the camera.
Overall, we employed an improved version of the TM calibration algorithm suggested in ref. 54. As shown in Supplementary Fig. 1, we construct a Michelson interferometer using a He-Ne laser (HNL050L, Thorlabs Inc.) with a SLM in the sample arm. After fixing the interferometer, we demagnify the beam by a factor of 3 × using a 4-f lens system in order to fit the clear aperture of the SLM (792 × 600 pixels; 20 μm pitch; X10468-01, Hamamatsu photonics K.K.) into the size of the aperture. We use 66 × 60 rectangular (N=3,960) low-pass Fourier modes as an input basis, which corresponds to a FOV of 5.28 × 4.00 mm, and a resolution of 80 × 67 μm; the output modes are 512 × 512 central camera pixels (M=262,144). For each p-th input mode, we take three interferograms I0,p, I1,p and I2,p, adding respective global phase delays of 0, and . Utilizing the phase retrieval algorithm, we retrieve the output field ti from the phase-shifted interferograms where R indicates the reference field, and Since the reference field also passes through the holographic unit, R also fluctuates along the space, which causes severe distortion, which is the result of Z matrix in equation (2). To compensate for the effect of the R, we take one additional image while the SLM arm is blocked, which is . Note that the phase part of R is unnecessary because it is automatically compensated for during the Z matrix calculation. As a result, we take a total of 3N+1 (11,881) images to calibrate the TM. Our TM calibration takes 20 min. Please refer Supplementary Methods for the detailed flow chart and full MATLAB codes.
Modified Gerchberg-Saxton (GS) iteration algorithm
The Fourier transform operator in the original GS iteration algorithm is replaced with a measured TM. Since we set the constraint and starting point , the rest of the procedure is identical to the field retrieval sequence of a single-constraint situation of an error-reduction iteration algorithm18. We stop the iteration at the f-th loop satisfying the condition that the correlation between and is above 0.999998. The iteration number widely ranged from 34 to 130 proportional to the noise signal (see Supplementary Fig. 2). The mean iteration numbers of the results in Figs 2 and 3 are 40 and 74, respectively. It took 1.75 s for a single iteration step with our computing power (3.50 GHz, intel core i5-4690 CPU; 32.0 GB RAM) using the MATLAB software. Please refer Supplementary Methods for the detailed flow chart and full MATLAB codes.
The data that support the findings of this study are available from the corresponding author upon reasonable request.
How to cite this article: Lee, K. and Park, Y. Exploiting the speckle-correlation scattering matrix for a compact reference-free holographic image sensor. Nat. Commun. 7, 13359 doi: 10.1038/ncomms13359 (2016).
Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Taylor, L. The phase retrieval problem. IEEE Trans. Antennas Propag. 29, 386–391 (1981).
Gabor, D. A new microscopic principle. Nature 161, 777–778 (1948).
Gabor, D. Microscopy by reconstructed wave-fronts. Proc. R. Soc. A 197, 454–487 (1949).
Millane, R. P. Phase retrieval in crystallography and optics. J. Opt. Soc. Am. A 7, 394–411 (1990).
Javidi, B. & Okano, F. Three-Dimensional Television, Video, and Display Technologies Springer (2002).
Tegze, M. & Faigel, G. X-ray holography with atomic resolution. Nature 380, 49–51 (1996).
Hariharan, P. Optical Holography: Principles, Techniques and Applications Cambridge University Press (1996).
Lee, K. et al. Quantitative phase imaging techniques for the study of cell pathophysiology: from principles to applications. Sensors 13, 4170–4191 (2013).
Shen, Q. Solving the phase problem using reference-beam X-ray diffraction. Phys. Rev. Lett. 80, 3268–3271 (1998).
Rodenburg, J. M. et al. Hard-X-ray lensless imaging of extended objects. Phys. Rev. Lett. 98, 034801 (2007).
Lee, K. & Park, Y. Quantitative phase imaging unit. Opt. Lett. 39, 3630–3633 (2014).
Shack, R. V. & Platt, B. Production and use of a lenticular Hartmann screen. J. Opt. Soc. Am. 61, 656–660 (1971).
Cui, X., Ren, J., Tearney, G. J. & Yang, C. Wavefront image sensor chip. Opt. Express 18, 16685–16701 (2010).
Paganin, D. & Nugent, K. A. Noninterferometric phase imaging with partially coherent light. Phys. Rev. Lett. 80, 2586–2589 (1998).
Hegerl, R. & Hoppe, W. Phase evaluation in generalized diffraction (ptychography). Proc. 5th Eur. Cong. Electron Microsc. 628–629 (1972).
Zheng, G., Horstmeyer, R. & Yang, C. Wide-field, high-resolution Fourier ptychographic microscopy. Nature Photon. 7, 739–745 (2013).
Gerchberg, R. W. A practical algorithm for the determination of phase from image and diffraction plane pictures. Optik 35, 237–246 (1972).
Fienup, J. R. Phase retrieval algorithms: a comparison. Appl. Opt. 21, 2758–2769 (1982).
Miao, J., Sayre, D. & Chapman, H. N. Phase retrieval from the magnitude of the Fourier transforms of nonperiodic objects. J. Opt. Soc. Am. A 15, 1662–1669 (1998).
Drémeau, A. et al. Reference-less measurement of the transmission matrix of a highly scattering material using a DMD and phase retrieval techniques. Opt. Express 23, 11898–11911 (2015).
Mosk, A. P., Lagendijk, A., Lerosey, G. & Fink, M. Controlling waves in space and time for imaging and focusing in complex media. Nature Photon. 6, 283–292 (2012).
Popoff, S. M. et al. Measuring the transmission matrix in optics: an approach to the study and control of light propagation in disordered media. Phys. Rev. Lett. 104, 100601 (2010).
Popoff, S., Lerosey, G., Fink, M., Boccara, A. C. & Gigan, S. Image transmission through an opaque material. Nat. Commun. 1, 81 (2010).
Choi, Y. et al. Scanner-free and wide-field endoscopic imaging by using a single multimode optical fiber. Phys. Rev. Lett. 109, 203901 (2012).
Čižmár, T. & Dholakia, K. Exploiting multimode waveguides for pure fibre-based imaging. Nat. Commun. 3, 1027 (2012).
Loudon, R. The Quantum Theory of Light OUP (2000).
Reed, I. On a moment theorem for complex Gaussian processes. IEEE Trans. Inf. Theory 8, 194–195 (1962).
Balakrishnan, A. V. Introduction to Random Processes in Engineering Wiley (2005).
Lemieux, P. A. & Durian, D. J. Investigating non-Gaussian scattering processes by using nth-order intensity correlation functions. J. Opt. Soc. Am. A 16, 1651–1664 (1999).
Bertolotti, J. et al. Non-invasive imaging through opaque scattering layers. Nature 491, 232–234 (2012).
Katz, O., Heidmann, P., Fink, M. & Gigan, S. Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations. Nature Photon. 8, 784–790 (2014).
Vellekoop, I. M. & Mosk, A. P. Universal optimal transmission of light through disordered materials. Phys. Rev. Lett. 101, 120601 (2008).
Shi, Z. & Genack, A. Z. Transmission eigenvalues and the bare conductance in the crossover to Anderson localization. Phys. Rev. Lett. 108, 043901 (2012).
Yu, H., Park, J.-H. & Park, Y. Measuring large optical reflection matrices of turbid media. Opt. Commun. 352, 33–38 (2015).
Yu, H. et al. Measuring large optical transmission matrices of disordered media. Phys. Rev. Lett. 111, 153902 (2013).
Dorokhov, O. N. On the coexistence of localized and extended electronic states in the metallic phase. Solid State Commun. 51, 381–384 (1984).
Mello, P. A., Pereyra, P. & Kumar, N. Macroscopic approach to multichannel disordered conductors. Ann. Phys 181, 290–317 (1988).
Martienssen, W. & Spiller, S. Holographic reconstruction without granulation. Phys. Lett. A 24, 126–128 (1967).
Pittman, T. B., Shih, Y. H., Strekalov, D. V. & Sergienko, A. V. Optical imaging by means of two-photon quantum entanglement. Phys. Rev. A 52, R3429–R3432 (1995).
Bennink, R. S., Bentley, S. J. & Boyd, R. W. ‘Two-Photon’ coincidence imaging with a classical source. Phys. Rev. Lett. 89, 113601 (2002).
Gatti, A., Brambilla, E., Bache, M. & Lugiato, L. A. Correlated imaging, quantum and classical. Phys. Rev. A 70, 013802 (2004).
Meyers, R., Deacon, K. S. & Shih, Y. Ghost-imaging experiment by measuring reflected photons. Phys. Rev. A 77, 041801 (2008).
Sen, P. et al. Dual photography. ACM Trans. Graph. 24, 745–755 (2005).
Duarte, M. F. et al. Single-pixel imaging via compressive sampling. IEEE Signal Process. Mag. 25, 83–91 (2008).
Bromberg, Y., Katz, O. & Silberberg, Y. Ghost imaging with a single detector. Phys. Rev. A 79, 053840 (2009).
Sun, B. et al. 3D computational imaging with single-pixel detectors. Science 340, 844–847 (2013).
Newman, J. A. & Kevin, J. W. Imaging optical fields through heavily scattering media. Phys. Rev. Lett. 113, 263903 (2014).
Naik, D. N. et al. Photon correlation holography. Opt. Express 19, 1408–1421 (2011).
Park, J.-H., Park, C., Yu, H., Cho, Y.-H. & Park, Y. Dynamic active wave plate using random nanoparticles. Opt. Express 20, 17010–17016 (2012).
Park, J.-H., Park, C., Yu, H., Cho, Y.-H. & Park, Y. Active spectral filtering through turbid media. Opt. Lett. 37, 3261–3263 (2012).
Park, J.-H. et al. Subwavelength light focusing using random nanoparticles. Nature Photon. 7, 454–458 (2013).
Park, C. et al. Full-field subwavelength imaging using a scattering superlens. Phys. Rev. Lett. 113, 113901 (2014).
Mobile science. Nature Photon 8, 745 (2014).
Yoon, J., Lee, K., Park, J. & Park, Y. Measuring optical transmission matrices by wavefront shaping. Opt. Express 23, 10158–10167 (2015).
This work was supported by KAIST, and the National Research Foundation of Korea (2015R1A3A2066550, 2014K1A3A1A09063027, 2014M3C1A3052567) and Innopolis foundation (A2015DD126).
K.L. and Y.P. are inventors on a patent describing a method for direct holography (Republic of Korea patent application number 10-2016-0043274).
Supplementary Figures 1&2, Supplementary Notes 1&2, Supplementary Methods and Supplementary References (PDF 795 kb)
Visualization of iteration and convergence. The iteration executed from x0 to xf. The amplitude (left) and wavefront (right) part of each iteration step is shown. Total 38 iterations have done, and total execution time is 66.6 seconds as displayed above the right figure. (MOV 921 kb)
About this article
Cite this article
Lee, K., Park, Y. Exploiting the speckle-correlation scattering matrix for a compact reference-free holographic image sensor. Nat Commun 7, 13359 (2016). https://doi.org/10.1038/ncomms13359
Photonics Research (2021)
Applied Optics (2021)
Nature Photonics (2021)
Applied Optics (2021)
Optics and Lasers in Engineering (2020)