Abstract
Holographic displays can generate light fields by dynamically modulating the wavefront of a coherent beam of light using a spatial light modulator, promising rich virtual and augmented reality applications. However, the limited spatial resolution of existing dynamic spatial light modulators imposes a tight bound on the diffraction angle. As a result, modern holographic displays possess low étendue, which is the product of the display area and the maximum solid angle of diffracted light. The low étendue forces a sacrifice of either the fieldofview (FOV) or the display size. In this work, we lift this limitation by presenting neural étendue expanders. This new breed of optical elements, which is learned from a natural image dataset, enables higher diffraction angles for ultrawide FOV while maintaining both a compact form factor and the fidelity of displayed contents to human viewers. With neural étendue expanders, we experimentally achieve 64 × étendue expansion of natural images in full color, expanding the FOV by an order of magnitude horizontally and vertically, with highfidelity reconstruction quality (measured in PSNR) over 29 dB on retinalresolution images.
Similar content being viewed by others
Introduction
Holography is the science of creating vivid scenery through carefully crafted interference patterns. This discipline has applications across domains, especially in virtual and augmented reality devices^{1,2}. While static holograms can be generated with a suitable recording medium, modern holographic displays typically employ spatial light modulators (SLM) that dynamically modulate the wavefront of a coherent beam^{3,4}. However, despite being the workhorse of holography, SLMs suffer from small diffraction angles caused by limitations of modern liquid crystal on silicon (LCoS) technology. Achieving dynamic control with LCoS induces several engineering challenges (e.g., display bandwidth, pixel crosstalk, power consumption), which have imposed a practical lower bound on the pixel pitch^{5}. Consequently, the étendue of holographic displays, which is the product of the viewing angle and the display size for tabletop displays and is the product of the fieldofview (FOV) and the eyebox size for neareye displays, is fundamentally limited when maintaining a compact SLM size^{6}. Holographic displays have to trade off FOV for display size, or vice versa, although both are critical for most display applications. For immersive virtual/augmented reality (VR/AR) devices, an FOV of at least 120^{∘} and an eyebox size greater than 10 × 10 mm^{2} is desired^{7}, where the eyebox or display size is defined as the region within which the eye must reside within to view a hologram. To reach the étendue needed for these specifications requires over one billion SLM pixels, which is two orders of magnitude more than what today’s LCoS technology achieves^{7}. Manufacturing such a display and dynamically controlling it is beyond modern fabrication and computational capabilities.
Several methods have been proposed to circumvent this problem, including dynamic feedback in the form of eye tracking^{8}, spatial integration with multiple SLMs^{9}, and temporal integration with laser arrays^{10} or fast switching digital micromirror devices^{11,12}. However, these approaches require additional dynamic components resulting in high complexity, large form factors, precise timing constraints, and incur additional power consumption. Timing is especially critical for eye tracking solutions as poor latency can result in motion sickness^{13}. Rewritable photopolymers are promising alternatives to SLMs for holographic displays, however, they are limited by low refresh rates^{14,15}. Microelectromechanical systems have low pixel counts and bit depth^{16}. Researchers have also proposed to make tradeoffs within the limited étendue of existing holographic displays, such as trading spatial resolution for depth resolution^{3} and optimizing for coherent versus incoherent interference^{17}, but these methods do not change the total étendue of the display.
Instead, researchers have explored expanding the display étendue by employing optical elements with randomized scattering properties in front of an SLM^{7,18,19,20,21,22}. The static nature of these elements facilitates fabrication of pixel areas at the micronscale, an order of magnitude lower than for a dynamic element such as the SLM, thus resulting in an enlarged diffraction angle^{23}. However, existing elements of this type exhibit randomized scattering that is agnostic to the optical setup and the images to be displayed. As modern SLMs have limited degrees of freedom for wavefront shaping, the random modulation delivered by these scattering elements results in lowfidelity étendue expanded holograms. Relaxing high frequencies beyond the human retinal resolution improves the fidelity, however the displayed holograms still suffer from low reconstruction fidelity^{7}. Furthermore, extensive calibration is necessary in the case where the scattering properties are unknown^{22,24,25,26,27}. Recent work explores using lenses and lenslet arrays^{28} to expand the fieldofview but this approach forces the eye pupil’s position to match that of the lenses or lenslets, thus shrinking the effective eyebox size, see Supplementary Note 5 for comparison experiments. Another line of work investigates tilting cascades^{29}, however, this system has a large physical footprint consisting of several 4F relays. Specifically, this system requires one 4F relay per 2 × factor of étendue expansion, thus three 4F relays are needed to achieve 8 × expansion along one axis, spanning roughly a meter in length. Moreover, the approach requires two parallel cascades to expand the étendue in both the horizontal and vertical directions, further increasing the physical form factor.
In this work, we lift these limitations with neural étendue expanders, a new breed of static optical elements that have been optimized for étendue expansion and accurate reproduction of natural images when combined with an SLM (Fig. 1). These optical elements inherit the aforementioned benefits of scattering elements. However, unlike existing random scattering masks, neural étendue expanders are jointly learned together with the SLM pattern across a natural image dataset. We devise a differentiable holographic image formation model that enables learning via firstorder stochastic optimization. The resulting learned wavefront modulation pushes reconstruction noise outside of the perceivable frequency bands of human visual systems while retaining perceptually critical frequency bands of natural images. We provide analysis that validates that the learned optical elements possess these properties. In simulation, we demonstrate étendueexpanded holograms at 64 × étendue expansion factor with perceptual display quality over 29 dB peak signaltonoise ratio (PSNR), more than one order of magnitude in reduced error over the existing methods. Furthermore, the expanders also facilitate highfidelity étendue expanded 3D color holograms of natural scenes, see Supplementary Information, and the elements are also robust to varying pupil positions within the eyebox. We experimentally validate the method with a 1Kpixel SLM to expand the FOV by 8 × in each direction.
Results
Neural étendue expansion
Holographic displays modulate the wavefront of a coherent light beam using an SLM to form an image at a target location. The étendue of a holographic display^{30} is then defined as the product of the SLM display area A and the solid angle of the diffracted light as
where \({\theta }_{s}={\sin }^{1}\frac{\lambda }{2{\Delta }_{s}}\) is the maximum diffraction angle of the SLM, λ is the wavelength of light, and Δ_{s} is the SLM pixel pitch. Most SLMs have large Δ_{s} resulting in small θ_{s} as shown in Fig. 1a. We enlarge the display étendue by placing a neural étendue expander in front of the SLM as a static optical element with pixel pitch Δ_{n} < Δ_{s}, see Fig. 1b. The smaller pixel pitch Δ_{n} increases the maximum diffraction angle θ_{n}, resulting in an expanded étendue \({G}_{n}=4A{\sin }^{2}{\theta }_{n}\).
To generate étendueexpanded highfidelity holograms, we propose a computational inversedesign method that learns the wavefront modulation of the neural étendue expander by treating it as a layer of trainable neurons that are taught to minimize a loss placed on the formed holographic image, see Fig. 1c. Specifically, we model the holographic image formation in a fully differentiable manner following Fourier optics. We relate the displayed holographic image I to the wavefront modulation of the neural étendue expander \({{{{{{{\mathcal{E}}}}}}}}\) as
where \({{{{{{{\mathcal{F}}}}}}}}\) is the 2D Fourier transform, \({{{{{{{\mathcal{S}}}}}}}}\) is the SLM modulation, U( ⋅ ) is zerothorder upsampling operator from the lowresolution SLM to the highresolution neural étendue expander, and ⊙ is the Hadamard product.
The differentiability of Eq. (2) with respect to the modulation variables \({{{{{{{\mathcal{E}}}}}}}}\) and \({{{{{{{\mathcal{S}}}}}}}}\) allows us to learn the optimal wavefront modulation of the neural étendue expander \({{{{{{{\mathcal{E}}}}}}}}\) by jointly optimizing the static neural étendue expander in concunction with the dynamic SLM modulation patterns \({{{{{{{\mathcal{S}}}}}}}}\). That is, for a given image, we optimize the optimal SLM pattern similar to conventional computergenerated holography^{31,32,33}, however, we also simultaneously optimize the neural étendue expander. The SLM and the neural étendue expander cooperate to generate an étendueexpanded highfidelity hologram. We formulate this joint optimization as
where \({{{{{{{{\mathcal{S}}}}}}}}}_{k}\) is the SLM wavefront modulation for the kth target image T_{k} in a naturalimage dataset with K training samples, * is the convolution operator, and f is the lowpass Butterworth filter for approximating the viewer’s retinal resolution as a frequencycutoff function^{7} as
where \({{{{{{{{\mathcal{F}}}}}}}}}^{1}\) is the inverse 2D Fourier transform, w is the spatial frequency, and c is the cutoff frequency. We set c to be \(\frac{{\Delta }_{n}N}{\sqrt{\pi }}\) where N is the SLM pixel count. This allows us to utilize the native bandwidth of the SLM for only the lower frequency features that the human viewer will perceive. We can then adjust the FOV and the eyebox size so that an angular resolution of at least 60 pixels/degree is achieved, see Supplementary Note 3 for details.
The optimization objective in Eq. (3) jointly optimizes a single static element \({{{{{{{\mathcal{E}}}}}}}}\) and a set of SLM patterns \({{{{{{{{\mathcal{S}}}}}}}}}_{\{1,...,K\}}\) so that the set of generated holograms matches the target set of natural images T_{{1, . . . , K}}. This objective function is fully differentiable with respect to the wavefront modulations of the SLM and the neural étendue expander. As such, we can solve this optimization problem by training the neural étendue expander and the SLM states akin to a shallow neural network by using stochastic gradient solvers^{34}. Our computational design approach is datadriven and requires a dataset of natural images. We used 105 highresolution training images of natural scenes. For testing, we used 20 natural images. We use grayscale images when designing neural étendue expanders for a monochromatic display, while the original RGB images are used when designing for a trichromatic display. The training procedure does not utilize any temporal multiplexing and only optimizes for the maximal reconstruction quality that can be achieved with a single SLM frame and a single expander.
Neural étendue expanded holographic display
We validate neural étendue expansion experimentally with a holographic display prototype. See Fig. 2a for a schematic of the hardware prototype and Supplementary Notes 9 and 10 for further details on the experimental setup. We fabricate neural étendue expanders with a pitch of 2 μm with resin stamping, see Supplementary Note 8 for fabrication details. The fabricated expanders are then placed at the conjugate plane of the SLM to establish pixelwise correspondence between the SLM and the expander. A DC block is further employed to filter out the undiffracted light from the SLM. To assess the proposed elements, we also compare to fabricated binary random expanders^{7} designed for 660 nm. Microscope images of both expanders are shown in Fig. 2b. We acquire holograms corresponding to conventional nonétendue expanded holography^{1}, 64 × étendue expanded full color holograms produced with the binary random expanders, and 64 × étendue expanded full color holograms produced with the neural étendue expanders. The illumination wavelengths are 450 nm, 520 nm, and 660 nm. We report captures in Fig. 2c and provide additional measurements in Supplementary Video 1 and in Supplementary Note 1. The captured holograms are tonemapped for visualization. For fair comparison we applied the same tonemapping scheme to all holograms, see Supplementary Note 1 for details.
The experimental findings on the display prototype verify that conventional nonétendue expanded holography can produce highfidelity content but at the cost of a small FOV. Increasing the étendue via a binary random expander will increase the FOV but at the cost of low image fidelity, even at the design wavelength of 660 nm, and chromatic artifacts. The étendue expanded holograms produced with the neural étendue expanders are the only holograms that showcase both ultrawideFOV and highfidelity. The captured holograms demonstrate high contrast and are free from chromatic aberrations. Fig. 2d reports the étendue expanded hologram produced with both expanders at each color wavelength. Since the binary random expander is, by design, only tailored to a single wavelength, in this case 660 nm, the étendue expanded holograms that are generated with it exhibit severe chromatic artifacts. In contrast, holograms generated with neural étendue expansion show consistent highfidelity performance at all illumination wavelengths. Notably, even at the wavelength of 660 nm that was used to design the binary random expander^{7} the hologram fidelity is higher for the holograms generated with neural étendue expansion, see Fig. 2c. For comparisons against a uniform random expander, where the phase profile is uniformly randomly selected from within [0, 2π], see Fig. 3 and Supplementary Note 2.
While our experimental prototype was built for a HOLOEYEPLUTO with 1Kpixel resolution, corresponding to a 1 mm eyebox with 77. 4^{∘} horizontal and vertical FOV, the improvement in hologram fidelity persists across resolutions. Irrespective of the resolution of the SLM, performing 4 × , 16 × , or 64 × étendue expansion with neural étendue expanders results in a similar margin of improvement over uniform and binary random expanders. This is because the improvement in fidelity depends only on the étendue expansion factor. We validate this by simulating an 8Kpixel SLM with 64 × étendue expansion and we verify that the improvement in fidelity is maintained. See Supplementary Note 6 for results and further details. Thus, neural étendue expansion enables high fidelity expansion for 64 × étendue expansion for 8Kpixel SLMs^{35}, providing étendue to cover 85% of the human stereo FOV^{36} with a 18.5 mm eyebox size, see Supplementary Note 3 for details.
Characterization of étendue expansion
Next, we analyze the expansion of étendue achieved with the proposed technique. To this end, suppose we want to generate the étendueexpanded hologram of only a single scene. Then, the optimal complex wavefront modulation for the neural étendue expander would be the inverse Fourier transform of the target scene, and, as such, we do not require any additional modulation on the SLM. The SLM therefore can be set to zerophase modulation. If we generalize this singleimage case to diverse natural images, the neural étendue expander is expected to preserve the common frequency statistics of natural images, while the SLM fills in the imagespecific residual frequencies to generate a specific target image. In contrast, existing random scatters used for étendue expansion do not consider any naturalimage statistics^{7,18,20,21}.
To assess whether the optimized neural étendue expander \({{{{{{{\mathcal{E}}}}}}}}\), shown in Fig. 1b, has learned the image statistics of the training set we evaluate the virtual frequency modulation \(\widetilde{{{{{{{{\mathcal{E}}}}}}}}}\), defined as the spectrum of the generated image with the neural étendue expander and the zerophase SLM modulation as
The findings in Fig. 1d confirm that the magnitude of the virtual frequency modulation \( \widetilde{{{{{{{{\mathcal{E}}}}}}}}}\) resembles the magnitude spectrum of natural images within the passband. Moreover, we observe that the virtual frequency modulation pushes undesirable energy outside of the passband of the human retina as imperceptible highfrequency noise.
To further understand this property of a neural étendue expander, we consider the reconstruction loss \({{{{{{{{\mathcal{L}}}}}}}}}_{T}\) for a specific target image T. Using the zerophase setting for the SLM as an initial point for the firstorder stochastic optimization and applying Parseval’s theorem places an upper bound on the reconstruction loss
where N is the pixel count of the neural étendue expander. Please see Supplementary Note 3 for further details of how this upper bound is found. Therefore, obtaining the optimal neural étendue expander, which minimizes the reconstruction loss \({{{{{{{{\mathcal{L}}}}}}}}}_{T}\), results in the virtual frequency modulation \(\widetilde{{{{{{{{\mathcal{E}}}}}}}}}\) that resembles the naturalimage spectrum \({{{{{{{\mathcal{F}}}}}}}}(T)\) averaged over diverse natural images. Also, the retinal frequency filter \({{{{{{{\mathcal{F}}}}}}}}(f)\) leaves the higher spectral bands outside of the human retinal resolution unconstrained. This allows the neural étendue expander to push undesirable energy towards higher frequency bands, which then manifests as imperceptible highfrequency noise to human viewers.
We investigate the image statistics preserved by the neural étendue expanders by visualizing \( \widetilde{{{{{{{{\mathcal{E}}}}}}}}}\). Fig. 3e visualizes the learned expander pattern \({{{{{{{\mathcal{E}}}}}}}}\) for increasing étendue expansion factors, specifically for 4 × , 16 × , 36 × , and 64 × expansion. Unlike uniform and binary random expanders, the learned expanders exhibit high and low frequency structures. Fig. 3f shows the corresponding virtual frequency \( \widetilde{{{{{{{{\mathcal{E}}}}}}}}}\) for each étendue expansion factor. We observe that the interwoven high and low frequency patterns on each learned expander correspond to a virtual frequency that pushes noise outside of the retinal frequency bands defined by \({{{{{{{\mathcal{F}}}}}}}}(f)\). Furthermore, the frequency structure within the passband resembles the frequency structure of the natural image training dataset, see Fig. 3c.
To characterize the hologram reconstruction with the proposed neural étendue expander we simulate a Fourier holographic setup that has been augmented with a neural étendue expander. Fig. 3a reports qualitative examples of trichromatic and monochromatic reconstructions achieved with neural étendue expanders, binary random expanders^{7}, photon sieves^{21}, and conventional holography^{1}. See Supplementary Note 2 for additional qualitative comparisons and for comparisons against uniform random expanders. The uniform random expander is constructed by assigning each pixel a phase that is uniformly randomly chosen within [0, 2π]. To ensure at least 2π phase is available for all wavelengths the [0, 2π] phase range is defined for 660 nm. Conventional holography is subject to a low display étendue that is limited by the SLM native resolution, thus resulting in a low FOV. Photon sieves^{21}, binary random expanders^{7}, and uniform random expanders have low reconstruction fidelity, resulting in severe noise and low contrast in the generated holograms. In the case of the trichromatic holograms, both uniform and binary random expanders do not facilitate consistent étendue expansion at all wavelengths, which results in chromatic artifacts. Although the uniform random expander provides at least 2π phase coverage for all wavelengths, the variation in refractive index across wavelengths results in differing phase profiles. Thus, although the uniform random expander has the same degree of quantization as neural étendue expanders, it does not enable étendue expanded trichromatic holograms. Photon sieves scatter light equally across wavelengths but their randomized amplitudeonly modulation does not allow for highfidelity reconstruction of natural images, see Fig. 3d for quantitative assessment and Supplementary Note 2 for qualitative examples and additional metrics. Neural étendue expansion is the only technique that facilitates highfidelity reconstructions for both trichromatic and monochromatic setups. We quantitatively verify this by evaluating the reconstruction fidelity on an unseen test dataset, where fidelity is measured in peak signaltonoise ratio (PSNR). Fig. 3b shows that neural étendue expanders achieve over 14 dB PSNR improvement favorable to other expanders when generating 64 × étendue expanded trichromatic holograms. For monochromatic holograms, neural étendue expansion achieves over 10 dB PSNR improvement. Thus, neural étendue expansion allows for an order of magnitude improvement over existing étendue expansion methods. See Fig. 3d for quantitative evaluations at different étendue expansion factors. See Supplementary Note 2 for further simulation details and more comparison examples.
We also investigate the robustness of neural étendue expansion to eye pupil movements. By initializing the learning process with a uniform random expander we bias the optimized solution towards expanders that distribute energy throughout the eyebox. As a result, the image quality of neural étendue expanded holograms is tolerant to variations in the shape and position of the viewer’s eye pupil. We find that alternative methods that use quadratic phase profiles^{28} concentrate the eyebox energy at fixed points and are not robust to eye pupil movements. See Supplementary Note 5 for findings. Finally, we also investigate 3D étendue expanded holograms. We find that neural étendue expansion also enables higher fidelity étendue expanded 3D color holograms. We note that existing methods on étendue expanded holography has focused on monochromatic 3D holograms^{7,28,29}. Photon sieves^{21} only realize 3D color holography for sparse points. See Supplementary Note 4 for a discussion of these findings.
Discussion
In this work, we introduce neural étendue expanders as an optical element that expands the étendue of existing holographic displays without sacrificing displayed hologram fidelity. Neural étendue expanders are learned from a natural image dataset and are jointly optimized with the SLM’s wavefront modulation. Akin to a shallow neural network, this new breed of optical elements allows us to tailor the wavefront modulation element to the display of natural images and maximize display quality perceivable by the human eye. As the first learned optics for étendue expansion, we achieve étendue expansion factor 64 × with over 29 dB PSNR reconstructions, an order of magnitude improvement over existing approaches. Furthermore, neural étendue expanders support multiwavelength illumination for color holograms. The expanders also support 3D color holography and viewer pupil movement. We envision that future holographic displays may incorporate the described optical design approach into their construction, especially for VR/AR displays. Extending our work to utilize other types of emerging optics such as metasurfaces may prove to be a promising direction for future work, as diffraction angles can be greatly enlarged by nanoscale metasurface features^{37} and additional properties of light such as polarization can be modulated using metaoptics^{38}.
Methods
Simulation
We used PyTorch to design and evaluate the neural étendue expanders. See Supplementary Notes 2 and 3 for details on the optimization framework, evaluation, and analysis.
Fabrication
The expanders are physically realized as diffractive optical elements (DOE). Fabricating the DOEs consists of several stages. The first stage consists of etching the negative of the desired pattern onto a substrate. This etching is performed with laser beam lithography. The etched substrate forms a stamp which is then pressed onto a resin mold that is mounted on a glass substrate. The resin itself contains the final pattern. The resin has a wavelength dependent refractive index that we incorporate into our design framework. For the resin we used, the refractive indices are 1.5081 for 660 nm, 1.5159 for 517 nm, and 1.5223 for 450 nm. See Supplementary Note 10 for details.
Experimental Setup
We evaluated the neural étendue expanders using a prototype holographic display. The prototype consists of a HOLOEYEPLUTO SLM, a 4F system, a DC block, and a camera for imaging the étendue expanded holograms. See Supplementary Notes 11 and 12 for details.
Data availability
The code and data used to generate the findings of this study are available at https://doi.org/10.5281/zenodo.10653321.
Code availability
The code and data used to generate the findings of this study are available at https://doi.org/10.5281/zenodo.10653321.
References
Shi, L., Li, B., Kim, C., Kellnhofer, P. & Matusik, W. Towards realtime photorealistic 3d holography with deep neural networks. Nature 591, 234–239 (2021).
Itoh, Y., Langlotz, T., Sutton, J. & Plopski, A. Towards indistinguishable augmented reality: A survey on optical seethrough headmounted displays. ACM Comput. Surv. 54, 120 (2021).
Dorrah, A. H. et al. Light sheets for continuousdepth holography and threedimensional volumetric displays. Nat. Photo. 17, 427–434 (2023).
Wakunami, K. et al. Projectiontype seethrough holographic threedimensional display. Nat. Commun. 7, 12954 (2016).
Moser, S., RitschMarte, M. & Thalhammer, G. Modelbased compensation of pixel crosstalk in liquid crystal spatial light modulators. Opt. Expr. 27, 25046–25063 (2019).
Lohmann, A. W., Dorsch, R. G., Mendlovic, D., Zalevsky, Z. & Ferreira, C. Space–bandwidth product of optical signals and systems. J. Opt. Soc. Am. A 13, 470–473 (1996).
Kuo, G., Waller, L., Ng, R. & Maimone, A. High resolution étendue expansion for holographic displays. ACM Trans. Graph. (TOG) 39, 66 (2020).
Choi, M.H., Ju, Y.G. & Park, J.H. Holographic neareye display with continuously expanded eyebox using twodimensional replication and angular spectrum wrapping. Opt. Expr. 28, 533–547 (2020).
Hahn, J., Kim, H., Lim, Y., Park, G. & Lee, B. Wide viewing angle dynamic holographic stereogram with a curved array of spatial light modulators. Opt. Expr. 16, 12372–12386 (2008).
Lee, B. et al. Wideangle speckleless dmd holographic display using structured illumination with temporal multiplexing. Opt. Lett. 45, 2148–2151 (2020).
Kim, M. et al. Expanded exitpupil holographic headmounted display with highspeed digital micromirror device. ETRI J. 40, 366–375 (2018).
Hellman, B. & Takashima, Y. Angular and spatial light modulation by single digital micromirror device for multiimage output and nearlydoubled étendue. Opt. Expr. 27, 21477–21496 (2019).
Wetzstein, G., Heidrich, W. & Luebke, D. Optical image processing using light modulation displays. Comput. Graph. Forum 29, 1934–1944 (2010).
Blanche, P.A. et al. Holographic threedimensional telepresence using largearea photorefractive polymer. Nature 468, 80–83 (2010).
Tay, S. et al. An updatable holographic threedimensional display. Nature 451, 694–698 (2008).
Ersumo, N. T. et al. A micromirror array with annular partitioning for highspeed randomaccess axial focusing. Light: Sci. Appl. 9, 1–15 (2020).
Yang, D. et al. Diffractionengineered holography: Beyond the depth representation limit of holographic displays. Nat. Commun. 13, 6012 (2022).
Buckley, E., Cable, A., Lawrence, N. & Wilkinson, T. Viewing angle enhancement for two and threedimensional holographic displays with random superresolution phase masks. Appl. Opt. 45, 7334–7341 (2006).
Huang, K. et al. Ultrahighcapacity nonperiodic photon sieves operating in visible light. Nat. Commun. 6, 7059 (2015).
Yu, H., Lee, K., Park, J. & Park, Y. Ultrahighdefinition dynamic 3d holographic display by active control of volume speckle fields. Nat. Photo. 11, 186–192 (2017).
Park, J., Lee, K. & Park, Y. Ultrathin wideangle largearea digital 3d holographic display using a nonperiodic photon sieve. Nat. Commun. 10, 1304 (2019).
Yu, P. et al. Ultrahighdensity 3d holographic projection by scatteringassisted dynamic holography. Optica 10, 481–490 (2023).
Genet, C. & Ebbesen, T. Light in tiny holes. Nature 445, 39–46 (2007).
Mosk, A. P., Lagendijk, A., Lerosey, G. & Fink, M. Controlling waves in space and time for imaging and focusing in complex media. Nat. Photo. 6, 283–292 (2012).
Conkey, D. B. et al. Superresolution photoacoustic imaging through a scattering wall. Nat. Commun. 6, 7902 (2015).
Vellekoop, I. M., Lagendijk, A. & Mosk, A. Exploiting disorder for perfect focusing. Nat. Photo. 4, 320–322 (2010).
Popoff, S. M. et al. Measuring the transmission matrix in optics: an approach to the study and control of light propagation in disordered media. Phys. Rev. Lett. 104, 100601 (2010).
Monin, S., Sankaranarayanan, A. C. & Levin, A. Analyzing phase masks for wide étendue holographic displays. In Proc. IEEE International Conference on Computational Photography (ICCP) 1–12 (2022).
Monin, S., Sankaranarayanan, A. C. & Levin, A. Exponentiallywide étendue displays using a tilting cascade. In Proc. IEEE International Conference on Computational Photography (ICCP) 1–12 (2022).
Chaves, J. Introduction to nonimaging optics (CRC Press, 2008).
Gerchberg, R. W. A practical algorithm for the determination of phase from image and diffraction plane pictures. Optik 35, 237–246 (1972).
Georgiou, A., Christmas, J., Collings, N., Moore, J. & Crossland, W. Aspects of hologram calculation for video frames. J. Opt. A: Pure Appl. Opt. 10, 035302 (2008).
Chakravarthula, P., Peng, Y., Kollin, J., Fuchs, H. & Heide, F. Wirtinger holography for neareye displays. ACM Trans. Graph. (TOG) 38, 213 (2019).
Kingma, D. & Ba, J. Adam: A method for stochastic optimization. In Proc. International Conference on Learning Representations (ICLR) (ICLR, 2015).
Sterling, R. Jvc dila high resolution, high contrast projectors and applications. In Proc. 2008 Workshop on Immersive Projection Technologies/Emerging Display Technologiges. 10 (2008).
Wheelwright, B. et al. Field of view: not just a number. In Digital Optics for Immersive Displays, vol. 10676, 1–7 (International Society for Optics and Photonics, 2018).
Scheuer, J. & Yifat, Y. Metasurfaces make it practical. Nat. Nanotechnol. 10, 296–298 (2015).
Dorrah, A. H., Rubin, N. A., Zaidi, A., Tamagnone, M. & Capasso, F. Metasurface optics for ondemand polarization transformations along the optical path. Nat. Photo. 15, 287–296 (2021).
Acknowledgements
E.T. acknowledges a Google PhD Fellowship. F.H. acknowledges an NSF CAREER Award (2047359), a Packard Foundation Fellowship, a Sloan Research Fellowship, a Sony Young Faculty Award, a Project X Innovation Award, and an Amazon Science Research Award. The authors acknowledge the use of Princeton’s Imaging and Analysis Center (IAC), which is partially supported by the Princeton Center for Complex Materials (PCCM), a National Science Foundation (NSF) Materials Research Science and Engineering Center (MRSEC; DMR2011750). The authors acknowledge Ilya Chugunov for providing natural image data. S.H.B. acknowledges NRF Korea (RS202300211658). Q.F. and W.H. acknowledge KAUST individual baseline funding. The authors acknowledge the use of the KAUST Nanofabrication Core Lab (NCL).
Author information
Authors and Affiliations
Contributions
E.T. and F.H. analyzed and designed the neural étendue expanders, performed the experiments, and led the manuscript writing. G.K., S.H.B., A.M., P.C., Q.F. assisted in analysis and design, experiments, and writing the manuscript. N.M., F.S., W.H. assisted in writing the manuscript. D.L. and F.H. supervised the project and assisted in writing the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Communications thanks Dongyeon Kim and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. A peer review file is available
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Tseng, E., Kuo, G., Baek, SH. et al. Neural étendue expander for ultrawideangle highfidelity holographic display. Nat Commun 15, 2907 (2024). https://doi.org/10.1038/s41467024469153
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41467024469153
This article is cited by

360° structured light with learned metasurfaces
Nature Photonics (2024)
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.