Neural étendue expander for ultra-wide-angle high-fidelity holographic display

Holographic displays can generate light fields by dynamically modulating the wavefront of a coherent beam of light using a spatial light modulator, promising rich virtual and augmented reality applications. However, the limited spatial resolution of existing dynamic spatial light modulators imposes a tight bound on the diffraction angle. As a result, modern holographic displays possess low étendue, which is the product of the display area and the maximum solid angle of diffracted light. The low étendue forces a sacrifice of either the field-of-view (FOV) or the display size. In this work, we lift this limitation by presenting neural étendue expanders. This new breed of optical elements, which is learned from a natural image dataset, enables higher diffraction angles for ultra-wide FOV while maintaining both a compact form factor and the fidelity of displayed contents to human viewers. With neural étendue expanders, we experimentally achieve 64 × étendue expansion of natural images in full color, expanding the FOV by an order of magnitude horizontally and vertically, with high-fidelity reconstruction quality (measured in PSNR) over 29 dB on retinal-resolution images.

cifications requires over one billion SLM pixels, which is two orders of magnitude more than what today's LCoS technology achieves 6 . Manufacturing such a display and dynamically controlling it is beyond modern fabrication and computational capabilities.
Several methods have been proposed to circumvent this problem, including dynamic feedback in the form of eye tracking 7 , spatial integration with multiple SLMs 8 , and temporal integration with laser arrays 9 . However, these approaches require additional dynamic components resulting in high complexity, large form factors, precise timing constraints, and incur additional power consumption. Timing is especially critical for eye tracking solutions as poor latency can result in motion sickness 10 . Rewritable photopolymers are promising alternatives to SLMs for holographic displays, however, they are limited by low refresh rates 11, 12 . Microelectromechanical systems have low pixel counts and bit depth 13 .
Instead, researchers have explored expanding the displayétendue by employing optical elements with randomized scattering properties in front of an SLM 6, 14-17 . The static nature of these elements facilitates fabrication of pixel areas at the micron-scale, an order of magnitude lower than for a dynamic element such as the SLM, thus resulting in an enlarged diffraction angle 18 . However, existing elements of this type exhibit randomized scattering that is agnostic to the optical setup and the images to be displayed. As modern SLMs have limited degrees of freedom for wavefront shaping, the random modulation delivered by these scattering elements results in low-fidelityétendue expanded holograms. Relaxing high frequencies beyond the human retinal resolution improves the fidelity, however the displayed holograms still suffer from low reconstruction fidelity 6 . Furthermore, extensive calibration is necessary in the case where the scattering properties are unknown [19][20][21][22] .
In this work, we lift these limitations with neuralétendue expanders, a new breed of static optical elements that have been optimized forétendue expansion and accurate reproduction of natural images when combined with an SLM (Fig. 1). These optical elements inherit the aforementioned benefits of scattering elements. However, unlike existing random scattering masks, neuralétendue expanders are jointly learned together with the SLM pattern across a natural image dataset. We devise a differentiable holographic image formation model that enables learning via first-order stochastic optimization. The resulting learned wavefront modulation pushes reconstruction noise outside of the perceivable frequency bands of human visual systems while retaining perceptually critical frequency bands of natural images. In simulation, we demonstrateétendueexpanded holograms at 64×étendue expansion factor with perceptual display quality over 29 dB peak signal-to-noise ratio (PSNR), more than one order of magnitude in reduced error over the existing methods. Our 64× neuralétendue expander combined with an 8K-pixel SLM 23 covers 85% of the human stereo FOV 24 with a 18.5 mm eyebox size, providing a high-fidelity, immersive VR/AR experience.

Neuralétendue expansion
Holographic displays modulate the wavefront of a coherent light beam using an SLM to form an image at a target location. Theétendue of a holographic display 25 is then defined as the product of the SLM display area A and the solid angle of the diffracted light as where θ s = sin −1 λ 2∆s is the maximum diffraction angle of the SLM, λ is the wavelength of light, and ∆ s is the SLM pixel pitch. Most SLMs have large ∆ s resulting in small θ s as shown in Fig. 1a. We enlarge the displayétendue by placing a neuralétendue expander in front of the SLM as a static optical element with pixel pitch ∆ n < ∆ s , see Fig. 1b. The smaller pixel pitch ∆ n increases the maximum diffraction angle θ n , resulting in an expandedétendue G n = 4A sin 2 θ n .
To generateétendue-expanded high-fidelity holograms, we propose a computational inversedesign method that learns the wavefront modulation of the neuralétendue expander by treating it as a layer of trainable neurons that are taught to minimize a loss placed on the formed holographic image, see Fig. 1c. Specifically, we model the holographic image formation in a fully differentiable manner following Fourier optics. We relate the displayed holographic image I to the wavefront modulation of the neuralétendue expander E as where F is the 2D Fourier transform, S is the SLM modulation, U (·) is zeroth-order upsampling operator from the low-resolution SLM to the high-resolution neuralétendue expander, and is the Hadamard product.
The differentiability of Eq. (2) with respect to the modulation variables E and S allows us to learn the optimal wavefront modulation of the neuralétendue expander E by jointly optimizing the static neuralétendue expander in concunction with the dynamic SLM modulation patterns S. That is, for a given image, we optimize the optimal SLM pattern similar to conventional computergenerated holography 26-28 , however, we also simultaneously optimize the neuralétendue expander. The SLM and the neuralétendue expander cooperate to generate anétendue-expanded high-fidelity hologram. We formulate this joint optimization as where S k is the SLM wavefront modulation for the k-th target image T k in a natural-image dataset with K training samples, * is the convolution operator, and f is the low-pass Butterworth filter for approximating the viewer's retinal resolution as a frequency-cutoff function 6 as where F −1 is the inverse 2D Fourier transform, w is the spatial frequency, and c is the cutoff frequency. In order to set the cutoff frequency to be beyond human perceptibility, it suffices to set c to be ∆nN √ π where N is the SLM pixel count. This is because an 8K-pixel SLM 23 can provide an angular resolution of 61 pixels/degree at an eyebox size of 18.5 mm, see Supplementary Note 3 for details, whereas the angular resolution of the human eye is limited to 60 pixels/degree 6 .
The optimization objective in Eq. (3) jointly optimizes a single static element E and a set of SLM patterns S {1,...,K} so that the set of generated holograms matches the target set of natural images T {1,...,K} . This objective function is fully differentiable with respect to the wavefront modulations of the SLM and the neuralétendue expander. As such, we can solve this optimization problem by training the neuralétendue expander and the SLM states akin to a shallow neural network by using stochastic gradient solvers 29 . Our computational design approach is data-driven and requires a dataset of natural images. We used 105 high-resolution training images of natural scenes. For testing, we used 20 natural images. We use grayscale images when designing neuraĺ etendue expanders for a monochromatic display, while the original RGB images are used when designing for a trichromatic display.

Neuralétendue expanded holographic display
We validate neuralétendue expansion experimentally with a holographic display prototype. See Fig. 2a for a schematic of the hardware prototype and Supplementary Notes 7 and 8 for further details on the experimental setup. We fabricate neuralétendue expanders with a pitch of 2 µm with resin stamping, see Supplementary Note 6 for fabrication details. The fabricated expanders are then placed at the conjugate plane of the SLM to establish pixel-wise correspondence between the SLM and the expander. A DC block is further employed to filter out the undiffracted light from the SLM. To assess the proposed elements, we also compare to fabricated random expanders 6 designed for 660 nm. Microscope images of both expanders are shown in Fig. 2b. We acquire holograms corresponding to conventional non-étendue expanded holography 2 , 64×étendue expanded full color holograms produced with the random expanders, and 64×étendue expanded full color holograms produced with the neuralétendue expanders. The illumination wavelengths are 450 nm, 520 nm, and 660 nm. We report captures in Fig. 2c and provide additional measurements in Supplementary Video 1 and in Supplementary Note 1. The captured holograms are tone-mapped for visualization. For fair comparison we applied the same tone-mapping scheme to all holograms, see Supplementary Note 1 for details.
The experimental findings on the display prototype verify that conventional non-étendue expanded holography can produce high-fidelity content but at the cost of a small FOV. Increasing thé etendue via a random expander will increase the FOV but at the cost of low image fidelity and chromatic artifacts. Theétendue expanded holograms produced with the neuralétendue expanders are the only holograms that showcase both ultra-wide-FOV and high-fidelity. The captured holograms demonstrate high contrast and are free from chromatic aberrations. Fig. 2d reports theétendue expanded hologram produced with both expanders at each color wavelength. Since the random expander can only be designed for a single wavelength, in this case 660 nm, theétendue expanded holograms that are generated with it exhibit severe chromatic artifacts. In contrast, holograms generated with neuralétendue expansion show consistent high-fidelity performance at all illumination wavelengths.

Characterization ofétendue expansion
Next, we analyze the expansion ofétendue achieved with the proposed technique. To this end, suppose we want to generate theétendue-expanded hologram of only a single scene. Then, the optimal complex wavefront modulation for the neuralétendue expander would be the inverse Fourier transform of the target scene, and, as such, we do not require any additional modulation on the SLM. The SLM therefore can be set to zero-phase modulation. If we generalize this single-image case to diverse natural images, the neuralétendue expander is expected to preserve the common frequency statistics of natural images, while the SLM fills in the image-specific residual frequencies to generate a specific target image. In contrast, existing random scatters used forétendue expansion do not consider any natural-image statistics 6,14,16,17 .
To assess whether the optimized neuralétendue expander E, shown in Fig. 1b, has learned the image statistics of the training set we evaluate the virtual frequency modulation E, defined as the spectrum of the generated image with the neuralétendue expander and the zero-phase SLM modulation as The findings in Fig. 1d confirm that the magnitude of the virtual frequency modulation | E| resembles the magnitude spectrum of natural images within the passband. Moreover, we observe that the virtual frequency modulation pushes undesirable energy outside of the passband of the human retina as imperceptible high-frequency noise.
To further understand this property of a neuralétendue expander, we consider the reconstruction loss L T for a specific target image T . Using the zero-phase setting for the SLM as an initial point for the first-order stochastic optimization and applying Parseval's theorem places an upper bound on the reconstruction loss where N is the pixel count of the neuralétendue expander. Please see Supplementary Note 3 for further details of how this upper bound is found. Therefore, obtaining the optimal neuralétendue expander, which minimizes the reconstruction loss L T , results in the virtual frequency modulation E that resembles the natural-image spectrum F(T ) averaged over diverse natural images. Also, the retinal frequency filter F(f ) leaves the higher spectral bands outside of the human retinal resolution unconstrained. This allows the neuralétendue expander to push undesirable energy towards higher frequency bands, which then manifests as imperceptible high-frequency noise to human viewers.
We investigate the image statistics preserved by the neuralétendue expanders by visualizing | E|. Fig. 3e visualizes the learned expander pattern E for increasingétendue expansion factors, specifically for 4×, 16×, 36×, and 64× expansion. Unlike random expanders, the learned expanders exhibit high and low frequency structures. Fig. 3f shows the corresponding virtual frequency | E| for eachétendue expansion factor. We observe that the interwoven high and low frequency patterns on each learned expander correspond to a virtual frequency that pushes noise outside of the retinal frequency bands defined by F(f ). Furthermore, the frequency structure within the passband resembles the frequency structure of the natural image training dataset, see Fig. 3c.
To characterize the hologram reconstruction with the proposed neuralétendue expander we simulate a Fourier holographic setup that has been augmented with a neuralétendue expander. Fig. 3a reports qualitative examples of trichromatic and monochromatic reconstructions achieved with neuralétendue expanders, random expanders 6 , photon sieves 17 , and conventional holography 2 . Conventional holography is subject to a low displayétendue that is limited by the SLM native resolution, thus resulting in a low FOV. Both photon sieves and random expanders achieve low reconstruction fidelity, resulting in severe noise and low contrast in the generated holograms. In the case of the trichromatic holograms, these expanders do not facilitate consistent etendue expansion at all wavelengths, which causes severe chromatic artifacts. Neuralétendue expansion is the only technique that facilitates high-fidelity reconstructions for both trichromatic and monochromatic setups. We quantitatively verify this by evaluating the reconstruction fidelity on an unseen test dataset, where fidelity is measured in peak signal-to-noise ratio (PSNR). Fig. 3b shows that neuralétendue expanders achieve over 14 dB PSNR improvement against the other expanders when generating 64×étendue expanded trichromatic holograms. For monochromatic holograms, neuralétendue expansion achieves over 10 dB PSNR improvement. Thus, neuralétendue expansion allows for an order of magnitude improvement over existingétendue expansion methods. See

Discussion
In this work, we introduce neuralétendue expanders as an optical element that expands theétendue of existing holographic displays without sacrificing displayed hologram fidelity. Neuralétendue expanders are learned from a natural image dataset and are jointly optimized with the SLM's wavefront modulation. Akin to a shallow neural network, this new breed of optical elements allows us to tailor the wavefront modulation element to the display of natural images and maximize display quality perceivable by the human eye. As the first learned optics forétendue expansion, we achievé etendue expansion factor 64× with over 29 dB PSNR reconstructions, an order of magnitude improvement over existing approaches. This means that expansion factor 64× combined with an 8K-pixel SLM can enable high-fidelity, ultra-wide-angle holographic projection of natural images with 126 • FOV and 18.5 mm eyebox size, covering more than 76% of the human FOV. Furthermore, neuralétendue expanders support multi-wavelength illumination for color holograms. We envision that future holographic displays may incorporate the described optical design approach into their construction, especially for VR/AR displays. Extending our work to utilize other types of emerging optics such as metasurfaces may prove to be a promising direction for future work, as diffraction angles can be greatly enlarged by nano-scale metasurface features 30 and additional properties of light such as polarization can be modulated using meta-optics 31 .

Methods
Simulation We used PyTorch to design and evaluate the neuralétendue expanders. See Supplementary Notes 2 and 3 for details on the optimization framework, evaluation, and analysis.
Fabrication The expanders are physically realized as diffractive optical elements (DOE). Fabricating the DOEs consists of several stages. The first stage consists of etching the negative of the desired pattern onto a substrate. This etching is performed with laser beam lithography. The etched substrate forms a stamp which is then pressed onto a resin mold that is mounted on a glass substrate. The resin itself contains the final pattern. The resin has a wavelength dependent refractive index that we incorporate into our design framework. For the resin we used, the refractive indices are 1.5081 for 660 nm, 1.5159 for 517 nm, and 1.5223 for 450 nm. See Supplementary Note 6 for details.

Experimental Setup
We evaluated the neuralétendue expanders using a prototype holographic display. The prototype consists of a HOLOEYE-PLUTO SLM, a 4F system, a DC block, and a camera for imaging theétendue expanded holograms. See Supplementary Notes 7 and 8 for details.

Data Availability
The code and data used to generate the findings of this study will be made public on GitHub.
Code Availability The code and data used to generate the findings of this study will be made public on GitHub.   Figure 1: Neuralétendue expansion for ultra-wide angle, high-fidelity holograms. a Conventional holographic displays suffer from lowétendue, resulting in either small FOV or eyebox size. Here, we illustrate a small FOV as θ s . b Introducing a neuralétendue expander into the display facilitates ultra-wide angle holograms, here we illustrate the increase in FOV as θ n . c We design the neuralétendue expanders via an end-to-end optimization algorithm that considers the SLM wavefront modulation and the human viewer's perceptual response. One SLM pattern is optimized for each training sample, while the neuralétendue expander learns a general structure that facilitates hologram generation of any natural image. d The learned neuralétendue expander preserves the major frequency bands of natural images within the frequency cutoff determined by the resolution of the human retina.  Figure 3:Étendue expander characterization. a The 64×étendue expanded holograms generated with neuralétendue expanders have the highest fidelity with respect to the target natural image, for both the trichromatic and monochromatic cases. In comparison, the holograms generated with random binary 6 or photon sieve 17 expanders show lower contrast and more speckle noise. A lowétendue hologram generated with conventional holography and no expander is included for comparison. b Quantitative performance comparison of each method evaluated over an unseen test set. c Virtual frequency modulation cross section. Neuralétendue expanders push reconstruction artifacts outside of the perceivable frequency bands of human vision while producing a natural image frequency spectrum within the passband as predicted by Eq. 6. In contrast, the random expander exhibits a flat spectrum which reduces the reconstruction quality within the passband. The cutoff frequency is indicated by c. d Quantitative reconstruction quality of each method for increasingétendue expansion factors for the monochromatic case. e Visualization of the learned expanders for increasingétendue expansion factors. We observe that the learned modulation structures contain both high and low frequency components. f Visualization of the corresponding virtual frequency modulation for each expander. In this document we provide additional discussion and results in support of the primary manuscript.

Supplementary Video 1: Experimental Assessment for Dynamic Scenes
The Supplementary Videos are experimental video captures of the étendue expanded holograms generated with neural étendue expanders. For comparison, étendue expanded holograms generated with random expanders 1 and non-étendue expanded holograms 2 are included. All results shown are captured on the same hardware prototype as described in the manuscript. We use time multiplexing CGH where we compute 3 independent holograms for each frame of each video. The 3 holograms are displayed and captured sequentially and each frame of each video is then computed as the average of the 3 holograms. These videos are saved at a framerate of 30 fps.

Supplementary Note 1: Additional Experimental Results
Next, we provide addition experimental findings that were acquired on the experimental setup. Supplementary Fig. S1 reports 64× étendue expanded color holograms. Supplementary Fig. S2 lists the reconstructions for individual color channels. Supplementary Fig. S3 and Fig. S4 display the corresponding color holograms at 16× étendue expansion. Supplementary Fig. S5 and Fig. S6 show 64× and 16× étendue expanded monochromatic holograms.
We use temporal multiplexing to reduce speckle for all results shown. Specifically, all experimental results shown are computed as the average of 20 holograms, where each hologram is generated with a different random seed. Using a different random seed for each hologram results in different speckle artifacts which are then eliminated through temporal multiplexing.
In addition, we performed white balancing on the captures so that the overall color ratio matches that of the target images. In the future, this color balancing step could have been performed in the hardware via laser power adjustment if we had a programmable laser source.
Lastly, all of the expanders leak a residual DC term similar to how the SLM also leaks a DC term. We suppressed this term by determining the hologram that corresponds to each expander's DC term and then subtracting it from the captured holograms. In the future, the DC term could be eliminated in hardware by using a tilted off-axis construction.  Figure S1: Experimentally captured color holograms at 64× étendue expansion. The wavelengths used are 660 nm, 517 nm, and 450 nm. For comparison, étendue expanded holograms generated with random expanders 1 and non-étendue expanded holograms 2 are included. These results supplement the experimental findings from Fig. 2 Figure S2: Experimentally captured color holograms at 64× étendue expansion. This figure shows the reconstructions for individual color channels for the holograms shown in Supplementary Fig. S1. These results supplement the experimental findings from Fig. 2 Figure S4: Experimentally captured color holograms at 16× étendue expansion. This figure shows the reconstructions for individual color channels for the holograms shown in Supplementary Fig. S3. These results supplement the experimental findings from Fig. 2 Figure S5: Experimentally captured monochromatic holograms at 64× étendue expansion. The wavelengths used is 660 nm. For comparison, étendue expanded holograms generated with random expanders 1 and non-étendue expanded holograms 2 are included. These results supplement the experimental findings from Fig. 2 Figure S6: Experimentally captured monochromatic holograms at 16× étendue expansion. The wavelengths used is 660 nm. For comparison, étendue expanded holograms generated with random expanders 1 and non-étendue expanded holograms 2 are included. These results supplement the experimental findings from Fig. 2 of the main manuscript.

Supplementary Note 2: Additional Synthetic Results
We provide addition simulation results that further validate the effects of neural étendue expansion. Supplementary Fig. S7 compares 64× étendue expansion with neural étendue expanders against étendue expansion with photon sieves 3 and randomized expanders 1 . Supplementary Fig. S8 reports the same comparison for only a single wavelength. Supplementary Figs. S9 and S10 report the same comparisons for 16× étendue expansion.
The wavelengths used are the same as in the physical experiment, specifically 660 nm, 517 nm, and 450 nm. The random expanders here are also designed for 660 nm as in the physical experiment. We assume 100% diffraction efficiency for both the SLM and the expander so no DC block is simulated.
Quantitative scores are shown in Table S1. The monochromatic evaluation uses a neural étendue expander designed specifically for a single wavelength whereas the trichromatic evalutions use a neural étendue expander designed simultaneously for all three wavelengths. The random expander 1 can only be designed for one wavelength and in this case it is designed for 660 nm. The photon sieve 3 is agnostic to wavelength because it only affects the amplitude component. The training and test datasets consists of images from personal photo collections, the DIV2K dataset 4 , and the INRIA Holiday dataset 5 .
The CGH algorithm used for our method is end-to-end gradient descent optimization with our fully differentiable image formation model 6 . We apply the same CGH algorithm when running baseline comparisons for the random expander. For the photon sieve, we follow the CGH algorithm described in their Supplementary Information. Specifically, we compute a phase-only inverse Fourier transform of the target image using IFTA 7 and then we directly sample the Fourier spectrum at the locations of the holes in the photon sieve.
We incorporate wavelength dependent dispersion into these simulations. As such, the full color holograms are generated by slightly reducing the FOV of the reconstructed holograms of the red channel (660 nm) than the green channel (520 nm). Specifically, the FOV for the red channel is scaled by a factor of 450 nm/660 nm and the FOV for the green channel is scaled by a factor of 520 nm/660 nm. For the monochromatic evaluation the full FOV of the red channel (660 nm) is used.    Figure S10: Simulated monochromatic holograms at 16× étendue expansion. The wavelength used is 660 nm. These experiments supplement the simulation results from Fig. 3 of the main manuscript.

Supplementary Note 3: Expander Analysis
In this section we provide additional analysis and derivations of the virtual frequency upper bound and the display specifications when our method is integrated with an 8K SLM 8 .

Derivation of Virtual Frequency Upper Bound
Here we provide further steps to show how we obtained the upper bound described by Equation 2 in the main manuscript. (S1) The inequality is obtained because the loss value that is obtained with the optimum setting for S must be less than or equal to the loss value when S is uses the zero-phase setting. This upper bound manifests itself in the learned frequency spectrum of the neural étendue expanders. See Supplementary Fig. S11 for a comparison between the virtual frequency modulation of the learned neural étendue expanders and the virtual frequency modulation of natural images.

Étendue-expanded Display Specifications
Here we provide the derivation of the FOV and eyebox sizes after étendue expansion. We first assume an 8K SLM 8 with 7680 × 4320 pixel resolution at 660 nm wavelength. We augment the SLM with a 64× étendue expander, resulting in 61440 × 34560 pixel resolution. Now the FOV and eyebox size are related to each other by where N n is the expanded pixel resolution 1 . By setting the eyebox to 18.5 mm we get a horizontal FOV of 126 • and a vertical FOV of 71 • . The stereo FOV provided is given by Stereo FOV = 4 arcsin(sin(Horizontal FOV/2) × sin(Vertical FOV/2)).
Thus, the total stereo FOV provided is 2.175 steradians. Human stereo FOV is 2.56 steradians 9 so this étendue expanded holographic display would provide a stereo FOV that covers 85% of the human stereo FOV. Note that this value could vary per individual. For this same system, the provided angular resolution is given by where N s is the native SLM resolution. Thus, the angular resolution is 61 pixels/degree for the setup assumed above.  Figure S11: Comparison of virtual frequency modulation of neural étendue expander and natural images. Most content for natural images resides within the lower frequency bands. The human visual system is also largely biased towards the lower frequency terms. We coarsely approximate the human visual response with low pass filters (top row). Our training algorithm teaches the expander to learn a virtual frequency modulation (middle row) that approximates the frequency modulation of a natural image dataset (bottom row). The center region insets correspond to the center eighth of each spectrum.

Supplementary Note 4: Étendue Expansion for 3D Holograms
Neural étendue expanders also facilitate the generation of 3D holograms. To demonstrate this, we optimize the SLM so that holograms are simultaneously produced at two different focus planes. Specifically, we solve a variant of Eq. from the main manuscript. This variant is given by where T near is the target image at the near plane, T far is the target image at the far plane, and Z is a z-offset phase given by where f is the z-distance of the defocus and λ is the wavelength of the laser source. Within the HOLOEYE software the distance f can be set with a slider s as where h is the height of the SLM, w is the width of the SLM, and ∆ SLM is the SLM pixel pitch. Solving Equation S5 will produce a two plane étendue expanded 3D hologram. Simulated results are shown in Supplementary Fig. S12. We used s = −0.08 for these results.  Figure S12: Simulated results for étendue expanded 3D holograms. Here we show monochromatic holograms generated at 660 nm at two different depth planes. The near plane corresponds to d = 0 mm and the far plane corresponds to d = 2 mm. Similar to the étendue expanded 2D hologram results, neural étendue expanders are capable of improving the fidelity of étendue expanded 3D holograms.

Supplementary Note 5: Étendue Expansion with Higher Order Filtering
The étendue expanded holograms are formed as the convolution of the far field wavefronts of the expander and the SLM 1 . Both of these far fields contain repeated copies or echoes as shown in Supplementary Figure S13. When these far fields are convolved the echoes are intertwined, resulting in undesirable copies within the étendue expanded hologram.
These copies do not noticeably degrade the quality of holograms of natural images. Nevertheless, if these copies must be removed then it can be performed by placing an amplitude block at the same location within the 4F system as the DC block. The amplitude block filters the higher order echoes coming from the SLM, as shown in Supplementary Figure S13. Simulated étendue expanded holograms with this block are shown in Supplementary Figure S14 and Figure S13: Removing undesired copies within étendue expanded holograms. Étendue expanded holograms contain undesirable copies because the SLM's far field wavefront contains higher order echoes. Placing a square amplitude block within the 4F system will remove the SLM's higher order echoes. This in turn removes the undesirable copies within the étendue expanded hologram.

With Echo Removal
Without Echo Removal Figure S14: Simulated results for étendue expanded holograms with and without higher order echo filtering, see text for detailed description.

Supplementary Note 6: Expander Fabrication
The expanders are physically realized as diffractive optical elements (DOE). The laser writer used for etching the stamp is capable of producing 8 levels of quantization. We set the heights of the DOE to have full 2π phase range modulation at 660 nm. Therefore,

Supplementary Note 7: Experimental Setup
We built an experimental setup to validate the neural étendue expanders. The setup consists of two main components: the SLM and the expander.
The holograms are physically realized by illuminating the SLM with a coherent, collimated laser. A single Fourier transforming lens is placed after the expander in order to produce the Fourier holograms. Finally, we use a camera to take pictures of the holograms. A schematic of this setup is shown in Supplementary Figure S19 and a picture of the physical prototype is shown in Supplementary Figure S18. A full list of parts is shown in Table S2. Note that our DC blocks are custom made by Frontrange Photomasks, a picture of the DC blocks are shown in Supplementary Figure S20.
The current prototype uses a 4F relay system to relay the SLM onto the expander which results in a bulky form factor. In the future a smaller form factor could be achieved by directly integrating the expander onto the SLM.   Figure S18: Schematic of experimental setup. Starting from the laser source, the collimating lens turns the laser source into a coherent, collimated beam. We use a half waveplate and a linear polarizer to adjust the polarization of the laser so that it matches the polarization of the SLM. The laser enters the beam splitter cube and the SLM reflects the laser light into the 4F system. The 4F system serves to filter out the SLM DC term via the DC block and also to relay the SLM onto the expander. Once the laser light passes through the expander it is then focused by the Fourier transforming lens to produce the Fourier hologram. This hologram is then imaged by the camera.  Figure S19: Picture of the physical prototype. We constructed this prototype using the parts described in Table S2. The physical realization of each component described in Supplementary Figure S18 is labeled. The neural étendue expander is mounted within the translation mount pointed to by the label. The étendue expanded hologram is formed in the space after the Fourier Transforming lens. The hologram cannot be seen in this picture as it can only be seen from the position of the Camera & Imaging Lens.
Picture of DC Block Figure S20: Picture of a DC block. We fill the center of a square piece of glass with a chrome filling. The chrome blocks out the undiffracted DC component of the SLM when placed at the Fourier plane of the 4F system. The diameter of the chrome filling shown in this picture is 1000 um. For the hardware prototype we used a DC block which has a chrome filling with a smaller diameter of 100 um.

Supplementary Note 8: Hardware Prototype Construction
In this section, we detail the calibration and alignment procedures we use for the proposed setup.

SLM Calibration
The voltage levels on the reflective SLM need to be calibrated for maximum performance. This is done via the HOLOEYE PLUTO-2 Configuration Manager. Within the configuration manager we set the voltage look-up table to maximize the optical power in the first order at the red wavelength (660 nm). The optical power was measured using a power meter (Thorlabs PM100). We also set the low level pixel voltage to 0.692 V and the high level pixel voltage to 1.286 V to produce a [0, 2π] phase range at 660 nm. Setting this phase range for the red wavelength allows for > 2π phase range for the green and blue wavelengths.
4F System Alignment We observed that the image quality significantly improves with better alignment. Hence, the following steps ensure that each element in the system is tightly bound and aligned to the main optical axis. Refer also to the equipment list in Table S2.
Set up the laser diode and the collimation lens. Use the shear interferometer to validate the collimation of the laser. Place the first 4F system lens after the beam splitter and the first pinhole. Ensure that the lens is mounted so that the optical axis is maintained through the other pinholes. This is done by looking at the laser dots formed on the other pinholes. If the alignment is off then a dot will be formed to the side of the pinhole. It is easiest to see these dots by affixing paper to the pinhole and working with the room lights off. Furthermore, it is also essential to inspect the dots formed via back reflections. Each lens is composed of several internal optical elements. If the alignment is off then internal reflections will occur within those elements and cause reflections back onto the first pinhole. Repeat for all other optical elements in the system. Note that this procedure must be conducted without the DC block in place.
Polarization Calibration Both the SLM and the laser diode are polarized. The maximum performance is achieved when the polarization states of the two elements are aligned. We will use a half-wave plate (HWP) to turn the polarization of the laser diode. We will use a linear polarizer (LP) after the HWP to eliminate stray polarization states. The goal of this stage is to align the HWP and the LP to the polarization state of the SLM. We used the following steps to align their polarization states.
First display the highest frequency blazed grating on the SLM. This pattern is designed by alternately setting each column of the SLM to be 0 or π. This pattern will produce a diffraction dot to the right of the main dot produced by the SLM's DC term. An echo of the diffraction dot also will appear to the left of the main dot. We will use the right and left diffraction dots to determine the correct polarization. We turn the LP until the area of the diffracted dots is maximized. The size of the dot should change as you turn the LP. Ignore the setting of the HWP for now. No matter how the HWP is set, changing the LP will always change the size of the diffraction dot. Once the area of the diffracted dots is maximized, we now know that the LP corresponds to the SLM's polarization state, see Figure S21a. Now we turn the HWP until the intensity of the entire diffraction pattern is maximized, see Figure S21b. After doing this we know that the HWP and LP must be aligned to the SLM.
As one additional note, turning the LP by 90 deg from the correct setting without changing the HWP will create the pattern shown in Figure S21c. Observe how the area of the diffracted dots changes from Figure S21b but the main DC dot is mostly unaffected.

Correct LP and Incorrect HWP Setting
Correct LP and Correct HWP Setting Incorrect LP with Correct HWP Setting a b c Figure S21: a Diffraction pattern when LP is set correctly while the HWP is set incorrectly. b Diffraction pattern when LP and HWP are set correctly. c Diffraction pattern when the LP is set incorrectly while the HWP is set correctly. Observe the size and intensity of the dots indicated by the red arrows.
Expander Alignment Procedure After the 4F system is constructed the expander needs to be placed at the precise location of the virtual SLM. This alignment step can be challenging because slight shifts in the expander position will result in a dramatic loss of contrast, oftentimes to the point where there is no discernible signal. Hence, the following steps are used to align the expander.
We construct a 6-axis adjustable translation and rotation stage by combining the Thorlabs KM100C with the Thorlabs Nanomax 300. This stage allows for micrometer-scale movement over the x,y,z axes and the roll, pitch, yaw axes. We mount the expander by placing it into the Thorlabs KM100C. We adjust the position of the expander until the illumination of the virtual SLM lies within the expander's boundaries.
We solve for an SLM pattern that will generate a hologram that contains a single dot. Do not use any binning on the SLM. We adjust the stage knobs until the dot is seen. Refer to Figure S22 for a capture of the dot. We then solve for an SLM pattern that will generate a hologram corresponding to a natural image. We display that SLM pattern on the SLM and we align the stage until the contrast of the hologram is maximized. Refer to Figure S23 for an example.
The reason that we begin with a dot hologram instead of going directly to a natural scene hologram is because the dot hologram is the most tolerant to alignment error. Therefore, even if the alignment is a little off the dot will be visible, whereas for natural scenes that will not be the case.
Note that the position of the relayed virtual SLM after the 4F system is slightly different for different wavelengths. Thus, the expander needs to be adjusted slightly for different colors for this version of the hardware prototype. Building a system that removes this wavelength variance is a next step.
Dot Hologram Captured By Camera Dot Hologram Displayed On SLM a b Figure S22: a SLM pattern that generates a single dot hologram produced through CGH. Display this pattern on the SLM when performing alignment. b Example of a dot hologram captured by the camera. Adjust the position of the expander until this dot hologram is shown. Note that the initial contrast of the dot may be low, further refinement of the alignment position is necessary to obtain higher contrast.
Natural Image Hologram Captured By Camera Natural Image Hologram Displayed On SLM a b Figure S23: a SLM pattern that generates a natural image hologram produced through CGH. Display this pattern on the SLM when performing alignment. b Example of a natural image hologram captured by the camera. Note that the initial contrast may be low, further refinement of the alignment position is necessary to obtain higher contrast.