Abstract
Deep learningbased image reconstruction methods have achieved remarkable success in phase recovery and holographic imaging. However, the generalization of their image reconstruction performance to new types of samples never seen by the network remains a challenge. Here we introduce a deep learning framework, termed Fourier Imager Network (FIN), that can perform endtoend phase recovery and image reconstruction from raw holograms of new types of samples, exhibiting unprecedented success in external generalization. FIN architecture is based on spatial Fourier transform modules that process the spatial frequencies of its inputs using learnable filters and a global receptive field. Compared with existing convolutional deep neural networks used for hologram reconstruction, FIN exhibits superior generalization to new types of samples, while also being much faster in its image inference speed, completing the hologram reconstruction task in ~0.04 s per 1 mm^{2} of the sample area. We experimentally validated the performance of FIN by training it using human lung tissue samples and blindly testing it on human prostate, salivary gland tissue and Pap smear samples, proving its superior external generalization and image reconstruction speed. Beyond holographic microscopy and quantitative phase imaging, FIN and the underlying neural network architecture might open up various new opportunities to design broadly generalizable deep learning models in computational imaging and machine vision fields.
Introduction
Digital holography provides unique advantages in microscopic imaging, by reconstructing the complex optical fields of input samples^{1,2,3,4,5,6,7,8,9,10,11,12}. Due to the missing phase information, various computational approaches have been developed to digitally reconstruct holograms^{13,14,15,16,17,18,19,20,21,22}. Recent work has also utilized deep neural networks^{23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45} to reconstruct the complex sample field from a hologram in a single forward inference step, achieving an image reconstruction quality comparable to iterative hologram reconstruction algorithms that are based on physical wave propagation. Some of the earlier results have also reported simultaneous performance of phase retrieval and autofocusing in a single network architecture, demonstrating holographic imaging over an extended depthoffield^{25,34,42}. In these earlier demonstrations, various deep network architectures, such as e.g., Unetbased convolutional neural networks (CNNs)^{23,25,27,33,37}, recurrent neural networks (RNNs)^{42,46}, as well as generative adversarial networks (GANs)^{31,37,42,43,47} have been proven to be effective for phase retrieval and hologram reconstruction for new (unseen) objects that belong to the same sample type of interest used during the training process. Stated differently, this earlier body of work has successfully demonstrated the “internal generalization” of the hologram reconstruction and phase retrieval networks to new objects of the same sample type as used in training.
On the other hand, “external generalization” to new objects from entirely new types of samples, never seen by the network before, remains a major challenge for deep neural networks, which might lead to image reconstruction degradation or hallucinations. Some studies have explored using transfer learning to address this challenge, which requires finetuning the network using a subset of the new types of samples^{48,49,50}. In addition to this external generalization issue, it is, in general, difficult for CNNbased image reconstruction networks to accurately reconstruct raw holograms of samples due to the limited receptive field of convolutional layers, which casts another challenge considering the relatively large scale of holographic diffraction patterns of samples. As a result, existing endtoend hologram reconstruction deep neural networks^{24,26,33,35,42} could only achieve decent reconstruction performance on relatively sparse samples. Alternatively, a preprocessing step, such as zero phasepadded free space propagation (FSP), has also been utilized to better deal with this issue^{23,25,26,40,42}, which requires a precise physical forward model with the correct estimate of the axial propagation distance.
Here we introduce an endtoend deep neural network, termed Fourier Imager Network (FIN), to rapidly implement phase recovery and holographic image reconstruction from raw holograms of new types of samples, achieving unprecedented success in external generalization. This framework takes in two or more input raw holograms captured at different sampletosensor distances without any preprocessing steps involved. By comprehensively utilizing the global spatialfrequency information processed by its trained spatial Fourier transform (SPAF) modules, FIN accurately reconstructs the complex field of the specimen, successfully demonstrating external generalization to new types of samples never used during its training. To experimentally demonstrate the success of this approach, we trained FIN models using human lung tissue samples (i.e., thin histopathology sections of connected tissue) and blindly tested the resulting trained FIN models on prostate and salivary gland tissue sections as well as Pap smear samples without compromising the image reconstruction quality. Compared to iterative hologram reconstruction algorithms based on wave propagation between different measurement planes, FIN is >27fold faster to reconstruct an image. Compared to existing CNNbased deep learning models, FIN exhibits an unprecedented generalization performance, and is also much faster in its inference speed. We expect FIN to be widely used in various image reconstruction and enhancement tasks commonly employed in the computational microscopy field. In addition to coherent imaging, FIN can be applied to other image reconstruction or enhancement tasks in different imaging modalities, including e.g., fluorescence and brightfield microscopy.
Results
FIN provides an endtoend solution for phase recovery and holographic image reconstruction, and its architecture is schematically presented in Fig. 1 (also see the Methods section). To acquire raw holograms of specimens, we used a lensfree inline holographic microscope, as detailed in the Methods section, to image transmissive samples, such as human tissue samples and Pap smears, using a set of sampletosensor distances, i.e., z_{2,i}, i = 1,⋯, M. The input images to FIN consist of M intensityonly raw holograms captured at z_{2,1} to z_{2,M} and the network outputs are the reconstructed real and imaginary images of the object, revealing the complexvalued sample field. The corresponding ground truth images for supervised learning of FIN are obtained using an iterative multiheight phase retrieval (MHPR) algorithm^{18} with M = 8 holograms acquired at different sampletosensor distances.
To demonstrate the success of FIN, we trained it using raw holograms of human lung tissue sections and tested the trained model on four different types of samples: (1) lung tissue samples from different patients never used in the training set (testing internal generalization), (2) Pap smear samples, (3) prostate tissue samples, and (4) salivary gland tissue samples, where (2,3,4) test external generalization, referring to new types of samples. The z_{2,i} distances that we used in these holographic imaging experiments were 300, 450, and 600 μm (M = 3). After its training, our blind testing results (Fig. 2) reveal that FIN can not only reconstruct new lung tissue sections from new patients (internal generalization) but also achieves a strong external generalization performance on new sample types never seen by the network before. Furthermore, compared to the output of the MHPR algorithm using the same input (raw hologram) data (M = 3), FIN is ~27.3 times faster in its inference speed per image (see Table 1) and delivers better reconstruction quality on all the test samples, as highlighted by the yellow arrows and the zoomedin regions in Fig. 2.
To further showcase the generalization ability of FIN, we separately trained four FIN models using lung tissue, prostate tissue, salivary gland tissue, and Pap smear hologram datasets (i.e., one type of sample for each network model), and blindly tested the trained models on unseen FOVs from four types of samples using M = 3 raw holograms for each FOV. Similar to our conclusions reported in Fig. 2, Fig. 3a also shows that the FIN model trained using each sample type can successfully generalize to other types of samples. Even when FIN was only trained using relatively sparse samples such as Pap smear slides, the resulting network model successfully generalized to reconstruct the raw holograms of connected tissue sections that significantly deviate from the structural distribution and sparsity observed in Pap smear samples. The success of the reconstruction performance of FIN was also quantified using the structural similarity index (SSIM)^{51} across all four FIN networks that were trained using different types of samples (see Fig. 3b), demonstrating the superior generalization of these FIN models regardless of the distribution shifts observed between the training and testing data.
Next, we evaluated the hologram reconstruction performance of FIN when only two input holograms were measured, i.e., M = 2. For this, we trained ten different FIN models from scratch using the same human lung tissue sections but with different sets of z_{2,1} and z_{2,2}, such that the sampletosensor distances for different FIN models were different. These trained FIN models were then blindly tested on new lung tissue sections from new patients (internal generalization); Fig. 4 reports the amplitude and phase root mean square error (RMSE) of the reconstructed holographic images generated by FIN (M = 2) and MHPR (M = 2) for different combinations of z_{2,1} and z_{2,2}. Both the amplitude and phase RMSE values show that FIN achieves a significant reconstruction quality improvement compared to MHPR (M = 2), and the RMSE values of FIN models are consistently better with different sampletosensor distances varying from 300 μm to 600 μm. The visualization of the reconstructed holograms shown in Fig. 4 further confirms the same conclusion that FIN achieves consistently better image reconstruction compared to MHPR for various combinations of z_{2,1} and z_{2,2}. We also confirmed that the same conclusions apply to the external generalization tests of FIN (M = 2).
In addition to MHPR based comparisons, we also extended our performance analysis to other deep learningbased phase retrieval and hologram reconstruction methods. For this additional set of comparisons, we used a stateoftheart deep learning model based on a recurrent convolutional neural network, termed RHM, that was developed for multiheight holographic image reconstruction^{42}. Using the same training hologram data, we trained FIN and RHM models for different M values, the blind testing results of which are compared in Fig. 5. As for the internal generalization performance shown in Fig. 5, both FIN and RHM can successfully generalize to new lung tissue samples from new patients. However, for the external generalization to new sample types (prostate and salivary gland tissue as well as Pap smear samples), FIN provides superior image reconstruction performance even though it uses a smaller number of trainable parameters compared to RHM (see Table 1); in comparison, RHM has reconstruction artifacts on external testing sets for both M = 3 and M = 4, also confirmed by the significantly lower SSIM values (for RHM reconstructions) reported in Fig. 5.
In addition to its superior generalization performance, FIN also has faster inference speed compared to deep learningbased or iterative phase retrieval algorithms. In Table 1, we compared the inference time of FIN, RHM, and MHPR algorithms. Noticeably, FIN has the shortest inference time among these methods using any number of raw input holograms. For the case of M = 3, FIN is ~9.3fold faster than RHM and ~27.3fold faster than MHPR, which highlights the computational efficiency of our network. We can further accelerate the inference speed of FIN by using parallelization, which reduces the computation time to 0.04 s/mm^{2} under an image batch size of 20 (see Table 1). We should also note that the number (M) of input holograms has a negligible impact on the inference time of FIN, since it uses a fixed channel size for most parts of the network model, and M only affects the first 1 × 1 convolutional layer. That is why the inference times of FIN (M = 3) and FIN (M = 4) are approximately the same as shown in Table 1. Refer to the Methods section for further details.
To further demonstrate the advantages of FIN, we performed a transfer learningbased comparison; for this, we transferred FIN and RHM models to unseen sample types to compare their generalization performance as shown in Table 2 (see the Methods). In this comparison, the original models of FIN and RHM were trained on lung tissue sections and then separately transferred to Pap smear, human prostate, and salivary gland tissue sections, creating 3 models for each method. The transferred FIN and RHM models were then tested on blind test sets of the corresponding sample types. Table 2 reveals that (1) FIN outperforms RHM in all cases, and (2) RHM gains significant performance improvement after transfer learning on all three types of samples that were never seen before. On the contrary, FIN only got differential improvement after the corresponding transfer learning step, due to its originally strong generalization to new types of samples (see Table 2).
Discussion
We demonstrated an endtoend phase retrieval and hologram reconstruction network that is highly generalizable to new sample types. FIN outperforms other phase retrieval algorithms in terms of both the reconstruction quality and speed. This method presents superior generalization capability to new types of samples without any prior knowledge about these samples or any finetuning of its trained model. This strong external generalization of our model mainly stems from the regularization effect of the SPAF modules in its architecture. In a lensfree holographic imaging system, the Fourier transforms of the fields at the sample plane and the measurement plane are related by a frequencydependent phasor, which can be effectively learned through the elementwise multiplication module in SPAF. The strong external generalization of FIN allows the model to be robust and effective on testing data distributions that are different from the training data distribution, which can result from e.g., imaging setup misalignments, poor signaltonoise (SNR) ratio, and sample related deformations or changes. The application of FIN for different imaging methods to potentially build crossmodality generalization is left as future work.
Besides, SPAF modules provide a global receptive field to FIN, in contrast to the limited, local receptive fields of common CNNs. The global receptive field helps the FIN model more effectively process the holographic diffraction patterns for various samples, regardless of the morphologies and dimensions of the objects. In fact, previous research has already shown that endtoend hologram reconstruction requires a larger network receptive field, which can be partially addressed by using e.g., dilated convolution^{42}. In our method, the Fourier transform intrinsically captures the global spatial information of the sample and thus provides a maximized receptive field for FIN, contributing to its performance gain over CNNbased hologram reconstruction models reported in Fig. 5. Like FIN, other deep neural networks^{52} have also utilized learnable spatial Fourier transform modules for inference, for example, to successfully map the initial and/or boundary conditions of partial differential equations (PDEs) and infer numerical solutions.
Unlike fully convolutional networks, in FIN architecture, the size of the input raw hologram FOV is fixed at the beginning, i.e., we cannot use a larger FOV in the testing phase because of the elementwise multiplication in our SPAF module. A larger FOV raw hologram can be reconstructed using FIN by dividing the hologram into smaller FOVs and running them through FIN in parallel. This parallelization of a large FOV hologram reconstruction is feasible since FIN has a significant speed advantage in its inference, and can reconstruct ~1 mm^{2} sample area within 0.04 sec using a standard GPU (see Table 1). The total inference time for an image FOV with ~4000 × 3000 pixels can be reduced to ~0.07 sec after parallelization, allowing realtime hologram reconstruction and imaging using FIN. This fast inference speed of FIN also opens up new opportunities to establish generalizable deep learning models for a broad spectrum of computational imaging tasks.
Materials and methods
Holographic Imaging
A lensfree inline holographic microscope was utilized to capture the raw holograms of the specimens. A broadband light source (WhiteLase Micro, NKT Photonics) and an acoustooptic tunable filter (AOTF) were used as the illumination source emitting 530 nm light. The image sensor was a complementary metaloxidesemiconductor (CMOS) RGB image sensor (IMX081, Sony). The light source, sample, and the CMOS image sensor were aligned vertically. The sample was directly placed between the light source and the sensor such that the sampletosource distance (z_{1}) was about 10 cm, and the sampletosensor distance (z_{2}) ranged from 300 to 600 μm. The CMOS sensor was placed on and controlled by a 6axis stage (MAX606, Thorlabs) to perform lateral and axial shifts. All hardware was connected to a computer and controlled by a customized LabVIEW program to capture holograms automatically.
All the human samples involved in this work were deidentified and prepared from existing specimens that were captured before this research. Human prostate, salivary gland, and lung tissue slides were provided by the UCLA Translational Pathology Core Laboratory (TPCL). Pap smear slides were prepared by the UCLA Department of Pathology.
Preprocessing
The captured raw holograms were firstly processed by a pixel superresolution algorithm^{18,53,54}. The 6axis stage was programmed to automatically capture inline holograms at 6 × 6 lateral positions with subpixel shifts. The superresolution algorithm estimated the relative shifts for each hologram and merged these holograms using a shiftandadd algorithm^{18}. The effective pixel size of the generated superresolved holograms decreases to 0.37 μm from the original CMOS pixel size of 2.24 μm. The resulting superresolved holograms were cropped into unique patches of 512 × 512 pixels, without any overlap. Hologram datasets of each sample type were partitioned into training and testing sets, at a ratio of 6:1, comprising ~600 unique FOVs in each training set and ~100 FOVs for the testing set. The testing FOVs were strictly obtained from different whole slides (new patients) excluded in the training sets.
The ground truth sample fields were retrieved by an iterative multiheight phase retrieval algorithm^{18}. At each sample FOV, M = 8 inline holograms were captured at different sampletosensor distances, which were later estimated by an autofocusing algorithm using the edge sparsity criterion^{55}. In each iteration, the estimated sample field is digitally propagated to each hologram plane using the angular spectrum propagation^{56}. The propagated complex field is updated according to the measurement at each hologram plane, by averaging the amplitude of the propagated field with the measured amplitude and retaining the new estimated phase. One iteration is completed after all the hologram planes are used, and this MHPR algorithm converges within 100 iterations.
Network structure
The FIN network architecture has a Residual in Residual architecture shown in Fig. 1, inspired by RCAN^{57} to have a deeper network structure and better information flow. Our network, FIN, consists of several SPAF modules with a long skip connection to form the largescale residual connection, in conjunction with two 1 × 1 convolutional layers at the head and tail of the network. Each SPAF group contains two recursive SPAF modules, which share the same parameters to improve the network capacity without significantly enlarging the size of the network. A short skip connection is introduced for every SPAF group to form the middlescale residual connection, and a smallscale residual connection is used to connect the inputs and outputs of each SPAF module. SPAF module, as shown in Fig. 1b, has a linear transformation applied to the tensor after it was transformed into the frequency domain using the 2D Discrete Fourier transform, following a similar architecture as in Ref. ^{52}; a half window size of k is applied to truncate the higher frequency signals, i.e.,
where \(F \in {\mathbb{C}}^{c,2k + 1,2k + 1}\) is the truncated frequency domain representation of the input to the SFAP module after performing the 2D Discrete Fourier Transform, \(W \in {\mathbb{R}}^{c,c,2k + 1,2k + 1}\) represents the trainable weights, c is the channel number, and k is the half window size. After this linear transformation, the inverse 2D Discrete Fourier transform is used to obtain the processed data back in the spatial domain, followed by a PReLU activation function.
where a is a learnable parameter.
To adapt the SPAF module to highresolution image processing in a deeper network, we shrank the matrix W allowing a significant model size reduction. The optimized linear transformation is defined as
where \(F \in {\mathbb{C}}^{c,2k + 1,2k + 1}\) is the truncated frequency components, and \(W^{\prime} \in {\mathbb{R}}^{c,2k + 1,2k + 1}\) represents the trainable weights.
To further optimize the network structure for highresolution holographic image reconstruction, a set of decreasing half window sizes (k) was chosen for the SPAF modules. Specifically, both of the SPAF modules in each SPAF group have shared hyperparameters, and we set a decreasing half window size k for the SPAF groups in the sequence of the network structure, which forms a pyramidlike structure. This pyramidlike structure provides a mapping of the highfrequency information of the holographic diffraction patterns to lowfrequency regions in the first few layers and passes this lowfrequency information to the subsequent layers with a smaller window size, which better utilizes the features at multiple scales and at the same time considerably reduces the model size, avoiding potential overfitting and generalization issues.
Network implementation
The networks are implemented using PyTorch^{58} with GPU acceleration and are trained and tested on the same computer with an Intel Xeon W2195 CPU, 256 GB memory, and NVidia RTX 2080 Ti GPUs. During the training phase, the input FOVs of 512 × 512 pixels were randomly selected from the training hologram dataset, and data augmentation was applied to each FOV, which includes random image rotations of 0, 90, 180, or 270 degrees.
The training loss is the weighted sum of three different loss terms:
where α, β, and γ are set as 0.5, 1, and 0.5, respectively. The MAE loss and complex domain loss can be expressed as:
where y is the ground truth, \(\hat y\) is the network’s output, n is the total number of pixels, and \({{{\mathcal{F}}}}\) stands for the 2D Discrete Fourier Transform operation. For the perceptual loss term^{59}, we used a pretrained VGG16 network as the feature extractor to minimize the Euclidean distance between the lowlevel features of the reconstructed images and the ground truth images.
The trainable parameters of the deep neural network models were learned iteratively using the Adam optimizer^{60} and the cosine annealing scheduler with warm restarts^{61} was used to dynamically adjust the learning rate during the training phase. Every model went through 1,000 epochs and we selected the best one with the lowest validation loss. When performing transfer learning to a new sample type (e.g., Table 2), we transferred FIN and RHM models using a smaller dataset containing 100 unique FOVs of the target sample type.
In the testing phase, a batch of test holograms with the same resolution (512 × 512 pixels) is fed to the network, and the inference time for one FOV at a time (batch size is set to 1) is 0.52 s/mm^{2}. Additionally, using the same Nvidia RTX 2080 Ti GPU, the inference can be parallelized with a batch size of 20, resulting in 0.04 s/mm^{2} inference time (Table 1).
References
Kou, S. S. et al. Transportofintensity approach to differential interference contrast (TIDIC) microscopy for quantitative phase imaging. Opt. Lett. 35, 447–449 (2010).
Popescu, G. Quantitative phase imaging of cells and tissues. (New York: McGrawHill, 2011).
Chhaniwal, V. et al. Quantitative phasecontrast imaging with compact digital holographic microscope employing Lloyd’s mirror. Opt. Lett. 37, 5127–5129 (2012).
Jericho, M. H. et al. Quantitative phase and refractive index measurements with pointsource digital inline holographic microscopy. Appl. Opt. 51, 1503–1515 (2012).
Su, T. W., Xue, L. & Ozcan, A. Highthroughput lensfree 3D tracking of human sperms reveals rare statistics of helical trajectories. Proc. Natl Acad. Sci. USA 109, 16018–16022 (2012).
Greenbaum, A. et al. Widefield computational color imaging using pixel superresolved onchip microscopy. Opt. Express 21, 12469–12483 (2013).
Greenbaum, A. et al. Widefield computational imaging of pathology slides using lensfree onchip microscopy. Sci. Transl. Med. 6, 267ra175 (2014).
Tian, L. & Waller, L. Quantitative differential phase contrast imaging in an LED array microscope. Opt. Express 23, 11394–11403 (2015).
Merola, F. et al. Tomographic flow cytometry by digital holography. Light Sci. Appl. 6, e16241 (2017).
Park, Y., Depeursinge, C. & Popescu, G. Quantitative phase imaging in biomedicine. Nat. Photonics 12, 578–589 (2018).
Barbastathis, G., Ozcan, A. & Situ, G. H. On the use of deep learning for computational imaging. Optica 6, 921–943 (2019).
Javidi, B. et al. Roadmap on digital holography [Invited]. Opt. Express 29, 35078–35118 (2021).
Fienup, J. R. Phase retrieval algorithms: a comparison. Appl. Opt. 21, 2758–2769 (1982).
Teague, M. R. Deterministic phase retrieval: a Green’s function solution. J. Optical Soc. Am. 73, 1434–1441 (1983).
Yang, G. Z. et al. Gerchberg–Saxton and Yang–Gu algorithms for phase retrieval in a nonunitary transform system: a comparison. Appl. Opt. 33, 209–218 (1994).
Allen, L. J. & Oxley, M. P. Phase retrieval from series of images obtained by defocus variation. Opt. Commun. 199, 65–75 (2001).
Marchesini, S. Invited article: a unified evaluation of iterative projection algorithms for phase retrieval. Rev. Sci. Instrum. 78, 011301 (2007).
Greenbaum, A. & Ozcan, A. Maskless imaging of dense samples using pixel superresolution based multiheight lensfree onchip microscopy. Opt. Express 20, 3129–3143 (2012).
Greenbaum, A., Sikora, U. & Ozcan, A. Fieldportable widefield microscopy of dense samples using multiheight pixel superresolution based lensfree imaging. Lab a Chip 12, 1242–1245 (2012).
Luo, W. et al. Synthetic aperturebased onchip microscopy. Light Sci. Appl. 4, e261 (2015).
Rivenson, Y. et al. Sparsitybased multiheight phase recovery in holographic microscopy. Sci. Rep. 6, 37862 (2016).
Luo, W. et al. Propagation phasor approach for holographic image reconstruction. Sci. Rep. 6, 22738 (2016).
Rivenson, Y. et al. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light Sci. Appl. 7, 17141 (2018).
Wang, H., Lyu, M. & Situ, G. H. eHoloNet: a learningbased endtoend approach for inline digital holographic reconstruction. Opt. Express 26, 22603–22614 (2018).
Wu, Y. C. et al. Extended depthoffield in holographic imaging using deeplearningbased autofocusing and phase recovery. Optica 5, 704–710 (2018).
Goy, A. et al. Low photon count phase retrieval using deep learning. Phys. Rev. Lett. 121, 243902 (2018).
Zhang, G. et al. Fast phase retrieval in offaxis digital holographic microscopy through deep learning. Opt. Express 26, 19388–19405 (2018).
Ren, Z. B., Xu, Z. M. & Lam, E. Y. Learningbased nonparametric autofocusing for digital holography. Optica 5, 337–344 (2018).
Rivenson, Y., Wu, Y. C. & Ozcan, A. Deep learning in holography and coherent imaging. Light Sci. Appl. 8, 85 (2019).
Liu, T. R. et al. Deep learningbased color holographic microscopy. J. Biophotonics 12, e201900107 (2019).
Wu, Y. C. et al. Brightfield holography: crossmodality deep learning enables snapshot 3D imaging with brightfield contrast using a single hologram. Light Sci. Appl. 8, 25 (2019).
Jo, Y. et al. Quantitative phase imaging and artificial intelligence: a review. IEEE J. Sel. Top. Quantum Electron. 25, 6800914 (2019).
Wang, K. Q. et al. YNet: a onetotwo deep learning framework for digital holographic reconstruction. Opt. Lett. 44, 4765–4768 (2019).
Byeon, H., Go, T. & Lee, S. J. Deep learningbased digital inline holographic microscopy for high resolution with extended field of view. Opt. Laser Technol. 113, 77–86 (2019).
Ren, Z. B., Xu, Z. M. & Lam, E. Y. M. Endtoend deep learning framework for digital holographic reconstruction. Adv. Photonics 1, 016004 (2019).
Li, H. Y. et al. Deep DIH: singleshot digital inline holography reconstruction by deep learning. IEEE Access 8, 202648 (2020).
Moon, I. et al. Noisefree quantitative phase imaging in Gabor holography with conditional generative adversarial network. Opt. Express 28, 26284–26301 (2020).
Zeng, T., So, H. K. H. & Lam, E. Y. RedCap: residual encoderdecoder capsule network for holographic image reconstruction. Opt. Express 28, 4876–4887 (2020).
Liu, T. R. et al. Deep learningbased holographic polarization microscopy. ACS Photonics 7, 3023–3034 (2020).
Deng, M. et al. Learning to synthesize: robust phase retrieval at low photon counts. Light Sci. Appl. 9, 36 (2020).
Yin, D. et al. Digital holographic reconstruction based on deep learning framework with unpaired data. IEEE Photonics J. 12, 3900312 (2020).
Huang, L. Z. et al. Holographic image reconstruction with phase recovery and autofocusing using recurrent neural networks. ACS Photonics 8, 1763–1774 (2021).
Yang, X. et al. High imaging quality of Fourier single pixel imaging based on generative adversarial networks at low sampling rate. Opt. Lasers Eng. 140, 106533 (2021).
Shimobaba, T. et al. Deeplearning computational holography: a review (invited). Front. Photonics 3, 854391 (2022).
Pirone, D. et al. Speeding up reconstruction of 3D tomograms in holographic flow cytometry via deep learning. Lab a Chip 22, 793–804 (2022).
Huang, L. Z. et al. Recurrent neural networkbased volumetric fluorescence microscopy. Light Sci. Appl. 10, 62 (2021).
Goodfellow, I. et al. Generative adversarial networks. Commun. ACM 63, 139–144 (2020).
Kim, S. J. et al. Deep transfer learningbased hologram classification for molecular diagnostics. Sci. Rep. 8, 17003 (2018).
Zhu, Y. M., Yeung, C. H. & Lam, E. Y. Digital holographic imaging and classification of microplastics using deep transfer learning. Appl. Opt. 60, A38–A47 (2021).
Huang, L. Z. et al. Fewshot transfer learning for holographic image reconstruction using a recurrent neural network. APL Photonics 7, 070801 (2022).
Wang, Z., Simoncelli, E. P. & Bovik, A. C. Multiscale structural similarity for image quality assessment. The ThirtySeventh Asilomar Conference on Signals, Systems & Computers, 2003. Pacific Grove: IEEE, 2003, 13981402.
Li, Z. Y. et al. Fourier neural operator for parametric partial differential equations. The 9^{th} International Conference on Learning Representations (ICLR21). https://iclr.cc/virtual/2021/poster/3281 (2021).
Bishara, W. et al. Lensfree onchip microscopy over a wide fieldofview using pixel superresolution. Opt. Express 18, 11181–11191 (2010).
Greenbaum, A. et al. Imaging without lenses: achievements and remaining challenges of widefield onchip microscopy. Nat. Methods 9, 889–895 (2012).
Zhang, Y. B. et al. Edge sparsity criterion for robust holographic autofocusing. Opt. Lett. 42, 3824–3827 (2017).
Goodman, J. W. Introduction to Fourier Optics. 3rd edn. (Greenwoood Village: Roberts & Company Publishers, 2005).
Zhang, Y. L. et al. Image superresolution using very deep residual channel attention networks. The 15th European Conference on Computer Vision. Munich, Germany: Springer, 2018, 294310.
Paszke, A. et al. PyTorch: an imperative style, highperformance deep learning library. Proceedings of the 33rd International Conference on Neural Information Processing Systems. Vancouver: ACM, 2019, 80268037.
Johnson, J., Alahi, A. & FeiFei, L. Perceptual losses for realtime style transfer and superresolution. The 14th European Conference on Computer Vision. Amsterdam: Springer, 2016, 694711.
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Print at https://doi.org/10.48550/arXiv.1412.6980 (2014).
Loshchilov, I. & Hutter, F. SGDR: stochastic gradient descent with warm restarts. Print at https://doi.org/10.48550/arXiv.1608.03983 (2016).
Acknowledgements
The Ozcan Research Group at UCLA acknowledges the support of NSF.
Author information
Authors and Affiliations
Contributions
A.O. conceived the research, H.C. and L.H. conducted the numerical experiments and analyzed the results. T.L. captured the experiment data, L.H. and T.L. processed and analyzed the data. All the authors contributed to the preparation of the manuscript. A.O. supervised the research.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no competing interests.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Chen, H., Huang, L., Liu, T. et al. Fourier Imager Network (FIN): A deep neural network for hologram reconstruction with superior external generalization. Light Sci Appl 11, 254 (2022). https://doi.org/10.1038/s41377022009498
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41377022009498
This article is cited by

Selfsupervised learning of hologram reconstruction using physics consistency
Nature Machine Intelligence (2023)

Randomness assisted inline holography with deep learning
Scientific Reports (2023)