New diagonal micropolarizer arrays designed by an improved model in fourier domain

The design of micropolarizer array (MPA) patterns in Fourier domain provides an efficient approach to reconstruct and investigate the polarization information. Inspired by Alenin’s works, in this paper, we propose an improved design model to cover both 2 × N MPAs and other original MPAs, by which an entirely new class of MPA patterns is suggested. The performance of the new patterns is evaluated through Fourier domain analysis and numerical simulations compared with the existing MPAs. Particularly, we analyze the reconstruction accuracy of the first three Stokes parameters and degree of linear polarization (DoLP) in detail. The experimental results confirm that the 2 × 2 × 2 MPA provides the highest reconstruction quality of s0, s1, s2 and DoLP in terms of quantitative measures and visual quality, while the 3 × 3 diagonal MPA achieves the state-of-the-art best results in case of single-snapshot systems. The guidance of this extended model and new diagonal MPAs show its massive potential for the division of focal plane (DoFP) polarization imaging applications.

At present, the most attractive polarization imaging instrument employs the way of division of focal plane (DoFP), in which a micropolarizer array (MPA) is placed in front of the camera's focal plane, showing the advantages of simple structure, compact size and the capability of real-time detection 1,2 . The first polarization imaging system which had been built using the conventional 2 × 2 repeating pattern of polarization analyzer, was introduced by Chun 3 in 1994 and has been a standard for almost 20 years. However, since each pixel within a 2 × 2 minimum periodic array can only collect the intensity information of its corresponding polarization orientation 4 , the spatial resolution is reduced, and the reconstructed polarization information is not accurate due to the instantaneous field of views (IFOV) 5,6 . Therefore, it is significant to enhance the quality of polarization reconstruction images for further applications.
Until now, various methods have been developed for the quality enhancement of polarization reconstruction images. Subsequently, several works have demonstrated that both the layout of MPAs and the reconstruction algorithms affect the quality of the reconstructed Stokes parameter images [7][8][9][10][11] . Furthermore, some unconventional MPA patterns 7,8 and interpolation methods 6,[9][10][11] have been proposed to achieve superior resolution and accurate polarization information. For example, in 2014, LeMaster and Hirakawa 7 creatively put forward a novel 2 × 4 MPA pattern, which demonstrated that the high reconstruction quality could be achieved over the traditional 2 × 2 MPA. For further exploration, in 2017, Alenin et al. 8 designed 2 × 3, 2 × 7 and 2 × 2 × 2 MPA patterns for linear DoFP polarimeters, and extended the 2 × 4 MPA into a family of 2 × N MPAs aiming to widen the optimization space in Frequency Domain. The results indicated that the maximized theoretical spatial resolution of the reconstructed Stokes parameters could be achieved by the 2 × 2 × N MPAs. Despite their previous superior efforts advanced the available set of MPA patterns and improved the reconstruction quality of Stokes parameters and degree of linear polarization (DoLP) images, however, these works were merely focused on the family of 2 × N patterns, and although the 2 × 2 × N MPAs have superior performance in reconstruction quality, these MPAs should be used in multiple-snapshots systems. Besides, to the best of our knowledge, the application of a generalized model to investigate new MPA layouts for improving reconstruction accuracy has not been reported. Therefore, for the purpose of a higher reconstruction quality in case of single-snapshot systems, designing higher-performance MPA patterns for linear DoFP polarimeters is required.
Motivated by the extraordinary design model 8  where I 0°, I 45°, I 90°, and I 135° represent the intensities measured through a linear polarizer in the directions of 0°, 45°, 90°, and 135°, respectively, and s 3 is always ignored for linear DoFP polarimeters 13 .
To derive Stokes parameters, the captured image I is given using a series of pre-determined analyzing polarization states, A N 8 , where I N represents the intensity of the polarized light passing through the linear polarizer, S represents the unknown polarization state, and n represents an additive detector noise. Then the unknown state's Stokes parameters can be revealed by, where S represents the calculated Stokes parameters and "+" in the upper right corner denotes the pseudo-inverse of a matrix. Because the layout of the MPA is periodic and defined by the square lattice, it is convenient to design polarization orientations of MPAs using an analysis vector. The conventional and most commonly used 2 × 2 MPA, as shown in Fig. 1a1, and the available set of 2 × N MPAs, as shown in Fig. 1b1-e1 can be expressed by using the analyzing vector 8 , where m and n are the horizontal and vertical pixel coordinates in a MPA pattern, respectively. The polarization orientations of each pixel can be calculated by changing the m and n, while a and b determine the positioning of carriers in horizontal and vertical direction, respectively. Inspired by the above design model, we propose a new generalized model, which breaks the limit of Eq. (4) and can be used to design a novel type of MPAs with N × N diagonal configuration. The permutation of three N × N diagonal MPAs, as s hown in Fig. 1e1-g1, belong to the following analyzing vector, where θ represents the orientation angle of analyzer at different location. According to the Eqs. (5) and (6), it is convenient to calculate the orientation angles of the MPAs designed by our proposed model.

MPA's performance evaluations.
As shown in Alenin's seminal work 8 , the Fourier transform of Eq. (5) can be denoted by the Frequency Phase Matrix (FPM) to describe the corresponding channel structure, which provides new insights into MPA patterns design, where a, b and c determine the positioning of carriers in Fourier domain. For the purpose of analyzing the proposed model theoretically, we use the Q formalism to analyze the Stokes polarimeters, which can reveal the relationship between the Stokes parameters and the set of channel structures 8,13 . By properly inverting process, the Stokes vector can also be reconstructed by, where Q + is the pseudo-inverse of Q, and C is the related channel. Accordingly, the parameters of MPAs designed by Eq. (5), the corresponding channel structures indicated by the Q matrix and its inverse are all shown in Table 1.
Based on the Alenin's theory 14 , to assess the performance of a MPA while taking the noise, bandwidth, and system error into consideration, using the equally weighted variance (EWV) inferred by Q + as an index is significant. It provides a handy way to evaluate the intrinsic properties of the channel structure. As shown in Table 1, the family of N × N diagonal MPAs has the same value of EWV as the family of 2 × N MPAs. In particular, we can also design the 2 × 2 × 2 MPA with the setting of a = 2 and b = c = 1 in our proposed model, whose EWV is equal to 8.5. Although this type of MPA extends the channel centers to the outer corners and provides a strong capability of noise resilience, this MPA cannot reconstruct the first three Stokes parameters in real-time 8 . To obtain all the mentioned Stokes parameters simultaneously, it must add a temporally modulated ferroelectric liquid crystal rotator. As an appropriate metric, the concept of EWV can be used to evaluate the performance I θ = (s 0 + s 1 cos 2θ + s 2 sin 2θ)/2, Table 1. Notable micropolarizer array patterns. The circles represent the coefficient's polar form of the frequency phase matrix, and the total number of circles is 9 × 4 = 36. The direction of the radius line in the circles represents the phase information as: right = + 1, up = + j, left = − 1, down = − j. The frequency phase matrix in Eq. (7) indicates that there have nine channels, which contain four empty channels and five nonempty channels. Empty circles indicate that the locations of the channels do not contain the information of Stokes parameter 8 . a CN is the condition number, which ensures the optimal configuration for W 12 . b σ 2 0,1,2,3 are the noise variances from Q, which is a 9 × 4 matrix; the EWV is the sum of σ 2 0,1,2,3 . www.nature.com/scientificreports/ and assess noise immunity. However, the bandwidth also has an effect on the quality of reconstruction, which improves when the higher frequency information is included. Therefore, the design of MPA patterns should take into account various factors comprehensively, and the effective analysis of the bandwidth of s 0 , s 1 and s 2 is significant for evaluating the MPA's performance. As shown in Fig. 1, it is noticeable that the spectra have several multiplex frequency components different by the location and the bandwidth in Fourier domain. White solid lines outline the channel bandwidths of s 0 , and white dotted lines outline the channel bandwidths of multiplex components of s 1 and s 2 . Note that by convention, the basebands are placed at the center in the second row, but all the DFT spectra are periodic in both horizontal and vertical directions. Based on the Fourier domain representation, it is intuitive and visual to analyze the location of multiplex frequency components, the extent of aliasing contaminations and relative bandwidths between different components. As shown in Fig. 1a2, we can easily discover that the conventional 2 × 2 MPA has a small theoretical bandwidth of s 0 , because the frequency components overlap on the horizontal and vertical axes. Obviously, as shown in Fig. 1b2-h2, the 2 × N MPAs and N × N diagonal MPAs remarkably reduce the risk of aliasing and improve the bandwidth of s 0 by further separating s 1 and s 2 carriers away from the s 0 carrier. Therefore, these MPAs can significantly reduce the crosstalk impact of different components and improve the reconstruction quality as the theoretical bandwidth of s 0 increases. Similarly, we can analyze the theoretical bandwidth of s 1 and s 2 with the same approach. Avoiding the frequency components situated on the horizontal and vertical axes, the reconstruction accuracy of s 1 and s 2 increases strikingly as the theoretical bandwidth increases. Based on these analyses, we can see that the 2 × 2 × 2 MPA provides the best s 0 , s 1 and s 2 reconstruction accuracy in case of multiple-snapshots systems, while the 7 × 7 diagonal MPA provides the best s 0 reconstruction accuracy in case of single-snapshot systems theoretically. Furthermore, the quality of the DoLP should be weighed depending on various factors comprehensively.

Experimental validation
Reconstruction method. To provide a quantitative comparison of different MPA patterns, numerical simulations are also performed to evaluate the imaging quality of reconstruction. Based on Junchao Zhang's database 15 , we simulated the reconstructed performance of Stokes parameters and DoLP through visual and quantitative measurement by comparing three new N × N diagonal MPA patterns with several mainstream MPA patterns such as 2 × 2 3 , 2 × 4 7 , 2 × 3 8 , 2 × 7 8 , 2 × 2 × 2 8 . The scheme for the reconstruction of Stokes parameters and DoLP images is described in Fig. 2, which contains four main steps: (1) by applying the true intensity images of I 0°, I 45°, I 90° and I 135°, we can calculate the Stokes parameters and DoLP images as the true samples; (2) by combining different MPA patterns as shown in Fig. 1, we generate the synthesized DoFP images from the true Stokes parameters images of s 0 , s 1 and s 2 ; (3) through Discrete Fourier Transform(DFT) and frequency domain filtering, we get three frequency components, respectively; (4) by employing the Inverse Fourier Transform (IFT), we can reconstruct the Stokes parameters and DoLP images in the end.
To reconstruct the Stokes parameters and DoLP from the synthesized image, the reconstruction algorithm and filtering operation must be applied 9,16 . As shown in Fig. 1, the appropriate band-pass filters must be used to estimate the frequency components from different locations and bandwidths. Like the color filter array (CFA) designing and reconstructing process 17,18 , we want to note that the algorithms also profoundly affect the reconstruction quality. Instead of developing extremely complex reconstruction algorithm, in this paper we focus on new diagonal MPAs design and the comparison of the MPA's inherent attribute. Considering that the practical bandwidths occupied by the frequency components are diverse for different scene targets and different MPAs, the filters based on the least-square method have more powerful abilities to be suited for various scenes and MPAs. Inspired by the reconstruction process for the Bayer CFA 19,20 , in this paper, we used the reconstruction algorithm based on the least-square filters to reconstruct Stokes parameters and DoLP. In order to estimate for the components s 0 , s 1 , and s 2 by the filtering operation, the corresponding filters h 1 , h 2 , h 3 were separately optimized according to the scenes and MPA patterns we mentioned. Formally, the least-square filters can be expressed by, As shown in Figs. 3i1 and 4i1, two target scenes were used to evaluate the reconstructed performance. Notably, the white rectangular regions of the ground-truth image were expanded and presented in Figs. 3a2-i2 and 4a2-i2. To facilitate the objective comparisons, the peak-signal-to-noise-ratio (PSNR) 21 was used as a quantitative evaluation criterion in this paper, which is the most common measure to assess the performance of spatial resolution enhancement. The calculation formula of PSNR is defined as: In order to give a comprehensive evaluation, the gradient magnitude similarity deviation (GMSD) 22 was also used to measure the quality of reconstructed images, which makes use of global variation of gradient based local quality map for the overall image quality prediction   www.nature.com/scientificreports/ where N represents the total number of pixels in the image. GMS represents the gradient magnitude similarity map and GMSM is the average of GMS. In general, the larger PSNR value is, the higher reconstruction accuracy will be, while a smaller GMSD value indicates that the reconstruction is of higher quality.

Results and discussions
The value of PSNR and GMSD used for comparison are shown in Table 2, and the best results in each row are in boldface and underlined among the proposed MPA layouts. For the single-snapshot systems, the best results in each row are highlighted in bold, to indicate the best reconstruction performance among the single-snapshot layouts. It is obvious from Table 2 that the 2 × 2 × 2 MPA provides the best s 0 , s 1 , s 2 accuracy and DoLP accuracy in all scenes, which is consistent with the theoretical analysis. Besides, the new N × N diagonal MPAs provide better Stokes parameters and DoLP reconstruction accuracy than the conventional 2 × 2 MPA in the building1 scene. In particular, the new 5 × 5 diagonal MPA provides the worst DoLP reconstruction accuracy in the build-ing2 scene, which indicates that the frequency structure is particularly associated with the MPA pattern and the scenes. In addition, according to the PSNR and GMSD results, the 2 × 3 MPA can reconstruct the s 0 with a higher accuracy than other single-snapshot layouts, and the 7 × 7 diagonal MPA comes second. Note that if we need to reconstruct Stokes parameters and DoLP images in case of single-snapshot systems, the 3 × 3 diagonal MPA provides the best s 1 , s 2 , and DoLP accuracy in all scenes, which demonstrates that our proposed new 3 × 3 diagonal MPA reconstructs state-of-the-art results. Moreover, the reconstructed DoLP images are shown in Figs. 3a1-i1 and 4a1-i1, and the enlarged views of the reconstructed images and ground-truth image are shown in Figs. 3a2-i2 and 4a2-i2 for visual comparison. As shown in Figs. 3 and 4, we can see that the reconstructed DoLP image by 2 × 2 × 2 MPA is much cleaner while preserving image details and edges, removing aliasing artifacts and reducing the zipper effect artifacts. However, in case of single-snapshot systems, it is seen that the quality of the DoLP image achieved by 3 × 3 diagonal MPA is much better. On the contrary, it can be seen clearly that the loss of image details and the generation of false grid exists in other reconstructed DoLP images. Thus, the novelty diagonal MPAs proposed in this paper can enhance the quality of the reconstruction images effectively, and our improved model has great value for DoFP polarization imaging. Finally, to compare the capability of denoising, we first add Gaussian noise to the original images. Then the corresponding reconstructed images are generated using the different MPAs. Similarly, the PSNR is used to evaluate the reconstruction performance, and the level of the additional noise represented by the signal-to-noise ratio (SNR) is increased by every 2 dB from 0 to 60 dB. The corresponding PSNR curves are drawn in Fig. 5. Experimental results show that the value of PSNR increases as the SNR increases. For a low SNR, all the MPAs behave a similar s 1 , s 2 and DoLP reconstruction performance, because all the filters are narrow and have a similar size. However, the 2 × 2 × 2 MPA provides the highest s 0 reconstruction accuracy due to the most bandwidth. As the SNR increases, the filters are widened to include more frequency information. The 2 × 2 × 2 MPA gives the best s 0 , s 1 , s 2 and DoLP reconstruction quality, and the 2 × 2 MPA exhibits the worst performance in the building1 scene. Unlike this trends, the 5 × 5 diagonal MPA exhibits the worst s 1 , s 2 and DoLP reconstruction performance in the building2 scene. It worth noting that the relative bandwidth may behave flexible size in Fourier domain for different scenes 8 , which means the proposed MPAs may highlight their advantages and properly be applied to reconstruction, when the s 0 , s 1 and s 2 channels require less or more bandwidth. As the SNR reaches a high enough level, it is expected that the PSNR curves keep stable. In this case, a larger filter can result in a grid effect, due to the high-frequency content mixing in the extractive information. In contrast, a narrower filter leads to excessive blurring because of the information missing. All in all, the 2 × 2 × 2 MPA provides the best s 0 , s 1 , s 2 and DoLP accuracy, while the 3 × 3 diagonal MPA designed by the improved model achieves state-of-the-art results in case of achieving the Stokes parameters s 1 , s 2 and DoLP in the single-snapshot systems, which demonstrates that our improved design model can efficiently find new MPA patterns for enhancing the quality of reconstruction images.

Conclusions
In conclusion, we propose an improved design model for constructing richer MPA patterns. This model is based on leveraging the Fourier domain and designing information carriers that yield optimal bandwidth, which can extend the 2 × N MPAs into an entirely new family of N × N diagonal MPAs. Both visual and quantitative measurements are applied to evaluate the reconstructed performance and denoising performance of s 0 , s 1 , s 2 and DoLP.
The experimental results show that the quality of s 0 , s 1 , s 2 and DoLP reconstruction is significantly improved in terms of quantitative measures and visual quality by 2 × 2 × 2 MPA. If we need to reconstruct the Stokes parameters s 1 , s 2 and DoLP simultaneously in case of a single-snapshot system, the 3 × 3 diagonal MPA achieves the state-of-the-art best results. This improved model widens the optimization space and can construct a number of available MPAs, which may further facilitate the practical applications of the DoFP technique.