A polarization-based image restoration method for both haze and underwater scattering environment

Existing polarization-based defogging algorithms rely on the polarization degree or polarization angle and are not effective enough in scenes with little polarized light. In this article, a method of image restoration for both haze and underwater scattering environment is proposed. It bases on the general assumption that gray variance and average gradient of a clear image are larger than those of an image in a scattering medium. Firstly, based on the assumption, polarimetric images with the maximum variance (Ibest) and minimum variance (Iworst) are calculated from the captured four polarization images. Secondly, the transmittance is estimated and used to remove the scattering light from background medium of Ibest and Iworst. Thirdly, two images are fused to form a clear image and the color is also restored. Experimental results show that the proposed method obtains clear restored images both in haze and underwater scattering media. Because it does not rely on the polarization degree or polarization angle, it is more universal and suitable for scenes with little polarized light.

A polarization-based image restoration method for both haze and underwater scattering environment Zhenming Dong 1,2 , Daifu Zheng 1,2 , Yantang Huang 1,2 , Zhiping Zeng 2 , Canhua Xu 1,2* & Tingdi Liao 3,4 Existing polarization-based defogging algorithms rely on the polarization degree or polarization angle and are not effective enough in scenes with little polarized light. In this article, a method of image restoration for both haze and underwater scattering environment is proposed. It bases on the general assumption that gray variance and average gradient of a clear image are larger than those of an image in a scattering medium. Firstly, based on the assumption, polarimetric images with the maximum variance (I best ) and minimum variance (I worst ) are calculated from the captured four polarization images. Secondly, the transmittance is estimated and used to remove the scattering light from background medium of I best and I worst . Thirdly, two images are fused to form a clear image and the color is also restored. Experimental results show that the proposed method obtains clear restored images both in haze and underwater scattering media. Because it does not rely on the polarization degree or polarization angle, it is more universal and suitable for scenes with little polarized light.
In traffic monitoring, remote sensing and ocean exploration, because of the scattering medium, such as fog, haze, turbid water and so on, the visibility and the contrast of images are reduced and the details of images are blurred [1][2][3] . Usually there are two types of imaging methods to enhance the image clearness 1 . One is built on image enhancement algorithm, such as histogram equalization 4 and Retinex algorithm 5 . The other is image restoration methods, which obtain unscattered object light based on specific physical models or priori hypotheses, such as dark channel prior method, Schechner's polarization defogging method, and Tan's single image defogging method. The image enhancement methods are straight forward and very effective to improve the contrast of blurred images. However, because these methods do not take into consideration of the image degeneration, they usually induce more image distortion and obtain fewer recovery details than image restoration methods. In addition, defogging methods based on using deep learning technologies become prevail in recent years 6,7 .
In this article, we present a defogging algorithm in the scope of image restoration method. Plenty of researches on image restoration techniques through scattering mediums have been conducted over the past decades. Classic methods are listed as following. In 2001, Schechner et al. [8][9][10] proposed an image defogging method based on polarization difference. They assumed that the atmospheric light is partially polarized and the object light is unpolarized. The difference between two polarization images ("I max " and "I min ") is regarded as atmospheric polarized light. And an Atmospheric Propagation Model is adopted to restore the image. In the next few years this method was extended to be used in the field of underwater imaging experiments 11 . In 2008, Tan 12 proposed a single image defogging method based on two assumptions of that clear images have larger contrast than foggy images and the airlight tends to be smooth in foggy weather. In the same year, Fattal 13 formulated a refined image formation model. They introduced the new variable "surface shading" and assumed that "surface shading" and transmission functions are locally statistically uncorrelated. Then the defogging is conducted with the transmission functions calculations. The well-known image dehazing method using dark channel prior 14 6 . In very recent years, Zhu et al. 15,16 proposed a novel fast single image dehazing algorithm based on artificial multiexposure image fusion and an image dehazing method by an artificial image fusion method based on adaptive structure decomposition. Vazquez-Corral et al. 17 proposed a method of physical-based optimization for non-physical image dehazing methods. In 2021, by using dual self-attention boost residual octave convolution, Zhu et al. 18,19 developed a defogging method suitable for remote sensing imaging. In general, benefit from more information obtained from multiple different polarization images, restoration method based on polarization imaging has more advantages and better details than single image dehazing methods. This technology had been intensively researched in past decades in the aspects of visibility enhancement 20 , active imaging system 21,22 , underwater target detection [23][24][25] , real-time measurement 26 , long-range polarization imaging 27 , utility of the polarization angle [28][29][30] . Moreover, Shao et al. 31 proposed a hazy image restoration method based on atmospheric light polarization tomography. Liu et al. 32 used Wavelet Transform to stratify images and removed haze of images. Fang et al. 33 proposed an image dehazing method using polarization effects of objects and airlight. In 2021, Liang et al. 34 proposed a low-pass filtering based polarimetric dehazing method for dense haze removal. So far plenty of methods have been used to restore the image effectively in the specific scene. The restoration effect depends on the accurate estimation of polarization degrees or polarization angles. However, it is difficult to calculate the polarization parameters exactly in a strong scattering scene with very weak polarized light. Large polarization parameters estimation error results in unsuccessful image restorations in these scenes. In comparison, calculation of gray variance is more accurate and practical in the various conditions. Based on the general assumption that gray variance and average gradient of a clear image are larger than those of an image through scattering media, a universal image restoration method is proposed in this article. Firstly, Stokes parameters are used to find polarimetric images with the maximum variance (I best ) and minimum variance (I worst ). Secondly, the transmittance can be estimated and be used to remove the scattering light from background medium of I best and I worst . Thirdly, two images are fused to form a clear image and the color is also restored. Experimental results show that the proposed method is applicable both in atmospheric and underwater scattering media. Moreover, it is still very effective in scenes with little polarized light.

Technical background
Stokes parameters. The polarization properties of light can be described by the Stokes parameters. The Stokes parameters of light are denoted as [S 0 , S 1 , S 2 , S 3 ] T , where S 0 is the total light intensity, S 1 is the light intensity difference between 0° and 90° polarization direction, S 2 is the light intensity difference between 45° and 135° polarization direction, and S 3 is the light intensity difference between left-handed and right-handed circularly polarized light. Generally, there is little circularly polarized light in the natural environment, so S 3 is ignored in this article. By measuring the intensities of light along three or four different angles, the first three components of the Stokes parameters can be obtained. For example, measure the intensities of light along four directions of 0°, 45°, 90°, and 135°, which are respectively expressed by I 0 , I 45 , I 90 , and I 135 . The first three components of the Stokes parameters can be calculated as 29 : By multiplying with Mueller matrix of a linear polarizor with axis at angle θ, the intensity image at any angle (denoted by I θ ) can be derived from the first three components of the Stokes parameters as Eq. (2) 30 , in which the polarimetric images with the maximum and minimum variance can be selected.
Atmospheric scattering model and underwater scattering model. In a scattering environment, the total light captured by a camera I total can be calculated from the sum of the attenuated object light L·t and the back scattered light A. Here, L is the clear object image and t is the transmittance of the scattering media. Generally, transmittance t is defined as Eq. (3), which relates the backscattered light A in the camera and the backscattered light from the infinite distance A ∞ .
So the restored image can be expressed by the Eq. (4) 35 . There are only two unknowns t and A ∞ , which need to be determined in the following restoration methods. (1)

Image restoration methods based on polarization imaging
In order to overcome the difficulty of polarization parameters estimation in a strong scattering scene with very weak polarized light, we adopted a restoration process based on the variance prior and illustrated it in Fig. 1. At first, the polarimetric images containing the most and least object light were selected among the calculation results of Eq. (2) with a variance prior, and named as I best and I worst ; then we calculated the backscattered light A w from I worst using a wavelet transform, and obtained the degenerated object light D b from the difference between I best and A w multiplying with several polarimetric factors; furthermore, the transmittance of the media t was calculated by the combination of D b , I best , I worst and the corresponding backscattered light; and the undegenerated object light in I best was calculated as L b = D b / t. Similar process was used to calculate the undegenerated object light in I worst and named as L w . In the end, the sum of L b and L w was taken as the final restored image in our method. The detailed restoration algorithm was described as the following.
Find the I best and I worst . The total light intensity received by the camera sensor is defined as I total , which is composed of the light from the object (defined as D) and the light from the background scattering medium (defined as A). This process is illustrated in Fig. 2.
In general, D and A have different polarization degrees and polarization angles. A linear polarizer is installed in front of an ordinary camera. Rotating the polarizer at different angles, the visibility of the image is different. At an angle θ, the light intensity I θ can be written as  www.nature.com/scientificreports/ The angle at which the image contains the most object light can be defined as θ best , and the corresponding polarization intensity is defined as I best . The angle at which the image contains the least object light is defined as θ worst and the corresponding intensity is I worst . At angle θ best , the value of D θ /I total is maximum. At angle of θ worst , the value of D θ /I total is minimum. In general, θ best is perpendicular to θ worst . In previous work, θ best and θ worst are usually determined by the polarization property of a specific region. For example, using the brightest angle 8-10 , the direction perpendicular to the incident surface 11,22 , the polarization angle of the sky region 28 , or two reference objects 36 to calculate θ best were reported in former researches. In order to establish a universally applicable algorithm for various scattering media, we use a variance prior based on the fact that clear images usually have larger gray-scale variance and larger average gradient than images through scattering media 12 . The scattering medium overwhelms the details, reduce the contrast, gray-scale variance and average gradient 29 . Therefore, the gray-scale variance is taken as the standard to find θ best and θ worst . Because the gray-scale variance does not depend on the polarization properties of a specific region of the image, it can be universally applied to various scattering media. In the specific calculation, the polarization image at any angle can be obtained by using Eq. (2). Each image at a different angle is divided into 10 × 10 blocks. The angle with maximum gray-scale variance in most regions is selected and regarded as θ best . θ worst is perpendicular to θ best . After θ best and θ worst are selected, I best and I worst can be calculated by Eq. (2). The θ best which maximizes the gray-scale variance of the image is automatically determined by the algorithm.

Obtain the object light (degenerated) of I best .
Usually, the light intensity in a scattering media consists of object light D and backscattered light A. In Eqs. (6) and (7), b and w are the subscripts representing the images at θ best and θ worst , respectively. After selecting the minimum variance region as background region, the polarization degree P can be derived from the background region of I best and background region of I worst 9 .
From Eq. (8), we obtain: By combining Eqs. (6) and (9), we obtain: A w and P are required to calculate D b . A w is the backscattered light of I worst . Because background scattering media are relatively smooth compared with objects 12 , object light has a much higher frequency than the backscattered light. So the low-frequency component of I worst (denoted by A′ w ) can be treated as an approximation of the backscattered light A w . Wavelet Transform is adopted to obtain the low-frequency component of I worst in this article. For example, we use wavelet transform (basis function "db8") to take the fourth layer of low-frequency components of the image I worst as A′ w . There is still some object light that has not been removed in A′ w . Two correction factors were introduced to estimate backscattered light more accurately.
First correction factor (denoted by y1). For each channel, the correction factor component y i (i = r, g, b) can be obtained by using the following formula: A w ∞ can be derived from the background region (average value of minimum variance region) of I worst , and note that A w ∞ is the backscattered light of I worst from an infinite distance.
The correction factor y1 is a matrix which is used to estimate the backscattered light of each pixel in A′ w . It can be described simply as the reciprocal of the difference between a pixel and A w ∞ . Less backscattered light in a pixel is corresponding to a smaller y1.
Second correction factor (denoted by y2). When the polarized light is weak, the result of Eq. (10) is near zero. In order to avoid the unreasonable result, we introduce a correction factor y2 (also a matrix), In order to make our method applicable to more scenes and not only depend on the polarization degree P. We set y = (1 − P)/(1 + P), P is in the range [0-1]. Thus, y is in the same range of [0-1]. In the actual algorithm, the image with maximum average gradient was calculated by searching the exact y in the range of [0-1]. The parameter y which maximizes the average gradient of the image is automatically determined by the algorithm.
In this section, the wavelet transform and two correction factors were used to calculate D b .
Calculate the transmittance. Because I best and I worst are two images of the same scene at two angles which are perpendicular to each other. We assume that they have the equivalent transmittance t. According to Eq. (4), where A b ∞ and A w ∞ can be respectively derived from the background region (average value of minimum variance region) of I best and I worst , and A w is unknown. t can be calculated by A b and A b ∞ . However, I best contains most of the object light and a small part of scattered light, I worst contains a small part of the object light and most of the scattered light. A b ∞ is little and as a denominator it causes the instability of the result. As an alternative, t is calculated by Eq. (17).
As mentioned above, D w is much smaller than D b . It can be neglected in Eq. (17). So after calculation of I best and I wors from the captured polarization images, A b ∞ and A w ∞ from the background region, D b from Eq. (14), the transmittance of the scattering media can be computed.
Obtain the object light (undegenerated) and fuse two images. Now we can obtain the object light L b of the image I best , which has not been degraded.
And the object light L w of the image I worst can be derived from Eq. (3): The fused object light L is obtained: We have used transmittance t to obtain object light L. In the last step of our method, the normalized first component of Stokes parameters S 0 is used for color restoration. L was multiplied by the gray value of S 0 to correct the intensity of each color channel.

Experimental results and discussion
We used a polarization camera (Luccid PHX050S, Canada) based on the technology of division of the focal plane, which was capable to capture four polarization images with different angles (0°, 45°, 90°, 135°) at the same time and can be applied to a moving object. The polarization camera is shown in Fig. 3(a). The smallest periodical cell in the camera focal plane is shown in Fig. 3(b). Micro-polarizer array and color filter array are in front of the sensor. The underwater experiments were arranged in a square glass container, and the background behind the glass container was arranged in black. The light source (LED white light source) was used for illumination. The intensity images (S 0 ), the results of dark channel prior dehazing, the results of DehazeNet 6 , the results of Schechner's method, the results of Ren's method 34 and the results of our proposed method are given in Fig. 4. The groups of 1-4 are images of outdoor buildings and hills in the foggy weather, the distance between the scene and the camera is within a few thousand meters. And the groups of 5-9 are indoor underwater experiments in the milk solution with different concentrations. The imaging objects were put in a transparent water tank made of glasses, and A 5 W unpolarized white LED light was taken as the illumination source in front of the tank.  Fig. 4, in group 1-4, the objects are buildings and hills in the foggy weather. (a) is the intensity image, (b)-(f) are the corresponding restored images using dark channel prior, DehazeNet, Schechner's method, Ren's method, and our method, respectively.
As we can see, in group 1, the building in intensity image (a) is very blurry. Buildings in images (b), (c) and (f) are clearer. For more details, we choose the same region (black frame) in images (a-f) and enlarge it in Fig. 5. Obviously, the proposed method obtains the clearest restored image with most details. It causes less noise and restored more details in the image. Same results can be found in group (1-3), even the buildings in group 3 are much farther away from the observer than those in group 1 and 2. Long distance scattering in fog causes  www.nature.com/scientificreports/ the depolarization of the light, which results in the restoration difficulty in a method using polarization prior. For comparison, images of polarization degree (P = (S 1 2 + S 2 2 ) 0.5 /S 0 ) are displayed in Fig. 6. As showed in Fig. 6, the object light is polarimetric and air light is not polarimetric. So Schechner's polarization assumption is not accurate here. Images (d) in group 1-3 in Fig. 4 show Schechner's method does not work very well. Our method does not depend on the specific state of polarization and is more universal. In group 4, hills are shrouded in fog. The color in the image (f) is closest to the real scene. It is dark green and little fog on the hill in front of the scene.
The concentration of the milk powder increases gradually from group 5 to 7. and from 8 to 9. In group 5 (10 mL milk in 6L water), 6(12.5 mL milk in 6L water) and 7(15 mL milk in 6L water), from images (c) and (f), we found the results of DehazeNet and proposed method are much better than intensity image (S 0 ). And severe blue shift happens in image (b), while the true color of the image is kept in our method. By comparison of restored images with different concentration of milk, we found that Schechner's method is better than the image (S 0 ) but not very effective with little polarimetric light. Especially in high-concentration scattering media, the degree of polarization is very small. But the proposed method still works well under such strong scattering condition.
The histograms can provide a more accurate description on the spectrum and details in an image. We calculate the histograms of the images in the group 5-7 in Fig. 4 and plotted in Fig. 7. Wider histogram relates to an image with higher quality and more details. We find that the histograms of both DehazeNet and our proposed method in Group 5-7 are wider than others in Fig. 7.
In order to observe the details, we choose the same region (black frame) and enlarge them in Fig. 8. In such a strong scattering, the intensity image (a) in group 9 is very blurry and we cannot see any details in this image. The restored images with Schechner's method and with Ren's method are better than the intensity image. But it is still blurry. The restored images with dark channel method, with DehazeNet and with the proposed method are much better.
Next, we evaluate the experimental results by four objective evaluation indexes: contrast 1 , gray standard deviation 37 , average gradient 38 and information entropy 7 . Contrast reflects the difference between adjacent pixels in the image. The better image has larger contrast. Gray standard deviation reflects the degree that the gray value deviates from the average value. The better image has larger standard deviation. Average gradient reflects the  Information entropy reflects the amount of information contained in an image. An image with larger information entropy contains more information.
In Table 1, the bold data indicates the maximum value of every group, and the percentage in brackets indicates the improvement of the proposed method compared with the intensity image. As can be seen from Table 1, compared with the intensity image, contrast, gray standard deviation, average gradient and information entropy of the restored images are improved evidently. In general, compared with intensity images, the standard deviation   www.nature.com/scientificreports/ and average gradient of restored images by the proposed method are enhanced by 50-100%. The information entropy is improved by 5-10%. And the contrast is enhanced by 100-300%. So far, our defogging algorithm has a performance as good as DehazeNet, both in foggy weather and underwater experiments. DehazeNet is a latest-developed dehazing method using a convolutional neural network 6 . It has an extremely high restoration accuracy, but requires a network training based on a large number of precollected image libraries. Our method can directly defog in a single image, so it will be more practical in the application of various scattering environment.

Conclusion
In this article, we proposed an image restoration method based on gray variance and average gradient assumption to overcome the difficulty of polarization parameter estimation under strong scattering condition. This technology is performed with a single-shot polarization camera based on the technology of division of the focal plane. Experimental results show that the proposed method obtains clear restored images. Same improvements can be found in the evaluation indexes for our method. Therefore, compared with previously reported methods, our method has some advantages of more restoration details, less noise, and more general applicability. It is independent of a specific polarization state in various scenes and keeps effective even in a scene only containing little polarized light.