Abstract
Recently, timeofflight LiDAR using the singlephoton detection approach has emerged as a potential solution for threedimensional imaging in challenging measurement scenarios, such as over distances of many kilometres. The high sensitivity and picosecond timing resolution afforded by singlephoton detection offers highresolution depth profiling of remote, complex scenes while maintaining low power optical illumination. These properties are ideal for imaging in highly scattering environments such as through atmospheric obscurants, for example fog and smoke. In this paper we present the reconstruction of depth profiles of moving objects through high levels of obscurant equivalent to five attenuation lengths between transceiver and target at standoff distances up to 150 m. We used a robust statistically based processing algorithm designed for the real time reconstruction of singlephoton data obtained in the presence of atmospheric obscurant, including providing uncertainty estimates in the depth reconstruction. This demonstration of realtime 3D reconstruction of moving scenes points a way forward for highresolution imaging from mobile platforms in degraded visual environments.
Similar content being viewed by others
Introduction
The Light Detection and Ranging (LiDAR) technique often uses the timeofflight (ToF) information of a reflected optical signal to determine the distance to an object^{1}. Compared to Radio Detection and Ranging (RADAR) approaches, LiDAR systems are generally capable of higher resolution imaging of objects at long ranges due to the use of much shorter wavelengths^{2}. The modular and compact nature of current LiDAR system designs has enabled them to be deployed in a variety of ranging and imaging applications, such as airborne platforms^{3,4,5} and vehicle navigation systems^{5,6,7}.
More recently, the timecorrelated singlephoton counting (TCSPC) technique has been employed in prototype threedimensional imaging LiDAR systems for sensing in demanding scenarios, due to its high sensitivity^{8,9}. While some conventional ToF LiDAR systems based on linear avalanche photodiode (APD) detector technologies are capable of providing highresolution images at long ranges in clear conditions, they lack however the shotnoise limited sensitivity provided by the timecorrelated single photon detection approach. The high sensitivity exhibited by singlephoton detectors, such as superconducting nanowire detectors (SNSPDs)^{10,11,12,13} or singlephoton avalanche diode (SPAD) detectors, allow the use of low average optical output power levels for scene illumination. Typically, both detector types are capable of picosecond timing, permitting excellent surfacetosurface resolution that is not readily achievable using an analogue optical detector approach^{12,14,15}. SPAD detectors are the most commonly used singlephoton detectors for remote sensing applications as they are capable of operation near room temperature, typically in Peltiercooled packages. Singlepixel SPAD detectors have been used to image a scene by scanning in a pointbypoint manner in the visible, nearinfrared and shortwave infrared regions^{16,17,18,19,20}. SPAD detector arrays have also been used in large pixel formats (e.g., 32 × 32, 128 × 64) that can offer high spatial resolution and rapid data acquisition^{14,21,22,23,24,25,26,27,28}. This recent emergence of high datarate singlephoton LiDAR systems, which employ SPAD detectors, has stimulated research into potential new applications that seem wellsuited to this detector technology such as autonomous navigation, environmental monitoring, and subsea mapping. SPADbased LiDAR systems have been successfully demonstrated in several challenging scenarios including longrange depth imaging^{29,30,31,32,33,34}, imaging through clutter^{35,36}, nonlineofsight detection of targets hidden from view^{37,38,39}, and imaging of targets in high levels of scattering media^{27,40,41,42,43,44,45}.
Although CMOSbased SPAD arrays offer advantages in terms of large format detector arrays, they are typically restricted to wavelengths of less than 1000 nm. The use of shortwave infrared (SWIR) wavelengths in LiDAR systems can have several advantages over shorter wavelengths, such as reduced inband solar background^{46}, higher atmospheric transmission in clear conditions^{47,48}, and improved transmission in some obscurants when compared to nearinfrared wavelengths^{36}. Most importantly, being outside the retinal hazard region (400–1400 nm)^{49}, the selection of 1550 nm as the operating wavelength enables the use of a higher average optical power illumination beam whilst remaining eyesafe when compared to visible and nearinfrared bands. The increased optical power levels afforded by use of the SWIR region can result in a greater maximum attainable LiDAR range and/or improvement in achievable depth resolution for shorter acquisition times.
The use of more conventional imaging approaches for “seeing” through degraded visual environments, such as those caused by natural and manmade obscurants (e.g., dust, fog, smoke, and haze), pose an impediment to situational awareness in scenarios such as airborne navigation, surveillance and reconnaissance^{50,51,52}. Previously, sensing technologies that have been used in the presence of obscurants include RADAR based sensors^{53,54}, and passive sensors based on thermal imaging^{55,56}. There has been some previous work on using LiDAR for imaging through obscurants, but this was mostly laboratorybased work or rangegated LiDAR techniques such as Burst Illumination LiDAR (BIL) systems^{43,44}. The singlephoton sensitivity and excellent depth resolution of TCSPC LiDAR systems have offered potential for highresolution 3D imaging through atmospheric obscurants^{41,42}. This paper reports the imaging through approximately 10 m of obscurant, of moving targets over distances of 50 and 150 m using fullfield singlephoton detection.
In recent years, there has been great interest in the implementation of image processing algorithms designed to reconstruct images from sparse photon data, and several algorithms have demonstrated good performance for data obtained in freespace scenarios where the return signal is very low and the background is relatively high^{29,57,58,59,60,61,62,63,64}. Currently, a major bottleneck in the use of singlephoton LiDAR systems is that these algorithms typically suffer from the disadvantage of long execution times (generally 10 s to 100 s of seconds), limiting their use in applications that rely on near instantaneous target analysis. Realtime reconstruction of 3D scenes from singlephoton data was achieved recently using highly scalable computational tools being run on a graphics processing unit (GPU)^{65}. However, this algorithm is not optimized for imaging through turbid media which contain high and nonuniform background levels, but instead was specifically designed for complex scenes typically containing more than one surface in each pixel. This paper presents a new approach for the realtime processing of singlephoton data acquired from imaging scenes through obscurants, which can deal with particularly high and nonuniform background levels. The proposed reconstruction algorithm combines ingredients from recent stateoftheart algorithms including the use of an advanced statistical modelling^{48,53,54}, the exploitation of spatiotemporal information, and combining nonlinear parameter estimation and filtering steps^{66,67} to deliver a robust estimation strategy. The resulting algorithm allows scene reconstruction from extremely noisy data with a nonuniform temporal (or depth) profile, as expected from propagation through high levels of obscurants, whilst quantifying the uncertainty of the depth and intensity reconstruction, which is essential for practical 3D imaging applications. This algorithm includes a new statistical formulation that exploits the multiscale and multitemporal information of singlephoton LiDAR data to improve robustness to noise and quantify the uncertainty of the estimates. In addition, this algorithm allows the use of latent variables that can be updated in parallel, improving computational costs. Finally, this model produces simple iterations that can be efficiently implemented.
In this paper we present an active imaging system based on the singlephoton ToF approach to obtain depth and intensity profiles of moving targets through high levels of obscurant. The bistatic system comprised a pulsed laser source with an operational wavelength of 1550 nm and a maximum average optical output power level of 220 mW, and an InGaAs/InP SPAD detector array which is highly efficient in the SWIR region.
The combination of this active imaging system with the proposed advanced algorithm allowed, for the first time, the reconstruction of depth and intensity profiles of static and moving targets, placed in high levels of atmospheric obscurant at standoff distances of up to 150 m in both indoor and outdoor environments.
Results
Experiment layout
The LiDAR transceiver was arranged in a bistatic optical configuration, and used a 32 × 32 InGaAs/InP SPAD detector array. A pulsed fibre laser source operating at a wavelength of 1550 nm was used to floodilluminate the scene using an average optical power of 220 mW at a pulse repetition rate of 150 kHz. The overall instrumental jitter was 485 ps full width at half maximum. The choice of 1550 nm wavelength operation meant that the transceiver system used in these measurements was characterised as being eyesafe at all distances between the system and target, i.e., the system had a nominal ocular hazard distance of zero metres. Further details of the transceiver are described in “Methods” below and in Supplementary Material 2.
Measurements were performed both indoors and outdoors in daylight for standoff ranges of 50 m and 150 m respectively with the setup as shown schematically in Fig. 1. For each measurement, a smoke gun was used to produce an oilbased vapour with droplet sizes on the order of a few microns. In order to contain the obscurant, it was released in a polyethylene tent or marquee with dimensions of 3 m (H) × 4 m (W) × 10 m (L), with 2 m × 2 m openings at either end. This was positioned in the lineofsight between the system and target area and was located at distances of approximately 35 m and 125 m from the LiDAR system for the for 50 m and 150 m ranges, respectively. For each measurement, the closed tent was filled with the oil vapour until a sufficient density was achieved. The tent doors were then opened and measurements of the target scene were made while the oil vapour slowly dispersed (see Fig. 2a). For the indoor range, two large fans were used to help control the dispersal of the obscurant for improved homogeneity. It is worth noting that during the indoor measurements at a range of 50 m the obscurant dispersed throughout the building, enveloping the target area. A depth calibration was performed using a target consisting of four flat 500 × 500 mm wooden panels separated at 100 mm increments in the direction of the beam path. These four flat surfaces were placed immediately adjacent to each other, as shown in Fig. 1 and the transceiver was aligned to incorporate all four target surfaces within the field of view. The target surfaces were painted matt white, which resulted in uniform scattering of the incident illumination, and were used for depth and reflectance calibration measurements. This 3D wooden panel target was included in the scene and used to evaluate the system performance when imaging through different densities of the oil vapour.
Estimation of attenuation
The number of attenuation lengths between the system and the target at the operating wavelength of 1550 nm was calculated from the number of photon counts collected in clear conditions (n_{0}) and the number of photons collected through obscurant (n), from the same area on the 3D wooden panel target, under otherwise identical operating conditions. The attenuation coefficient (α) for the level of obscurant present in the chamber, and hence the number of attenuation lengths (N_{AL}), for the oneway distance (d) between the transceiver and target, was then calculated from the BeerLambert law^{68} as follows:
where, due to the low return signal through high levels of the obscurant, n and n_{0} were calculated from the photon counts summed over a 9 × 9 pixel neighbourhood for a 1 s data acquisition in order to acquire an accurate measurement of the number of attenuation lengths.
A transmissometer was also used to perform a second independent measurement of the number of attenuation lengths, in the visible region of the spectrum, corresponding to the level of obscurant. The transmissometer operated at a wavelength of λ = 637 nm and was set up to one side of the LiDAR system location, approximately 10 m closer to the target. The transmissometer laser beam was directed through the marquee to a corner cube retroreflector (located near the target position) and reflected back to a fibrecoupled Si photodiode. The transmissometer measurements were synchronised with those of the LiDAR system but it is important to note that the two systems did not follow the exact same optical path, although care was taken that the transmissometer data were as representative as possible of the optical attenuation experienced by the bistatic LiDAR system. A comparison of the number of attenuation lengths over the duration of a measurement set for wavelengths of 1550 nm and 637 nm is shown in Fig. 2b. This figure clearly demonstrates the advantage of λ = 1550 nm operation for this type of obscurant, with significantly less attenuation compared with the visible wavelength shown.
Observation model
3D imaging through obscurants raises several challenges due to backscattered photons leading to high and nonuniform background level in the observed histograms. This limits the use of the classical matched filter (or crosscorrelation) strategy for depth estimation and increases the requirement for statistical confidence guarantees regarding the reconstructed scene. In addition, for 3D video representation, the system acquires successive data cubes, highlighting the need for an online robust processing approach to account for temporal data correlations while dealing with the high volume of acquired data. In the following, we describe the observation model and proposed reconstruction algorithm, which offers a solution to previous challenges, i.e., it allows robust and online processing of 3D LiDAR imaging data through obscurants while providing uncertainty estimates of the depth value.
The TCSPC system provides \(K\) successive data cubes composed of two spatial dimensions (i.e., pixel locations) and one timeofflight dimension (related to depth). The kth cube/frame is denoted by \({y}_{t,n,k}\) and contains histograms of photon counts at pixel location, \(n \in \left\{1, \dots , N\right\}\), timeofflight bin \(t \in \left\{1, \dots , T\right\}\) and cube number \(k \in \left\{1, \dots , K\right\}\). Figure 1 shows an example of a raw data cube.
Assuming at most one surface perpixel, each photon count can be assumed to be drawn from the Poisson distribution \(P\left(.\right)\) as follows^{57,61}:
where \(f\) represents the system impulse response assumed to be known from a calibration step, \({d}_{n,k}>0, {r}_{n,k}>0\) denote the distance from the sensor and reflectivity of the object for the \(k\) th data, \({b}_{t,n,k}>0\) gathers the background and dark counts of the detector. To account for obscurants, several studies showed that the background level might vary with respect to the depth observation window such as described by Satat et al.^{41}, which approximated the noise using a Gamma shaped distribution. In this paper, we assume the signal is located in the decreasing tail of the background distribution leading to the approximation \({b}_{t,n,k} = max({a}_{n,k}{exp}^{{c}_{n,k}t} ,{\tilde{e }}_{n,k})\), where \({a}_{n,k}\) and \({c}_{n,k}\) respectively represent the amplitude and decreasing rate of the exponential, and \({\tilde{e }}_{n,k}\) is a constant background level perpixel. The observation model is finally given by (see Fig. 3 for an example of a real timing histogram obtained by the system using a gate duration of 20 ns (i.e., 80 timing bins)).
Our goal is to adaptively and robustly estimate the vector \(\Theta =({\Theta }_{s }, {\Theta }_{n})\) that includes the target \(\Theta =(D,R)\) and noise \({\Theta }_{n}=(A,C,\tilde{E })\) parameters using the successively observed histograms \({y}_{t,n,k}, \forall t, n, k\), and exploiting their statistics in Eq. (3).
Reconstruction algorithm
The proposed solution deals with previous challenges by adopting a threestep strategy. The first preprocessing step is used for the estimation of noise parameters using an efficient median estimator for the constant level, \({\tilde{e }}_{n,k}\), and approximate analytical estimators for the exponential parameters. This step allows the approximate unmixing of signal and background counts and can be easily adapted to different background distributions to deal with different scenarios.
The second step represents the core of the algorithm and aims at the robust estimation of depth and reflectivity images by adopting a hierarchical Bayesian model. This approach introduces prior distributions for the unknown parameters to account for their known properties, i.e., positivity, and multiscale/multitemporal correlations. The combination of the measurement statistics summarised in the likelihood and the parameters prior distributions leads to a posterior distribution on the parameters. The latter summarises the probability information regarding each parameter allowing the extraction of its estimate and the quantification of its uncertainty. A main contribution relies on the appropriate choice of prior that accounts for the parameter properties while leading to fast parameter estimates. Multiscale information will be considered as it accounts for spatial correlations between pixels and has shown its importance in many restoration algorithms^{57,62} especially in extreme conditions due to a sparsephoton regime or high noise levels. For added robustness, the proposed algorithm also accounts for multitemporal information by considering previously estimated depth and reflectivity frames to restore the current frame. These priors are accounted for through the introduction of a depth latent variable denoted x that decouples the multiscale spatial and multitemporal information of the depth allowing parallel and fast parameter estimation. To preserve edges between distinct surfaces, we adopt a Laplace prior for x as it promotes depth sparsity through the implicit \(\ell_{1}\)norm^{69}, that has demonstrated good results for several depth reconstruction applications^{70}.
The reflectivity is assumed spatially smooth and this is introduced by exploiting multiscale information. Depth uncertainty is represented using a variance parameter for x, which is assigned a conjugate noninformative scale prior. The resulting posterior distribution is exploited by considering maximum aposteriori estimators for the parameters. The latter are obtained using a coordinate descent algorithm that iteratively maximize the parameters conditional distributions. The resulting algorithm alternates between robust nonlinear parameter estimation (efficient weighted median^{71}) and a filtering step (generalised softthresholding), which are commonly observed steps in several stateoftheart algorithms^{65,66,67} and optimisation algorithms^{72}.
The third optional step relates to data superresolution to improve the spatial quality of the images. Inspired from the depth maximum aposteriori estimate, superresolution is performed using a combination of a weighted median operator with a point cloud filtering step.
The main steps of the proposed algorithm, named Medianbased Multiscale Restoration of 3D images (M2R3D), are summarised in Fig. 4 and are described with additional detail in the Supplementary Information. The acquired histograms have nonuniform timing bins due to timing issues with the detector array readout circuitry (see section ID of the Supplementary Information for further information). This effect is corrected before applying the proposed strategy and the correcting procedure is denoted as ‘histogram corrections’ in this paper.
Depth and intensity reconstruction at 50 m range
A series of measurements were performed indoors using a scene containing the 3D wooden panel target positioned at a range of 50 m (100 m roundtrip) from the system. The results were reconstructed using both a simple pixelwise crosscorrelation algorithm (i.e., matched filter of the raw histogram with the impulse response of the system) and the proposed M2R3D algorithm. The 3D panel chart was floodilluminated with 220 mW average optical output power from the laser source. The field of view (FoV) of the SPAD detector array camera setup was approximately 0.53 × 0.53 m meaning that only the central region of the 1 × 1 m target was imaged at this range. Attenuation results for this measurement obtained by the transmissometer (for λ = 637 nm) and the depth imaging system (for λ = 1550 nm) are shown in Fig. 2b. The attenuation results are shown from 200 s until the end of the measurement (590 s) since the transmissometer was unable to obtain reliable attenuation values at the very high densities of obscurant observed at the beginning of a measurement cycle. These results indicate that λ = 1550 nm light has considerably lower attenuation than that at λ = 637 nm demonstrating a clear benefit in the use of SWIR illumination over shorter illumination wavelengths for this type of obscurant. Figure 5 shows reconstructed depth profiles of the target at the standoff distance of 50 m with varying obscurant density corresponding to 4.0, 4.5, 5.0 and 5.5 attenuation lengths between the LiDAR transceiver and target, measured at the illumination wavelength of 1550 nm. The data was acquired over a time of 1 s in each case, at an average optical output power level of 220 mW. As mentioned previously regarding measurements on the 50 m range, the obscurant dispersed throughout the target area and fully submerged the target. In order to investigate the consequence of obscurant density on the reconstruction quality we investigate several scenarios as indicated in the rows of Fig. 5. In rows from top to bottom: (a) the crosscorrelation was first applied to noncorrected histograms; (b) the crosscorrelation was applied to corrected histograms; (c) the crosscorrelation was applied to corrected histograms while accounting for an exponential background; and finally (d) the M2R3D algorithm was used to reconstruct depth profiles and (e) the depth uncertainty, as quantified by the standarddeviation of the depth conditional distribution. Note that the M2R3D algorithm provides superresolved depth profiles composed of 128 × 128 pixels. A depth threshold was set such that all depths outside a predetermined distance around the target (in this case 0.6 m) were considered to be inaccurate estimates, disregarded, and presented in the reconstructed depth profiles as empty pixels for clarity (shown as white pixels in Fig. 5). In addition, the signalbackgroundratio (SBR) estimated from M2R3D is given for each corresponding attenuation length. SBR is defined as the ratio of the average signal photonsperpixel (S_{ppp}) and the average background photonsperpixel (B_{ppp}). More details on these parameters are given in the supplementary material.
The results in Fig. 5 show that the 3D panel target can only be partially reconstructed using the crosscorrelation algorithm at up to 4.5 attenuation lengths with no histogram corrections nor background consideration (Fig. 5a), and similarly with only histogram correction (Fig. 5b). When both histogram correction and an exponential background are considered, using the crosscorrelation provides a better reconstruction at 4.5 attenuation lengths, and a partial reconstruction can be achieved for obscurant densities up to 5.0 attenuation lengths (Fig. 5c). However, using the M2R3D algorithm (Fig. 5d), the 3D panel target can be partially reconstructed at 5.0 attenuation lengths and fully reconstructed at 4.5 attenuation lengths. In fact, a partial reconstruction was also made using the M2R3D algorithm at up to 5.5 attenuation lengths. Significantly, M2R3D also quantifies depth uncertainty as represented in (Fig. 5e) by showing the standard deviation of the estimated depth statistical distribution. It can be seen that higher uncertainty is observed around object edges and in the presence of high obscurant densities. In particular, standard deviations (of a Laplace distribution) greater than 4 cm are observed for badly estimated pixels at greater than 5.0 attenuation lengths.
True positive percentage values, which represent the percentage of pixels satisfying a given depth within a given error^{61}, were calculated for the results obtained with each algorithm are shown in red in Fig. 5. As an example, for the 5.0 attenuation lengths results the true positive percentage values satisfying a depth absolute error (DAE) of 5.6 cm are 1%, 0%, 35%, and 75% for each algorithm respectively, illustrating a significant quantitative improvement in the reconstructed depth profiles by using both the exponential background and the proposed M2R3D algorithm. To compute the DAE, a 32 × 32 pixel reference depth map was generated using data obtained in the absence of fog. These values were calculated using the lower resolution depth maps due to the unavailability of a highresolution ground truth. A more detailed description of the true positive percentage, DAE, and further analysis are supplied in the Supplementary Information. We also provide video 1 which displays the results of M2R3D and crosscorrelation algorithm (with all corrections) when processing this data at different time instances.
Depth and intensity reconstruction at outdoor 150 m range
A second set of measurements was performed outdoors in daylight conditions at a range of 150 m (300 m roundtrip) from the transceiver location. The 3D wooden panel target was housed within an intermodal container to help shield it from adverse weather conditions. An actor was also situated directly behind the 3D wooden panel target. Since the horizontal and vertical FoV was three times larger at 150 m than at 50 m, the resultant data contained information from the entire 3D wooden panel target, the actor, as well as superfluous information from the back wall of the container. As with the 50 m range measurements shown above, the depth profiles of the scene were reconstructed using crosscorrelation with no corrections applied, with only the histogram corrections applied, with both histogram corrections and an exponential background, and with the proposed M2R3D algorithm, as shown in Fig. 6.
The results shown in Fig. 6a–d indicate that a partial depth reconstruction can be made at up to 5.0 attenuation lengths using both crosscorrelation and the proposed M2R3D algorithm at a standoff distance of 150 m. It should however be noted that a high depth uncertainty is observed for most regions at attenuation lengths of greater or equal to 5.0 as highlighted in Fig. 6e. A full reconstruction of the target scene could be made at 4.5 attenuation lengths for both the crosscorrelation with both histogram and exponential background correction (Fig. 6c) and the M2R3D algorithm (Fig. 6d). In this case, the depth profile reconstructed using corrected crosscorrelation is noisier than that of the M2R3D algorithm. However, the reconstruction using the M2R3D algorithm does tend to oversmooth the fine target details, such as the arms and face of the actor, while assigning higher depth uncertainty levels for these regions. These results demonstrate the potential of the system for rapid threedimensional imaging outdoors in high levels of obscurants, in high levels of solar background, and adverse weather conditions using SWIR wavelengths. Since the human target was moving in this measurement set, no ground truth could be obtained and therefore no true positive percentage values could be calculated for this data.
Realtime processing of moving 3D scenes in obscurant
This section describes a further set of measurements using a moving target at a distance of 50 m through the oilbased vapour obscurant. The target scene included the 3D wooden panel target and an actor walking from side to side through the scene holding alternating objects (a box and a plank of wood). The measurements were acquired using the same system configuration and parameters as previously described. For these measurements 15,000 successive binary frames, each with an acquisition time of approximately 6.7 µs, were aggregated to create a 10 frames per second video with an overall duration of several minutes. Figure 7 shows three stationary depth profiles obtained with crosscorrelation (with histogram corrections and assuming no background), and the proposed M2R3D algorithm. These stationary frames from the video correspond to the point in time when there were approximately 3 attenuation lengths at λ = 1550 nm between the system and the target.
These results highlight the benefit of the M2R3D algorithm in removing false detections and improving spatial resolution. As expected, higher depth uncertainty is estimated for pixels located at object edges, around small object features (e.g., the man’s hat) or in regions where the pixels of the SPAD array camera demonstrate lower singlephoton detection efficiency (e.g., the region at the right in the depth map). The processing time of the M2R3D algorithm is around ~ 90 ms per frame enabling real time processing (latency lower than one frame). Video 2 shows the results of the M2R3D algorithm and the crosscorrelation algorithm (with all corrections) when processing this data at different time instances. Other data and comparison algorithms have also been considered in the supplementary materials, which shows the advantages of the proposed algorithm.
Discussion
This paper presents depth and intensity profiling of static and moving targets in high levels of obscurant at standoff ranges of 50 m and 150 m outdoors using the timecorrelated singlephoton counting technique. The eyesafe timeofflight imaging system comprised a 32 × 32 InGaAs/InP SPAD detector array and a picosecond pulsed fibre laser with an illumination wavelength of 1550 nm and a maximum average optical output power of 220 mW. This was used to perform threedimensional imaging at up to 5 attenuation lengths for a range of 50 m and 4.5 attenuation lengths at 150 m. A comparison of light propagation through the oilbased vapour (i.e., artificial fog) was also made between shortwave infrared wavelengths and visible wavelengths, illustrating a significant wavelength dependence in the attenuation due to increased scatter at the shorter wavelengths with this particular obscurant, as seen in our previous work^{42}. Of course, wide scale implementation of this approach means that attention must also be paid to the development of costeffective LIDAR sources, since eyesafe implementation in the SWIR region permits much higher optical powers than possible in the nearinfrared.
This paper presented an advanced image processing algorithm, which is capable of realtime reconstruction of depth and intensity profiles of moving targets in high levels of obscurants. In contrast to other algorithms^{57,60,65}, the M2R3D uses an observation model that accounts for the presence of obscurants which typically can exhibit a nonuniform depth profile, allowing a robust reconstruction of the scene while providing uncertainty estimates of the depth measurements using multiscale and multitemporal information. In the Supplementary Materials, it is shown that the proposed M2R3D algorithm performs better than several stateoftheart singlephoton depth and intensity reconstruction algorithms^{57,60,65} especially in the presence of high or nonuniform background levels, which are often observed in presence of obscurants^{41}. The realtime processing allowed the demonstration of depth imaging of moving targets, a critical attribute necessary for future implementations on mobile platforms. A key aspect of this algorithm is that the processing time is not proportional to the number of photon events in a histogram, which is particularly important for the specific case of imaging in obscurants which necessarily incurs an unusually high level of backscattered photon events from the illumination. More details and comparison with other algorithms are given in Supplementary Information.
The results presented in this paper demonstrate the potential for the implementation of singlephoton counting approaches using InGaAs/InP SPAD detector arrays in modern embedded systems (e.g., driverless cars). While the high timing resolution, longrange capabilities, and low optical power levels inherent to the TCSPC approach fulfil many of the requirements of current systems, further investigation is necessary to evaluate the system and proposed algorithm in more challenging scenarios. Therefore, future work will include measurements made in natural fog environments and high levels of precipitation, and the investigation of targets travelling at higher velocities at longer ranges. Generalising the proposed model to account for alternative background profiles, and to account for pixels without target information are also important points that will be investigated. Also, the use of parallel processing tools and graphical processing units (GPU) will be investigated to decrease frame processing times to a duration of a few milliseconds in order to utilise next generation, larger format detector arrays. We have demonstrated depth image reconstruction using a LiDAR system with a limited optical field of view, and must acknowledge that many applications will require an extended FoV requiring optical scanning in at least one dimension, which will, in turn, reduce pixel dwell time. Future work will concentrate on the implementation of a scanning strategy and examining the tradeoffs between FoV, spatial resolution, and dwell time.
Methods
System description
The LiDAR system was based on an InGaAs/InP 32 × 32 SPAD detector array camera manufactured by Princeton Lightwave. The pixel elements of the detector array were on a 100 μm square pitch, resulting in the active area of the sensor having dimensions of ~ 3.2 mm × 3.2 mm. This sensor can be used for singlephoton and lowlight detection in the SWIR wavelength range of 1400–1620 nm. The sensor was operated at a frame rate of 150 kHz, and this provided the synchronised trigger for the pulsed laser source. The SPAD detector array was configured to use a 250 ps timing bin width (the smallest available), which corresponds to a 3.75 cm depth resolution. For the measurements reported here, a 20 ns gate duration was selected, which corresponds to a total of 80 histogram bins and is equivalent to a measurement depth range of 3 m. A 300 mm effective focal length objective lens operating at f/3.5 was attached to the sensor unit to collect photons scattered back from the target scene and resulted in a FoV of approximately 11 mrad horizontally and vertically. This means that the sensor imaged an area of approximately 0.53 × 0.53 m at a standoff distance of 50 m, and 1.6 × 1.6 m at a standoff distance of 150 m, which corresponded to each individual pixel covering an area of approximately 16 × 16 mm and 50 × 50 mm, respectively.
The LiDAR system was implemented using a bistatic configuration and a schematic diagram of the setup is included in Supplementary Information Fig. 10. The illumination channel (to which the laser source was fibrecoupled) and SPAD detector array camera channel were mounted side by side. The illumination source was a pulsed fibre laser with a central operating wavelength of 1550 nm, and run at a repetition rate of 150 kHz (clocked by the camera control electronics), which resulted in a pulsewidth of 413 ps, and a maximum average optical output power of approximately 220 mW. Further details on the system setup can be found in Supplementary Material 2.
Algorithm implementation
The algorithm performs weighted median filtering using the C ++ efficient implementation proposed in^{71}. All other steps were implemented using MATLAB R2019a on a computer with an Intel(R) Core(TM) i74790 CPU running at 3.60 GHz and with 32 GB RAM.
Data availability
The data will be made available upon publication of this work. Algorithm available on request.
References
McManamon, P. LiDAR Technologies and System (SPIE Press, 2019).
Molebny, V., McManamon, P. F., Steinvall, O., Kobayashi, T. & Chen, W. Laser radar: Historical prospective—from the East to the West. Opt. Express 56, 031220 (2016).
Cunningham, D., Grebby, S., Tansey, K., Gosar, A. & Kastelic, V. Application of airborne LiDAR to mapping seismogenic faults in forested mountainous terrain, southeastern Alps, Slovenia. Geophys. Res. Lett. 33, L20308 (2006).
Clifton, W. E., Steele, B., Nelson, G., Truscott, A., Itzler, M. A., Entwistle, M. Medium altitude airborne Geigermode mapping LIDAR system, in Proc. SPIE 9465, Laser Radar Technology and Applications XX; and Atmospheric Propagation XII, 946506 (2015).
Itzler, M.A., Entwistle, M., Wilton, S., Kudryashov, I., Kotelnikov, J., Jiang, X., Piccione, B., Owens, M., and Rangwala, S., GeigerMode LiDAR: from airborne platforms to driverless cars, in Imaging and Applied Optics, ATu3A.3 (2017).
Hecht, J. Lidar for selfdriving cars. Opt. Photonics News 29(1), 26–33 (2018).
Wallace, A. M., Halimi, A. & Buller, G. S. Full waveform LiDAR for adverse weather conditions. IEEE Transactions on Vehicular Technology 2, 1–1, (2020).
Buller, G. & Wallace, A. Ranging and threedimensional imaging using timecorrelated singlephoton counting and pointbypoint acquisition. IEEE J. Sel. Top. Quantum Electron. 13(4), 1006–1015 (2007).
Massa, J. S. et al. Timeofflight optical ranging system based on timecorrelated singlephoton counting. Appl. Opt. 37, 7298–7304 (1998).
Korzh, B. A. et al. Demonstrating sub3 ps temporal resolution in a superconducting nanowire singlephoton detector. in arXiv: 1804.06839 (2018).
Warburton, R. E. et al. Subcentimeter depth resolution using a singlephoton counting timeofflight laser ranging system at 1550 nm wavelength. Opt. Lett. 32, 2266–2268 (2007).
McCarthy, A. et al. Kilometerrange, high resolution depth imaging via 1560 nm wavelength singlephoton detection. Opt. Express 21, 8904–8915 (2013).
Buller, G. S. & Collins, R. J. Singlephoton generation and detection. Meas. Sci. Technol. 21, 012002 (2010).
Ren, X. et al. Highresolution depth profiling using a rangegated CMOS SPAD quanta image sensor. Opt. Express 26, 5541–5557 (2018).
Taylor, G. G. et al. 2.3μm wavelength single photon LIDAR with superconducting nanowire detectors. in Conference on Lasers and ElectroOptics paper JTh2A.93 (2019).
Lacaita, A., Francese, P. A., Zappa, F. & Cova, S. Singlephoton detection beyond 1 μm: Performance of commercially available germanium photodiodes. Appl. Opt. 33, 6902–6918 (1994).
Pellegrini, S. et al. Design and performance of an InGaAsInP singlephoton avalanche diode detector. IEEE J. Quantum Electron. 42, 397–403 (2006).
Sun, M.J. & Zhang, J.M. Singlepixel imaging and its application in threedimensional reconstruction: A brief review. Sensors 19, 732 (2019).
Sun, M.J. et al. Singlepixel threedimensional imaging with timebased depth resolution. Nat. Commun. 7, 12010 (2016).
Kuzmenko, K. et al. 3D LIDAR imaging using GeonSi singlephoton avalanche diode detectors. Opt. Express 28, 1330–1344 (2020).
Albota, M. A. et al. Threedimensional imaging laser radar with a photoncounting avalanche photodiode array and microchip laser. Appl. Opt. 41, 7671–7678 (2002).
Niclass, C., Rochas, A., Besse, P. & Charbon, E. Design and characterization of a CMOS 3D image sensor based on single photon avalanche diodes. IEEE Journal of SolidState Circuits 40, 1847–1854 (2005).
Zappa, F., Tisa, S., Tosi, A. & Cova, S. Principles and features of singlephoton avalanche diode arrays. Sens. Actuators, A 140, 103–112 (2007).
Shin, D. et al. Photonefficient imaging with a singlephoton camera. Nat. Commun. 7, 12046 (2016).
Chan, S. et al. Longrange depth imaging using a singlephoton detector array and nonlocal data fusion. Sci. Rep. 9, 8075 (2019).
Itzler, M. A. et al. Advances in InGaAsPbased avalanche diode single photon detectors. J. Mod. Opt. 58, 174–200 (2011).
Maccarone, A., Rocca, F. M. D., McCarthy, A., Henderson, R. & Buller, G. S. Threedimensional imaging of stationary and moving targets in turbid underwater environments using a singlephoton detector array. Opt. Express 27, 28437–28456 (2019).
Laurenzis, M. Single photon range, intensity and photon flux imaging with kilohertz frame rate and high dynamic range. Opt. Express 27, 38391–38403 (2019).
Pawlikowska, A. M., Halimi, A., Lamb, R. A. & Buller, G. S. Singlephoton threedimensional imaging at up to 10 kilometers range. Opt. Express 25, 11919–11931 (2017).
Li, Z.P. et al. Alltime singlephoton 3D imaging over 21 km. in Conference on Lasers and ElectroOptics SM1N.1 (2019).
Li, Z.P. et al. Superresolution singlephoton imaging at 8.2 kilometers. Optics Express 28, 4076–4087 (2020).
Li, Z.P. et al. Singlephoton computational 3D imaging at 45 km. Photonics Res. 8, 1532–1540 (2020).
McCarthy, A. et al. Longrange timeofflight scanning sensor based on highspeed timecorrelated singlephoton counting. Appl. Opt. 48, 6241–6251 (2009).
Henriksson, M., Allard, L. & Jonsson, P. Panoramic singlephoton counting 3D lidar. in ElectroOptical Remote Sensing XII, vol. 10796 (2018).
Henriksson, M., Larsson, H., Grönwall, C. & Tolt, G. Continuously scanning timecorrelated singlephotoncounting singlepixel 3D lidar. Opt. Eng. 56, 031204 (2016).
Tobin, R. et al. Longrange depth profiling of camouflaged targets using singlephoton detection. Opt. Express 57, 031303 (2017).
Velten, A. et al. Recovering threedimensional shape around a corner using ultrafast timeofflight imaging. Nat. Commun. 3, 745 (2012).
Chan, S. et al. Nonlineofsight tracking of people at long range. Opt. Express 25, 10109–10117 (2017).
Laurenzis, M., Velten, A. & Klein, J. Dualmode optical sensing: threedimensional imaging and seeing around a corner. Opt. Eng. 56(3), 031202 (2016).
Maccarone, A. et al. Underwater depth imaging using timecorrelated singlephoton counting. Opt. Express 23, 33911–33926 (2015).
Satat, G., Tancik, M. & Raskar, R. Towards photography through realistic fog. IEEE International Conference on Computational Photography 1–10 (2018).
Tobin, R. et al. Threedimensional singlephoton imaging through obscurants. Opt. Express 27, 4590 (2019).
Christnacher, F. et al. Influence of gating and of the gate shape on the penetration capacity of rangegated active imaging in scattering environments. Opt. Express 23, 32897–32908 (2015).
Laurenzis, M., Christnacher, F., Monnin, D. & Scholz, T. Investigation of rangegated imaging in scattering environments. Opt. Eng. 51(6), 061303 (2012).
Lindell, D. B. & Wetzstein, G. Threedimensional imaging through scattering media based on confocal diffuse tomography. Nat. Commun. 11, 4517 (2020).
Bird, R. E., Hulstrom, R. L. & Lewis, L. J. Terrestrial solar spectral data sets. Sol. Energy 30, 563–573 (1983).
Kim, I. I., Mitchell, M. & Korevaar, E. J. Measurement of scintillation for freespace laser communication at 785 nm and 1550 nm. Optical Wireless Communications II(3850), 49–63 (1999).
Nadeem, F., Javornik, T., Leitgeb, E., Kvicera, V. & Kandus, G. Continental fog attenuation empirical relationship from measured visibility data. Radioengineering 19, 596–600 (2010).
Youssef, P. N., Sheibani, N. & Albert, D. M. Retinal light toxicity. Eye (Lond) 25, 1–14 (2011).
Kruse, P. W. Why the military interest in nearmillimeter wave imaging?. Millimeter Optics 0259, 94–99 (1981).
Soan, P. et al. Comparative assessment of different active imaging technologies for imaging through obscurants. in ElectroOptical Remote Sensing XII 10796 (2018).
Sevcik, K. W., Kuntz, N. & Oh, P. Y. Exploring the effect of obscurants on safe landing zone identification. J. Intell. Rob. Syst. 57, 281–295 (2010).
Daniel, L. et al. LowTHz radar, lidar and optical imaging through artificially generated fog. in International Conference on Radar Systems 1–4 (2017).
Fritz, J., Gasiewski, A. J. & Zhang, K. 3D surface imaging through visual obscurants using a subTHz radar. in Degraded Visual Environments: Enhanced, Synthetic, and External Vision Solutions 2014 vol. 9087 908702 (2014).
Martin, C., Lovgerg, J., Clark, S. & Galliano, J. Real time passive millimeterwave imaging from a helicopter platform. in 19th DASC. 19th Digital Avionics Systems Conference. Proceedings (Cat. No.00CH37126) vol. 1 2B1/1–2B1/8 vol.1 (2000).
Pulpea, G. B. et al. Thermal systems capabilities using infrared obscurants. J. Optoelectron. Adv. Mater. 19, 16–26 (2017).
Rapp, J. & Goyal, V. K. A few photons among many: unmixing signal and noise for photonefficient active imaging. IEEE Transactions on Computational Imaging 3, 445–459 (2017).
Kirmani, A. et al. Firstphoton imaging. Science 343, 58–61 (2014).
Lindell, D. B., O’Toole, M. & Wetzstein, G. Singlephoton 3D imaging with deep sensor fusion. GraphicsACM Transactions on Graphics 37, 1–12 (2018).
Tachella, J. et al. Bayesian 3D reconstruction of complex scenes from singlephoton lidar data. Technol. Journal of Imaging Science and Technology 12, 521–550 (2019).
Altmann, Y., Ren, X., McCarthy, A., Buller, G. S. & McLaughlin, S. Lidar waveformbased analysis of depth images constructed using sparse singlephoton data. IEEE Trans. Image Process. 25, 1935–1946 (2016).
Halimi, A. et al. Robust restoration of sparse multidimensional singlephoton LiDAR images. IEEE Transactions on Computational Imaging 1–1 (2019).
Rapp, J., Tachella, J., Altmann, Y., McLaughlin, S. & Goyal, V. K. Advances in SinglePhoton Lidar for Autonomous Vehicles: Working Principles, Challenges, and Recent Advances. IEEE Signal Process. Mag. 37, 62–71 (2020).
Peng, J. et al. PhotonEfficient 3D Imaging with A Nonlocal Neural Network. in Computer Vision – ECCV 2020, vol. 12351 225–241 (2020).
Tachella, J. et al. Realtime 3D reconstruction from singlephoton lidar data using plugandplay point cloud denoisers. Nat. Commun. 10, 1–6 (2019).
Romano, Y., Elad, M. & Milanfar, P. The Little Engine That Could: Regularization by Denoising (RED). SIAM J. Imaging Sci. 10, 1804–1844 (2017).
Venkatakrishnan, S. V., Bouman, C. A. & Wohlberg, B. PlugandPlay priors for model based reconstruction. in IEEE Global Conference on Signal and Information Processing, 945–948 (2013).
Swinehart, D. F. The BeerLambert law. J. Chem. Educ. 39, 333 (1962).
Figueiredo, M. A. T. Adaptive sparseness for supervised learning. IEEE Trans. Pattern Anal. Mach. Intell. 25, 1150–1159 (2003).
Ferstl, D., Reinbacher, C., Ranftl, R., Ruether, M. & Bischof, H. Image guided depth upsampling using anisotropic total generalized variation. in 2013 IEEE International Conference on Computer Vision, 993–1000 (2013).
Zhang, Q., Xu, L. & Jia, J. 100+ Times Faster Weighted Median Filter (WMF). in 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2830–2837 (2014).
Boyd, S., Parikh, N., Chu, E., Peleato, B. & Eckstein, J. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3, 1–122 (2011).
Acknowledgements
This work was supported by UK Defence Science and Technology Laboratory (Projects DSTLX1000108233 and DSTLX1000095645R), the UK Engineering and Physical Sciences Research Council (projects EP/N003446/1, EP/T00097X/1 and EP/S026428/1), the UK Royal Academy of Engineering Research Fellowship Scheme (Project RF/201718/17128), and Defence and Security Accelerator (project DSTLX1000147844). The authors wish to thank the team at Dstl Porton Down, Salisbury for their help in conducting the field trials.
Author information
Authors and Affiliations
Contributions
G.S.B. and P.S. planned and supervised the project. The bistatic setup was designed by A.M. and the LiDAR system was developed by A.M., R.T., and G.S.B. The data acquisition using the bistatic imaging system was performed by R.T. and A.M. and the transmissometer measurements were performed by P.S. A.H. and R.T. performed the data analysis with contributions from the other authors. A.H. designed, developed, and implemented the M2R3D algorithm. All authors contributed to the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Tobin, R., Halimi, A., McCarthy, A. et al. Robust realtime 3D imaging of moving scenes through atmospheric obscurant using singlephoton LiDAR. Sci Rep 11, 11236 (2021). https://doi.org/10.1038/s41598021905878
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41598021905878
This article is cited by

Fundamental limits to depth imaging with singlephoton detector array sensors
Scientific Reports (2023)
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.