Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

# Automatic extraction channel of space debris based on wide-field surveillance system

## Abstract

In the past few years, the increasing amount of space debris has triggered the demand for distributed surveillance systems. Long exposure time can effectively improve the target detection capability of the wide-area surveillance system. Problems that also cause difficulties in space-target detection include large amounts of data, countless star points, and discontinuous or nonlinear targets. In response to these problems, this paper proposes a high-precision space-target detection and tracking pipeline that aims to automatically detect debris data in space. First, a guided filter is used to effectively remove the stars and noise, then Hough transform is used to detect space debris, and finally Kalman filter is applied to track the space debris target. All experimental images are from Jilin Observatory, and the telescope is in star-tracking mode. Our method is practical and effective. The results show that the proposed automatic extraction channel of space debris can accurately detect and track space targets in a complex background.

## Introduction

Since mankind launched the first satellite in 1957, a new era of space exploration has ushered in. However, more and more space activities have also brought a lot of space debris, including failed spaceflights as large as several meters in diameter, rocket wreckage, and projectiles in space missions, as small as a few centimeters and a few millimeters of solid rocket combustion and spacecraft in orbit collision disintegration of the debris1,2, and such pose serious threats to human space activities and the normal operation of satellites. Therefore, it is necessary to monitor the movement and orbit information of space debris to achieve effective prediction of its activities and avoid accidents. And space-target detection and tracking technology is the focus of our study3,4.

In many wide-field surveillance systems, detecting and tracking dim space targets from optical images has always been a problem. Owing to the long distance between the space target and the CMOS sensor, the target on the focal plane is characterized by weak intensity and small area and is often easily submerged by background and noise5,6. Therefore, many methods have been proposed to solve the problem of dim-target detection in images, such as the template matching method7,8, morphological algorithm9,10 and neural network11. Reed12 proposed 3D matched filtering to detect small moving targets with strong background clutter. This method has an excellent detection effect when the target has the same moving speed. However, the detection performance of targets with unknown speeds decreases sharply. Bai13 et al. used mathematical morphological operators to eliminate noise and background changes and finally merged them into a median image to obtain the position of the object. Blostein14,15 et al proposed a multistage hypothesis testing method, this algorithm first introduced a tree structure to represent the trajectory of the target, which can detect multiple targets at the same time. In order to reduce missed alarms, the establishment of multiple candidate trajectory starting points may lead to an exponential increase in subsequent branches, which seriously affects the performance of the algorithm.

## Methods

### Image-processing channel

The astronomical image data used in the paper was collected by CMOS telescope. The specific parameters of the telescope are listed in Table 1. The time interval between exposures of the CMOS telescope camera is relatively short, and therefore it can be observed at a high frame rate. This camera has a relatively large field of view and a long detection range, and thus the captured astronomical images are relatively complex. These images were taken on the ground in star-tracking mode, in which the telescope was mounted on a turntable to counteract the speed of Earth’s rotation. Space objects appear as streaks, and stars appear as points in the image. Figure 1 shows the space exploration telescope components of the Changchun observatory. We have multiple telescopes that can cover a larger search area. Figure 2 shows the original astronomical image obtained by using this device.

The sequence image taken by the large field-based photoelectric telescope is a superposition of stars, space targets, and noise. It can be modeled as follows:

$$f(x,y,n) = O(x,y,n) + S(x,y,n) + B(x,y,n) + N(x,y,n)$$
(1)

where (x, y) represents the pixel coordinates in the star map. n represents the image frame number. f(x, y, n) represents the gray value of the image. O(x, y, n) represents the target gray value of the image. S(x, y, n) represents the star gray value of the image. B(x, y, n) represents the image background gray value. N(x, y, n) represents the noise gray value of the image, which may include atmospheric noise, space radiation, image generation noise, flicker noise, salt and pepper noise, multiplicative noise, and dark current16,17.

Based on the above principles, we proposed schematic diagram of the debris information extraction algorithm is shown in Fig. 3. It is divided into three stages. The first step is to denoise the image and eliminate uneven background such as skylight. We obtain the median value of the multi-frame sequence star image to estimate the background intensity of the star image. Subsequently, we introduce guided filtering algorithms to further remove stars and isolated noise and then perform binary quantization. The second stage is space debris detection. First, binarization is applied to further process the image to reduce the amount of image data. Then apply Hough transform to detect space debris. The third part is space debris tracking. Kalman filter is applied for tracking to feedback debris information at all times.

### Image preprocessing

We first perform median filtering. The algorithm determines the gray level of the center pixel by sorting the pixels in the neighborhood. The median filter processing result will be largely affected by the set filter window size18,19. Small filter window can better protect the details of the original image, but the noise suppression capability will be limited. Large filter window can enhance the noise suppression capability, but the details of the original image will become blurred. We have improved the median filter algorithm to minimize the impact of the filter window size setting on the image processing results.

First, 5 × 5 median filter is applied to process the original image, and the result of the processing is called the roughly denoised image. Calculate the average value of the rough noise-reduction image, the next step is to subtract the corresponding pixels of the original image from the average value, and the difference can be used as a threshold to judge the contaminated pixels in the image. The improved median filter only processes pixels that are considered to be contaminated by impulse noise by setting a threshold. Reducing the number of pixels processed by the filter window also reduces the influence of the filter window size setting on the image processing result. The contaminated pixels are processed with a median filter with size of 5 × 5, and the pixels that are not contaminated with noise remain unchanged, and the final improved median filter processing result is obtained. As shown in Fig. 4b, the median image was then analyzed for background estimation.

In order to detect the dim target in the image, we need to accurately estimate the uneven background. In astronomical images, we treat all pixels except the target as the background. The background analog-to-digital unit (ADU) value is the sum of photons from the sky plus the effects of various noises20,21,22. When processing the actual star map background, we take the median value of five consecutive star maps as the sky background. As shown in Fig. 4c, our algorithm accurately obtains most of the background, and the algorithm has the advantage of a small amount of calculation. We subtract the background image from the median filtered image to get an image containing space debris, isolated noise and stars, as shown in Fig. 4d, the image contains brighter stars and noise points, so in the next step we continue to eliminate these factors for the extraction target.

Guided filtering is an image smoothing filter based on a local linear model. The basic idea of the algorithm is to assume a linear relationship between each pixel in the image and its neighboring pixels, and obtain a linear model of each image, so as to obtain a basic image similar to the original image gradient23. In our extraction channel, the background suppressed image g is used as the input image, and g is also used as a guide image. When g is used as an input image and a guide image at the same time, the output of the guide filter can ensure the edge information of the space target. Its mathematical model is:

$$\begin{array}{*{20}{c}} {q_i = a_kg_i + b_k} & {\forall _i \in w_k} \end{array}$$
(2)

in the formula 2, ak and bk are the linear coefficients in window wk, g is the guide picture, wk is the window with radius h, and the constraint equations of ak and bk in window wk can be expressed as:

$$E(a_k,b_k) = \mathop {\sum}\limits_{i \in w_k} {((a_kg_i + b_k - p_i)^2 + \varepsilon a_k^2)}$$
(3)

in which ε is the regularization parameter, used to prevent the coefficient ak from being too large. p represents the input picture. The coefficients ak and bk are respectively:

$$a_k = \frac{{\frac{1}{{\left| w \right|}}\mathop {\sum}\nolimits_{i \in w_k} {p_ig_i - \overline {p_k} u_k} }}{{\sigma _k^2 + \varepsilon }}$$
(4)
$$b_k = \overline {p_k} - a_ku_k$$
(5)

in the formula, uk is the mean value in the window wk, $$\sigma _k^2$$ is the variance value in the window wk, $$\overline {p_k}$$ represents the mean value of the input image p in the window wk, and |w| represents the number of pixels in the window wk. As shown in Fig. 4e, we have obtained an image containing space debris and a very small number of noise points (which basically does not affect the extraction of debris).

It can be seen from the above formula that if the regularization parameter is set to a fixed value, the size of the filter window h will affect the quality of the star map. We select a star map with a large starlight background and noise to illustrate. Figure 5 shows the result of processing with multiple window sizes. When a smaller window is used (h = 0.1H, H represents the height of the image), there will be residual isolated noise and brighter stars, as shown in Fig. 5b. But h also cannot be too large, and the debris target (h = 0.6H) will be lost, as shown in Fig. 5d. In the experiment of this article, h = 0.3H is used, and the regularization parameter ε is set to 0.04, which has a good effect.

### Space debris detection

After the first stage of processing, we obtain candidate space targets through denoising algorithms and background suppression algorithms. We carry out the binarization process, which can reduces the amount of data, but also improve the precision of the Hough transform.

Binarization applies the maximum between-class variance method, which is the best algorithm for threshold selection. The threshold is the value with the largest variance between the target and background images. We use the Hough transform to process the image. This process converts the detection problem of a given curve in the original image into a problem of finding a peak in the parameter space. If fringes exist in the image, we map them to the parameter space and detect the peak value in it, as shown in the Fig. 6b. The Hough transform algorithm can detect the striped target in the image, so the starting point and ending point of the debris target can be obtained24.

When the line of common points in the image space is perpendicular to the x-axis, the slope is infinite, and the intersection point cannot be found in the Hough space. Therefore, the polar coordinates can be used. Similar to rectangular coordinates, the Hough transform in polar coordinates also transforms points in the image space into the parameter space. At this time, the obtained ρ and θ are the polar coordinate parameters of the straight line. For a straight line passing through any point (x, y) in the image space. Its polar coordinate space expression is:

$$\rho = x \ast \cos \theta + y \ast \sin \theta$$
(6)

in the formula 6, ρ is the vertical distance from the straight line to the origin, and θ represents the direction of the straight line. Once ρ and θ are determined, further analysis finds the endpoints and center of each line segment.

Figure 6c shows the curve distribution of the parameter space in the polar coordinate. The two space targets in the image have the most intersection points in the parameter space. Therefore, the points of collinear lines in the image space are mapped to the intersection points of the sine curves in the parameter space.

### Space debris tracking

When the space-debris target was detected in the first few frames of the images, we used the Kalman filter for follow-up tracking. Its small calculation amount and real-time calculation are more suitable for tracking space debris with a large field of view and a large amount of data. This method is an optimal linear recursive filtering method based on the minimum mean-square error, based on the state and the observation equations25. According to the movement characteristics of the debris in the large field of view optical observation system, it can be assumed that its movement approximately conforms to a uniform linear movement in adjacent frames. The initial position of the target is given by the detection result of the Hough transform, and the target speed can be given by the first two frames. The filter in the correction stage corrects the predicted value obtained in the prediction stage to obtain a new predicted value closer to the true value.

Prediction:

$$\begin{array}{l}{\mathop {x}\limits^{\Lambda ^\prime }} _n = A\,{\mathop {x}\limits^\Lambda} _{n - 1} + Bu_n\\ p_n^\prime = Ap_{n - 1}A^T + Q\end{array}$$
(7)

Correction:

$$\begin{array}{ll}k_n = p_n^\prime H^T(Hp_n^\prime H^T + R)^{ - 1}\\ {\mathop {x}\limits^\Lambda} _n = {\mathop {x}\limits^{\Lambda ^\prime }} _n + K_n{\mathop {Z}\limits^\Lambda} _n\\ p_n = (I - K_nH)P_n^\prime \end{array}$$
(8)

in which $${\mathop {x}\limits^\Lambda} _n$$ and $${\mathop {x}\limits^\Lambda} _{n - 1}$$ represent the true value, $${\mathop{x}\limits^{\Lambda\prime}}_n$$ represents the Kalman estimated value; pn is the Kalman error covariance matrix; $$P_n^\prime$$ is the Kalman estimation error covariance matrix; kn is the Kalman gain matrix; $$\mathop {Z}\limits^\Lambda _n$$ is the measured value; A represents State transition matrix; B represents the input control matrix; p represents the error matrix; Q represents the predictive noise covariance matrix; R represents the observation noise covariance matrix; H represents the observation matrix. Figure 7b shows the Kalman tracking graph. It can be seen from the figure that the algorithm can track the detected debris targets well without losing the weaker targets. Figure 7c shows the algorithm tracking trajectory graph, and there is no target loss.

### Experiment

In this part, we will conduct some experiments to verify the performance of the algorithm. The images used in the experiments are all obtained by the large field of view telescope in the star-tracking mode. The background stars in the image are dense, and it is necessary to accurately extract the streak target, and the image size is 4096 × 4096 pixels. For ease of understanding and clarity, some of the experimental images have been reduced. In this summary, the pseudocode of debris extraction channel and the detection effect of real astronomical images are illustrated.

### Procedure

As shown in Table 2, in this section we give the pseudo-program of the detection algorithm.

### Real astronomical image detection experiment

The real astronomical image used in this experiment to explain the detection and tracking effect of the method. In order to fully test the algorithm proposed in this paper, the targets are named T1 and T2 as shown in the figure, and the space target detection results are shown. The red circles display the detected targets. Figure 8 shows the experimental results, b-1 to b-4 are the detection and tracking of the target T1, and c-1 to c-4 display the experimental result of the target T2.

It can be seen from the detection results that the algorithm detects the target in four consecutive frames of images, the algorithm can continuously track target with high precision, which shows the practicability and stability of the algorithm. Figure 9 shows the estimation and tracking of the centroid coordinates of continuous multi-frame images of our algorithm. We can see that the algorithm has a better tracking effect. This shows the excellent detection performance and real-time tracking capability of the debris extraction channel.

To further demonstrate the performance of the proposed space debris extraction approach, we apply the multistage hypothesis testing algorithm (MHT)14, new top-hat algorithm (NTH)26, improved maximum value projection algorithm (IMVP)27 and our extraction channel to extract space debris from the same actual star image, and analyze the probability of debris extraction, false alarm rate and time-consuming process. The evaluation results of the images are given in Table 3. In order to compare the fairness of the results, the selected comparison methods are all carried out using standard published parameters. In the MHT method, detection is performed on all space target trajectories in the star map. However, with the increase of noise and the number of targets, the trajectory of targets increases sharply, resulting in a large amount of computational cost. In the wide-area monitoring system, the NTH detection result is affected by structural elements, which seriously affects the performance of the algorithm. The IMVP algorithm can detect moving targets well and track their trajectories accurately, but the algorithm comes at the cost of computation.

Our algorithm has low false alarm rate and simple calculation. The channel was evaluated for 200 real images. Figure 10 shows the detection results of four images with different backgrounds. The images from a-1 to a-3 and b-1 to b-3 have more stars and noise, the images from c-1 to c-3 have less noise, and the images from d-1 to d-3 are not only composed of a large number of the stars and noise are also disturbed by clouds. Our algorithm has obtained better detection and tracking effects in these types of images. Figure 11 shows the debris trajectory diagrams of Fig. 10a, b. The target trajectory is circled in red. The proposed debris extraction channel can extract space debris very well and takes less time.

## Results

The space target information automatic extraction pipeline is research in this paper. The algorithm is suitable for the detection of space stripe targets with complex background. The median value of multiple frames of images is taken to suppress the influence of the background, and guided filtering is introduced to eliminate isolated noise points and stars. The Hough transform and Kalman filter were used for target detection and tracking. We conducted tests on the original star map. The test results show that our space debris extraction channel can not only effectively detect debris targets, but also accurately track debris information.

## Discussion

In our approach, the multi-frame median method is used to suppress the influence of the background, and guided filtering is introduced to eliminate isolated noise points and stars. Hough transform is used to detect space debris, and finally Kalman filter is applied to track the space debris target. We propose a channel to detect the presence of space debris without the need to use auxiliary space targets and orbital data. We did not obtain information in advance to help the target detection algorithm. Furthermore, the focus of our work is not only the debris detection technology itself, but the space target information extraction that combines space debris detection and astronomical positioning.

### Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.

## Data availability

All image data in the article are raw data, not obtained from public databases, if necessary, all data in the article can be obtained from the corresponding author upon reasonable request.

## References

1. Schildknecht, T. et al. Optical surveys for space debris. Astron. Astrophys. Rev. 14, 41–111 (2007).

2. Sun, R. Y., Zhang, J. W. & Zhao, C. Y. Improving the precision of astrometry for space debris. Astron. J. 147, 58–67 (2014).

3. Sun, R. Y., Zhang, J. W. & Zhao, C. Y. Algorithms and applications for detecting faint space debris in GEO. Acta Astronaut. 110, 9–17 (2015).

4. Liu, R., Li, X., Han, L. & Meng, J. Track infrared point targets based on projection coefficient templates and non-linear correlation combined with kalman prediction. Infrared Phys. Technol. 57, 68–75 (2012).

5. Du, B., Zhang, Y. X., Zhang, L. P. & Tao, D. C. Beyond the sparsity-based target detector: a hybrid sparsity and statistics-based detector for hyperspectral images. IEEE Trans. Image Process 25, 5345–5357 (2016).

6. Rufino, G. & Accardo, D. Enhancement of the centroiding algorithm for star tracker measure refinement. Acta Astronaut. 53, 135–147 (2003).

7. Liu, R., Lu, Y., Gong, C. & Liu, Y. Infrared point target detection with improved template matching. Infrared Phys. 55, 380–387 (2012).

8. Murphy, T. S., Holzinger, M. J. & Flewelling, B. Space object detection in images using matched filter bank and bayesian update. J. Guidance, Control, Dyn. 40, 497–509 (2016).

9. Wei, M. S., Xing, F. & You, Z. A real-time detection and positioning method for small and weak targets using A 1D morphology-based approach in 2D images. Light. Sci. Appl. 7, 97–106 (2018).

10. Sun, R., Zhao, C. & Ping, Y. Use of mathematic morphological operators for processing the smeared images. Chin. Astron. Astrophys. 36, 340–352 (2012).

11. Jia, P., Liu, Q. & Sun, Y. Detection and classification of astronomical targets with deep neural networks in wide-field small aperture telescopes. Astronomical J. 159, 212–222 (2020).

12. Reed, I. S., Gagliardi, R. M. & Stotts, L. B. Optical moving target detection with 3-D matched filtering. IEEE Trans. Aerosp. Electron. Syst. 24, 327–336 (1988).

13. Bai, X., Zhou, F., Xie, Y. & Jin, T. Enhanced detectability ability of point target using adaptive morphological clutter elimination by importing the properties of the target region. Signal Process 89, 1973–1989 (2009).

14. Blostein, S. D. & Huang, T. S. Detecting small moving objects in image sequences using sequential hypothesis testing. IEEE Trans. Signal Process. 39, 1611–1629 (1991).

15. Li, M., Yan, C., Hu, C., Liu, C. & Xu, L. Space target detection in complicated situations for wide-field surveillance. IEEE Access 7, 123658–123670 (2019).

16. Sun, Q., Niu, Z., Wang, W., Li, H. & Lin, X. An adaptive real-time detection algorithm for dim and small photoelectric GSO debris. Sensors 19, 4026–4048 (2019).

17. Castronuovo, M. M. et al. Active space debris removal-A preliminary mission analysis and design. Acta Astronaut. 69, 848–859 (2011).

18. Guo, Z. & Zhang, L. A completed modeling of local binary pattern operator for texture classification. IEEE Trans. Image Process 19, 1657–1663 (2010).

19. Xi, J. B., Wen, D. S. & Song, Z. X. A maximum projection and particle filtering algorithm for space debris detection. J. Inf. Computational Sci. 12, 161–169 (2015).

20. Liu, D. et al. Space target detection in optical image sequences for wide-field surveillance. Int. J. Remote Sens. 41, 7846–7867 (2020).

21. Thirumavalavan, S. & Jayaraman, S. An improved teaching–learning based robust edge detection algorithm for noisy images. J. Adv. Res. 7, 979–989 (2016).

22. Xi, J. D., Wen, D. S. & Yi, H. W. Space debris detection in optical image sequences. Appl. Opt. 55, 7929–7940 (2016).

23. Wang, E. D., Jiang, P. & Hou, X. K. Infrared stripe correction algorithm based on wavelet analysis and gradient equalization. Appl. Sci. 9, 1–21 (2019).

24. Moyer, L. R., Spak, J. & Lamanna, P. A multi-dimensional Hough transform-based track-before-detect technique for detecting weak targets in strong clutter backgrounds. IEEE Trans. Aerosp. Electron. Syst. 47, 3062–3068 (2011).

25. Leong, P., Arulampalam, S. & Lamahewa, T. A Gaussian-sum based cubature Kalman filter for bearings-only tracking. IEEE Trans. Aerosp. Electron. Syst. 49, 1161–1176 (2013).

26. Bai, X. & Zhou, F. Analysis of new top-hat transformation and the application for infrared dim small target detection. Pattern Recognit. 43, 2145–2156 (2010).

27. Yao, D., Wen, D., Xue, J., Chen, Z. & Wen, Y. Maximum projection and velocity estimation algorithm for small moving target detection in space surveillance. Image Process. Anal. 9675, 96752–96762 (2015).

## Acknowledgements

The research, supported by the National Natural Science Foundation of China (12003052, U2031129), used astronomical star maps provided by the Changchun Satellite Observatory.

## Author information

Authors

### Contributions

P.J. is responsible for data analysis and paper writing, C.L. and W.Y. are responsible for experimental design and experimental analysis, and Z.K., C.F., and Z.L. are responsible for data analysis and experimental design.

### Corresponding author

Correspondence to Chengzhi Liu.

## Ethics declarations

### Competing interests

The authors declare no competing interests.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Reprints and Permissions

Jiang, P., Liu, C., Yang, W. et al. Automatic extraction channel of space debris based on wide-field surveillance system. npj Microgravity 8, 14 (2022). https://doi.org/10.1038/s41526-022-00200-z

• Accepted:

• Published:

• DOI: https://doi.org/10.1038/s41526-022-00200-z