Introduction

Internal pipe corrosion within water distribution systems has a critical impact on water quality. The materials of the drinking water distribution pipes include substances such as polyvinyl chloride, steel, and cast iron, with cast iron pipes being most commonly used worldwide1. In drinking water distribution systems, cast-iron pipes undergo aging over time, leading to corrosion easily. The resulting corrosion scales take the form of porous deposits of iron oxide or oxyhydroxide phases2,3,4,5. As corrosion progresses, these scales gradually release iron into the drinking water, primarily in the form of ferrous oxide, resulting in the formation of red water. Additionally, the gradual buildup of corrosion products in the form of corrosion scales provides environments where harmful and opportunistic bacteria can thrive, leading to infections in the gastrointestinal system, skin, and lymph nodes6,7,8,9,10. Therefore, continuous and non-destructive inspection of pipelines in drinking water systems is essential, and research on iron corrosion has been actively conducted in the past.

Water distribution systems are pivotal infrastructures in our society, ensuring a consistent supply of clean water daily. Pipelines constitute the largest component of these systems, making the maintenance of pipe integrity a priority. In newer systems, pipes are commonly made from materials like ductile iron and galvanized steel, while older systems frequently use cast iron11,12. These metallic materials are prone to corrosion, a process that can significantly impact the infrastructure, water quality, and safety. Corrosion can lead to pipe weakening through wall thinning or pitting, which may result in leaks or bursts. It can also impede water flow by accumulating on the pipe’s inner walls and introduce contaminants into the water supply, thereby raising water quality concerns13,14. These contaminants can further damage hydraulic components, including pressure regulators, valves, and mechanical seals. Therefore, continuous monitoring and assessment of pipe corrosion are an essential maintenance strategy to maintain the integrity and functionality of water distribution systems15.

The most precise methods currently employed include analyzing the chemical composition of the water flowing through the pipes, coupon testing, or visual inspection inside the pipes. However, these approaches are considered destructive as they require taking water samples, inserting metal coupons into pressurized pipes, or halting the system for visual inspections. Another option involves installing inline fluid corrosivity monitoring devices for continuous corrosion monitoring, but this also necessitates system downtime for device installation, which significantly limits its practicality16. Consequently, continuous and non-destructive testing (NDT) methods are recommended for their ability to monitor system integrity without disrupting water supply15,17,18.

There are various NDT methods and studies to investigate the pipe’s defect in non-invasive way. The most used NDT methods are magnetic flux leakage inspection (MFL) and eddy current (EC) technique. MFL testing is one of the NDT methods for inspecting ferromagnetic materials. The interaction between magnetic field and defects is used in MFL testing19. It has high efficiency and reliability. However, the defect size must be quantified to determine if it is seriously damaging. Thus, a lot of effort is needed to study of the quantification of defects, which is a classic problem20. The EC technique is the conventional electromagnetic methods, which is utilized for the inspection of conductive materials such as copper, aluminum, or steel21. It requires that the materials being tested must be electrical conductors where eddy currents can flow. Also, EC testing has sensitivity and robustness. However, EC NDT testing is very sensitive to the lift-off effect. The lift-off variations can be caused by varying coating thickness, irregular sample surfaces, or the operator’s movement22. The disadvantage of inspecting magnetic materials is that permeability changes generally have a much greater effect on eddy current response than conductivity variations.

The ultrasound method, which offers the advantages of being non-destructive and allowing continuous measurements, has been widely utilized in NDT23,24,25. In particular, the use of array transducers is predominant, and the ease of measurement using ultrasound B-scan has made it the most commonly employed method26. Due to the nature of ultrasound, higher frequencies offer better resolution, allowing for more precise measurements. However, producing array transducers at higher frequencies (over 15 MHz) during the manufacturing process is a challenging task. To overcome these problems, we utilized scanning acoustic microscopy (SAM) in this study. SAM operates on an ultrasound-based system, employing a single ultrasound transducer to scan a defined area while acquiring reflected ultrasound signals from the object under investigation. These ultrasound signals are primarily used for imaging C-scans within objects27. SAM finds applications in various fields of NDT, such as identifying internal defects in materials like semiconductors and inspecting automotive components. It is also utilized in cell imaging. The significant advantage of SAM lies in its use of a single ultrasound transducer, enabling the utilization of high-frequency transducers, unlike array transducers. Due to the nature of ultrasound, higher frequencies result in better resolution. Indeed, when using ultrasound transducers with frequencies exceeding 200 MHz, resolutions reach as fine as 7 \(\mu m\)28,29. Due to its fine resolution, SAM has an advantage of higher accuracy compared to other previously mentioned NDT methods.

The amount of iron oxide inside a pipe is related to red water and is an important factor in water quality assessment30,31. But in the past, there were no studies on obtaining information inside the pipes using NDT. We aim to utilize the advantage of ultrasonic non-destructive measurement by securing a large amount of ultrasonic signals within the pipes and analyzing them through convolution neural network (CNN). Our previous studies32,33,34,35 demonstrated that the efficacy of visual geometry group (VGG)-like CNN models for extracting cell physical properties (e.g., deformability and size) from acoustic reflection signals obtained from individual cells. Despite focusing on single biological cell (about 20 μm in diameter) reflection signals, these studies demonstrated the ability of CNN models to capture generalized characteristics of cell reflection signals over the variety of living cells. Therefore, in this study, we extend this notion to assume that CNN models can discern the concentration levels of iron oxide particles impacting the amplitudes of acoustic reflection signals within pipes. This extension involves overcoming the varied signal characteristics arising from the unspecified distribution of iron oxide particles present in the flowing water within the pipes. This study aims to explore appropriate CNN architectures for analyzing the reflection signals, although our previous studies focused on assessing the effectiveness of the CNN models in extracting physical properties from acoustic reflection signals of target objects. Thus, we conducted time-frequency domain analysis for the reflection signals with four different types of widely used CNN models, including VGGNet, InceptionNet, ResNet, and EfficientNet, which are the basic models to compare the differences in model architecture. The analysis results showed that the combination of high-frequency transducers and CNN models is effective for discerning physical properties of particles within fluids, expanding the applicability beyond living cells to encompass particles in fluids within pipes.

In this study, we have developed a dual-mode approach for the assessment of pipe corrosion in water distribution system, leveraging the integration of ultrasound technology and CNN. This multifaceted approach enables simultaneous evaluation of pipe corrosion levels through NDT and the quantification of iron oxide particle concentrations within the water enclosed in the pipes through CNN. We achieved this by conducting B-mode analyses utilizing high-frequency transducers to measure pipe thickness and gauge the extent of corrosion. Concurrently, we carried out experiments involving the classification of the quantity of iron oxide particles present in the water within the pipes, employing A-scan data which was processed utilizing CNN techniques.

Results

Pipe corrosion assessment system

Schematic diagram of pipe corrosion assessment in water distribution system is demonstrated in Fig. 1. The syringe pump (NE-4000, New Era Pump Systems Inc., USA) was used to maintain a consistent water flow, which included iron oxide particles, at a steady velocity of 0.3 m per s. In this experiment, we used SAM (Ohlabs Table SAM V2, Ohlabs Corp., KOR) system. The primary objectives of this equipment utilization were twofold: firstly, to acquire precise measurements of pipe corrosion thickness within the pipe, and secondly, to collect A-scan data for the purpose of its application in CNN models, as visually depicted in Fig. 2. The SAM was configured with the following parameters: a sampling rate of 250 MS per s, a gain setting calibrated to 20 dB, and a pulse repetition frequency (PRF) of 200 Hz. For measurements of pipe corrosion thickness, a 20 MHz ultrasound transducer (V316-SU, Olympus Inc., USA) was employed. Additionally, in the acquisition of A-scan data from within the pipe, we utilized a 5 MHz ultrasound transducer (V307-SU, Olympus Inc., USA).

Fig. 1: Pipe corrosion assessment system.
figure 1

a Schematic diagram of internal pipe corrosion assessment method in water distribution system. b Photograph of the experimental system.

Full size image
Fig. 2: Schematic diagram of dual-mode water distribution pipe corrosion assessment system.
figure 2

a The measurement of pipe corrosion thickness. b A cross-sectional view of the pipe. c A classification scheme for determining iron oxide particle concentration.

Full size image

Pipe corrosion simulation

Corrosion can cause the walls of a pipe to thin, leading to several serious risks. As the pipe wall gets thinner, stress concentrations can develop in specific areas. This weakens the pipe’s ability to handle internal pressures and increases the likelihood of mechanical failure. Such failure can cause leaks of the transported fluid, which can be especially dangerous if the fluid is toxic, flammable, or hazardous in any way. It’s well known that as pipe thickness decreases, hoop stress increases. Hoop stress, the main stress component in a pressurized pipe, is described by the thin-walled pipe equation shown in Eq. (1).

$${{\rm{\sigma }}}_{{\rm{h}}}=\frac{{{\rm{p}}}_{{\rm{i}}}{\rm{r}}}{{\rm{t}}}$$
(1)

where σh is the hoop stress, pi is internal pressure, r is the internal radius of the pipe, and t is the wall thickness. The equation for a thick-walled pipe is more complex, as shown in Eq. (2). While the equation alone is less straightforward, it clearly demonstrates that hoop stress increases as thickness decreases when expressed graphically.

$${{\rm{\sigma }}}_{{\rm{h}}}={{\rm{p}}}_{{\rm{i}}}\frac{{{\rm{r}}}_{i}^{4}+2{r}_{i}^{3}t+{r}_{i}^{2}{t}^{2}+{r}_{i}^{2}{r}^{2}}{{{\rm{r}}}^{2}(2{r}_{i}t+{t}^{2})}$$
(2)

where ri is inner radius.

While thin-walled and thick-walled pipe stress equations can illustrate the inverse relationship between stress and wall thickness, finite element analysis (FEA) can calculate stress in a pipe more accurately. Therefore, we conducted FEA using Ansys simulation. During the meshing process, we divided the pipe thickness into five layers of elements to improve accuracy in capturing through-thickness behavior. We incorporated this detailed discretization into an axisymmetric analysis to ensure a thorough and precise representation of the pipe’s structural response. Figure 3a shows the FEA result for stress analysis of a pipe with an outer diameter of 6.625 inches and a thickness of 0.28 inches under an internal pressure of 300 psi. Figure 3b shows the result for a pipe with a thickness of 0.14 inches, while other conditions remain the same. The result shows that as thickness reduces by half, the stress increases from 22.53 MPa to 44.25 MPa, which is approximately a 96% increase.

Fig. 3: Pipe corrosion FEA simulation using Ansys.
figure 3

a FEA result for stress analysis in pipe with original thickness and b with thickness reduced by half.

Full size image

Pipe thickness measurement

Six different brass pipes were used in the experiments, and corrosion measurements were conducted using a 20 MHz transducer and SAM. The pipe thicknesses used in the experiments were 958, 889, 784, 683, 531, and 477 µm. These measurements were obtained using a digimatic indicator (DI) with a resolution of 1 µm (ID-C150XB, Mitutoyo, Japan), as shown in Fig. 4. When comparing the thickness measured by SAM with the thickness measured by the DI, the error range was approximately 7 to 48 µm, indicating a small margin of error. All pipes exhibited an error within 10%. The use of a high-frequency transducer confirmed the capability to measure small-scale pipe corrosion in µm units.

Fig. 4: The results of measuring the thickness of the pipes using SAM and DI.
figure 4

L represents the thickness of the pipe. The red labels indicate the measurements of pipe thickness as conducted by SAM, whereas the black letters below the images indicates the measurements obtained using a DI.

Full size image

Pipe corrosion measurement

Three different pipes with thicknesses of 958, 531, and 528 μm were used to make the corroded pipes, resulting in reductions of 76, 69, and 80 μm, respectively. The difference between the thickness measured by DI and the thickness measured by SAM ranged from 2 to 36 μm, indicating a high level of precision in the measurements (Fig. 5).

Fig. 5: Pipe corrosion measurement.
figure 5

a Photograph of control and corroded pipe. b The results of measuring the thickness of the control and corroded pipes using SAM and DI. The L represents the thickness of the pipe. The red labels indicate the measurements of pipe thickness as conducted by SAM, whereas the black letters below the images indicates the measurements obtained using a DI.

Full size image

Effectiveness of CNNs for determining iron oxide particle concentration

We collected 10,000 acoustic reflection signals from corroded pipes with each of six distinct levels of iron oxide particle concentration. We segmented the total of 60,000 signals into five folds, preserving the sample distribution across the six classes. Then, a five-fold cross-validation approach was employed, wherein each fold served as test data once while the remaining folds constituted the training data. The objective was to assess the efficacy of the proposed system in accurately classifying the entire dataset. Evaluation of classification accuracy was based on four metrics: accuracy, precision, recall, and F1 score. The metrics can be formulated as:

$${accuracy}=\frac{{TP}+{TN}}{{TP}+{FN}+{TN}+{FP}}\,,{precision}=\frac{{TP}}{{TP}+{FP}}\,,{recall}=\frac{{TP}}{{TP}+{FN}}\,,\,{F}_{1}=2\frac{P\times R}{P+R}\,$$
(3-6)

where TP, FP, FN, and TN represent the number of true positive, false positive, false negative, and true negative samples, respectively. Table 1 shows the mean and standard deviation of the accuracy metrics derived from the performance of the four CNN classifiers across the five folds. Additionally, Supplementary Figure 1 to 4 show the confusion matrices illustrating the classification outcomes of the four classifiers on each fold.

Table 1 CNN Model Performance: Comparison of four models for classifying six levels of iron oxide concentration
Full size table

All the four CNN backbones exhibited high accuracy. Across these models, the F1-scores consistently surpassed 0.92 on average, exhibiting minimal standard deviation, typically below 0.04. Also, the discrepancy between their precision and recall measures remained marginal, consistently below 0.01 across each CNN backbone. These results indicate that (i) acoustic signals reflected from pipes include discernible features related to particle concentrations in fluids within the pipes and (ii) CNN architectures effectively extract and interpret these distinctive features from the reflection signals.

Although the CNN backbones demonstrated consistently high accuracy, performance deviations emerged among the four types. Notably, ResNet significantly outperformed the other architectures across all accuracy metrics, boasting an average accuracy exceeding 0.99 with a low variance, typically below 0.007. Specifically shown in Supplementary Figure 2, ResNet achieved perfect classification across Folds 2 to 5, with a singular instance of misclassification observed in Fold 1, where 10% of samples with a concentration of 15 mg per L were misattributed to 1 mg per L. By comparing ResNet with the other backbones, we can assume that correlations between frequency components that are far apart in time or frequency band are important features in analyzing physical properties of objects with their acoustic reflection signals. Consequently, the implication arises that the analyzing physical property with acoustic reflections requires large receptive fields.

EfficientNet had the third-highest average accuracy, surpassing 0.96, with very small deviations from VGGNet, which had the second-highest accuracy. Notably, their standard deviations were largely comparable, consistently exceeding 0.30 and surpassing those of ResNet and InceptionNet. EfficientNet exhibited high accuracy on Folds 2 to 5 as ResNet. However, compared to ResNet, it exhibited a higher frequency of misclassifications, notably assigning a greater number of 15 mg per L samples to 1 mg per L, and 5 mg per L samples were also incorrectly classified as 15 mg per L, as shown in Supplementary Figure 4. EfficientNet displayed a diminished capacity compared to ResNet in managing noisy samples, potentially associated with the 15 mg per L class, within Fold 1. From these results, we can assume that among the factors determining expressive power of CNNs (e.g., depth, width, and resolution), depth is the most significant due to the required receptive fields for discovering correlations between distant frequency components.

VGGNet and EfficientNet showed similar performances, positioning between ResNet’s superior accuracy and InceptionNet’s lower performance. Both backbones had similar average accuracy (nearby 0.97) and variance (nearby 0.04). However, distinctive tendencies were noticeable in their respective confusion matrices, as depicted in Supplementary Figure 1 and 4. Although errors of VGGNet occurred by classifying 5 mg per L and 20 mg per L samples into 300 mg per L class on overall folds, misclassifications of EfficientNet were concentrated on 5 mg per L and 15 mg per L classes on Fold 1 and 2. Despite the higher performance of EfficientNet in Folds 3 to 5, we can assume that VGGNet is more robust to noisy outliers than EfficientNet. Considering two points (i) VGGNet was a simple but efficient model, which simply consists of convolution layers and pooling layers, and (ii) our EfficientNet and VGGNet backbones resultingly have similar number of layers, these results also support our assumption that satisfying the required receptive fields would be the key point in physical property analysis with acoustic reflections.

Furthermore, the CNN backbones commonly showed lower performance on the low concentration levels than the high. This tendency suggests that low concentrations of particles yield less discernible features within acoustic reflection signals compared to their higher-concentration counterparts. Therefore, these results indicate that VGGNet’s proficiency in effectively addressing the challenges posed by the indistinctiveness in low particle concentration signals, while other architectures have difficulties in extracting general and distinctive features from low concentration classes. We assume that VGGNet’s straightforward architecture and reduced parameter complexity contribute significantly to its ability to prevent overfitting and acquire general features.

InceptionNet had the lowest performance among the four CNN backbones. The average accuracy is about 0.03 behind VGGNet and EfficientNet, with the smallest standard deviation of 0.03, indicating that InceptionNet is consistently underperforming. As shown in Supplementary Figure 3, different from ResNet and EfficientNet, which faced difficulties in low concentrations on only a few folds, InceptionNet exhibited misclassifications among low concentration classes across all folds. Although we expected that multiple sizes of convolution kernels could be effective for extending receptive fields of CNN backbones, multi-scale analysis exacerbated the difficulties in accurately classifying low concentration levels. The accuracy and loss history of the four CNN models on the test data are shown in Fig. 6.

Fig. 6
figure 6

Plots of the four CNN models’ accuracy and loss on the test data according to epochs.

Full size image

Conclusively, we discovered two points from the experimental results. First, physical property analysis with acoustic reflection signals requires enough sizes of receptive fields, which depend on sufficient depth of CNN backbones. Second, features associated with low particle concentrations, as reflected in acoustic signals, lack distinctiveness, and efforts to prevent overfitting, such as reducing model complexity, are required to bring out their general features. Third, multi-scale analysis in time-frequency domain is not effective for revealing correlations between magnitude of frequency components, which are related to particle concentrations in fluids.

Discussion

In this study, we conducted experiments involving the measurement of pipe corrosion thickness using B-scan image analysis and quantitative analysis of particles within pipes using A-scan raw data. This approach allowed us to simultaneously measure pipe corrosion levels and assess water quality. B-scan analysis of pipes has been widely researched and established as a well-utilized technology in the past. However, we introduced an innovative method by combining particle analysis, specifically the analysis of iron oxide particle concentrations in water, with CNN technology, enabling us to assess water quality based on suspended particulate matter standards. Notably, ultrasound-based particle analysis has gained significant attention in recent years, with some noteworthy studies focusing on clay particle and concentration classification, cell type classification, and other related areas33,34,35,36,37,38. The primary advantage of ultrasound technology lies in its non-invasive and continuous measurement capabilities. Leveraging this characteristic, we proposed the possibility of creating a continuous and non-invasive method for water quality prediction.

Traditionally, SAM found its applications primarily in image analyses, such as B-scans and C-scans. However, in our study, SAM was harnessed for B-scan imaging specifically geared towards the precise measurement of pipe thickness. Leveraging a high-frequency ultrasonic transducer, we achieved superior resolution in B-mode images within the pipeline. Moreover, SAM was adapted to procure a substantial volume of A-mode ultrasound signals, facilitating the classification of iron oxide particle concentration. This abundant data resource plays a pivotal role in preventing overfitting, a challenge which is addressed by employing a CNN model39.

In this experiment, the concentration groups classified through CNN and NDT were very small, such as 1 mg per L, 5 mg per L, and 15 mg per L. By classifying groups with such small concentration differences, highly precise water quality measurement is possible. When corrosion occurs in a pipe, its thickness can either decrease or increase. If the thickness increases, iron oxide particles can be released due to the water flow force, contaminating the water quality. If the thickness decreases, the stress on the pipe increases and the internal pressure rises, potentially leading to mechanical failure. The technology used in this experiment is particularly effective at detecting both increases and decreases in pipe thickness, allowing for real-time thickness measurement. Real-time pipe thickness measurements can be integrated with analytical equations or FEA methods to estimate stress in real time. Techniques like autoregressive integrated moving average (ARIMA), regression models, long short-term memory (LSTM), and CNN can be used to analyze how stress evolves with corrosion progression and predict the remaining time until mechanical failure, incorporating appropriate safety factors. This approach can be developed into a predictive maintenance method for water distribution systems using an ultrasound transducer as a single NDT device.

Previously, ultrasound-based particle measurement methods were widely used due to their non-invasive and non-intrusive advantages. ultrasound can penetrate opaque materials and offer a wide frequency range, making them effective for various applications40. Notably, these include the IB coefficient, attenuation measurement, and acoustic emission41,42,43,44,45,46. However, previous measurement techniques are comparatively less precise due to their reliance on a single characteristic as the primary feature for analysis. In this experiment, by replacing these traditional methods with CNN, we achieved higher accuracy and faster measurements. Especially, ResNet displayed very high accuracy. Therefore, we were able to confirm the potential applicability of this method for measuring water quality inside water distribution pipes. The outcomes of this research have not only validated the efficacy and merits of the SAM-CNN hybrid system but have also paved the way for promising avenues of further investigation in the future.

This study is subject to a notable limitation, namely the challenges and time-intensive nature of generating real-world samples featuring corrosion within water supply pipes. To address this constraint, the research utilized brass pipe specimens characterized by varying thicknesses spanning the hundred-micrometer range and accelerated testing to make corroded pipes. In addition, the contemporary water supply infrastructure predominantly employs cast iron pipes rather than brass. Both cast iron and brass, along with many other metals, display similar acoustic properties such as acoustic impedance, speed of sound, and density, demonstrating analogous ultrasonic characteristics47.

Methods

Ultrasound transducer

In this dual-mode measurement system, we used two ultrasound transducers: one with a frequency of 5 MHz and the other with a frequency of 20 MHz. The time-domain and frequency-domain characteristics of these two ultrasound transducers are illustrated in Fig. 7. The 5 MHz transducer operates with a voltage peak-to-peak (Vpp) of 3.0 V, a center frequency of 5.4 MHz, and a bandwidth of 37.6%. The 20 MHz transducer has an approximate Vpp of 1.6 V, a center frequency of 19.0 MHz, and a -6 dB fractional bandwidth of 110.3%.

Fig. 7: Performance evaluation of transducers.
figure 7

a Time and frequency domain of the 5 MHz transducer and b the 20 MHz transducer.

Full size image

Preparation of pipes and \({{\boldsymbol{Fe}}}_{{\boldsymbol{2}}}{{\boldsymbol{O}}}_{{\boldsymbol{3}}}\)

In this experiment, brass pipes were utilized as models for water distribution system, with thicknesses ranging from 477 to 958 μm. Additionally, accelerated testing was conducted by immersing the pipes in a 1 M solution of nitric acid and maintaining them in an oven at 60 °C for 24 h48,49. To simulate iron oxide detached from corroded pipes, \({{Fe}}_{2}{O}_{3}\) powder (Magerial Science, US) with a particle median size of 10 μm was employed. The iron oxide concentration standards were set at 1, 5, and 15 mg per L, following water resources management information system (WAMIS) water quality environmental standards. To cover a broader range of measurements, concentrations were increased twenty-fold to 20, 100, and 300 mg per L, resulting in a total of six experimental groups. We obtained a total of 60,000 ultrasound signals, with 10,000 data for each group.

Determining iron oxide particle concentration with CNNs

Prior research32,33,34,35 has shown that CNN models similar to VGG can extract physical attributes of cells, such as deformability and size, using acoustic reflection signals from the cells. Although the previous studies analyzed reflection signals from captured single cells, the CNN models could acquire general properties of cell reflection signals over the variety of living cells. Therefore, in this study, we assume that the CNN models can determine the concentration levels of iron oxide particles that affect the amplitudes of acoustic reflection signals from pipes by overcoming the variety of signal characteristics caused by the unspecified distribution of iron oxide particles in the water flowing through the pipes.

The transmitted ultrasound waves pass through water and iron oxide particles within the pipe, with some being reflected while others are absorbed or transmitted further. Therefore, the ultrasonic reflection signal varies depending on the concentration of iron oxide particles. The purpose of CNNs is to classify the concentration of iron oxide particles by analyzing the arrival time and intensity of the reflected signal measured by the ultrasonic transducer. For the acoustic reflection signals, we first applied short-term Fourier transform (STFT) to analyze both time-domain and frequency-domain characteristics of the signals. The concentration of iron oxide particle mainly affects the amplitudes of the reflection signals, and our previous study34 exhibited that the time-domain and frequency-domain features are synergistic in capturing the changes in amplitudes. In this study, we performed the STFT with specific parameter settings, where the time window size was set to 32, the hop length was set to 4, and the frequency resolution was established as 1024. After data preprocessing, we can analyze the localized frequency information within each time window, which provides accurate frequency characteristics at specific segments of the signal. Therefore, extracted as a result of STFT, we can detect and track fine frequency variations presented in the ultrasonic reflection signal with CNNs to classify the iron oxide concentration in flowing water.

We aim to analyze the performance deviation due to the architectural differences of CNN backbones (i.e., filter size, multiple kernels, depth, parameter optimization) in classifying acoustic reflection signals for changes in iron oxide particle concentration. Therefore, we applied four types of CNN backbones, namely VGGNet, InceptionNet, ResNet, and EfficientNet, to the spectrogram obtained from STFT to analyze the correlation between the characteristics of the CNN backbones and their effectiveness. VGGNet50 is one of the most widely used CNN architectures, characterized by its simplicity. This model consists of several VGG blocks, and each block performs a few times of feature extraction with convolution filters and dimension reduction with max pooling. We used \(3\times 3\) convolution filters and \(2\times 2\) max pooling. We added batch normalization and dropout after each convolution layer to avoid overfitting and to make the model focus on general features of acoustic reflection signals. This approach was applied to all the CNN backbones. Finally, we stacked one convolution layer, three VGG modules, another convolution layer, pooling layers in between, and four FC layers. VGGNet using \(3\times 3\) filters allows us to capture high-resolution features, meaning that fine temporal structures or high-frequency components in the spectrogram can be extracted more sensitively. Additionally, the architecture was determined through empirical experiments while preserving the basic structures of each model.

InceptionNet51 consists of multiple inception modules that apply various sizes of convolution filters to feature maps from previous layers. This allows the inception modules to analyze temporal changes in the magnitude of frequency components at multiple scales. In other words, InceptionNet can extract features at multiple scales simultaneously, enabling the model to capture both local and global patterns present in the spectrogram. We used the inception module proposed by Szegedy et al.51, which performs \(1\times 1\), \(3\times 3\), and \(5\times 5\) convolutions and \(3\times 3\) max pooling in parallel in each layer. By comparing the effectiveness of VGGNet with InceptionNet, we show whether multi-scale analysis is effective in capturing changes in the amplitude of acoustic signals caused by particle concentrations in fluids. We constructed the top layers of InceptionNet by stacking three consecutive 3\(\times\)3 convolutional layers followed by max-pooling layers to reduce dimensionality. Below this, we further employed a sequence of 1\(\times\)1 and 3\(\times\)3 convolutional layers, downscaling the dimensions once more. Then, we implemented five different types of inception modules within our architecture. The first module is consistent with the proposal of Szegedy et al.51. The second and third modules use 1\(\times\)1 convolutions followed by 3\(\times\)3 and 1\(\times\)1 followed by 7\(\times\)7 convolutions, respectively. Additionally, the fourth module employs 1\(\times\)1, 3\(\times\)3, and 7\(\times\)7 convolutions, while the last module employs 1\(\times\)1 and 3\(\times\)3 convolutions. The model architecture was constructed by vertically stacking the first module over three layers, the second module as a single layer, and replicating the third module four times. Subsequently, the last and fourth modules were positioned in sequence, and finally, the last module was reused, resulting in a total composition of 11 inception modules in a row without branching for subtasks.

ResNet52 is characterized by residual connections that allow stacking a larger number of convolution layers by reminding initial features. One residual block consists of two convolution layers and one residual connection, which adds the input feature maps of the block to the output feature maps of the second convolution layer. We used \(3\times 3\) convolution filters and \(2\times 2\) max pooling. Since the spectrograms have higher resolution (\(1024\times 250\times 1\)) than ordinary image data, we supposed that a larger number of layers will be required to extend the receptive fields of the model to discover correlations between distant frequency components in terms of time and frequency. We stacked 8 residual blocks with a stride of 2. Using residual blocks, ResNet can enhance the depth of the network. Deep networks effectively capture complex relationships and dependencies within the spectrogram data, leading to a more comprehensive understanding of the collected reflection signal.

EfficientNet53 is a model designed to improve the efficiency of CNN backbones by optimizing their depth (i.e., the number of layers), width (i.e., the number of channels), and resolution (i.e., the size of feature maps). The model consists of MBConv (Mobile inverted Bottleneck Convolution) blocks, as proposed by Tan et al.53. EfficientNet’s MBConv block can capture the generalized features of the spectrogram inputs by simultaneously adjusting the network’s depth, width, and resolution. MBConv consists of a \(1\times 1\) convolution layer, \(3\times 3\) (or \(5\times 5\)) depth-wise convolution layer, another \(1\times 1\) convolution layer, and a residual connection, consecutively. This block does not apply an activation function at the last convolution layer to preserve information in the dimensional reduction and uses depth-wise convolution and inverted residuals, which increases the width (the number of channels) in the convolution layers and reduces it before the residual connection, to reduce the computational and spatial complexity. In addition, each MBConv block has a Squeeze-and-Excitation (SE) block, which is designed to condense critical channel information into a feature map through a squeezing process, followed by the computation of channel-wise dependencies through an excitation process. The SE block architecture includes successive stages: global average pooling, a fully connected layer, a rectified linear unit (ReLU), another fully connected layer, and finally a sigmoid layer, sequentially. This sequence allows the scaling of channels according to their importance, which is achieved by applying the sigmoid function point by point after the linear transformation of the channel descriptor vector. We implemented the EfficientNet by stacking 14 MBConv blocks. We compare the EfficientNet with the ResNet to investigate whether a sufficiently large receptive field is required to extract the physical properties of objects from acoustic reflection signals, or whether the balance between depth, width and resolution can compensate for this.

All CNN backbones extract feature vectors from the spectrograms, and we fed these feature vectors into fully connected (FC) layers to perform iron oxide particle concentration classification. The activation functions of the FC layers were Softmax on the output layer and ReLU on the other layers, with the Adam optimizer (Fig. 8). For the iron oxide particle concentration classification, the objective of the classification models is to minimize the cross entropy between the ground truth class and the output predicted class, given by

$$L\left(X,\,c\right)=\,-\log \left(\frac{\exp \left(p\left(X,c\right)\right)}{\sum _{a}\exp \left(p\left(X,a\right)\right)}\right)$$
(7)

where \(p\left(X,a\right)\) is the probability that the classification model assigns the class \(a\) to the input reflection signal \(X\), and \(c\) is the ground truth class.

Fig. 8: Architecture of the four CNN backbones.
figure 8

VGGNet, InceptionNet, ResNet, and EfficientNet.

Full size image