Self-normalized density map (SNDM) for counting microbiological objects

The statistical properties of the density map (DM) approach to counting microbiological objects on images are studied in detail. The DM is given by U\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^2$$\end{document}2-Net. Two statistical methods for deep neural networks are utilized: the bootstrap and the Monte Carlo (MC) dropout. The detailed analysis of the uncertainties for the DM predictions leads to a deeper understanding of the DM model’s deficiencies. Based on our investigation, we propose a self-normalization module in the network. The improved network model, called Self-Normalized Density Map (SNDM), can correct its output density map by itself to accurately predict the total number of objects in the image. The SNDM architecture outperforms the original model. Moreover, both statistical frameworks—bootstrap and MC dropout—have consistent statistical results for SNDM, which were not observed in the original model. The SNDM efficiency is comparable with the detector-base models, such as Faster and Cascade R-CNN detectors.

www.nature.com/scientificreports/ is an example of the Bayesian-inspired approach. In contrast, the bootstrap technique is an excellent example of the non-Bayesian method 19 , which statisticians have successfully exploited for years. Presently, it is widely adapted for shallow and deep NN 20 . This paper aims to consider one of the practical (microbiological) applications of DL systems-a problem of counting microorganisms on images of Petri dishes. Our goal is to perform the error analysis, which will allow us to understand the validity of the counting system and let us propose the significant model's improvements. Indeed, our main achievement is a proposal of the self-normalization mechanism, implemented as additional modules in the network architecture. A counting system with self-normalization modules works more efficiently, accurately, and in the broader domain than the vanilla model.
There are many different approaches to object counting on images. An obvious way to face this type of problem is to build a regression model 21,22 . The biggest advantage of this method is that it is enough to label each image with the number of objects in it. If a dataset includes more detailed annotation with a bounding box for every object in an image, it is feasible to leverage detectors for object counting [23][24][25] . Eventually, an autoencoder can be used to estimate the density map (DM) based on a given image, which can be later integrated to obtain the number of objects [26][27][28][29][30] . It requires objects to be labeled with the coordinates of their center. The latter two methods performed better on the dataset we worked on in our experiments. The main goal of this study is to perform a comprehensive statistical analysis for a deep learning solution. Thus, we decided to focus on the density map approach because the architecture of autoencoders is usually less complex than recent neural networks for object detection, and the methods introduced in this paper are more transparent.
Counting microbial colonies on Petri dishes is an essential step in a microbiological laboratory to evaluate the cleanliness of the samples. Traditionally, this task is done manually or semi-automatically using traditional computer vision methods 31,32 . However, recent studies prove that a DL-based methodology accelerates the process 23,25,29,[33][34][35] . We shall utilize the DM method to predict the number of microbial colonies on a Petri dish. The DM system is defined by U 2 -Net DL model 36 . To test the DM models, we utilize the AGAR (Annotated Germs for Automated Recognition) dataset 23 , which includes several thousand pictures of Petri dishes with five species of microbial colonies grown.
The vanilla U 2 -Net network accurately predicts the number of microbes up to about 60 colonies but underestimates the counts above. It also overestimates its prediction for some samples with just a few colonies. We adapt two known statistical approaches to perform an error analysis of the U 2 -Net predictions. Namely, we implement the bootstrap and MC dropout frameworks, respectively. The analysis of the model's uncertainties indicates that the main problem with the performance of U 2 -Net is not the localization of microorganisms itself, but rather the normalization of density maps for a large range of possible outcomes (from 0 to 100 colonies on a Petri dish). Thus, we propose a modification of the U 2 -Net, called Self-Normalized Density Map (SNDM) architecture, which includes extra normalization layers significantly improving the system's performance in the previously failing domain. Moreover, the SNDM works as efficiently as detector-base approaches, such as Faster 37 and Cascade 38 R-CNN.
The paper is organized as follows: in second section the DM and U 2 -Net architecture is introduced and the AGAR dataset is described; third section reviews two statistical approaches for neural networks designed to estimate the uncertainties, namely, bootstrap and MC dropout; in fourth section we present our numerical results. The SNDM is introduced, and its performance is studied in detail. Conclusions are drawn in the last section.
Density map and microbiological data U 2 -net as density map. The DM, denoted by M , transforms the given input image, x , into a density map target Note that M is parametrized by weights and hyperparameters, which we intentionally omit in notation because they are not relevant to our discussion. In our approach, the total number of objects in the image is obtained by summing over the entries (the pixels) of the density map y.
The convolutional neural network (CNN) has been designed to face problems such as image recognition, image classification 39,40 , and segmentation. Hence the CNN-based models are also proposed to transform the image into a density map. Among several types of CNNs, architecture U-Net-type 41 is the most successful in reproducing the density map of the objects 30 . In the present paper, we discuss the results of the analysis in which the U 2 -Net 36 is utilized.
The U 2 -Net has a form of a two-level nested U-shaped structure as presented in Fig. 1. The top-level is built of eleven U-Net blocks, connected by pooling (encoder branch) or upsampling layers (decoder branch), with additional concatenation connections. On the bottom level, each U-Net block itself also has an encoder-decoder structure, within additional residual connections in the U-Net block.
The analysis is performed on the AGAR dataset, including images of Petri dishes with bounding box annotations for each colony. To generate density maps, we assume that the center of objects lies in the center of the bounding box, and corresponding coordinates are added to an empty map in the form of single pixels. Next, the map is blurred with a Gaussian filter, which leads to a smooth density map normalized so that the sum over all pixels gives the total number of colonies. In that way we create the annotation map for each input image. The autoencoder is trained to transform a real image into a density map. Note that each density map is normalized so that the sum over all pixels gives the total number of objects. Hence the network predicting the density map for an input picture also provides information about the total number of objects.
(1) y = M(x). www.nature.com/scientificreports/ AGAR dataset. As mentioned before, we utilize the AGAR dataset 23 to test the properties of the U 2 -Net in the MC dropout and bootstrap approaches. It is a huge microbiological database including 18 000 highresolution images of Petri dishes with cultured colonies of 5 standard microbes taken under diverse lighting conditions. Professional microbiologists manually annotated microbial colonies by precisely labeling each colony using a bounding box. AGAR is diverse data collected to automate microbial colony counting. The data covers edge cases, including highly crowded plates with overlapping colonies, cases with extensive and tiny colonies, and cases where colonies are hardly visible, e.g., when located close to the dish edge. Typical plates captured in the various setups (three different illumination conditions) with microbial colonies of different shapes and sizes are shown in Fig. 2. Note that AGAR mainly contains images with fewer than 50 colonies, about 85% of samples (see Fig. 3), or even empty dishes, which are essential for testing the models' tendency to generate false-positive counts.  www.nature.com/scientificreports/

Uncertainties in DM
Some methods of estimation of uncertainties in the predictions of the DM, for the U-Net architecture, have recently been discussed by Eaton-Rosen et al. 42 . Our paper focuses on two approaches: bootstrap and MC dropout adapted for U 2 -Net. We briefly review both of them in the following two subsections.

Bootstrap method.
A bootstrap method for error estimation is one of the oldest and the most popular methods of statistics 19 . An extensive review can be found in the work of Hastie et al. 43 , whereas Tibshirani 44 discusses the application of the bootstrap technique in the neural network analyses. Two types of bootstrap approaches are distinguished, namely, pairs sampling and residual sampling algorithms. We adapt the first option.
The main idea of the bootstrap is to train the same model on some number of data subsets sampled (with replacement) from the original set. Then, as a result, one obtains the ensemble of models. In the inference mode, one makes predictions of all models in the ensemble. The bootstrap model prediction is then given by the mean over the model predictions in the ensemble. Additionally, the variance over the models' outcomes gives 1σ uncertainty.
The procedure of taking the average over the models is robust to overfitting. In our applications, we consider B = 20 bootstrap training subsets. The same network architecture is trained for each subset, and the best model is selected using the validation subset. If the noise of the data is random and not correlated (from sample to sample), then the total error obtained for the ensemble of models is B times smaller than the error of one model. For the detailed explanation, see Bishop 45 . Note that the bootstrap approach is not a Bayesian approach, but if it is combined with early stopping, it is interpreted as approximate inference 46 .
The known strengths of the bootstrap approach are the following: • It is simple in implementation and relatively fast in execution, i.e., it is usually enough to run the analysis for B = 10 to 20. • The obtained model is given by the ensemble of networks of the same architecture but with a different configuration of network parameters. • The predictions of the bootstrap model are similar to the Gaussian process (GP) in data range 20 . Indeed, the GP model is defined by some density distribution used to sample the model. The best model prediction is given by the average over the outcomes of the model samples.
The mean overall models give the response of the bootstrap model. Therefore the bootstrap model should not overfit the data. Indeed, a single fit obtained for a particular subset of the data might be overtrained, but the mean over the models is not. Hence the bootstrap model should be characterized by a good generalization ability.
The main steps of the bootstrap approach: (i) consider the training data set D train with the total number of samples equals N; (ii) obtain B subsets C i ⊂ D train with an equal number of images. C i is obtained by sampling from the data D (with replacement) b images. Note that in our experiments B = 20 , moreover b ≈ 0.63 N as it is recommended in the literature 43 ; (iii) consider network M and train it on C i data subset, as the result one gets the model M B i ≡ M(C i ); (iv) to make the predictions for given input x compute the outcome of each network M B i and take the mean, namely:  MC dropout. In 1995, Neal proved that a class of neural networks with a single hidden layer and an infinite number of units converge to the Gaussian process 15 . Moreover, Rasmussen and Williams 47 studied Gaussian process methods in supervised learning. In both cases, the Bayesian statistics stand a background for all derivations. It is believed that the GP approach allows accessing the model prediction uncertainties with reasonable accuracy. Motivated by the Bayesian statistics as well as the success of GP techniques in the estimation of model uncertainties, Gal and Ghahramani studied the dropout technique 18,48 . They showed that a deep neural network with a dropout layer after each weight layer can be understood as the Gaussian process.
The main idea of the approach is to keep dropout layers active in the training and inference modes. It means that, for a given input x , every prediction is computed with a slightly different configuration of active units. Therefore, to obtain the MC dropout prediction for a given neural network, one has to run it N-times. Then, the mean over the predictions gives the best value while the variance estimates the uncertainty of the predictions.
The main steps of the MC dropout: (i) Consider a network M dropout ; (ii) Train the M dropout on the data set D train . The training is run with regularization-we apply Adam algorithm with weight decay µ = 0.0005 . Note that during the training the dropout layers are active. (iii) To make a prediction for given input vector x , run the network M dropout , r = 20 times, keeping dropout layers active. Then the dropout network prediction is given by the where M i dropout (x) denotes the i-th run prediction of the network. The 1σ uncertainty is given by the square root of the variance:

Numerical results
This section examines the properties of the U 2 -Net architecture as a density map predictor. The results of these studies allow us to propose the self-normalization mechanism in the U 2 -Net model. We justify the relevance of this approach in the following sections.
To study the U 2 -Net properties, we consider the higher-resolution subset of the AGAR dataset, represented by RGB images of Petri dishes with a resolution of about 4000 × 4000 pixels. We limit our discussion to images containing no more than 100 colonies. Because, from the microbiological point of view, the accuracy of automatic counting is less relevant for more populated samples, i.e. determining whether the sample is contaminated or not is the most important.
The analyzed data is divided into three subsets: training (5656 images), validation (706 images), and test (706 images) data subset. As introduced earlier, we consider two types of statistical approaches for estimating uncertainties in predictions of deep neural networks: the bootstrap and the MC dropout. In both methods, we exploit the same network architecture shown in Fig. 1. Moreover, the same training scheme is adapted for both models-the Adam optimization algorithm with learning rate of 0.0001, weight decay equals 0.0005, and the size of the minibatch of two. The training takes at least 100 epochs.
We ran the computations on NVIDIA TITAN RTX GPU with 24 GB of memory. The training time was about 40 h for MC dropout. The bootstrap experiments took longer, and the training lasted about four days, performed in parallel on five GPUs.

Performance of vanilla U 2 -Net as a density map predictor.
To access the quality of both discussed models, we compare the predictions of the network versus the ground truth computed on the test data set; see the left column in Fig. 4. Each predicted number of colonies is given with 1σ uncertainty. Note that both statistical models work well for the samples with the number of colonies smaller than 60. However, for images with the number of colonies lower than 10, the DM count is overestimated. Predictions with tiny uncertainties characterize the MC dropout for these cases. Consequently, the model disagrees with the data within the 1σ confidence level. In contrast, the bootstrap model predictions have larger uncertainties. Hence, the model agrees with the ground truth in the 1σ level. From that perspective, the bootstrap approach works better than the MC dropout.
The detailed examination of our results shows that the uncertainty of the model's predictions depends on the mean size of colonies for a given dish. It is illustrated in Fig. 5 (left panel). Interestingly, for the images with www.nature.com/scientificreports/  www.nature.com/scientificreports/ smaller colonies, the uncertainties for the bootstrap approach are larger than for the dropout model. However, when the size of the colonies increases, the effect is the opposite. We noted that the uncertainties of the model's prediction (normalized to the number of colonies) weakly depend on the dish coverage (the fraction of the dish covered by colonies), as shown in Fig. 6 (left panel). It is interesting to note that the MC dropout uncertainties are systematically larger than for the bootstrap approach, see Table 1, where the average (over the test dataset) uncertainties are given.
In Fig. 7, we showed four panels. Each contains an input picture, a map of annotations, network prediction, and a map of uncertainties computed for each output pixel. We show the results for two dishes that contain 16 and 72 bacterial colonies, respectively. The DM prediction agrees with the ground truth in the first case (first panel). In contrast to the second case (third panel), the DM underestimates the count. However, in the latter case, the DM correctly reproduces spots representing the objects in the image, but the most significant uncertainties are consistently observed in the spot positions. Therefore, we conclude that the DM reconstructs the map of spots (objects) correctly but has a problem finding the correct normalization for the images with a larger number of colonies. Therefore, we propose a modification of the network architecture so that the network can correct by itself the normalization of the output.

Self-normalized DM (SNDM).
To propose the normalization module in the network, we must explain how the loss function is constructed in the U 2 -Net model.
Let us denote by E U 2 the loss function U 2 -Net. It is a function of the data and model parameters. From the statistical perspective, it refers to the logarithm of the likelihood function. In practice, E U 2 has a quite complicated structure 36 : where P M and P G denote predicted density map by M-th U-Net block and the ground truth, respectively.
The L M is the standard binary cross entropy function: where i, j denotes the pixel indexes in the figure of the width W and height H. The U 2 -Net generates six output maps, denoted by S 1,...,6 , from the encoder ( S 6 ) and five from the decoder branch ( S 1,...,5 ), respectively. Outputs from smaller blocks ( S 2,...,6 ) are next upsampled to fit the size of the output map. Note that S 1 already has the proper 1008 × 1008 size. After upsampling (we still denote them by S i ), each of them gives the density map and P i = σ (S i ) , where σ is the sigmoid activation function. All maps P 1..6 contributes to the loss function. Moreover, S 1..6 maps are fused with a concatenation operation followed by a 1×1 convolution layer (giving S 0 ). Then P 0 = σ (S 0 ) is the final density map output of the U 2 -Net. The P 0 contributes also to the loss function.
Our idea is to allow the network to correct the normalization of output pixels in S i maps. Since every output pixel ranges from 0 to 1, one can not simply re-scale the P i -outputs by some normalization number. Therefore, we propose the rescaling of pixels normalization in S i map. We use the property that sigmoid is a linear function on neighborhood, namely, σ (x) ≈= 1 2 + x 2 + . . . . Hence, the rescaling of the sigmoid argument changes the total normalization of the sigmoid almost linearly in this range. In practice, we rescale each S i output by a factor β , so www.nature.com/scientificreports/ The normalization value β might depart from one, and it can be a function of the distribution of colonies, the total coverage, etc. Therefore, we assume that β depends on the input image and is an additional network outcome. To limit the number of parameters that define β we connect it with the smallest U-Net block of U 2 -Net, the encoder part, see Fig. 1. In the rest of the paper, the U 2 -Net architecture with our modification will be called a self-normalized density map (SNDM).
In principle, the system should work without the need for corrections, β = 1 , hence we expect that β should take values around one. To keep the control β-dependence, we add to the loss term a penalty term: In general β can take any positive number, but, in practical applications, we assume that β =σ (. . . ) = σ (. . . ) to speed up the process of training. It guarantees that normalization ranges from 0 to 1.5. In our experiments, we tested a sigmoid with = 2 and σ = ReLU . However, the most optimal results have been obtained for the sigmoid activation function with = 1.5.
In Fig. 4 (right column of panel), we plot the predictions of SNDM versus ground truth. We see that modifying U 2 -Net architecture by introducing the self-normalization module significantly improves the network's predictions. Indeed, the SNDM works well for images containing less than 60 objects and samples with the number of objects larger than 60. Interestingly, in the improved model, the bootstrap and MC dropout predictions are in agreement. The estimated uncertainties for both approaches became similar, see Figs. 5 and 6 (right panel), as well as Table 1. Moreover, the bootstrap uncertainties for SNDM are reduced with respect to the original model.  www.nature.com/scientificreports/ The normalization parameter β depends on the features of the input image, such as the number of colonies. It is illustrated in Fig. 8. For both statistical approaches, β varies from 0.96 to 1.06 with the tendency to take lower values for less crowded images. On the other hand, for the images containing more than 50 objects, β varies less. It is also worth mentioning that β takes a wide range of values for a given number of colonies, which varies with the number of colonies, which translates to coverage and other features like mean colony size. It is visible in Fig. 9 where we plot the dependence of β on the dish coverage. When we group the samples by the mean colony size (indicated by different colors of points in Fig. 9), one can observe that β almost linearly depends on coverage. Moreover, the slope of the fitted lines increases gradually with the mean size of colonies. Figure 7, shows density maps together with pixel uncertainties for two input pictures with 16 and 72 objects, respectively. The predictions for the SNDM are shown in the second and the fourth panel, respectively. We see Figure 8. Dependence of the normalization parameter β on the number of colonies. Uncertainties of the β parameter calculated using bootstrap method (left) and MC dropout (right). Results for the test subset. Table 1. Average uncertainties of predictions for the test dataset within the DM and the SNDM models in the bootstrap and MC dropout approaches (as presented in Fig. 4). Uncertainties averaged in a group of less (more) crowded dishes with ≤ 50 ( > 50 ) colonies and overall. Eventually, we note that, overlapping objects is one of the difficulties the counter system must face. In Fig. 7, we see an example of colonies that overlap. In this case, the amplitude of density maps is naturally enlarged. But the interesting thing is that also uncertainties have higher values there-see the pixel uncertainty maps in the last column of Fig. 7. Therefore, we may conclude that these areas have a more substantial impact on counting uncertainties. However, if one compares DM and SNDM pixel uncertainty in the overlapping regions, then in the SNDM case, uncertainties are slightly reduced compared to DM.
The detailed inspection of the results presented in Fig. 4 shows that there are input figures for which the SNDM network overshoots. In Fig. 10 we plot the predicted map with uncertainties for such a case. We see that the network's main problem is the presence of colonies of different sizes on the same dish. It complicates the inference of the normalization parameter, which also depends on the size of the colonies, see in Fig. 9. The blue boxes in Fig. 10 represent ground truth labels showing the precise position and size of the colonies. It should be also noted that the pixel uncertainties for small colonies in the small box area are much larger than for big colonies located in bigger boxes.
We close the discussion of the numerical result by comparing the SNDM model predictions with the other popular counting systems. In one of our previous papers 23, 25 we discussed some of them. For the comparison, we choose two of the most successful approaches (in our experiments) to count AGAR microbiological colonies, namely, Faster R-CNN detector 37 and Cascade R-CNN detector 38 . Note that, for both models, the algorithm first performs the detection of interesting objects and then counts them.
To quantitatively compare various models predictions, we consider a standard Mean Absolute Error: and less common in usage, symmetric Mean Absolute Percentage Error: where N is a number of all samples, n i is true count of microbe colonies present on i-th image, and ñ i is predicted number of colonies. Note that the sMAPE shows the relative error with respect to the overall number of colonies. Both metrics are obtained for the higher-resolution subset of the AGAR dataset and the four models: Faster and Cascade detectors, as well as DM and SNDM in the MC dropout approach. The metrics are given in Table 2. We see that counting by detection gives better results than the standard DM model. But the SNDM gives comparable or even slightly better counting results than the detectors. Further discussion of other deep learning detectors, e.g., well-known YOLO, can be found in our previous paper 24 . Note that preparing annotations for the density map models are less expensive than for detector models. Moreover, the training of the detector systems is more resources-demanding and time-consuming than the density map models. Therefore, for the data for which the density map approaches work with the same or better efficiency as the detectors, the density map systems as SNDM are recommended.

Summary
We have shown that the detailed analysis of the uncertainties in the deep neural network model leads to a deeper understanding of its limitations. It allows us to assess how uncertain the predictions are. More profound knowledge of the model allowed us to modify the DM model by introducing the Self-Normalization mechanism. The SNDM significantly improves the DM predictions. Indeed, before the modification, the model underestimated the number of colonies for highly populated dishes. In contrast, for some images with a small number of colonies, the Figure 10. Density maps of one of the outliers from Fig. 4 predicted with the SNDM. The model predicts 29 ± 7.30 colonies, while the sample contains 77 colonies of two types: 19 P. aeruginosa (big ones), and 58 C. albicans (small ones). The blue boxes represent all the colonies' ground truth position and size. Uncertainties calculated using the bootstrap method.

Data availability
The AGAR dataset used during the current study is available from https:// agar. neuro sys. com/ on reasonable request.