A new method to control error rates in automated species identification with deep learning algorithms

Processing data from surveys using photos or videos remains a major bottleneck in ecology. Deep Learning Algorithms (DLAs) have been increasingly used to automatically identify organisms on images. However, despite recent advances, it remains difficult to control the error rate of such methods. Here, we proposed a new framework to control the error rate of DLAs. More precisely, for each species, a confidence threshold was automatically computed using a training dataset independent from the one used to train the DLAs. These species-specific thresholds were then used to post-process the outputs of the DLAs, assigning classification scores to each class for a given image including a new class called “unsure”. We applied this framework to a study case identifying 20 fish species from 13,232 underwater images on coral reefs. The overall rate of species misclassification decreased from 22% with the raw DLAs to 2.98% after post-processing using the thresholds defined to minimize the risk of misclassification. This new framework has the potential to unclog the bottleneck of information extraction from massive digital data while ensuring a high level of accuracy in biodiversity assessment.

In the context of accelerating human impacts on ecosystems 1 , the capacity to monitor biodiversity at large scale and high frequency is an urgent although challenging goal 2 . This urgency resonates with the ambition of international initiatives like the Group on Earth Observations Biodiversity Observation Network (GEO BON) and the call for monitoring Essential Biodiversity Variables (EBVs) 3,4 .
Remote sensors are rapidly transforming biodiversity monitoring in its widest sense from individuals 5 to species and communities of species 6 . In the last decade, satellites 7,8 , drones 9, 10 , camera traps 6 , or underwater cameras 11,12 have been extensively deployed to record pictures or videos of aquatic and terrestrial organisms. For instance, satellite data can be used to track whale shark movements 13 or detect whales 14 while photos from airborne or underwater vehicles can deliver accurate density estimations of vulnerable organisms like mammals or sharks 15,16 .
Such massive records are also used by citizen science programs with for example public tools like inaturalist. org which share pictures and associated metadata, or fishpix (https ://fishp ix.kahak u.go.jp) which offers the possibility to upload individual fish images that are then identified by experts at the species level.
However, processing photos or videos to identify organisms is a highly demanding task, especially in underwater environments, where some particular contexts add many difficulties (e.g., visual noise due to particles and small objects, complex 3D environment, color changing according to depth, etc.). For instance, identifying all fish individuals on videos may take up to 3 h of expert analysis per hour of video 17 . Under the avalanche of new videos and images to analyse, alternatives to fish identification by humans and trained experts must be found.
Recently, an effort to use machine learning methods 18,19 and deep learning algorithms (DLAs) for ecological analysis have been made, thanks especially to computer-vision challenges on public databases of annotated open 1 MARBEC, Univ of Montpellier, CNRS, IRD, Ifremer, Montpellier, France. 2  www.nature.com/scientificreports/ photos or videos (e.g. for fish, Fish4Knowledge database (https ://group s.inf.ed.ac.uk/f4k/) and Seaclef challenge (https ://www.image clef.org/lifec lef/2017/sea)). The last generation of DLAs offer much promise for passing the bottleneck of image or video analysis through automated species identification [20][21][22][23] . DLAs, and particularly convolutional neural networks (CNNs), simultaneously combine the automatic definition of image descriptors and the optimization of a classifier based on these descriptors 24 . Even though DLAs usually have a high accuracy rate, they do not provide information on the confidence of the outputs. Hence, it remains difficult to identify and control potential misclassifications which limits their application.
Misclassification of images has two types of consequences for biodiversity monitoring. On one hand, if all individuals of a given species occurring in a given community are erroneously labelled as another species also occurring in the community, this species will be incorrectly listed as absent (false absence). The risk of missing present species because of misclassification is the highest for rare species, i.e. those with the lowest abundance in terms of the number of individuals per unit area. Yet missing these rare species can be critical for ecosystem heath assessment since some play important and unique roles like large parrotfishes on coral reefs 25 while others are invasive like the lionfish in Eastern Mediterranean Sea 26 . In addition, since most species in a community are represented by a few individuals 27 , such misclassifications could significantly lead to the underestimation of species richness. The other error associated with misclassification is when an individual of a given species is mistaken for another species not present in the community (false presence). Such misclassifications could lead to an overestimation of the abundance or geographical range of a species as well as it could artificially increase species richness, unless a species is consistently mistaken for another. Since biodiversity monitoring should be as accurate as possible, automated identification of individuals on images should provide high correct classification rates (close to 100%) even if a subset of images has not been classified by the algorithm with sufficient confidence and must be identified by humans a posteriori.
Chow 28 was the first to introduce the concept of risk for a classification algorithm. For instance, a clustering algorithm classifying an object placed in the center of a given cluster would present a low risk of misclassification, while classifying an object placed on the edge of a cluster would be highly risky. Chow 28 proposed a classification framework, which contains n + 1 channels as outputs, n channels for the n classes considered and an additional channel called the "rejection" channel. When the risk of misclassification is too important, the algorithm rejects the classification.
Applied to machine learning, a first method consists in learning a rejection function during the training, in parallel to the classification learning [29][30][31] . Another method, called a meta-algorithm, uses two algorithms, one being a classifier, and the other one analyzing the classifier outputs, to distinguish predictions with a high risk of misclassification from those with a low risk 32 . A recent comparative study suggests that meta-algorithm-based methods are the most efficient 33 .
An extension of meta-algorithms to control the risk of misclassification is to calibrate models obtained through Machine Learning and Deep Learning algorithms. Machine Learning methods usually produce wellcalibrated models for binary tasks 34 . The calibration consists of a matching between the score predicted by the machine-learning model and the real probability of true positives. While Deep Learning models produce more accurate classifications than other Machine learning models, these models are not well calibrated, and thus need a re-calibration to be used for real-world decisions 35 . Several propositions have been made to improve the calibration of Machine Learning models through the post-processing of outputs. The Platt scaling 36 , the Histogram binning 37 , the Isotonic Regression 38 and the Bayesian Binning into Quantiles 39 are mapping the model outputs to real accuracy probabilities. More recently, Temperature Scaling, an extension of the Platt Scaling, was used to calibrate Deep Learning models using a single parameter for all classes 35 . This parameter is used, instead of the traditional softmax function, to convert the vector output from the neural network into a real probability.
However, such calibration methods are based on a discretization of the Deep model outputs into bins. Many bins are not useful as they only contain a few outputs with low values, whereas many high values fall in the same bin and are thus not discriminated. Moreover, the choice of the number of bins is left to the user, and therefore is not optimized to the Deep model nor to a specific application 40 .
In this paper, we present a simple, yet efficient method that accounts for uncertainty in the classifier outputs. Unlike calibration methods, our approach is not changing algorithm outputs. Instead, we simply assess the behaviour of the model thanks to a validation dataset. We can then set-up a fine tuned threshold per class, allowing us to take into account that the model confidence can be highly variable between "easy" classes and "difficult" classes. Then, through the addition of a new class "unsure", corresponding to predictions with scores lower than the predicted class threshold, we can control the coverage (total amount of images automatically identified) and misclassification rates. We applied this framework to classify 20 species of coral reef fishes in underwater images and assessed its efficiency for 3 real-case scenarios.

Material and methods
Data. We decided to build our own dataset instead of using existing datasets (e.g. Fish4Knowledge: https :// group s.inf.ed.ac.uk/f4k/), to be in phase with quality of videos currently used by marine ecologists. We used 3 independent fish images datasets from the Mayotte Island (Western Indian Ocean) to train and test our CNN model and our post processing method. For the 3 datasets, we used fish images extracted from 175 underwater high-definition videos which lasted between 5 and 21 min for a total of 83 h. The videos were recorded in 1920 × 1,080 pixels with GoPro Hero 3 + black and Hero 4 + black. The videos were recorded between 2 and 30 m deep, with a broad range of luminosity, transparency, and benthic environment conditions on fringing and barrier reefs. www.nature.com/scientificreports/ We extracted 5 frames per second from these videos. Then, we cropped images to include only one fish individual with its associated habitat in the background. Thus, images of the same species differed in terms of size (number of pixels), colors, body orientation, and background (e.g. other fish, reef, blue background) (Fig. 1).
We used 130 videos for the training dataset, from which we extracted a total 69,169 images of 20 different fish species ( Supplementary Fig. S1). We extracted between 1,134 and 7,345 images per species.
In order to improve our model, we used data augmentation 41 on native biodiversity and ecosystem. Each "natural" image yielded 4 more images: 2 with increased contrast (120% and 140%) and 2 with decreased contrast (80% and 60%) ( Supplementary Fig. S2). We then horizontally flipped all images to obtain our final training dataset (T0) composed of 691,690 images (Supplementary Table S1).
We then used two independent datasets made of different videos recorded on different days and on different sites than videos used to build the training dataset. The first dataset (T1) contained 6,320 images from 20 videos with at least 41 images per species, and the second (T2) contained 13,232 images from 25 videos with at least 55 images per species (Supplementary Table S1). We then used dataset T1 to tune the thresholds and T2 as the test dataset. This method ensures that our results are not biased by similar acquisition conditions between the training, tuning and testing dataset and hence that algorithm performance was evaluated using a realistic full cross-validation procedure.
Building the convolutional neural network. Convolutional neural networks (CNNs) belong to the class of DLAs. For the case of species identification, the training phase is supervised, which means that the classes to identify are pre-defined by human experts while the parameters of the classifier are automatically optimized in order to accurately classify a "training" database 24 . CNNs are composed of neurons, which are organized in layers. Each neuron of a layer computes an operation on the input data and transfers the extracted information to the neurons of the next layer. The specificity of CNNs is to build a descriptor for the input image data and the classifier at the same time, ensuring they are both optimized for each other 42 . The neurons extracting the characteristics from the input data in order to build the descriptors are called convolutional neurons, as they apply convolutions, i.e. they modify the value of one pixel according to a linear weighted combination of the values of the neighbor pixels. In our case, each image used to train the CNN is coded as 3 matrices with numeric values describing the color component (R, G, B) of the pixel. The optimization of the parameters of the CNN is achieved during the training through a process called back-propagation. Back-propagation consists of automatically changing parameters of the CNN through the comparison between its output and the correct class of the training element to eventually improve the final classifications rate. Here we used a 100-layer CNN based on the TensorFlow 43 implementation of ResNet 44 . The ResNet architecture achieved the best results on ImageNet Large www.nature.com/scientificreports/ Scale Visual Recognition Competition (ILSVRC) in 2015, considered as the most challenging image classification competition. It is still one of the best classification algorithms, while being very easy to use and implement. All fish images extracted from the videos to build our datasets were resized to 64 × 64 pixels before being processed by the CNN. Our training procedure lasted 600,000 iterations; each iteration processed a batch of 16 images, which means that the 691,690 images of the training dataset were analyzed 14 times each by the network on average. We then stopped the training to prevent from overfitting 45 , as an over fit model is too restrictive and only able to classify images that were used during the training.
Assigning a confidence score to the CNN outputs. The last layer of our architecture, as in most CNNs, is a "softmax" layer 44 . When input data passing through the network reaches this layer, a function is applied to convert the image descriptors into a list of n scores S i , with i = {1, .., n}, and n the number of learned classes (here the 20 different fish species), with the sum of all scores equal to 1. A high score means a "higher chance" for a given image to belong to the predicted class. However, a CNN often outputs a class with a very high score (more than 0.9) even in case of misclassification. To prevent misclassifications, the classifier should thus be able to add a risk or a confidence criterion to its outputs.
Assessing the risk of misclassification by the CNN. For a given input image, a CNN returns a predicted class, in our case a fish species. As seen in the previous section, the CNN outputs a decision based on the score, without any information on the risk of making an error (i.e. a misclassification). Following De Stefano et al. 32 , we thus propose to apply a post-processing step on the CNN outputs in order to accept or reject its classification decision. The hypothesis is that the higher the similarity between an unknown image and the images used for the training, the stronger the activation in the CNN during the classification process (i.e. the higher the score is), and thus, the more robust the classification is.
For this method, the learning protocol is thus made of two consecutive steps performed on 2 independent training datasets.
• In the first phase, a classification model is built by training a CNN on a given database T0 (Fig. 2a) • Then, the second phase consists of tuning a risk threshold τ i specific to each class (i.e. each species in our case), noted i, with i ∈ {1, ..., n} , using a second and independent database noted T1 (Fig. 2b).
Computing the confidence thresholds. After the phase 1 (model training phase), for an image X of the threshold tuning dataset processed by the classifier, we obtain an output {C(X), S(X)} , where C(X) is the class (i.e. species, belonging to the trained set of species) with the highest classification score S(X). For this image, we know the ground truth Y in {1, .., n} belonging to the same set of species classes.
So with C(X) being the output class, Y the ground truth class, and #(.) the enumeration function, the standard definition for Correctly Classified images (or true positives) rate of a class i is: We can also write the standard definition of Misclassified images rate (or false negatives) of a class i as: Then, we can extend the Correct Classification rate (CC) and Misclassification (MC) rate of a species i by introducing the thresholds τ i and by adding the Unsure Classification (UC) rate: For each species we have: We can also note that the standard coverage definition (COV, the rate of images for which a classification is given) of a species i can be extend with the introduction of thresholds as threshold τ as: www.nature.com/scientificreports/ We then used this threshold to post-process outputs of the CNN model. More precisely, for a given image, the classifier of the CNN returns a score for each class (here for each fish species). The most likely class C(X) for this image is the one with the highest score S(X) We then compared this highest score S(X) with the computed confidence threshold for this species (τ C(X) ) obtained in the second phase. If the score was lower than the computed threshold that is S(x) > τ C(X) , then the input image was classified as "unsure". Otherwise, we kept the CNN classification. www.nature.com/scientificreports/ The question is now to select "optimal" thresholds {τ i } i=n i=1 based on the dataset T1. This is not straightforward as is it up to user specific objective, such as minimizing MC, maximizing CC, minimizing UC… In the following, we analyze three different goals corresponding to some standard protocols in marine ecology: • The first goal G1 consists of keeping the best correct classification rate while reducing the misclassification error rate. For this, we used two steps. First, we identified the threshold(s) τ which maximizes CC i (τ ) . Since several thresholds could reach this maximum, we get a set of threshold(s) Se g1 . Then, we selected the threshold with the lower MC i (τ ) . This can be mathematically written as: • The second goal G2 consists in constraining the misclassification error rate to an upper bound of 5% while maximizing the correct classification rate. Reaching this goal requires to first find Se g2 the set of threshold(s) such as MC i (τ ) < 5%. If there is none, we considered Se g2 as the set of threshold(s), which minimize MC i . Then we defined the optimal threshold τ i by choosing the one in Se g2 which maximizes CC i : • The third goal G3 consists of keeping the lowest misclassification rate while raising the correct classification error rate (implying a lower coverage). First, we defined Se g3 as the set of threshold(s) τ that minimizes MC i (τ ) . If there were several thresholds with the same minimal value, we chose τ i as the one which maximizes CC i : For a given image X in the test dataset, the classification and post-process is sequential as follows (Fig. 2c): Figure 3. Impact of the post-processing framework on classification of images for a given species and a given threshold. Usually, the classification of an image of class i can either be correct, if the model classifies it as i, or wrong, if the classifier classifies it as j with j = i (a). We propose a post processing to set a confidence threshold for each class to obtain 3 types of results, correct, misclassified, and unsure (b). The goal is then to transform as many misclassifications as possible as "Unsure", while preventing to transform too many correct classifications "Unsure". www.nature.com/scientificreports/ • First, the image is given to the CNN, which outputs a list of scores, including S(X) the highest score obtained by a class. • Second, for the class C(X) (i.e. the class with the highest classification score), the post-processing step estimates the risk of classifying the image as belonging to the class C(X) . If (X) < τ j , the prediction is changed to "Unsure", otherwise, it is confirmed as the class j (Fig. 2c).
The misclassification rate for a species Y = i after post-processing thus equals: and the unsure classification rate equals: First, to assess the effectiveness of our framework, we processed all the images contained in T2 through the DL algorithm, without post processing (threshold tuning + threshold application).
Second, we assessed whether a unique threshold for all the classes was sufficient to separate correct classifications from misclassifications for all species. For this test, we computed the distribution of correct classifications and misclassifications over scores for each species. During this study, we multiplied the softmax scores, which ranged from 0 to 1, by 100, for an easier reading.
Then, to study the impact of the post-processing method in an hypothetical ideal condition, we selected the thresholds based on the dataset T2 and we applied them to the same dataset T2. For this experiment and the following, we also measured both the Correct Classification rate and the Accuracy, defined for a species i as The accuracy varies from 0 to 1, and increases when the number of false positives decreases and the number of true positives increases. Meanwhile, the CC rate varies from 0 to 100, and increases when the number of false negatives decreases and the number of true positives increases.  www.nature.com/scientificreports/  www.nature.com/scientificreports/ Finally, to ensure that the post-processing method was relevant for any real-life application, i.e. that thresholds are defined and tested on independent datasets, we used the dataset T1 for the threshold-setting phase and the dataset T2 for the testing phase. To assess the robustness of our method, we repeated the same experiment while switching the roles of T1 and T2. Note that we limited our experiments to the use of T1 and T2, but that it could be interesting in further work to assess the robustness of this method with datasets composed of less data.

Results
Results of the CNN model classification. The mean rate of correct classification of fish images inT2 by the raw CNN was of 78.0%, with rates of correct classifications per species ranging from 54.4% to 99.1% (sd = 15.16) ( Table 1). These results are the baseline for our following experiments.
Images obtained softmax scores between 41 and 100 with 80% of images classified with a score between 60 and 100 (Fig. 4a). The distribution of correct classifications and misclassifications among scores was highly variable among species (Fig. 4b,c, Table 1).

Benchmark of the threshold fine-tuning method.
For each species i, we computed CC i , MC i , UC i values while varying the threshold. We computed and applied the thresholds on T2, according to Eqs. 8-16. As the score varied from 0 to 99.9, the misclassification rate decreased to 0.9% (Fig. 5). This decrease was mainly compensated by the increasing rate of unsure classifications between 0 and 99.9 of classification scores.
Indeed, the rate of correct classifications experienced little variation along this distribution of threshold scores, remaining between 74 and 78% for threshold scores between 0 and 99.8 and decreasing to 61% for threshold scores > 99.8. However, correct, wrong, and unsure classification rates were highly variable among species (Supplementary Table S2).
The accuracy of the goals G1, G2, and G3 were, on average, higher than the raw accuracy (0.53) with respectively 0.72, 0.89 and 0.94 ( Table 2).
The thresholds showed higher variations among species for G1, with values ranging from 33.46 to 99.97, than for G3 for which values ranged from 99.86 to 99.98 among the 20 species (Supplementary Tables S2, S3).
Application of the method. For a real cross-validation experiment, thresholds were set using T2 and then applied on T1. The correct, wrong and unsure classification rates were very close (difference < 2.6%) to those obtained with the benchmark situation (Supplementary Table S5).
The proposed post-processing was able to decrease the misclassification rate by at least 10.05%, for all goals, and 19.02% at most compared to the raw output of the Deep Learning model (Fig. 6b). The accuracy followed the same tendency, with an average accuracy for G1, G2 and G3 respectively equal to 0.74, 0.81 and 0.92 (Table 3).

Discussion
Biodiversity monitoring is experiencing a revolution with the emergence of new sensors (light, noise, image, environmental DNA) that generate massive datasets and require powerful and accurate treatment tools. Indeed, species misclassifications must be controlled and limited to avoid false negatives or absences i.e., missing species that are actually present and false positives or presences i.e., detecting species that are actually absent.
In this paper, we demonstrated that the risk of misclassification by CNN algorithms can be measured and controlled in a post-processing step to provide more accurate identification of species on pictures. Such postprocessing can be applied with any classifier as long as the output is a vector of scores. Reducing the misclassification rate is at the detriment of the correct classification rate and increases "unsure" classifications, which implies a low coverage and a greater human effort needed to identify unclassified individuals. Hence, there is a trade-off between a more secure (less misclassifications) or a more automatic (more classifications) method so species thresholds can be set according to the goal or priority of the study or the availability and time of experts. Here Table 2. Accuracy of the models without post-processing, and with post processing according to our goals, with thresholds tuned and applied on T2. Each line shows the result for a species, with: the species name, the accuracy of the model without post processing, and the accuracy of the model with post processing according to the 3 goals defined earlier. www.nature.com/scientificreports/ we define three main goals which represent archetypal study cases. The first goal, maximizing the correct classification rate but not limiting misclassifications, can be applied when avoiding false negatives is more important than detecting false positives. This can be the case for monitoring invasive species, since the priority is to detect the first occurrence of such invasive individuals with potential deleterious consequences on native biodiversity and ecosystem functioning 47 particularly on islands 48,49 . For instance, the Indo-Pacific predator lionfish (Pterois volitans and P. miles) has invaded most reefs of the Western Atlantic and depleted many native prey populations, and are starting to spread in the Eastern Mediterranean Sea 12 . To better anticipate the impact of such species, ecosystem mangers needs to be aware of the first occurrence on reefs and can thus accept having "false alarms". The same constrains applies for detection particular or emblematic individuals, like Whale Sharks, through photoidentification 50 where the primary goal is to avoid missing an occurrence. In both ecological cases, experts will eventually validate the few false positive identifications of targeted organisms by the algorithm to discard them. The second goal, maximizing the correct classification rate while limiting misclassifications at 5% maximum per species, can be applied when avoiding false negatives and false positives are both important. This is the tradeoff scenario that requires the least human effort and that can process massive datasets with few errors. It can be recommended to analyze long videos (> 2 h) for monitoring biodiversity metrics that are weakly influenced by undetected species (rare or classified as "unsure"), like the assessment of taxonomic or functional diversity 25 , and that can feed initiatives like the Group on Earth Observations Biodiversity Observation Network (GEO BON) and provide robust estimates of Essential Biodiversity Variables (EBVs) 3,4 .
The third goal, minimizing the misclassification rate, can be applied when detecting false positives is more problematic than avoiding false negatives, which creates many "unsure" classifications. This can be the case when priority is to accurately analyze a relatively small dataset with the support of many experts who can help to identify species on potentially a high number of "unsure" images. For instance, assessing abundance of all species within a given area to explain ecosystem functioning (e.g. 51 ) or to monitor changes in species relative abundances (e.g. 52 ) requires a minimum number of misclassifications.
Whatever the goal, our framework is highly flexible and can be adapted by tuning the species thresholds regulating the trade-off between classification robustness and coverage in an attempt to monitor biodiversity through big datasets where species are unidentified. To unclog the bottleneck of information extraction about organism forms, behaviors and sounds from massive digital data, machine learning algorithms, and particularly the last generation of deep learning algorithms, offer immense promises. Here we propose to help the users to control their error rates in ecology. This is a valuable addition to the ecologist's toolkit towards a routine and robust analysis of big data and real-time biodiversity monitoring from remote sensors. With this control of error rate in the hands of users, Deep Learning Algorithms can be used for real applications, with acceptable and Table 3. Accuracy of the model without post-processing, and with post processing according to our goals, on the cross-validation, with thresholds tuned on T1 and applied on T2. Each line shows the result for a species, with: the species name, the accuracy of the model without post processing, and the accuracy of the model with post processing according to the 3 goals defined earlier. www.nature.com/scientificreports/ controlled error rates, lower than any state of the art fully automatic process, while fixing the effort by human experts to correct algorithm mistakes.