Introduction

Recently, hyperspectral imaging (HSI), as a rapid and non-invasive measurement ability, has generated much interest due to its unique potential to monitor food quality and safety1,2,3. Among various food materials, fish is considered one of the most perishable foods. Consumption of deteriorated fish can seriously affect consumer health. Therefore, to evaluate and monitor the quality and safety of this valuable and at the same perishable seafood, the rapid and non-destructive detection of freshness is a necessary task4,5. Hyperspectral imaging method in combination with different chemometric analysis has been applied to evaluate several freshness indicators, such as Total Volatile Basic Nitrogen (TVB-N), Trimethylamine (TMA)5,6,7, Thiobarbituric acid reactive substances (TBARS)8,9,10, total viable count (TVC)11,12,13, sensory factors5,14,15 and, etc.

Volatile compounds such as trimethylamine, ammonia and dimethylamine are considered as total volatile basic nitrogen (TVB-N), produced as a result of destructive activities of microorganisms and are considered as one of the most important freshness indicators to monitor the quality and safety of seafood products16. During storage time, the change of the TVB-N value of fish fillets caused several chemical variations that can be shown in the hyperspectral imaging data and be applied to measure the TVB-N value.

Although the capability of HSI for evaluating the TVB-N value of fish fillet has already been proved, attempts to improve the prediction power of HSI is still considered as an attractive subject. One of the most important efforts to enhance the performance of hyperspectral imaging technique could be to focus on new chemometric methods. Many data analysis methods such as partial least squares (PLS), random forest, principal component analysis (PCA), support vector machine (SVM), artificial neural network (ANN) and so on have been developed to deal with the large volume of data.

Deep learning, as a powerful machine learning tool, is a promising method to be used in various fields such as medical science, remote sensing, robotic and food safety, etc. This technique dealing with huge data sets, and obtaining suitable prediction power and accuracy. Deep learning is a kind of representation-learning method that refines multilevel representation by the deep artificial neural network composed by multiple layers of neurons. Because of the strong feature learning capability, deep learning method can solve many complicated problems in a rapid and effective manner17. Deep learning models exhibit powerful ability in classification and regression tasks18. There are several studies, using deep learning techniques in the recent literature, which indicates the superiority of this class of machine learning techniques in the food industry by analyzing RGB images and spectra images of food. For example, Yu et al. (2017) used visible and near-infrared hyperspectral imaging (HSI) technique combined with stacked auto-encoders (SAEs) algorithm to classify shrimp into fresh and stale groups according to their TVB-N contents. This method achieved satisfactory total classification accuracy of 96.55 (for the calibration set with 116 samples) and 93.97% (for the prediction set with 116 samples) freshness grade of shrimp19. Al-Sarayreh et al. (2018) compared the performance of HSI data to evaluate adulteration in red-meat by the support vector machines and deep convolution neural networks (CNN) algorithm. Results confirmed that the CNN model has the best prediction power (94.4% classification accuracy)20. Moreover, Yu et al. (2019) used the hyperspectral imaging (HSI: 900–1700 nm) technique to predict the TVB-N value of Pacific white shrimp. They used Successive projections algorithm (SPA) and deep-learning-based stacked auto-encoders algorithms to choose more informative wavelengths. Least-squares support vector machine (LS-SVM), partial least squares regression (PLSR) and multiple linear regression (MLR) were applied to predict TVB-N content. The results demonstrated that the SAEs-LS-SVM was the best model (RP2 = 0.921, RMSEP = 6.22 mg N [100 g]−1 and RPD = 3.58)21 for prediction of this index. Overall, in these researches, deep learning was applied to classify the samples or select optimal wavebands. It is sometimes necessary to evaluate the numerical output of a product's freshness index for better decision making. Therefore, the use of a regression framework can be preferred over a classification one. To the best knowledge, no research has been reported yet on the detection of fish quality by a regression framework of deep learning. Moreover, systematization and meta-analysis of the data extracted from previous studies can help to obtain conclusive results about the best predictive model for evaluating the TVB-N content of fish fillets. Therefore, the discussion of results was established based on meta-analysis and systematic review. Therefore, the specific aim of the study is to (1) investigate the potential of visible and near-infrared (VIS/NIR) hyperspectral imaging technique coupled with deep learning model to predict the TVB-N content of rainbow trout fish, and (2) compare the performance of deep learning algorithm with PLSR and LS-SVM models established in current study and (3) meta-analysis of previous researches on the prediction of the TVB-N value in meat products using hyperspectral imaging coupled with various chemometric algorithms.

Results and discussion

TVB-N value

Changes in TVB-N values of 210 subsamples (30 fish fillets per day) during storage were presented in Table 1. The initial TVB-N content of the rainbow trout fillets was 8.70 ± 0.86 N/100 g, which significantly increased during storage time and finally reached to 36.79 ± 4.38 N/100 g, which this data is comparable with previous study results for rainbow trout fish fillets22,23,24,25.

Table 1 Descriptive statistics for TVB-N content of samples, measured by the conventional methods.
Full size table

The threshold limit of acceptability for the TVB-N of rainbow trout, as a freshwater fish is considered 20 N/100g26,27,28. Based on this critical value, the acceptable shelf-life for analyzed rainbow trout was 8 days. Furthermore, as shown in Table 1. the variation range of TVB-N for calibration and prediction set were 38.8 and 35.48 mg N/100 g, respectively. Therefore, the differences between the fresh and stale samples were highlighted during 12 days storage which was helpful to establish a suitable and robust calibration model for predicting the total volatile basic nitrogen and consequently estimating shelf-life and quality of rainbow trout fish subsamples29.

Spectral feature analysis

The mean reflectance spectra plot (400–1000 nm) of fish fillet with different TVB-N values are illustrated in Fig. 1. The spectral reflectance curves of samples in various storage days followed an almost similar trend. However, overtime, there was an increase in spectral reflectance across the whole investigated waveband range. The amplitude of variation of spectral reflectance was recognizable on the spectra plot of samples. it was mainly related to the chemical and physical variations in fish muscle during the freshness loss30,31 due to microbial and enzyme activity. Hence, the fresh samples with lower TVB-N content showed the curves with lower reflectance and vice versa.

Figure 1
figure 1

Extracted mean reflectance spectra of rainbow trout fish fillets during cold storage.

Full size image

Overall, the bands in the visible range, 400–700 nm can be connected to the change of fish color. The significant absorption regions were observed around near infrared range (700–1000 nm) may be related to overtones of several chemical bonds, such as N–H (760–840 nm: protein), C-H (930 nm: protein compound), O–H (690–720 nm and 970 nm: water and lipid oxidation compound)and S–H (930 nm: methylene) stretching8,13,14,32,33,34.

Optimal wavelength selection

In the current study, the SPA method was used to choose the most important wavebands related to the TVB-N content of fish quality from the full spectral range. Nine wavelengths (459, 552, 616, 629, 695, 760, 896, 956 and 986 nm) were considered as the optimal variable which covered the whole spectral range. Figure 2 showed the frequency of various waveband ranges selected by different methods in previous studies. As seen in Fig. 2 the optimal wavelengths almost covered the full spectral range. More than half of these wavebands (5 out of 9 wavebands) were located in the visible region of the spectrum (400–750 nm). The changes of the chemical compounds (e.g. protein, fat, water, etc.) occurring during freshness loss of fish can directly reflect in fish fillet color and result in the spectral variations in visible region35. These results were agreed with several previous studies (Fig. 2 and Table 3). Moreover, the results of the meta-analysis (Fig. 2) showed that the most frequent waveband range was located at 400–500 (25%) and 501–600 (20%) nm. However, in the current study, most of the selected wavebands were located at the main absorption range of 601–700 nm which is related to the variation of H2S produced by microbial activity36. The waveband of 890 nm is ascribed to the C–H and N–H stretching that is associated with protein, methylene group of lipid11. Water in the fish fillet is the major component and finally, the selected waveband 950 nm is assigned to the second overtone O–H stretching in water and the third overtone C–H and C–H2 stretching of fat11,37. The waveband was observed around 1000 nm (990 nm) which are mainly related to the NH stretching of proteins11,21.

Figure 2
figure 2

Frequency of optimal spectral range in previous studies.

Full size image

Evaluation of TVB-n value based on deep learning regression framework

In this work, hyperspectral imaging (HSI) coupled with linear deep neural network, as a chemometric algorithm, has been applied to evaluate the TVB-N content of rainbow-trout fish fillets. It is still very hard to obtain a big dataset for TVB-N value of fish samples, due to the use of manual, time consuming and destructive information acquisition tools. Therefore, the main characteristic of the available data is the small sample size (210 samples) which restricts the functionality of the machine learning tools which are used for prediction. Therefore, the use of the LDNN model was proposed for resolving this problem. Because by using the linear activation function, the deep learning neural network will not overfit to small sample size and the use of any number of layers was allowed for such DNN to be trained without the concern of overfitting to the data. The performance of this model in calibration, cross-validation and prediction sets for prediction of TVB-N were presented in Table 2. The result showed that the LDNN model exhibited acceptable performance for the prediction of TVB-N content (R2p = 0.853; RSMEP = 3.159 and RDP = 3.001). Moreover, in order to better compare the performance of LDNN algorithm, two well-known models including PLSR and LS-SVM were also established. Although, all of the chemometric models exhibited good performance in the prediction of the TVB-N value (0.82 < R2p < 0.9 and 2.5 < RDP < 3). In the calibration set, the PLSR and LS-SVM models showed better performance in comparison with LDNN algorithm (lower RMSC and higher R2C). However, in the prediction set, the lowest RMSEP and difference between RMSEP and RMSEC as well as the highest RDP obtained in LDNN model (Table 2). This can be considered as a reason for more stability of the LDNN model to predict TVB-N values of rainbow trout fillets. Therefore, Deep learning as a state-of-the-art method for processing large and complicated datasets, showed a promising performance to resolve regression problems and evaluation of TVB-N value of fish fillets. In this regard, Yu et al., (2017) used HSI combined with a deep learning algorithm (stacked auto-encoders (SAEs)) followed by logistic regression (LR) to classify the fresh and stale shrimp based on TVB-N value during cold storage and reported the results showed that the established SAEs-LR model is satisfactory for discriminating freshness grade of the shrimp (R2P = 0.858 and RMSEP = 0.19 and RPD = 2.64)19. Yu, Wang, Wen, Yang, and Zhang (2019) also correlated the hyperspectral data (900–1700 nm) for determining total volatile basic nitrogen (TVB-N) content in shrimp. They compared Successive projections algorithm (SPA) and deep-learning-based stacked auto-encoders (SAEs) algorithm to select spectral features. The SAEs-LS-SVM and SPA-LS-SVM showed a suitable performance with RPD values of 3.58 and 3.11 respectively which compared with our findings. Deep learning method can learn representational features from the dataset during the training process, and show stronger ability than traditional methods in the current study (RDP > 3).

Table 2 Calibration, cross-validation and prediction results of the TVB-N values of rainbow-trout samples by hyperspectral imaging system.
Full size table

Based on Table 3 and Fig. 3, the performance of LDNN model was comparable with the results of previous works established for prediction of TVB-N value of various meat and seafood products based on hyperspectral imaging systems5,6,7,33.

Table 3 Prediction results of the TVB-N values of several meat products by hyperspectral imaging system.
Full size table
Figure 3
figure 3

The effect of various chemometric models on (A) R2P; (B) RMSEP; Similar small letters indicate non-significant difference between models (p > 0.05).

Full size image

Figure 3a,b showed the effect of various chemometric algorithms on the predictive power of hyperspectral imaging system. The results of meta-analysis indicated that although linear models averagely showed a higher R2P value, the lowest RMSEP was obtained for non-linear model (R2P(linear model) = 0.895 ± 0.0417 vs R2P(non-linear model): 0.876 ± 0.0406; RMSEP(linear model) = 2.945 ± 1.36 vs. RMSEP(linear model) = 2.648 ± 1.032; P > 0.05). The result was in agreement with the LDNN finding. The results of the meta-analysis did not show any significant difference between the prediction power of various chemometric models. However, the highest R2P and RMSEP value was obtained for PLSR model. Since the LS-SVM model showed a relative high R2P beside a lower RMSEP value, this model can be considered as more stable than the PLSR one to estimate TVB-N content on fish and meat products. It should be noted that the performance of a chemometric model is a function of several factors: the number of samples and variables; the optimal waveband selection method; the type of samples and chemical structure of them; the hyperspectral imaging wavelength range and etc. In this regard, Cheng et al. (2016) reported that when the SPA method was used for optimal wavelength selection, the best method was obtained for the LS-SVM model, while the GA method was provided the best performance for the MLR model33. However, Khoshnoudi-Nia and Moosavi-Nasab et al. (2019) ranked the performance of the various chemometric models to predict TVB-N content of rainbow trout based on R2P, RMSEP and RDP as follow: LS-SVM > PLSR > MLR > BP-ANN5.

The difference between RMSEP and RMSEC of LDNN model showed that this model provided a stable and reliable model for prediction of TVB-N value (RMSEP-RMSEC = 0.2 mg N/100 g). Khoshnoudi-Nia and Moosavi-Nasab et al. (2019) also introduced the BP-ANN model (RMSEP-RMSEC = 0.326 mg N/100 g) as a suitable model to evaluate rainbow fish fillet freshness. Moreover, as shown in Fig. 4 LDNN model subjected to some under-fitting and overfitting in the lower and higher value of TVB-N. However, for moderate values (10–20 mg N/100 g) LDNN model showed great potential for TVB-N prediction.

Figure 4
figure 4

Regression results of LDNN for 70 samples of prediction set.

Full size image

Materials and methods

Fish fillets preparation

Forty fresh rainbow-trout (Oncorhynchus mykiss) fish (0.75–1.2 kg) were caught in a local aquaculture pond (Bajgah, Shiraz, Fars, Iran) in December 2016 and immediately transported to the Seafood-Processing Research Group laboratory (Shiraz, Iran). Total transport time was 15 min. After rigor mortis, the rainbow trout fishes were filleted and washed with cold water. Fish fillets were cut into a small size and consequently, 210 subsamples were obtained (8.0 × 4.0 × 1.0 cm). All the subsamples were labeled, packaged by and stored at 4 ± 2 ◦C for 12 days. To exhibit the suitability of the model, 70% of subsamples were classified in training (calibration) group (140 subsamples) and 30% was used to test the model (prediction set: 70 subsamples)1.

Acquisition and calibration hyperspectral images

The region of 430–1010 nm was considered for analysis. The main components of the HSI system consisted of a mobile platform (stepper motor), hyperspectral imaging unit (Hyper Spectral Imaging (1000) spectrograph, Opt Co., Kashan, Iran), an illumination unit (two tungsten (100 W) light sources and two daylight fluorescent (36 W) lamps at two sides of the stepper motor). The speed of the stepper motor was adjusted on 0.6 mms-1 and exposure time of 25 ms, controlled by a computer system (LabVIEW 2011, National Instruments CO. Austin, USA)13,32.

Subsamples were placed on the stepper motor to scan by a line-scanning hyperspectral imaging system. The obtained images corrected by black and white reference images to reduce the effect of illumination and “dark” current of the hyperspectral imaging units2,13,28.

Determination of TVB-N content

Immediately after hyperspectral image collection, the samples were analyzed for TVB-N value by a micro Kjeldahl distillation unit (Kjeltec PDU-500, PECO food Co., Iran) as explained by Goulas & Kontominas (2005) using the following equation:40.

$$ {\text{TVB-N }}\left( {{\text{mg}}/{1}00{\text{ g fish muscle}}} \right) = \frac{{\left( {V_{1} - V_{2} } \right) \times c \times 14}}{{{\raise0.7ex\hbox{${\left( {m \times 5} \right)}$} \!\mathord{\left/ {\vphantom {{\left( {m \times 5} \right)} {100}}}\right.\kern-\nulldelimiterspace} \!\lower0.7ex\hbox{${100}$}}}} \times 100 $$
(1)

V1 is the titration volume for the tested sample (mL); V2 is the titration volume of blank sample (mL); c is the actual concentration of HCl (mol L−1); m is the weight of minced muscle (g)12,40.

ROI identification and extraction of spectral data

The region of interests (ROIs) of hyperspectral imaging were identified manually by the software of hyperspectral imaging system (LabVIEW 2011, National Instruments CO. Austin, USA). A mean spectrum for each fish fillet was obtained and used as input data for evaluation TVB-N values of the samples based on a trained deep learning algorithm. Savitzky–Golay (S–G) algorithm was used to decline the noises of extracted average spectrum (by: Unscrambler 10.4; CAMO, Trondheim, Norway)5.

Optimal wavelength selection

Data extracted from hyperspectral images of each fish fillet sample comprises hundreds of contiguous wavebands. However, most of these wavelengths are poorly correlated with TVB-N content. Successive projections algorithm (SPA), as a forward selection method, was used to select the most informative wavelengths. This algorithm starts with one waveband and incorporates a new one at each replication until a specified number of wavebands with minimum redundancy is obtained41. The procedure of SPA was conducted in Matlab 2016a software (The MathWorks Inc., Mass, USA).

Deep regressor model for TVB-N prediction

Our objective is to predict the TVB-N value of fish fillets based on deep learning method. The main approaches for regression analysis include analytical methods and neural network (NN) based methods. The former assumes a mathematical equation and aims at finding the optimum parameters for this equation describing the relationship between variables, while the latter train a NN as a black box with the input predictor and the output outcome variables, to estimate their relationship.

Artificial neural network (ANN)

An ANN is based on artificial neurons (i.e., a collection of connected units) which showed in Fig. 5.

Figure 5
figure 5

Diagram of an artificial neuron as building block of a ANN. For a given artificial neuron, let there be m + 1 inputs with signals x0 through xm and weights wo through wm. The output y of the neuron acts like an input to the neurons in the next layer42.

Full size image

Each connection can transmit a signal to other neurons. The output y will be computed as42:

$$y= \varphi (\sum_{j=1}^{m}{w}_{j}{x}_{j})$$
(2)

In Fig. 5, the activation functions \(\varphi (.)\) determine the type of the function which ANN approximates. If all activation functions are selected to be linear, then we will have a linear ANN (LANN), while by choosing nonlinear activation functions in ANN the ANN provides a nonlinear function approximator. Moreover, if the number of network layers proceeds from some threshold (e.g., 5) we call the NN a Deep Neural Network (DNN).

In this research, DNNs was used for regression of the TVB-N content of fish fillet by a set of existing data. The advantage of using NNs over analytical methods include their efficient inference step and their power in handling the noise in the data.

The training (learning) phase of NNs is performed considering examples, and without being programmed with task-specific rules.

Network structure

Because of the small sample size, training nonlinear NN models for our problem encounters overfitting to the training data. Therefore, we are restricted to linear regression, to find a line (or a more complex linear function) that most closely fits the data according to a specific mathematical criterion (e.g., MSE). The linear models do not overfit to the data because of their simple structure. A seven layers linear Deep Neural Network (MLP) was designed for regression of TVB-N content of fish fillets with the structure was presented in Fig. 6. LDNN model was designed using Keras library and trained it using 140 samples, as the training data and test it using 70 samples as the test data.

Figure 6
figure 6

Network structure for first LDNN used in the experiments for regression of TVB-N feature. (a) Schematic diagram, (b) detailed structure.

Full size image

PLSR and LS-SVM models

The calibration between the spectral and TVB-N value was established by PLSR and LS-SVM models to compare the results of them with LDNN performance. The linear models were conducted in Unscrambler 10.4 × software (CAMO, Trondheim, Norway) and LS-SVM model was established in MATLAB R2016a (The Mathworks Inc., Natick, MA, USA).

Model evaluation

The spectral data selected by the SPA algorithm were considered as input for training (calibration set) and testing (prediction set) the deep learning models. The assessment factors include the adjusted determination coefficient (R2C(adj), R2CV(adj), and R2P(adj)), the root mean square error of them (RMSEC, RMSECV and RMSEP) and residual predictive deviation (RDP)43.

Generally, a suitable prediction model can be introduced based on the following principles: (i) high value of determination coefficient (poor model: R2 < 0.82; good model: 0.82 \(\le \) R2 \(\le \) 0.9; and excellent model: R2 > 0.9); (ii) high RPD value (RPD < 1.5: very poor model; 1.5 < RPD < 2.0: poor model; 2 < RPD < 2.5: fair model; 2.5 < RPD < 2.5: good model; RPD > 3: very good model and R > 5: excellent model) (iii) low RMSEs values and (iv) a small difference between RMSEC and RMSEP value1,44,45. The building, validation, and evaluation processes of LDNN model were carried out in MATLAB R2016a (The Mathworks Inc., Natick, MA, USA)5,32.

Statistical analysis

TVB-N analysis was conducted in 30 replicate and the data were reported as mean ± standard division (SD). Statistical analyses were carried out by one-way ANOVA. Tukey’s test was applied to determine the significant differences between the means (P ≤ 0.05). Statistical calculations were performed in Minitab Version 17 statistical software (Minitab Inc. Pennsylvania, USA).

Meta-analysis

Meta-analysis was done based on published studies reporting the evaluation of TVB-N content of meat products based on HSI system (400–1000 nm). The articles with experimental study design (original research), searching on meat and seafood, HSI system (400–1000 nm), TVB-N content (based on regression) and English language were considered as eligible article to review. On the other hands the letters, conference abstracts, patients and review articles as well as the researches classified the meat samples based on TVB-N content were excluded. Thus, eight articles (26 case studies: some articles had compared the two or more algorithms) were selected. A comprehensive literature search was carried out between 20 December 2019 to 20 January 2020 on google scholar databases. The following search terms were applied as keywords and Boolean operator: “hyperspectral imaging” + meat, TVB-N + “hyperspectral imaging”. The search was limited to English language articles. The meta-analysis was carried out based on descriptive analysis (one-way ANalysis Of VAriance (ANOVA)) and comparative analysis (frequency data). Statistical analysis was performed using the Minitab 17.1.0 (Minitab Inc. PA, USA)46. The further details were provided in supplementary materials (Supplementary Fig S1 and Supplementary Table S1).

Statement

All experiments and methods were performed in accordance the approved guidelines of the Shiraz University. All experimental protocols were approved by the Ethics Committee of Shiraz University of Medical Sciences and all experiments were conducted in accordance with the approved guidelines of Iran Veterinary Organization.

The methods were carried out in accordance with the approved guidelines of the University of Sydney Ethics Committee. All experimental protocols were approved by the University of Sydney Ethics Committee.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Conclusions

In several previous works, the potential of various deep learning algorithms to classify of seafood and meat products in two freshness grades (fresh and stale) was investigated. The results demonstrated that the hyperspectral imaging system coupled with the deep learning method had great potential for classifying of freshness quality of meat products with a total classification accuracy of more than 90%. However, sometimes the evaluation of freshness index based on numerical output and in a regression, framework is helpful for better decision making and management. As a result, the present study is the first time that a deep learning algorithm was applied to predict a freshness indicator in regression framework. In the calibration set, the PLSR and LS-SVM models showed better performance than the LDNN algorithm. However, in the prediction set, the findings of this study demonstrate that the combination of linear deep learning neural network and the hyperspectral system gave reasonable accuracy for the prediction of TVB-N content in fish fillets (R2P = 0.853; RMSEP = 3.159 and RDP = 3.001). Based on the meta-analysis, the results of established prediction system were comparable with the hyperspectral imaging system based on the traditional chemometric analysis. In order to make the DLNN model with higher accuracy and ability, a large amount of data to train the system is necessary. it is the most important challenge for evaluating food product quality based on a deep-learning algorithm and experimental data. Therefore, there is still much work to be done and the results obtained by the SPA-LDNN method would encourage more research efforts on using deep learning as a novel chemometric method for evaluating the freshness quality of food products. Establishing a comprehensive database for a certain fish freshness index, the extraction of modeling features (e.g., optimal wavebands) based on a deep-learning method and the use of other deep-learning algorithms and compare their performance, are some suggested solutions to enhance the accuracy of a hyperspectral system coupled with deep learning algorithms.