Introduction

Literature review and problem statement

Concrete has been commonly used in construction and architecture due to its favourable engineering properties. Concrete has the characteristics of rich raw material, low price, and high compressive strength and good durability1. Concrete comprises four primary components: coarse aggregate, fine aggregate, cement, and water. Concrete's economic value allows it to be widely used in constructions and the accessibility to the material available in the local market. It also demonstrates excellent benefits over other construction materials such as steel, and concrete can be produced with minimum effort. In certain instances, supplementary materials like fly ash (PFA)2,3, blast furnace slag (GGBS)4, silica fume5, and other industrial waste/by-products are added in concrete to enhance the mechanical properties of the concrete4. The introduction of industrial waste/by-product6,7 into concrete offers environmental benefits while increasing the longevity and resiliency of concrete structures.

Among the various concrete property indices, compressive strength is the most critical because it is directly related to the structural safety and is required for determining the performance of structures throughout their life, from new structural design to old structural assessment8.

When dealing with concrete materials, one of the difficulties in selecting the appropriate materials and predicting the mechanical properties of the concrete, i.e., compressive strength, is due to cost and the availability of local material9. It is vital to have robust and reliable predictive models based on existing input and output data at the early stage to drive down the cost of making further experiments and reduce the cost associated with the risk of non-compliance concrete during construction5. With the use of suitable models, it can lead to success in finding combination inputs that can achieve meaningful outcomes and, at the same time, saves considerable time and money. However, empirical, and statistical models, such as linear and nonlinear regression, have been widely used. However, these models require laborious experimental work to develop, and can provide inaccurate results when the relationships between concrete properties and mixture composition and curing conditions are complex10.

ML is a sub-class of AI that self-learning through algorithms and improves its performance based on previous datasets/experience. The distinction between AI, ML, and DL is illustrated in Fig. 1. With minimal human input, ML algorithms will automatically learn and improve over time6. ML has been widely applied in the field of engineering to solve a variety of problems i.e., predict outages, estimate angular velocity, components failure prognostics and prediction of fatigue life11,12,13,14. In civil engineering, AI and ML have been previously employed to tackle problems in various structural engineering fields15. ML application is also used in building structural design & performance assessment, improving finite element modelling of structures, and enhancing concrete properties prediction & assessment9,16,17,18,19,20.

Figure 1
figure 1

Artificial Intelligence sub-classes.

Full size image

Given the popularity of machine learning, especially in concrete technology, various studies have been conducted using ML/DL approaches10. Table 1 below shows the summary of concrete compressive strength prediction for various types of concrete using various ML and DL models. Many empirical and statistical models, i.e., linear and nonlinear regression algorithms, were employed to predict the properties of concrete10. Multiple Linear Regression (MLR)21, Support Vector Machine (SVR)22,23, Multilayer Perceptron (MLP)24, and Gradient Boosting25,26 are most used ML algorithms to predict the mechanical and chemical properties of concrete. In general, the compressive strength prediction was undertaken for several type of concrete i.e., ordinary concrete8,10, high-performance concrete25,27,28,29,30, ultra-high-performance concrete 20, and green concrete with supplementary cementitious material i.e., fly ash16,31,32, blast furnace slag4 and recycled aggregates6. ML/DL is also used to predict other mechanical and chemical properties of concrete, i.e., prediction of concrete shear strength15,24, 30], tensile strength33, flexural strength5, the thermal conductivity of concrete34, and chloride concentration of concrete35.

Table 1 Summary of previous studies on concrete strength prediction.
Full size table

For the DL model, Artificial Neural Network (ANN)4,7,29,36,37,38 was widely used in most previously reported studies. The use of boosting algorithms is not extensively reported in any previous studies except the GBR models. The proposed boosting algorithms were chosen based on their popularity and frequency in other research areas such as biomedical and construction hazard analysis, which reports that the BML models have higher prediction accuracy than other ML and DL models39,40,41,42. We implemented and analysed the accuracy and error of compressive strength prediction for five different boosting algorithms, namely LBGM, CATB, GBR, ADAB, and XGB. Additionally, the BML models are enhanced using the Random Search (RS) optimization process, which involves tuning the hyper-parameters of the BML algorithms.

Objectives

The study's objective is to identify the best performing BML models, i.e., LBGM, CATB, GBR, ADAB, and XGB to predict the HPC with high volume GGBS using BML algorithms, i.e., LBGM, CATB, GBR, ADAB, and XGB. The BML models were then optimised using the Random Search (RS) optimisation process by tuning the hyper-parameters of each BML model function. Additionally, comparison studies were also conducted using commonly used ML models, i.e., linear regression, decision tree, random forest, etc., to evaluate the performance of the BML model in predicting the concrete strength.

The fundamentals behind BML algorithms models are defined in Sect. 2, followed by the statistical properties analysis of the dataset & modelling approach, findings from the optimised BML model, comparison studies between other ML models, and model validation results are provided in Sect. 3. The findings of each model’s prediction accuracy and modelling errors are concluded in Sect. 4.

Methodology

BML algorithms

Light gradient boosting machine (LBGM)

LGBM is a gradient boosting framework that uses tree-based learning algorithms developed by Microsoft50. LBGM uses two innovative sampling techniques: Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB). GOSS excludes a substantial fraction of data instances with small gradients and uses the remainder to estimate the information gain. Since data instances with large gradients contribute more to the computation of information gain, GOSS can generate a highly accurate estimate of information gain with a significantly smaller data set.

EFB allows for the grouping of mutually exclusive features, hence reducing the number of features. It also demonstrates that while determining the optimal bundling of exclusive features, a greedy approach can reach an approximation ratio of relatively high. It was reported that LGBM speeds up the training process of conventional GBDT by up to over 20 times while achieving almost the same accuracy, and it is six times faster than XGBoost50.

CAT boost regressor (CATB)

CATB is an open-sourced machine learning algorithm developed by Yandex in 2017. CATB is a decision tree algorithm based on gradient boosted decision trees. The algorithms in CATB models are a series of decision trees constructed sequentially, with each new tree having a lower loss than the prior trees. The starting parameters determine the number of trees generated, and overfitting is avoided using an overfitting detector. The processes of tree construction for a single tree in CATB algorithms include computing splits in advance, converting categorical features to numerical features, selecting the tree structure, and calculating values in leaves.

Generally, CATB employed greedy algorithms in optimising the prediction accuracy. The features of CATB models are ordered according to their splits and are then substituted in each leaf. The depth of the tree and other constraints for structure selection is specified with pre-modeling parameters, and a random permutation of classification/regression objects is conducted before the construction of each new tree. CATB models validate the model performance with a metric that indicates the direction in which the function should be improved further when deciding the construction of the next tree. CATB model surpasses leading GBR packages and achieves new state-of-the-art performance on common benchmarks51,52.

Gradient boosting regressor (GBR)

Friedman presented the GBR model as an ensemble method for regression and classification in 1999. The gradient boosting approach compares each iteration of the randomly chosen training set to the base model. In the GBR model, the lower the training data fraction, the faster the regression, as the model fits smaller data each iteration. GBR model requires the following tuning parameters: ntrees and shrinkage rate, where ntrees is the number of trees to be grown, and the shrinkage parameter, often referred to as the learning rate applied to each tree in the expansion25,53.

This algorithm's fundamental foundation is 'boosting.' The boosting process aids in transitioning prediction from a 'weak' learner via the additive training process. The essential advantage of GBR algorithms is that it avoids overfitting and makes efficient use of computational resources by using an objective function. Besides improving output performance, GBR algorithms reduce the selected error function further54.

Adaboost regressor (ADAB)

ADAB, an acronym for Adaptive Boosting, is a meta-algorithm for statistical categorization developed in 2003 by Yoav Freund and Robert Schapire. It can be combined with a variety of other types of learning algorithms to enhance performance. The output of the other learning algorithms, i.e., 'weak learners,' is combined into a weighted sum representing the boosted classifier's final output. ADAB is adaptive because it adjusts succeeding weak learners favoring instances misclassified by previous classifiers/regressors. It is less prone to overfitting than other learning algorithms in some cases29.

The individual learners in ADAB algorithms may be ineffective. Still, if their performance is marginally better than random guessing, the final model can be demonstrated to converge to a powerful learner. This technique benefits from a single best-fit decision model formed from the outcomes of several decision trees, each of which is constructed using a random selection of base features, i.e., decision factors from a training dataset55,56.

Extreme gradient boosting (XGB)

Extreme Gradient Boosting (XGB) or XGBoost is a decision tree-based ensemble ML algorithm that uses gradient boosting to make predictions for unstructured data. Tianqi Chen and Guestrin developed XGBoost, and the method uses the conventional tree gradient boosting algorithm45 to create state-of-the-art algorithms, the ‘extreme gradient boosting’23. The multiple Kaggle competition winner ‘XGBoost’ is a highly effective ML algorithm due to its scalable tree boosting system and sparsity-aware algorithm in modelling structured datasets. The algorithm has been the source of countless cutting-edge applications, and it has been the driving force behind many of these recent advances. It’s been widely used as industrial solutions such as customer churn prediction57, applicant risk assessment58, malware detection59, stock market selection60, classification of traffic accidents61, diseases identification40, and even in predicting the death of patience during SARS-COV-2(Covid-19) treatment42. The most significant benefit of XGBoost is its scalability across any condition62. In general, the XGBoost algorithms are the evolution of decision tree algorithms that were improved over time. Figure 2 below shows the development of decision tree-based algorithms to XGBoost.

Figure 2
figure 2

The evolution of XGBoost.

Full size image

Model structure

For the most part, we utilised the Python programming language on Google's Colab platform to analyse the data and create the models. An open-source, low-code machine learning library ‘PyCaret’ was used in research63. Figure 3 illustrates the step-by-step procedure for training, optimising, and validating the BML models in predicting the concrete compressive strength. Seven key processes are involved in the development of the optimised BML model, and each stage is described in detail below, with brief explanations:

  1. i.

    Data Collection – This entails collecting data from the laboratory and compiling it appropriately.

  2. ii.

    Data Pre-Processing – To correctly identify and arrange the acquired data, it is necessary to sort out the missing values and then normalise the dataset in preparation for model building.

  3. iii.

    Model Selection—For prediction and evaluation in this research, BML algorithms, i.e., LBGM, CATB, GBR, ADAB, and XGB, were utilised.

  4. iv.

    Hyper-parameter Optimisation – The RS approach was employed in each of the five proposed BML algorithms, and the results are compared to the original models.

  5. v.

    Model Validation—Validation and testing of the models were performed using the k-fold cross-validation approach, which randomly splits the dataset and minimises overfitting.

  6. vi.

    Model Evaluation – All the models are compared, and the best performing algorithms are selected based on evaluation metrics, i.e., R2, RMSE, MAE, MSE, RMSLE, MAPE.

  7. vii.

    Analysis and Reporting – The findings in the case study are reported based on comparing various ML models, optimisation parameters, and evaluation metrics.

Figure 3
figure 3

Step by step BML modelling approach.

Full size image

Data collection and pre-processing

Overview

A total of 152 data of HPC compressive strength data were gathered from concrete trial mix conducted at a laboratory in Selangor, Malaysia. In general, the dataset is composed of seven concrete components: fine aggregate, coarse aggregate, ordinary Portland cement (OPC), ground granulated blast-furnace slag (GGBS), silica fume (SF), water, admixture, and moisture content (MC). The dataset also contains concrete compressive strength of a Grade 80 HPC, and the compressive strength results are available for 7, 28, 56, and 91 days. On average, each batch of concrete contains around 246 kg of GGBS and OPC, respectively.

The proportion of cementitious content in each batch is around 45% of GGBS, 45% of OPC, and 10% of SF. Similarly, the ratio of fine to coarse aggregate is 1:1, equating to 70% of total concrete volume with a 0.25 water-to-cement ratio, or 138 kg of water in each batch of concrete. Additionally, the moisture content of fine and coarse aggregate was included as input parameters since the water content in each concrete batches was adjusted according to the moisture content in the aggregates. Details of statistical metrics are listed in Table 2 below.

Table 2 Summary of statistical analysis of the concrete material composition.
Full size table

Data distribution analysis

The distribution correlations between the input parameters and the compressive strength are shown in Fig. 4. It illustrates the correlation between the data points by including the relative frequency distribution of each input parameter. Generally, the distribution of input parameters suggests that the dataset is appropriately distributed and fit for machine learning modelling.

Figure 4
figure 4

Distribution correlation of input parameters and strength.

Full size image

Correlation coefficient analysis

Along with statistical and distribution analysis, a correlation coefficient study was performed to analyse the dataset and prepare for modelling. Pearson’s correlation coefficient approach indicated in Eq. 1 was used to calculate the correlation coefficient64. Pearson's correlation coefficient is a test statistic that shows the statistical link between two continuous variables. It is based on the covariance approach, in which the best method is considered for determining the relationship between two variables of interest. It reveals both the size of the association or correlation and the direction of the relationship. The correlation between all parameters was analysed for this research and visualized in Fig. 5 as a Pearson’s correlation heatmap.

Figure 5
figure 5

Pearson’s correlation heatmap.

Full size image
$$Pearson\, Correlation\, Coefficeint, r = \frac{{\sum \left( {x_{i} - \overline{x}} \right)\left( {y_{i} - \overline{y}} \right)}}{{\sqrt {\sum \left( {x_{i} - \overline{x}} \right)^{2} \sum \left( {y_{i} - \overline{y}} \right)^{2} } }}$$
(1)

where;

\(r\) = correlation coefficient.

\(x_{i}\)  = values of the x-variable in a sample.

\(\overline{x}\) = mean of the values of the x-variable.

\(y_{i}\) = values of the y-variable in a sample.

\(\overline{y}\) = mean of the values of the y-variable.

As shown in Fig. 5 above, it can be observed that the correlation between input and output parameters is relatively low and generally in the range of -0.62 to 0.64. The range of the correlation coefficients indicates that the input variables can be considered low to moderately correlated to the compressive strength.

Data split and normalisation

The dataset's modelling proportion was randomly partitioned into two sets, i.e., training and testing dataset. Around 70% of the dataset was utilized for training the BML models, whereas 30% were used to test the models65, 65 Before training BML models, pre-processing data is required. To prevent training from being dominated by one or a few features with large magnitude, features should be normalised so that their range is consistent. The Z-score normalisation method was used in this study to normalise all values in a dataset so that the mean of all values is 0 and the standard deviation is 1. Equation 2 below shows the formula to perform a z-score normalization on every value in a dataset:

$${\text{Z}} - {\text{Score}} = \left( {{\text{x }}{-} \, \mu } \right)/\sigma$$
(2)

where:

x: Original value.

μ: Mean of data.

σ: Standard deviation of dataset.

Model validation using K-Fold cross-validation

Validation and testing of the models were performed using the k-fold cross-validation method illustrated in Fig. 6. In this study, a total of ten folds or k value of 10 were used. The dataset is randomly separated into test and training data and divided into k groups, using this cross-validation procedure. Validation of the model is performed on one of the groups, and training is performed on the remaining groups. The process is performed k times more until each distinct group is used as the validation set. The ultimate performance of the model is determined using test data that the model ‘not seen’ during training.

Figure 6
figure 6

K-fold cross validation method.

Full size image

K-fold cross-validation enables the model to be trained and verified several times, resulting in a more accurate model with less overfitting. With the more traditional hold-out strategy, the dataset is partitioned into training, validation, and test sets, which reduces number of samples for model training. The model's performance is contingent upon a random selection of samples for the training, validation, and test sets.

Model evaluation

In this paper, six separate statistical measurement parameters were used to calculate the prediction efficiency of the BML models. In simpler terms, the evaluation parameters estimate the accumulated error in predictions concerning actual observations. The statistical parameters are: coefficient of determination (R2), mean absolute error (MAE), root mean squared error (RMSE), mean squared error (MSE), root mean squared logarithmic error (RMSLE), and mean absolute percentage error (MAPE). These mathematical formulations are defined in Eqs. 3–8; in this case, n is the total number of test dataset records while y′ and y are the predicted and measured values, respectively. The values of R2 would range from 0 to 1 – the closer the value is to 1, the higher fitting optimisation of the model is. The values MAE, RMSE, MSE, RMSLE, and MAPE are used to evaluate modelling error—the smaller the value, the lesser the difference between the predicted and measured values.

$$R^{2} = 1 - \frac{{\mathop \sum \nolimits_{i = 1}^{n} (\widehat{{y_{i} }} - y_{i} )^{2} }}{{\mathop \sum \nolimits_{i = 1}^{n} \left( {y_{i} - \overline{y}_{i} } \right)^{2} }}$$
(3)
$$MAE = \frac{1}{n} \mathop \sum \limits_{i = 1}^{n} \left| {y - y^{\prime}} \right|$$
(4)
$$RMSE = \sqrt {\frac{1}{n} \mathop \sum \limits_{i = 1}^{n} \left| {y - y^{\prime}} \right|^{2} }$$
(5)
$$MSE = \frac{1}{n} \mathop \sum \limits_{i = 1}^{n} \left| {y - y^{\prime}} \right|^{2}$$
(6)
$$RMSLE = \sqrt {\frac{1}{n} \mathop \sum \limits_{i = 1}^{n} (\log \left( {y_{1} + 1} \right) - log\left( {y^{\prime} + 1} \right)^{2} }$$
(7)
$$MAPE = \frac{1}{n} \mathop \sum \limits_{i = 1}^{n} \left| {\frac{{y - y^{\prime}}}{y}} \right|$$
(8)

Results and discussion

Initial modelling

The LBGM, CATB, GBR, ADAB, and XGB algorithms were initially modelled using their default hyper-parameter settings. Each model's performance is measured in terms of prediction accuracy and error rates, i.e., R2, MSE, RMSE, MAE, RMSLE, and MAPE. The findings of the initial modelling are summarised in Table 3 below.

Table 3 Summary of initial modelling.
Full size table

LBGM predicted the compressive strength of concrete with the highest prediction accuracy and the least prediction errors of all five BML models. The initial modelling of LBMG reached 0.86 and 0.94 for the training and testing prediction scores, respectively. GBR and XGB models also performed well, with prediction accuracy of 0.93 and 0.92 on the test dataset. The evaluation metrics in the LGBM model was the lowest in comparison to other BML models, with an MAE of 3.29, an RMSE of 4.10, and an RMSLE of 0.03 for test dataset. The GBR model was the second-best model in prediction errors with MAE and RMSE values of 3.24 and 4.22, respectively. The distribution of predicted results against actual results for all BML models are visualized in Fig. 7, along with the best fit line for the prediction distribution. The initial modelling suggests a reasonable prediction result; however, it is further improved by using the RS algorithm, which is discussed in detail in the following section.

Figure 7
figure 7

Best fit line for prediction distribution (RS model).

Full size image

In terms of prediction distribution, the LBGM and CATB models have the highest training scores of 0.86 and 0.85, respectively. In contrast, their prediction scores for the test dataset are significantly higher at 0.94 and 0.93. It suggests that both models concentrated on optimising the test score to get the maximum possible prediction accuracy. The prediction distribution indicates that the LBGM, GBR, and XBG results are closely spaced along the best fit lines compared to the CATB and ADAB models.

Model optimisation with RS algorithm

The RS algorithm focuses on the use of random combinations to optimise the hyperparameters of a model. It measures random combinations of a set of values to optimise decent outcomes, with the function tested at any number of random combinations in the parameter space. The chances of discovering the optimal parameter are relatively higher in RS algorithms compared to Grid Search algorithm due to various search patterns in the model being trained on the optimised parameters without aliasing. RS algorithms are best for lower dimensional data as this method takes less time and iterations to find the right parameter combination67. Numerous hyperparameters were optimised in this study, including n_estimator, learning_rate, max_depth, and subsample and min_sample_split. A total of 1000 iteration was performed to identify the performing model and the optimum hyperparameters for each BML models. Table 4 below shows the hyperparameters and values used for all the model, before and after optimisation process.

Table 4 Summary of hyperparameter tuned values.
Full size table

Based on the optimised BML models, the GBR model achieved the highest prediction accuracy of 0.96 for the test dataset followed with LBGM and CATB model with R2 of 0.95. In comparison, the optimised GBR model had the lowest prediction errors for test errors, with an MAE of 2.73, an RMSE of 3.40, respectively. For training dataset, the CATB model recorded lowest prediction error and highest prediction accuracy of 0.89. Table 5 shows the summary of prediction accuracy and evaluation metrics for the optimised BML models.

Table 5 Summary of RS optimised models.
Full size table

The comparison of the training and test datasets for both the initial and optimised BML models is shown in Fig. 8. In general, the RS algorithm improves prediction accuracy and reduces the modelling error for the training dataset of all BML models. However, the optimised ADAB model show a minor deficiency compared to the training results. The overall performance of BML models with RS optimisation shows that the GBR model is the best performing model with highest prediction accuracy and lowest modelling errors while the LGBM model are the best model without any optimisations with highest prediction accuracy and lowest modelling errors.

Figure 8
figure 8

Comparison between BML and RS optimised models.

Full size image

The prediction distributions of the optimised BML models appear to have a similar pattern for both the training and test datasets, with only a minor difference in prediction scores. The RS algorithms optimise the BML models to obtain a high prediction accuracy and a low error rate by tuning the hyperparameters for both the training and test datasets while simultaneously improving model performance. As presented in Fig. 9, the LBGM, CATB, and GBR all suggest a closed space between best fit and the identity line, demonstrating that the model's predictions are highly accurate.

Figure 9
figure 9

Best fit line for prediction distribution (RS model).

Full size image

Features importance analysis

The explainability and interpretability of ML models are active areas of research that seek to understand why and how an ML model predicts output values. Numerous techniques, including feature importance analysis (FIA), are frequently used to explain and interpret ML models68. The permutation FIA techniques are model-dependent, which means they evaluate model predictions rather than the actual data. The explainability and interpretability metrics reveal how well ML model predictions correspond to physical knowledge. Additionally, it enables the discovery of hidden correlations between targets and features that are not readily visible in the data by allowing ML models to make correct predictions.

The original dataset is updated for each feature by randomly shuffling the feature values. The model's evaluation metric for the updated dataset is computed and compared to the original dataset's evaluation metric. This procedure is repeated numerous times for each feature to get the mean and standard deviation of the permutation importance score.

In this research, the permutation FIA was performed in all BML models to understand the influence of each feature/component of concrete in predicting the compressive strength of concrete. Figure 10 below displays all the features used in the compressive strength prediction model and their relative importance. ‘Days’ are an essential feature for all BML models, and SF is the least important feature in GBR, ADAB, and XGB models. It demonstrates that changes to the day's value in the dataset substantially affect the concrete compressive strength results. In contrast, changes in SF value have a considerably low impact on the strength prediction.

Figure 10
figure 10

Feature importance analysis of BML models.

Full size image

Comparison between various ML algorithms

Subsequently, the initial BML models without optimisation was compared to 14 commonly used ML algorithms, including linear regression (LR), decision trees (DT), random forests (RF), and extra trees (ET), etc. Table 6 shows the summary of prediction accuracy and the evaluation metrics for 14 comparison ML models. For Table 6, only test dataset values were provided as the purpose of this section is to make comparison between BML model and other conventional models. Generally, all comparison models exhibit much lower prediction scores and more significant prediction errors than BML models. The comparison models show that the ET and RF models were the best performing model with an R2 of 0.78. Similarly, both models produced prediction errors, i.e., MAE and RMSE of 4.17 and 5.0, respectively. Overall, the comparison models demonstrate that the initial BML model outperforms all other machine learning models.

Table 6 Summary of comparison between various ML models.
Full size table

Conclusion and recommendation

Comparing all 5 BML models, the GBR model has outperformed the LBGM, CATB, ADAB, and XGB models. The GBR model optimised with RS algorithms achieved the highest prediction accuracy of 0.96 and the least prediction errors, with an MAE of 2.73, an RMSE of 3.40, and an RMSLE of 0.03. Notably, the RS algorithms optimisation technique improved the model prediction accuracy and reduced the modelling errors in all 5 BML models. Simultaneously, the evaluation of 14 commonly used ML models also suggests that the BML models have superior prediction accuracy and minimum prediction errors. These studies conclude that the optimised BML models, i.e., the GBR model are the best choice to predict the compressive strength of concrete, mainly for HPC and concrete with high volume GGBS replacements. For future research, a comparison study between ANN models with BML models or hyperparameter tuning with different optimisation algorithms, i.e., Grid Search, can be evaluated and compared with the proposed BML model's performance.