Abstract
Early prediction of neonates' survival and Length of Stay (LOS) in Neonatal Intensive Care Units (NICU) is effective in decision-making. We developed an intelligent system to predict neonatal survival and LOS using the "Case-Based Reasoning” (CBR) method. We developed a web-based CBR system based on K-Nearest Neighborhood (KNN) on 1682 neonates and 17 variables for mortality and 13 variables for LOS and evaluated the system with 336 retrospectively collected data. We implemented the system in a NICU to externally validate the system and evaluate the system prediction acceptability and usability. Our internal validation on the balanced case base showed high accuracy (97.02%), and F-score (0.984) for survival prediction. The root Mean Square Error (RMSE) for LOS was 4.78 days. External validation on the balanced case base indicated high accuracy (98.91%), and F-score (0.993) to predict survival. RMSE for LOS was 3.27 days. Usability evaluation showed that more than half of the issues identified were related to appearance and rated as a low priority to be fixed. Acceptability assessment showed a high acceptance and confidence in responses. The usability score (80.71) indicated high system usability for neonatologists. This system is available at http://neonatalcdss.ir/. Positive results of our system in terms of performance, acceptability, and usability indicated this system can be used to improve neonatal care.
Similar content being viewed by others
Introduction
Neonatal mortality is considered one of the main health indicators. Mortality within the first month of life is 18 cases per every 1000 live births. Based on statistics, 2.4 million newborns (47%) in 2019 died within the first month of life1. A systematic review indicates that more than half of neonatal deaths occur within the first three days of life, and death within the first day is about two-thirds of this neonatal mortality rate2. Many of these deaths can be prevented through high-quality care and specialized post-delivery care for mothers and newborns, as well as taking care of sick or premature newborns1. Thus, neonatal mortality is considered very important and is known as a quality indicator for comparing the quality of care provided in Neonatal Intensive Care Units (NICUs)3.
Premature or sick newborns receive specialized care in NICUs, whose care period in this unit is another important challenge. The Length of Stay (LOS) of these newborns is six times as large as that of normal newborns, and in most cases, they require extensive as well as advanced care4,5. Their LOS elevation in NICUs increases the burden of healthcare services4, the level of morbidity6, and hospitalization costs7.
Accurate estimation of the risk of in-hospital mortality of newborns is necessary for the management of healthcare quality and logical use of resources8. Further, precise estimation of the number of days requiring intensive care and thus costs of hospitalization is important to inform parents, healthcare specialists, insurance companies, and governmental organizations for planning, and policymaking. Indeed, the physician taking care of the newborn should be able to make decisions within a short time after childbirth5,9. Nevertheless, decision-making on this issue is very complex, because various diagnostic and clinical factors complicate the prediction and decision-making by physicians on neonatal mortality and LOS in NICUs. In this regard, the physician should consider various factors simultaneously including the newborn status at the time of childbirth such as Birth Weight (BW), Gestational Age (GA), Apgar score, type of delivery, the maternal health condition at the time of delivery, and so on4,10,11. Moreover, regarding the use of resources in NICUs, there should be a combination of information about the neonates who have died or have been eventually discharged alive, whereby consideration of only one of mortality or LOS does not reflect a general view of the care provided in NICUs12. Nevertheless, many studies in this regard considered only one of these challenges7,13.
Clinical decision-making requires combining observations, and the experience of physicians with new diagnostic and specialized methods, the available clinical guidelines, and novel therapeutic strategies14. This can be achieved through Artificial Intelligence (AI) methods such as Case-Based Reasoning (CBR). This method has high flexibility and offers suitable performance in solving new problems based on the solutions for similar problems, and more importantly, these systems are well accepted by physicians15.
Many studies have been performed to predict neonatal mortality (or survival) or LOS using statistical or AI methods; for example, predicting neonates' LOS in NICUs using multivariate regression models16, a fuzzy expert system for predicting the risk of neonatal death13, K-Nearest Neighborhood (KNN), Random Forest (RF) and Bayesian network to classify the causes of fetal death17, decision tree (C5.0) and Artificial Neural Network (ANN) to predict neonatal mortality and preterm births in twin pregnancies18; However, studies on the use of CBR to predict both neonatal survival and LOS are rare. Therefore, this study was conducted to develop and evaluate a web-based CBR system that simultaneously predicts neonatal survival and LOS in NICUs.
Related work
Case-based reasoning
CBR is a problem-solving approach, which can employ specialized knowledge of previously-solved problems (cases). The new problem is solved by finding similar cases in the past and reusing their solutions to solve a new situation19,20.
The cycle of CBR is summarized in four steps:
-
1.
Retrieval: the aim of “retrieval” is finding a case that has the greatest similarity (the most suitable) to the new problem20. Retrieval occurs based on applying a similarity index between the new case and all available cases in the case library. Thus, the set of cases is called retrieved cases that are ranked based on the similarity index21,22.
-
2.
Reuse: at this step, the case or cases standing at the top of the list of the previous step are reused and their solutions are adapted21. If the situation of the new problem is exactly similar to the retrieved case, it is known as the most successful solution20,23. Otherwise, adoption should be used for the new problem, which can be done manually or automatically20. The outcome of the reuse step is creating a solution for the new case, which is called the "solved case"21.
-
3.
Revise: this step involves testing the solution in a real setting or an assessment by a supervisor, a specialist, or a modeler/simulator. At the end of this step, the solved cases are considered tested or revised cases, since the system should remember only valid cases with a proper solution24.
-
4.
Retaining the new case: the important feature of CBR is its learning. When a problem is solved successfully, its experience is retained in the system to solve similar problems in the future20; when the "revise" step creates a new case, the case base is updated with the new case (learned) to solve future problems20,21.
Previous research
A Nationwide cohort study in the Netherlands used a multiple logistic regression model to estimate the risk of neonatal mortality within 28 days after birth and indicated an Area Under Curve (AUC) of 0.8325. Cooper et al.26 developed (6499 cases) and validated (3552 cases) a superleaning algorithm to predict 30-day neonatal postoperative mortality. The superlearning algorithm (14 machine learning and regression algorithms) was performed on demographic preoperative clinical data. According to the results, the superlearning algorithm outperformed all individual algorithms with regard to AUC which was 0.91 for the development and 0.87 for the validation.
In another study, researchers applied KNN, RF, and Bayesian network algorithms to classify the causes of fetal death using 49 features. The results showed that KNN and RF had the best performance (accuracy of 81.38% and 81.84%, respectively)17. Performance evaluation of a fuzzy expert system designed to predict neonatal mortality risk showed an accuracy (90%), sensitivity (83%), and specificity (97%)13.
Another study predicted the mortality and LOS for neonatal admissions to a private hospital NICU in Southern Africa using a logistic regression model. The proposed model for predicting neonatal mortality had a good fit (AUC: 0.85 and accuracy: 86.4%), but the low positive predictive value of this model reduced its performance. Furthermore, the Poisson log-linear model had a good fit (R2 = 0.70) for predicting LOS9. Coimbra et al.27 introduced a decision support system to predict the LOS for preterm infants using 284 cases by the CBR approach. The proposed model led to an improved "retrieval step" using optimization and the logical programming method and reduced the computational time by about 21.3% and an accuracy of 84.9%.
Rodriguez et al.28 proposed a prediction model for pediatric mortality risk by combining CBR, fuzzy set theory, and ANN models. After problem-solving by using the fuzzy ANN model, the CBR made justification of the solutions and stored experiences in the case base by using the KNN method (K = 3). The model was developed with 1079 cases and 33 features. Eventually, 99 cases were evaluated by seven pediatricians, which resulted in 89.89% classification accuracy.
Adawiyah et al.29 suggested a prediction system to detect preterm births based on CBR. For system development, 18 variables were selected and local and global similarity was determined based on Minkowsky distance. Twenty cases were used to evaluate the system in which in 18 cases, the system responses and the recorded results were similar. Hence, the system was able to detect premature infants with 90% accuracy.
Our contribution
Generally, a variety of studies have dealt with providing a decision support system in the neonatal care domain by benefiting from artificial intelligence methods13,17,18. Studies on mortality, as well as LOS, have mostly focused on statistical methods and providing models3,9,16,26,30,31. On the other hand, we proposed a system to predict the survival and LOS of newborns in NICUs, which can be updated by adding new data (new cases) to its case base. Then, based on all data (whether at the model development or its usage) it can perform the prediction. Furthermore, the system design in this study deals with survival and LOS simultaneously. In other words, considering the importance of simultaneous prediction of survival and LOS for physicians and families of newborns, this system deals with predicting both outcomes concurrently, while previous studies have dealt with only one of these issues alone3,13,16,26,27,31.
Meanwhile, most previous studies have only focused on the evaluation using retrospective data without any implementation in a real setting or external validation and consideration of system acceptability and usability3,9,13,16,17,18,26,27,31,32,33. On the other hand, in this study in addition to the evaluation of the system with retrospective data, the system was implemented in a hospital setting and its external validation was evaluated prospectively. In addition, the usability and acceptability of proposed responses were also evaluated in a clinical setting.
Results
Development phase
The dataset for developing the case base contained 1346 records, which included 1225 alive (91.01%), and 121 dead (8.99%) neonates with an average LOS of 15 days (0–191 days). The distribution of the selected qualitative and quantitative features in the dataset is presented in Supplementary Table S1.
We developed the case-based system using MySQL-V5.2. The CBR process was also implemented according to the proposed architecture by weighted Euclidean distance function and KNN algorithm with PHP programming language. We also normalized the collected data using the maximum-minimum normalization method.
In the proposed CBR system, the problem-solving cycle consists of four steps; retrieval of the similar case(s) to the new problem (retrieval), using the retrieved solution to answer the new problem (reuse), reviewing the new suggested solution (revise), and maintaining the new case and using it for the future problems (retain).
-
Retrieval: To find a similar case(s), the weighted Euclidean distance similarity function and the KNN algorithm were applied.
-
Reuse: During the CBR cycle, the suggested solutions are presented as "Approved" or "Unapproved". In this step, the user can apply the system suggestions as a solution to the new problem ("Approved" case).
-
Revise: When the proposed solution was not approved by the neonatologist, the new case is considered "Unapproved" and resolved by the neonatologist.
-
Retain: Since the important part of the CBR cycle is learning from the previous cases, the solved problems are maintained in temporary tables. For this purpose, after finding and displaying the system response to the user, this response is temporarily stored in a temporary table, and after the determination of the final neonate's status in the real environment (alive/dead and LOS), the outcome is recorded in the system and transferred from the temporary table to the permanent case base.
We finally developed the web-based CBR prediction system for neonatal survival and LOS, which is available at www.neonatalcdss.ir. Figures 1 and 2 show the data entry and output views for survival and the LOS prediction system. The details related to this system are presented in sections "Evaluation phase" and "Acceptability and confidence evaluation".
(a) Data entry for survival prediction system, (b) data entry for LOS prediction system.
(a) Output view for survival prediction system, (b) output view for LOS prediction system.
Evaluation phase
Retrospective evaluation
The original dataset for the evaluation included 336 records which contained 323 alive (96.13%) and 13 dead (3.87%) neonates with an average LOS of 8.5 days (0–86 days). The distribution of qualitative and quantitative variables of these neonates is presented in Supplementary Table S2.
The performance evaluation on the unbalanced case base regarding neonatal survival showed that the accuracy (97.02%), precision (98.15%), specificity (53.84%), sensitivity (98.76%), F-score (0.984), Matthews Correlation Coefficient (MCC) (0.57), and Kappa coefficient (0.624). In addition, the results of system performance on the neonatal LOS showed the RMSE was 4.79 days (Table 1). The results for the balanced case base showed an improved performance. Supplementary Tables S3–S7 shows the confusion matrix.
External validation
During the implementation period for the external validation, 92 neonates were admitted and included in the analysis. 74 (80.43%) neonates were finally alive, and 18 (19.57%) were dead. The average LOS was 11.39 days (1–90 days). The characteristics of these neonates regarding the selected qualitative and quantitative features are shown in Supplementary Table S8.
External validation on the unbalanced case base showed the accuracy and specificity measures were 97.82% and 88.88%, respectively. Furthermore, the kappa coefficient was 0.928 which indicated a very good agreement between the system predictions and the real outcome. In addition, the external validation for the neonatal LOS showed the RMSE was 3.49 days (Table 2). Furthermore, the system performance for the balanced case base indicated an improved performance compared to the original case base (Table 2). Given that, the external validation was performed in another hospital using prospectively collected data, the improved results show that the system can be used accurately in other healthcare centers. The confusion matrix is presented in Supplementary Tables S3–S7.
Acceptability and confidence evaluation
The acceptance and confidence levels were evaluated by using a questionnaire with a 5-Likert scale ranging from one to five (more details in “Acceptability and confidence evaluation”). The Physicians’ acceptance of survival prediction system outputs was higher than the LOS prediction system. For the survival prediction system, the mean score for acceptability and confidence were 4.88 and 4.25, respectively. Furthermore, the physicians’ acceptance and confidence in LOS prediction system responses were 4.96 and 3.96, respectively.
Usability evaluation
Table 3 shows the completed tasks and the mean and standard deviation of the completion time for each task per second. All users performed all tasks successfully.
As presented in Table 3, the longest time to perform a task was related to “registration in the system” (202.6 s). It was followed by “inputting the data related to a new case in the neonatal survival system and retrieving similar cases” (111.6 s).
Analysis of think-aloud data indicated that a set of 17 problems were identified. We categorized these problems into groups of "interface design problems", "notifications and guides", as well as "editing the elements". The interface design problems were related to adjustments to the interface and the customized profile. The problem with notifications was related to displaying the notifications pending to be verified by the system administrator for confirming the new users and the new cases. Another suggestion was adding a guideline for the data entry into the system. Further, the “editing the elements” was related to the modification of some terminologies, clarifying the measurement scales of variables, and using a single protocol at the time of data entry.
We categorized these problems and suggestions according to the Nielsen severity scale. A score of 0 means this is not a usability problem at all (2 issues), a score of 1 means cosmetic problems that do not need to be fixed unless extra time is available on the project (8 issues), a score of 2 indicates minor and low priority usability problems but are important to be fixed (4 issues), score 3 means major usability problems which are important to be fixed, so should be given high priority (3 issues), and score 4 reflects usability catastrophic problems that are imperative to be fixed before the product can be released (0 issues)34. In other words, more than half of the usability problems (12 issues out of 17) were due to appearance problems with low usability priority. Further, ten positive comments were provided by the users concerning the system functionalities. Furthermore, the final score of the participants for the SUS was 80.71, suggesting the high usability of the system.
Discussion
We developed, implemented, and evaluated a CBR system to predict neonatal deaths and LOS in NICUs. Our evaluation on the retrospective data with the balanced case base indicated 0.986 and 4.78 days for F-score and RSME, respectively. The results of external validation on the balanced case base indicated 0.993 for the F-score and 3.27 days for the RMSE. In another study that was conducted on the same dataset, different feature selection methods like neonatologists' opinions, and statistical and machine learning methods were performed, and the results showed that the neonatologists' opinions resulted in a better performance35. Therefore, we implemented the CBR system based on features selected by neonatologists.
In another study, the accuracy of a CBR system to detect preterm labor and premature births was 90%29. Jaskari’s study36 for predicting neonatal mortality had a good AUC (0.922) and an F-score (0.477) for the RF classifier. Cooper et al.26 introduced a postoperative mortality risk prediction; their model AUC for the development and validation phases was 0.91 and 0.87, respectively. Beluzos et al.37 developed a new decision-support method for classifying neonates based on neonatal mortality risk and obtained accuracy and AUC of 93% and 0.965, respectively. Other researchers developed a fuzzy expert system to predict neonatal mortality risk with 90% accuracy13. A decision tree-based decision support system (DSS) was reported with 63.24% sensitivity and 99.95% specificity for predicting mortality after 10 min, and 63.24% sensitivity and 91.97% specificity for twin pregnancy deaths18. A pediatric death risk prediction was introduced by combining CBR, ANN, and fuzzy methods with 89% accuracy28. The accuracy, F-score, and sensitivity measures for predicting mortality in these studies were lower than in our study. In Sheikhtaheri’s study35, which was conducted on the same data as the current study, ANN outperformed other machine learning models; however, their ANN has a lower performance compared to our suggested CBR system.
The accuracy of the LOS prediction system provided by Coimbra et al. was 84.9%27. Pepler’s suggested model had an AUC and accuracy of 0.85 and 86.4%, respectively for death prediction and R2 = 0.70 for LOS prediction9. Among the reviewed studies, only in one study28, the system has been implemented in a real environment and performed external validation. Table 4 summarizes the related studies.
One of the factors affecting the approval of DSSs is the users’ acceptance and a usable user interface. In this regard, cognitive methods have gained popularity for identifying system usability problems. Meanwhile, an important prerequisite in designing an effective user interface is minimizing cognitive demands38. In this study, according to the Nielsen scale, more than half of the usability problems (12 out of 17 cases) were due to the appearance of the system with low priority. Further, the SUS score (80.71) indicates that the system is considered usable and user-friendly by neonatologists.
In addition, the evaluation of the system acceptability suggested that neonatologists were satisfied and confident in the outputs of the system for most cases. Other researchers evaluated the usability of a DSS for antibiotic prescription in NICUs39, mobile applications for perinatal period health40, and mobile applications for pregnant women41 and highlighted the importance of system usability in this field. For example, in line with our results, physicians indicated the importance of the appearance and design of the user interface of a DSS for prescribing antibiotics42. Other usability evaluations of CBR systems indicated the importance of learnability, memorability43, ease of use, and confidence in these systems44. These studies suggest that CBR systems are accepted by physicians if they are designed properly.
The main audience of this study is neonatologists, who through the system can make decisions on determining the outcomes of neonates at the time of admission in NICUs. In addition, the use of the system to identify neonates at risk of death allows for developing a specialized team for better decision-making and providing advanced care for neonates at higher risk. Moreover, healthcare providers and hospital managers would be able to allocate and plan properly for hospital resources to manage NICU beds and workload. Application of this system can lead to an improved notification to parents about the duration required for their babies’ hospitalization in NICU as well as the healthcare costs plus notification to them about the neonate’s outcome as well as their psychological preparation.
Some limitations should be considered. The number of dead neonates was far lower than the live ones. Despite creating artificial data for the mortality class, “specificity” was low for the retrospective data. However, external validation suggested an improved performance of the system for predicting neonate survival and LOS. Further, the small number of samples for the external validation was due to the implementation of this system in only one hospital in 3 months which is another limitation of this study. It is suggested that the system should be implemented in more hospitals and evaluated with more samples. In addition, we applied simple methods (mean, median, and mode replacement) to impute the missing values, it is suggested that other researchers apply more advanced imputing methods in future studies.
In conclusion, we introduced a web-based CBR system for predicting neonatal death and LOS in NICUs. The evaluation showed that the system has a good performance in predicting neonatal survival and LOS based on similar cases. Moreover, the system outputs are mostly acceptable and trustable.
Methods
Study design and settings
This study was performed in Tehran, Iran on NICU-admitted neonates. To design the system, we used the data available in the "Maternal, Fetal, and Neonatal Research Center" as an academic center. External validation of the system was performed prospectively with the admitted neonates to the NICU of “Yas” hospital affiliated with Tehran University of Medical Sciences (TUMS).
Development phase
Feature selection and weight calculation
To identify risk factors, we conducted a systematic review published elsewhere11. We also considered another systematic review on LOS predictors in NICUs published in 20154, and other studies published after this review3,12,16,31,45,46,47,48,49,50 to identify LOS predictors.
We asked the neonatologists about the importance of these risk factors using a 5-point scale questionnaire (lowest to highest importance). The questionnaire consisted of 47 identified risk factors for neonatal death and 37 ones for LOS. It was distributed among all 30 neonatologists in Tehran, Iran and finally was completed by 22 individuals (73% response rate). A minimum of 60 percent agreement on the importance (very important and important) of each variable was considered as the selection criteria. Based on our analysis, 20 and 14 risk factors were considered important for the prediction of neonatal mortality and LOS, respectively, but some of them were not recorded in the dataset (1 variable: mother infection) and were excluded. In addition, 3 variables (septic shock, septicemia, and prematurity) had high missing values (more than 70%); therefore, we did not extract data about these variables.
Accordingly, risk factors for neonatal mortality include 17 items including steroid use, surfactant administration, mother chronic disease, prenatal care, preterm birth, BW, small for gestational age (SGA), respiratory distress syndrome (RDS), asphyxia, GA, sepsis, congenital heart disease (CHD), mechanical ventilation, intubation, intraventricular hemorrhage (IVH), necrotizing enterocolitis (NEC) and pulmonary hemorrhage. Furthermore, 13 selected risk factors for LOS prediction in NICU included steroid use, prenatal care, BW, GA, SGA, RDS, sepsis, CHD, mechanical ventilation, NEC, NEC therapy, pulmonary hemorrhage, and multiple gestations.
Moreover, we considered the mean value of each variable as the weights assigned by the neonatologists to design the CBR system. We normalized these weights using the linear normalization method between 0 and 151,52. Table 5 lists the neonatal mortality and LOS predictors along with the normalized weights.
Dataset description and data preprocessing
To develop the system, we used data related to NICUs discharged neonates (alive/dead) from a neonatal registry database from "The Maternal, Fetal, and Neonatal Research Center", TUMS. The inclusion criteria were admitted neonates to the NICUs who were born in the same hospital; therefore, transferred cases to the hospital due to surgery and other reasons, and readmission cases were excluded. The extracted data was from 6 July 2017 to 6 July 2018 and included 1682 records. We randomly selected 80% of the data to develop the case base (1346 records, 1225 alive, and 121 dead).
We developed the survival and LOS case bases separately based on the identified features. The selected features consisted of quantitative and qualitative data. Quantitative variables included BW and GA, and other variables were in the form of "Yes" or "No" values. In general, the missing data were seen in 16 of the variables; the highest rate was for NEC therapy (1.07%) and the lowest rate was for RDS (0.05%). Imputing the missing value was done according to the distribution of that variable in that class. It means that the mean for the death class and the alive class was applied separately, not the mean of the entire dataset. Also, if the data distribution was normal, we used the mean, otherwise, the median was used. As for Boolean data, we considered the distribution of values (ratio) in that variable in each class (not the ratio in the entire dataset). Details on the frequency of missing data in different features are presented in Table 6.
Furthermore, the frequency of records in each class was imbalanced, so that the "dead" to "alive” class ratio was 1 to 10 (only 121 records for the dead class). Therefore, we balanced the data using the synthetic minority over-sampling technique (SMOTE) method to develop the system53.
Proposed CBR architecture
After reviewing the existing articles and architectures for CBR systems, we developed an initial architecture54,55,56. The proposed architecture was discussed in an expert panel meeting with neonatologists, and accordingly, the final architecture for CBR predicting system for neonatal survival and LOS at NICUs was developed (Fig. 3).
CBR system architecture for predicting neonatal survival and LOS in the NICU.
As shown in Fig. 3, the final architecture consists of three modules: data acquisition module, CBR module, and management module. In the data acquisition module, similar to the proposed architecture by Lin et al.57, developing the database is performed based on the determined variables for mortality and LOS and experts assign the weights. After the data normalization process, we developed the case base of the CBR system which includes the previously solved cases. The system administrator can update the case base through the data acquisition module.
After a user enters a new case, the CBR module retrieves similar cases based on the previous cases in the case base (search process in the case base) using the weighted Euclidean distance similarity function (Eq. 1) and KNN algorithm (K is determined by the user).
The percentage of similarity between each of the retrieved cases with the new case is also shown to the user. In these conditions, similar to various studies58,59,60, two solution states "solved" and "unsolved" are considered for the new problem. If based on the similarity percentage and the specialist opinion, the retrieved cases match the new problem, one of the system solutions (previously solved cases) is determined as the solution to the new problem, and then that solution is regarded as verified. On the other hand, if the retrieved solutions cannot be used for the new problem, the new case solution is considered unapproved. Then, similar to some studies55,61, the specialist predicts the status of the hospitalized newborn (alive/dead and LOS) according to his/her knowledge and experience, and after his/her verification, the solution to the new problem is stored in a temporary table. Eventually, once the conditions of the neonate become clear (real discharge status: alive/dead and LOS), similar to previous studies23,54,55,57,58,61,62, the system administrator registers the final status of the new cases based on the neonate medical record in the system. With his/her final verification, these cases would be transferred from the temporary case base to the main case base and would be stored there. In the management module, cases can be added, deleted, or edited by the administrator.
Evaluation phase
Retrospective evaluation
Before the web-based system implementation, the system performance was evaluated using retrospective data. For this purpose, we used 20% of the total collected data (336 records) and compared the results of the system with the actual patients’ outcomes.
Prospective evaluation and external validation
To perform an external evaluation, we implemented the system for 3 months (20 April 2020 to 20 July 2020) in the NICU of "Yas" hospital affiliated with TUMS. During this period, 92 neonates were admitted and their data from admission time was entered into the system by their physicians and the system predictions were recorded into the system. We followed up the neonates until discharge, and after determining the neonatal status at discharge, we recorded the actual outcomes and compared them with the system prediction.
Acceptability and confidence evaluation
During the system implementation, we evaluated the acceptability of the system responses by the physicians in the NICU of "Yas" hospital (including three neonatologists and four pediatricians). We evaluated the acceptance level using a researcher-made questionnaire that consisted of two questions to assess physicians' acceptance and confidence level with the system's responses. Questions were on a 5-Likert scale ranging from one (very low) to five (very high). After using the system for each neonate, physicians recorded the system predictions and also their acceptance and confidence level in the system predictions.
Usability evaluation
We evaluated the system usability both quantitatively and qualitatively. For the qualitative evaluation, we applied the think-aloud method. The users were asked to perform the tasks developed by the researchers and a neonatologist based on specific scenarios and then expressed their opinions through the think-aloud method. Completion of the scenarios and users’ opinions was recorded audio-visually using a recorder, video capturing camera, and Camtasia software.
During the usage of the system, the researcher also asked questions from the users to encourage them to continue thinking aloud. The scenarios included five tasks (Supplementary Table S9). Each task involved a set of subtasks ranging between three to five activities. If all subtasks related to the main task were performed completely, it would be considered a successful task, while performing some parts of a task was not acceptable.
We also used the System Usability Scale (SUS) questionnaire to measure the general system usability. SUS includes 10 items as five options, with the response to them, ranked from 1 (absolutely disagree) to 5 (absolutely agree)63. The reliability and validity of the SUS have already been confirmed64.
Data analysis
We used Statistical Package for the Social Sciences (SPSS) software version 20 to analyze the data. We analyzed the data using a confusion matrix and performance measures including accuracy, precision, sensitivity, specificity, F-score, MCC, and Kappa coefficient based on nearest neighbor (K = 1)65 to determine the performance of the neonatal survival system, and Root Mean Square Error (RMSE) measure for LOS prediction system. We applied standard formulas to calculate these metrics (Supplementary Tables S3–S7).
The acceptability of the system was calculated using the mean of scores. We scored responses from one (very low) to five (very high) and calculated the mean of scores obtained from users.
We also applied descriptive statistics to analyze quantitative data from the SUS questionnaire based on standard methods. The final score is between zero and 100. A score of 70 or higher indicates that the system is usable, with 50 to 70 as the boundary point and less than 50 indicates that the system is unacceptable in terms of usability64,66.
We transcribed and analyzed the participants’ opinions from think-aloud using the content analysis method and organized the opinions. Furthermore, we used the Nielsen severity scale34 to classify the usability problems. To this end, all of the identified problems were discussed and classified by the research team. Finally, we calculated the time duration of the performed actions.
Ethics declarations
This study received ethical approval from the research ethics committee of Iran University of Medical Sciences, Tehran, Iran (IR. IUMS.REC1396.9321481001). The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and the research regulations of the country.
Considering the retrospective nature of the study, Informed consent was waived by the ethics committee of the Iran University of Medical Sciences.
Data availability
The data are not available because of the confidentiality policy of the neonatal registry of the Maternal, Fetal, and Neonatal Research Center. Data are however available from the authors upon reasonable request and with permission of the Maternal, Fetal, and Neonatal Research Center. The corresponding author should be contacted in case one needs to access the data.
References
Sharrow, D., Hug, L., Yang, L., & You, D. Levels and trends in child mortality (Report 2020). (Unicef, World Health Organization, World Bank, UN-DESA Population Division, 2020).
Sankar, M. J. et al. When do newborns die? A systematic review of timing of overall and cause-specific neonatal deaths in developing countries. J. Perinatol. 36(Suppl 1), S1–S11 (2016).
Manktelow, B. N., Draper, E. S. & Field, D. J. Predicting neonatal mortality among very preterm infants: A comparison of three versions of the CRIB score. Arch. Dis. Child Fetal Neonatal. Ed 95, F9-13 (2010).
Seaton, S. E. et al. What factors predict length of stay in a neonatal unit: A systematic review. BMJ Open 6, 1–10 (2016).
Shetty, S., Kennea, N., Desai, P., Giuliani, S. & Richards, J. Length of stay and cost analysis of neonates undergoing surgery at a tertiary neonatal unit in England. Ann. R Coll. Surg. Engl. 98, 56–60 (2016).
Bhatti, K. M. et al. Factors responsible for the prolonged stay of surgical neonates in intensive care units. Sultan Qaboos Univ. Med. J. 15, e91–e97 (2015).
Temple, M. W., Lehmann, C. U. & Fabbri, D. Predicting discharge dates from the NICU using progress note data. Pediatrics 136, e395–e405 (2015).
Ezz-Eldin, Z. M., Hamid, T. A. A., Youssef, M. R. L. & Nabil, H. E.-D. Clinical risk index for babies (CRIB II) scoring system in prediction of mortality in premature babies. JCDR 9, SC08–SC11 (2015).
Pepler, P. T., Uys, D. W. & Nel, D. G. Predicting mortality and length-of-stay for neonatal admissions to private hospital neonatal intensive care units: A Southern African retrospective study. Afr. Health Sci. 12, 166–173 (2012).
Chow, S. et al. A selected review of the mortality rates of neonatal intensive care units. Front. Public Health 3, 1–18 (2015).
Kermani, F., Sheikhtaheri, A., Zarkesh, M. R. & Tahmasebian, S. Risk factors for neonatal mortality in Neonatal Intensive Care Units (NICUs): A systematic literature review and comparison with scoring systems. J. Pediatr. Neonat. Ind. Med. 9, 1–15 (2020).
Hinchliffe, S. R. et al. Modelling time to death or discharge in neonatal care: An application of competing risks. Paediatr. Perinat. Epidemiol. 27, 426–433 (2013).
Safdari, R., Kadivar, M., Langarizadeh, M., Nejad, A. F. & Kermani, F. Developing a fuzzy expert system to predict the risk of neonatal death. Acta Inform. Med. 24, 34–37 (2016).
Mack, E. H., Wheeler, D. S. & Embi, P. J. Clinical decision support systems in the pediatric intensive care unit. Pediatr. Crit. Care Med. 10, 8–23 (2009).
Lu, J., Bai, D., Zhang, N., Yu, T. & Zhang, X. Fuzzy case-based reasoning system. Appl. Sci. 6, 1–14 (2016).
Lee, H. C., Bennett, M. V., Schulman, J., Gould, J. B. & Profit, J. Estimating length of stay by patient type in the neonatal intensive care unit. Am. J. Perinatol. 33, 751–757 (2016).
Huerta, L., Huesca, J., Cabrera, N., Ruiz, J. & Hernandez, L. On the data mining process for classification of fetal death causes. IJCSIS 13, 1–8 (2015).
Gunaratnam, M. A. A web-based perinatal decision support system framework using a knowledge-based approach to estimate clinical outcomes: neonatal mortality and preterm birth in twins pregnancies, Carleton University, (2013).
Hüllermeier, E. Case-based approximate reasoning. Vol. 44 (Springer, 2007).
Richter, M. M. & Weber, R. O. Case-based reasoning. (Springer, 2016).
Bichindaritz, I. & Montani, S. Advances in case-based reasoning in the health sciences. Artif. Intell. Med. 51, 75–79 (2011).
Andritsos, P., Jurisica, I. & Glasgow, J. I. in Springer Handbook of Bio-/Neuroinformatics (ed Kasabov N) 207–221 (Springer, 2014).
Begum, S., Ahmed, M. U., Funk, P., Xiong, N. & Von Schéele, B. A case-based decision support system for individual stress diagnosis using fuzzy similarity matching. Comput. Intell. 25, 180–195 (2009).
Aamodt, A. & Plaza, E. Case-based reasoning: Foundational issues, methodological variations, and system approaches. AI Commun. 7, 39–59 (1994).
Ravelli, A. C. et al. Antenatal prediction of neonatal mortality in very premature infants. Eur. J. Obstet. Gynecol. Reprod. Biol. 176, 126–131 (2014).
Cooper, J. N., Minneci, P. C. & Deans, K. J. Postoperative neonatal mortality prediction using superlearning. J. Surg. Res. 221, 311–319 (2018).
Coimbra, A. et al. in Intelligent Decision Technologies 2016 Intelligent Decision Technologies 2016 (eds Ireneusz Czarnowski, Alfonso Mateos Caballero, Robert J. Howlett, & Lakhmi C. Jain) 115–128 (Springer Cham, 2016).
Rodríguez, Y. et al. in Mexican International Conference on Artificial Intelligence (eds Gelbukh A & Morales E) 472–481 (Springer, Zaragoza, Mexico, 2008).
Adawiyah, R. Implementation of the Minkowsky distance method for detection of preterm births based on case-based reasoning. JIKO 3, 36–41 (2020).
Wen, J. G. et al. Prediction of neonatal outcomes in extremely preterm neonates. Pediatrics 132, e876–e885 (2013).
Hintz, S. R. et al. Predicting time to hospital discharge for extremely preterm infants. Pediatrics 125, e146–e154 (2010).
López, B., Gay, P., Pla, A. & Pous, C. in 2nd International Workshop on Practical Applications of Computational Biology and Bioinformatics (IWPACBB 2008) (eds Corchado J, Paz J, Rocha M, & Fernández Riverola F) 45–54 (Springer, 2009).
Rezaeian, A., Nasimi, F. & Pooralizadeh Moghadam, F. Predicting mortality rate of preterm infants in neonatal intensive care unit using artificial neural network model. J. Mazandaran Uni. Med. Sci. 26, 85–94 (2015).
Nielsen J. Usability engineering. (Morgan Kaufmann, 1994).
Sheikhtaheri, A., Zarkesh, M. R., Moradi, R. & Kermani, F. Prediction of neonatal deaths in NICUs: Development and validation of machine learning models. BMC Med. Inform. Decis. Mak. 21, 1–14 (2021).
Jaskari, J. et al. Machine learning methods for neonatal mortality and morbidity classification. IEEE Access 8, 123347–123358 (2020).
Beluzos, C. E. et al. Towards neonatal mortality risk classification: A data-driven approach using neonatal, maternal, and social factors. Inform. Med. Unlock. 20, 100398 (2020).
Khajouei, R., Hasman, A. & Jaspers, M. W. Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system. Int. J. Med. Inform. 80, 341–350 (2011).
Sheehan, B., Kaufman, D., Bakken, S. & Currie, L. M. Cognitive analysis of decision support for antibiotic ordering in a neonatal intensive care unit. Appl. Clin. Inform. 3, 105–123 (2012).
Connor, K., Wambach, K. & Baird, M. B. Descriptive, qualitative study of women who use mobile health applications to obtain perinatal health information. J. Obstet. Gynecol. Neonatal. Nurs. 47, 728–737 (2018).
Van Beukering, M. et al. Usability and usefulness of a mobile health app for pregnancy-related work advice: Mixed-methods approach. JMIR Mhealth Uhealth 7, 1–13 (2019).
Akhloufi, H. et al. A usability study to improve a clinical decision support system for the prescription of antibiotic drugs. PLoS ONE 14, e0223073 (2019).
Ehtesham, H. et al. Developing a new intelligent system for the diagnosis of oral medicine with case-based reasoning approach. Oral. Dis. 25, 1555–1563 (2019).
Nordstoga, A. L. et al. Usability and acceptability of an app (SELFBACK) to support self-management of low back pain: Mixed methods study. JMIR Rehabil. Assist. Technol. 7, e18729 (2020).
Altman, M., Cnattingius, S. & Norman, M. Moderately preterm infants and determinants of length of hospital stay. Arch. Dis. Child Fetal Neonatal. Ed 94, F414–F418 (2009).
Bender, G. K. D. et al. Neonatal intensive care unit: Predictive models for length of stay. J. Perinatol. 33, 1–15 (2013).
Berry, M. A., Brouillette, R. T. & Hellmann, J. Predictors of mortality and length of stay for neonates admitted to children’s hospital neonatal intensive care units. J. Perinatol. 28, 297–302 (2008).
Kurek Eken, M. T. A. et al. Evaluation of the maternal and fetal risk factors associated with neonatal care unit hospitalization time. J. Matern. Fetal Neonatal. Med. 29, 3553–3557 (2016).
Lee, H. C., Schulman, J. & Gould, J. B. Accounting for variation in length of NICU stay for extremely low birth weight infants. J. Perinatol. 33, 872–876 (2013).
Rozanska, A.W.-M.J. et al. Infections and risk-adjusted length of stay and hospital mortality in Polish neonatology intensive care units. Int. J. Infect. Dis. 35, 87–92 (2015).
Lakshmi, M., Vetriselvi, K., Joshuva-Anand, J. & Prasanna-Venkatesan, A. A Study on different types of normalization methods in adaptive technique for order preference by similarity to ideal solution (TOPSIS). Int. J. Eng. Res. Technol. 4, 1–7 (2016).
Pal, S. K. & Shiu, S. C. Foundations of soft case-based reasoning. (John Wiley & Sons, 2004).
Chawla, N. V., Bowyer, K. W., Hall, L. O. & Kegelmeyer, W. P. SMOTE: Synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002).
Choy, K. L. T. et al. An intelligent case-based knowledge management system for quality improvement in nursing homes. VJIKMS 48, 103–121 (2017).
Tyagi, A. & Singh, P. ACS: Asthma care services with the help of case base reasoning technique. Proc. Comput. Sci. 48, 561–567 (2015).
Yin, Z. et al. A clinical decision support system for the diagnosis of probable migraine and probable tension-type headache based on case-based reasoning. J. Headache Pain 16, 1–9 (2015).
Lin, R.-H. An intelligent model for liver disease diagnosis. Artif. Intell. Med. 47, 53–62 (2009).
Rueangsirarak, W., Atkins, A., Sharp, B., Chakpitak, N. & Meksamoot, K. in 5th International Conference on Software, Knowledge Information, Industrial Management and Applications (SKIMA) 1–5 (IEEE, Benevento, Italy, 2011).
Kovasznai, G. in 6th IEEE International Symposium on Applied Computational Intelligence and Informatics (SACI) 505–509 (IEEE, Timisoara, Romania 2011).
Wongpun, S. & Guha, S. in 2nd Advanced Information Technology, Electronic and Automation Control Conference (IAEAC) 548–552 (IEEE, Chongqing, China 2017).
Chakraborty, B., Srinivas, S. I., Sood, P., Nabhi, V. & Ghosh, D. in GCC Conference and Exhibition 132–135 (IEEE, Dubai, United Arab Emirates 2011).
Chattopadhyay, S., Banerjee, S., Rabhi, F. A. & Acharya, U. R. A Case-Based Reasoning system for complex medical diagnosis. Expert. Syst. 30, 12–20 (2013).
Brooke, J. SUS—A quick and dirty usability scale. Usabil. Eval. Ind. 189, 4–7 (1996).
Bangor, A., Kortum, P. T. & Miller, J. T. An empirical evaluation of the system usability scale. Int. J. Hum. Comput. Interact. 24, 574–594 (2008).
El-Sappagh, S., Elmogy, M. & Riad, A. M. A fuzzy-ontology-oriented case-based reasoning framework for semantic diabetes diagnosis. Artif. Intell. Med. 65, 179–208 (2015).
Bangor, A., Kortum, P. & Miller, J. Determining what individual SUS scores mean: Adding an adjective rating scale. J. Usabili. Stud. 4, 114–123 (2009).
Acknowledgements
We appreciate the "Maternal, Fetal, and Neonatal Research Center", Tehran University of Medical Sciences, Tehran, Iran, to provide data for this study.
Funding
The study was supported by the Iran University of Medical Sciences (grant No: 9321481001).
Author information
Authors and Affiliations
Contributions
A.S. and F.K. had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: A.S. and F.K. Acquisition, analysis, or interpretation of data: A.S., F.K., M.Z., and M.V. Drafting of the manuscript: A.S. and F.K. Critical revision of the manuscript for important intellectual content: A.S. and F.K. Statistical analysis: A.S. and F.K. Administrative, technical, or material support: A.S., F.K., and M.V. Study supervision: A.S. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Kermani, F., Zarkesh, M.R., Vaziri, M. et al. A case-based reasoning system for neonatal survival and LOS prediction in neonatal intensive care units: a development and validation study. Sci Rep 13, 8421 (2023). https://doi.org/10.1038/s41598-023-35333-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41598-023-35333-y
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.
