Recalibration and validation of the Charlson Comorbidity Index in an Asian population: the National Health Insurance Service-National Sample Cohort study

Weights assigned to comorbidities in predicting mortality may vary based on the type of index disease and advances in the management of comorbidities. We aimed to develop a modified version of the Charlson Comorbidity Index (CCI) using an Asian nationwide database (mCCI-A), enabling the precise prediction of mortality rates in this population. The main data source used in this study was the National Health Insurance Service-National Sample Cohort (NHIS-NSC) obtained from the National Health Insurance database, which includes health insurance claims filed between January 1, 2002, and December 31, 2013, in Korea. Of the 1,025,340 individuals included in the NHIS-NSC, 570,716 patients who were hospitalized at least once were analyzed in this study. In total, 399,502 patients, accounting for 70% of the cohort, were assigned to the development cohort, and the remaining patients (n = 171,214) were assigned to the validation cohort. The mCCI-A scores were calculated by summing the weights assigned to individual comorbidities according to their relative prognostic significance determined by a multivariate Cox proportional hazard model. The modified index was validated in the same cohort. The Cox proportional hazard model provided reassigned severity weights for 17 comorbidities that significantly predicted mortality. Both the CCI and mCCI-A were correlated with mortality. However, compared with the CCI, the mCCI-A showed modest but significant increases in the c statistics. According to the analyses using continuous net reclassification improvement, the mCCI-A improved the net mortality risk reclassification by 44.0% (95% confidence intervals (CI), 41.6–46.5; p < 0.001). The mCCI-A facilitates better risk stratification of mortality rates in Korean inpatients than the CCI, suggesting that the mCCI-A may be a preferable index for use in clinical practice and statistical analyses in epidemiological studies.


Results
Baseline characteristics of the entire cohort. The baseline characteristics of the entire cohort, development cohort and validation cohort are listed in Table 1 of the 570,716 participants, 46.3% of the patients were men. The overall mortality rate was 3.83%.
In total, 74.43% of the subjects had one or more comorbidities. Among the 17 comorbidities, the most prevalent comorbidity was chronic pulmonary disease (47.56%), followed by ulcer disease (37.1%) and mild liver disease (24.05%) ( Table 2). Figure 1 shows the adjusted hazard ratios (HRs) and weights of each comorbidity. All comorbidities were signifi- Table 1. Baseline characteristics of the development and validation cohort. a Family income ratio was divided into the following 11 groups: medical aid with group 0 and the income decile with 10 equal -sized groups according to the rank of the gross household income and registered National Health Insurance (based on 2010).  www.nature.com/scientificreports/ cantly associated with mortality. Metastatic solid tumors were assigned the highest weight, followed by AIDS, moderate or severe liver disease, and any tumor, consecutively (Table 3). Compared with the weights in the CCI, in the mCCI-A, the updated weights for cerebrovascular disease, myocardial infarction (MI), congestive heart failure (CHF), dementia, any tumor and moderate or severe liver disease increased; the updated weights for acquired immunodeficiency syndrome (AIDS) and metastatic solid tumors decreased; and the updated weights for diabetes without complication, diabetes with end organ damage, hemiplegia, and moderate or severe renal disease did not change (Fig. 1).

Development of the New Comorbidity Index (mCCI-A) for the prediction of mortality.
The mCCI-A scores were calculated by summing the updated weights. The scores were applied to each patient in the development cohort. Based on the CCI and mCCI-A scores, we categorized both scores into the following 4 risk groups: ≤ 50th percentile, 50th-80th percentile, 80th-90th percentile and > 90th percentile. To determine the cut-off values of the comorbidity scores in each risk group, we generated a histogram to represent the distributions of the scores (Fig. 2). The cut-off values of CCI scores corresponding to the 50th, 80th and 90th percentiles were 1, 3 and 5, respectively, whereas those of the mCCI-A scores were 1, 4 and 6, respectively. Figure 3 shows the survival curves obtained using the Kaplan-Meier method in the development cohort, which were differentiated by the 4 risk groups of the CCI and mCCI-A. The risk groups in each index could discriminate the survival rates, indicating that increasing comorbidity index scores were associated with lower cumulative survival.
Application and validation of the mCCI-A. The baseline characteristics of the validation cohort (n = 171,214) are listed in Table 1. Males represented 46.26% of the subjects. The overall mortality was 3.83%. In total, 74.37% of the subjects had one or more comorbidities. Among the 17 comorbidities, the most prevalent comorbidity was chronic pulmonary disease (47.44%), followed by ulcer disease (37.17%) and mild liver disease (24.1%) ( Table 2).
To assess the discrimination ability of each index, the c statistic and cNRI were calculated after adjusting for confounders in the following two analyses: univariate and multivariate. Significant differences were observed in the c statistics between the CCI and the mCCI-A in the univariate and multivariate analyses (Table 4). Additionally, a significant risk reclassification improvement was observed in the mCCI-A in the univariate and multivariate analyses using cNRI. In the multivariate analysis, compared with the CCI, the mCCI-A significantly improved the net mortality risk reclassification by 44.0% (95% CI 41.6-46.5; p < 0.001), indicating that the mCCI-A facilitates a better risk stratification of mortality in Korean inpatients than the CCI. Figure 3 shows the survival curves obtained using the Kaplan-Meier method in the validation cohort, which were differentiated by the 4 risk groups of the CCI and mCCI-A.
Application and validation of the mCCI-A to specific diseases. To assess the discrimination ability of each index, the c statistic and cNRI were calculated after adjusting for confounders in multivariate analysis in subgroups such as mild liver disease, chronic pulmonary disease, diabetes mellitus and moderate/severe renal disease patients composed of the total cohort. Significant differences were observed in the c statistics and cNRI between the CCI and the mCCI-A in multivariate analyses (Table 5). Table 3. Weights for comorbidities in the development cohort. a Adjusted for age (10 groups), sex, region, family income ratio, and all comorbidities.

Discussion
In this study, we modified the CCI using a nationwide population-based database and to developed a comorbidity index that provides better risk predictions in the general inpatient population. To the best of our knowledge, this study is the largest study to modify the CCI using a nationwide population-based database and develop a comorbidity index that provides better risk prediction in a general inpatient Asian population. Since 1987, the CCI has been used as a comorbidity index throughout the medical community, and there have been efforts to predict the outcome of the general inpatient population using the CCI since the 1980s to 2000s. In 1996, the CCI was applied to 33,940 inpatients with ischemic heart disease 12 . In 1992, another study involving 27,111 patients who underwent lumbar spine surgery was conducted 13 . In both of the above mentioned studies, the diagnostic information used was based on International Classification of Disease, 9th revision codes (ICD-9). Using ICD codes may be useful in exploratory data analyses 14 . In the 2000s, a retrospective cohort study was conducted using National Health Insurance claims data (2001)(2002) to compare the performance of three comorbidity measurements (Elixhauser, Charlson/Deyo, and Charlson/Romano method) among inpatients hospitalized for Chronic Obstructive Pulmonary Disease and Acute Myocardial Infarction in Taiwan 15 . However, these studies had two limitations. First, these studies investigated specific disease groups rather than general patients. Second, the purpose of these studies was to demonstrate superiority among the existing methods or to apply the methods to the general population.
Previous studies have directly applied the CCI to patients using a weight equal to that in the original index for each CCI index. However, the recalibration and validation of the weights of the CCI index diseases are needed for several reasons. First, the original CCI does not reflect the significant progress achieved in the treatment of each comorbidity and medical advancements over the previous 30 years. Second, the extent to which the original CCI reflects long-term outcomes considered important is unclear because the "training" population used to develop the original CCI was created based on the one-year mortality rate in general inpatients. Third, the original CCI was developed based on a relatively small number of patients.
In 1996, a study was conducted to update the CCI and scores using USA administrative databases of 6,326 patients who underwent bypass surgery, and the new index exhibited superior performance over the original CCI (c = 0.74 vs. 0.70) 16 . A new comorbidity index was developed by assigning specific weights to the original CCI in this study. However, AIDS was excluded because no patients with AIDS were included in the sample. In 2011, a study updated the CCI and scores using Canadian administrative databases of 55,929 patients admitted to a medical facility in the Calgary region (Alberta, Canada) (population 1.3 million) 17 . The authors excluded the 5 comorbidities found to have no statistical correlation with the mortality rates among the 17 comorbidities. The new index exhibited superior performance over the original CCI (c = 0.825 vs. 0.808). Although these two studies reflect the recent advances in medical management, there are limitations in that study. The evaluation of several comorbidities was limited (particularly AIDS), and there is still a need for further evidence regarding whether the index can be directly applied to Asians.
In Asia, the CCI has also been applied to various disease groups to evaluate the various outcomes of patients. However, the usefulness of the original CCI remains controversial. A Japanese study revealed that the CCI had www.nature.com/scientificreports/ been used to predict the overall survival of patients with solid tumors; however, the CCI is not considered a significant predictor 18 . The authors emphasized the necessity for developing scales that can more accurately predict patient outcomes. Previously, we updated the CCI and scores in hemodialysis and peritoneal dialysis patients using the National Health Insurance dataset in Korea 10,11 . The first study involved 24,738 people who first started hemodialysis between 2005 and 2008 10 . We developed the mCCI-IHD, which included 14 comorbidities with reassigned severity weights. In the validation cohort, compared with the CCI, the mCCI-IHD showed modest but significant increases in the c statistics at 6 months and 1 year. Compared with the CCI, the analyses using cNRI revealed that the mCCI-IHD improved the net mortality risk reclassification by 24.6%, 26.2% and 42.8% at 6 months and 1 and 2 years, respectively. The second study involved 7,606 people who first started peritoneal dialysis between 2005 and 2008 11 . We developed the mCCI-IPD, which included 11 comorbidities with reassigned severity weights. In the validation cohort, compared with the CCI, although the mCCI-IHD showed no differences in the c statistics, the analyses using cNRI revealed that the mCCI-IHD provided a 38.2% improvement in mortality risk assessment.
Based on the results of the previous two studies, we extended this study to the entire inpatient population. In this study, we modified the CCI using administrative data that included nearly all Korean inpatients who were cohort for mCCI-A. CCI in development cohort, < 50th percentile (n = 220,258, scores 0-1); 50th-80th percentile (n = 103,849, scores 2-3); 80th-90th percentile (n = 40,558, scores 4-5) and > 90th percentile (n = 34,837, score ≥ 6). mCCI-A in the development cohort, < 50th percentile (n = 215,768, scores 0-1); 50th-80th percentile (n = 113,413, scores 2-4); 80th-90th percentile (n = 31,535, scores 5-6) and > 90th percentile (n = 38,786, score ≥ 7). CCI in the validation cohort, < 50th percentile (n = 94,378, scores 0-1); 50th-80th percentile (n = 44,517, scores 2-3); 80th-90th percentile (n = 17,467, scores 4-5) and > 90th percentile (n = 14,852, score ≥ 6). mCCI-A in the validation cohort, < 50th percentile (n = 92,507, scores 0-1); 50th-80th percentile (n = 48,490, scores 2-4); 80th-90th percentile (n = 13,626, scores 5-6) and > 90th percentile (n = 16,591, score ≥ 7). CCI, Charlson Comorbidity Index; mCCI-A, modified version of the Charlson Comorbidity Index for Asian populations.   www.nature.com/scientificreports/ and CHF) and nonmetastatic tumors are likely associated with the increased prevalence of these diseases. This tendency is consistent with recent studies investigating populations with specific diseases in Korea 10,11 . Additionally, recent advances in the effectiveness of the treatment of metastatic cancer has led to a decreased mortality rate. This change is likely due to the decreased weights of metastatic cancer in this study. The second implication is that this study, based on the overall cohort of inpatients, overcame the limitations associated with the use of small sample sizes in previous studies. For example, AIDS was not considered in a previous study modifying the CCI due to the small sample size. However, in this study, obtaining the weights of AIDS was possible because of the large sample size. This study has several limitations. First, in contrast to collecting data through chart reviews, the determination of the prevalence of diseases is generally problematic using administrative data 19 . Second, although the use of administrative data has advantages, such as the conservation of time and resources and consistency in diagnosis, the diagnoses may be inappropriate due to physician preferences regarding diagnostic codes. Third, administrative data do not include biochemical parameters such albumin and hemoglobin, which could affect the survival rate. Fourth, new values for the weight of each comorbidity were derived in this study, but no expert agreement on them could be derived and will need to be considered later. Fifth, because this study was conducted for patients from 2002 to 2013, there is a limitation to the application of hospitalized patients after 2013. Last, although there were interactions between variables and comorbidities that we considered in our analysis, the interaction test was not conducted in our study. 2% of the target population) were randomly selected until 2013 and were followed until 2013 (for 12 years). All traceable identifiers were removed before publishing to protect patient confidentiality.In this study, 578,547 patients (56.4% of the NHIS-NSC) who were hospitalized at least once were analyzed longitudinally. Except for 7,831 patients who died immediately or same month as hospitalization after admission, the total number of patients in our cohort was 570,716.

Study variables.
Using the International Classification of Diseases, 10th revision (ICD-10), we identified the Charlson comorbidities among the secondary diagnoses based on all diseases that were diagnosed from both inpatient and outpatient services before discharge 22 . To develop the modified index, we used the same comorbid conditions covered by the CCI, including MI, CHF, peripheral vascular disease, cerebrovascular disease, dementia, chronic pulmonary disease, connective tissue disease, ulcer disease, mild liver disease, diabetes, hemiplegia, diabetes with end-organ damage, any tumor (including leukemia and lymphoma), moderate to severe liver disease, metastatic solid tumors and AIDS. We retrieved all records of each patient from the National Health Insurance database prior to the date of the discharge to identify the comorbidities. A patient was considered to have a comorbid condition if the condition was present in the index admission records. The outcomes included allcause long-term mortality within the follow-up period after admission. Additionally, the National Health Insurance claims databases were used to identify mortality. Death occurring between January 1, 2002 and December 31, 2013, was considered in the development and validation cohorts. However, patients with admission and death occurring in the same month were excluded. The demographic information of both cohorts included age, sex, region, and family income.
Statistical analysis. First, to analyze the baseline characteristics of the study population, we merged the demographic and medical utilization data. A Cox regression analysis adjusted for age (quartile), sex, region and family income ratio was performed to develop new weights for the comorbidities. We obtained the adjusted HRs and 95% confidence intervals (CIs) after adjusting for age, sex, region, family income ratio, and all 17 comorbidities. The prognostic weights of the mCCI-A were computed by dividing the HRs associated with each comorbidity by the lowest HRs 23 . Then, the relative weights were truncated to integer values rounded to zero decimal places. The comorbidity score of each patient was calculated by summing the weights. Kaplan-Meier survival curves 24 were generated to compare the performance of the CCI to the performance of the mCCI-A.
We performed internal-validation partitioning of the main data set into two sets comprising 70% of the sample for training and 30% of the sample for testing using a random sampling function with the variable of death as a reference parameter. In total, 70% (n = 399,502) of the sample was used for training to develop new comorbidity weights, and the remaining 30% (n = 171,214) of the sample was used as a validation cohort.
To assess the capacity of discriminating between the indices, a c statistic was calculated using the area under the receiver-operator curve 25 . To determine the statistical significance and 95% CIs of the c statistic, a Mann-Whitney test was performed with a contrast test. The continuous net reclassification improvement (cNRI) score obtained by performing logistic regression models was calculated to evaluate the reclassification 26 . For the binary response, i.e., death, the function improveprob in R was used to determine whether the predictions obtained from the model of mCCI-A significantly differed from those obtained from the model of the original CCI 27 . The cNRI total is the sum of cNRI Event and cNRI Non-event , indicating the sum of the net proportions of subjects who died (Event) and who did not die (Non-event) were correctly reassigned a predicted risk. www.nature.com/scientificreports/ The data were analyzed by using SAS 9.4 for Windows software (SAS Institute, Cary, NC, USA) and R software version 4.0.0 (Comprehensive R Archive Network: https ://cran.r-proje ct.org). In all analyses, p < 0.05 was considered statistically significant.
Ethical approval. This study was conducted in accordance with the principles of the Declaration of Helsinki. The study was approved by the Institutional Review Board (IRB) of Seoul National University Hospital (1610-038-797), and the study protocol was approved by the IRB. Under IRB approval, the informed consent waived.

Data availability
The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.