Development and validation of a risk-score model for opioid overdose using a national claims database

Opioid overdose can be serious adverse effects of opioid analgesics. Thus, several strategies to mitigate risk and reduce the harm of opioid overdose have been developed. However, despite a marked increase in opioid analgesic consumption in Korea, there have been no tools predicting the risk of opioid overdose in the Korean population. Using the national claims database of the Korean population, we identified patients who were incidentally prescribed non-injectable opioid analgesic (NIOA) at least once from 2017 to 2018 (N = 1,752,380). Among them, 866 cases of opioid overdose occurred, and per case, four controls were selected. Patients were randomly allocated to the development (80%) and validation (20%) cohort. Thirteen predictive variables were selected via logistic regression modelling, and a risk-score was assigned for each predictor. Our model showed good performance with c-statistics of 0.84 in the validation cohort. The developed risk score model is the first tool to identify high-risk patients for opioid overdose in Korea. It is expected to be applicable in the clinical setting and useful as a national level surveillance tool due to the easily calculable and identifiable predictors available from the claims database.

Opioids are considered crucial for pain relief; however, they are associated with significant harms such as overdose 1 . In the USA, along with the rise in opioid prescription since 1999, opioid-related overdose admissions had quadrupled by 2010 2 . During 1999-2017, 702,568 drug-related deaths occurred, of which two-thirds were opioid-related 3 .
US regulatory authorities declared the opioid crisis a national public health emergency 4 . To encourage safe use of opioids, the Centers for Disease Control and Prevention published guidelines on safe opioid prescription practice in primary care settings 5,6 . Risk evaluation and mitigation strategies and prescription drug monitoring programs were implemented to encourage safe opioid use. Consequently, a decrease in opioid prescription was observed from 2012, and death by prescription opioids had decreased by 13.5% in 2017-2018 7 . However, overdose deaths involving opioids are still on the rise, and the proportion of drug-related overdose deaths due to illicit opioids has risen sharply since 2017 8 . This phenomenon might be related to the unexpected interruption of opioid prescription due to the increased prescription burden of physicians. Thus, according to experts, prescription restriction policies should be applied with caution, and each patient's situation should be considered 9,10 . For instance, patients inevitably in need of opioids, such as those with malignant pain, should receive opioid prescriptions. Therefore, to prevent opioid-related adverse outcomes, it is important to assess the individual patient risk rather than to regulate opioid prescriptions according to uniform standards.
In this context, several models for predicting opioid overdose have been reported. These included a risk index to identify patients with a risk of overdose or serious opioid-induced respiratory depression 11 and a machine learning algorithm to predict opioid overdose 12 . However, most such studies were conducted in North America [11][12][13][14][15][16][17] . Considering that the healthcare system is key in prescription practice and healthcare utilisation 18,19 , and that ethnicity influences opioid-related negative outcomes 20 , these tools should be developed or validated in domestic populations.
Although Korea has had few issues regarding opioid overdose because opioid use is lower than in other countries, a significant increase in opioid prescription trends has been observed in recent years (347.5 prescriptions/1000 person in 2009 vs. 531.3 prescriptions/1000 person in 2019) 21 . Also, the prevalence of chronic use of Study participants and outcome definition. First, adult patients who initiated NIOA use during 2017-2018 (N = 1,752,380) were defined as the NIOA cohort, and the date of the first prescription was the cohort entry date. We excluded patients with NIOA prescriptions in 2016. We defined NIOA as oral or transdermal opioid analgesics (ATC code: N02A and R05DA04), excluding tramadol, and excluded injectable formulations.
Among them, the case group with opioid overdose events was defined as follows: (A) those with occurrence of an opioid-related poisoning event, (B) those with occurrence of emergency department (ED) visits involving a naloxone injection and active NIOA prescriptions within 180 days before the event (those with naloxone injection for the first developed stroke or within the context of a surgical procedure were not included). When a patient had more than one event, we gave a higher priority to outcome (A) than to (B). When outcome (A) occurred, we set the index date as the date of occurrence; otherwise, the date of occurrence of outcome (B) was set as the index date. Additionally, we classified the cases as severe when central respiratory depression was concurrently confirmed (Supplementary Table 1). We excluded patients with diagnosis codes for (A) or naloxone injection before the entry date to identify a new event. Patients who had naloxone injection after the entry date, but who were not classified as the case group, were also excluded.
In a ratio of 1:4, control groups were selected among patients without any aforementioned event, using the exact-match method, based on cohort-entry date and follow-up duration (duration between the date of initial opioid prescription and the date of the last medical record).
Variables. The dependent variable for prediction was opioid overdose at the index date. Variables previously identified as risk factors or that were likely to be related to opioid overdose were selected as candidate predictors 14,17,26 .
The assessed variables were demographics (age, sex, and insurance status), baseline comorbidities (mental health disorders and other medical diagnoses), cause of pain at the initiation of NIOA, healthcare utilisation at baseline (hospitalisation period, number of ED visits), NIOA use pattern during the month prior to the index date (main ingredient, number of extended-release and long-acting [ER/LA] opioids, persistence of NIOA use, mean daily milligrams of morphine equivalents [MME]), other medications used during the month prior to the index date (anxiolytics, anticonvulsants, antidepressants, antipsychotics, benzodiazepines, gabapentinoids, muscle relaxants, non-opioid analgesics, naltrexone, other hypnotics, stimulants, and tramadol) (Supplementary Table 2), number of prescribers, and number of prescriptions for NIOA from cohort-entry date to index date. Individual mean daily MME was calculated as the sum of the total MMEs prescribed 1 month prior to the index date, divided by the number of days covered with NIOA prescriptions. Based on the continuity of the NIOA use 1 month before the index date, we classified the persistence of NIOA use as follows: (1) none: no active NIOA prescription for 2 months (− 60 to − 1 days), (2) new: new active NIOA prescription 1 month (− 30 to − 1 days), (3) past: active NIOA prescription ended before 1 month (− 60 to − 31 days), and (4) persistent: active NIOA prescription for 2 months (− 60 to − 1 days) prior to the event.
We assumed no missing values for the measured variables. Since we utilised a claim database, it was not possible to determine whether the absence of records, such as prescription or diagnosis records, was due to missing data. Therefore, we deemed the absence of record as the absence of a corresponding condition. We confirmed that there were no missing values for demographic factors, such as age, sex, and insurance status.
Derivation of risk-score model. First, we randomly split the data into the development cohort (80%) and validation cohort (20%), stratified by outcome. To select the variables to be included in the prediction model, the following process was performed. The frequencies of candidate predictors were evaluated, and items with a prevalence of less than 1% were excluded. The excluded items were moderate to severe liver disease, sleep www.nature.com/scientificreports/ apnoea, stimulant use, and naltrexone use. We examined the multicollinearity between variables using the variance inflation factor. Next, variables with α ≤ 0.1 in the univariable logistic regression analysis were selected as potential predictors of opioid overdose. During this process, sex, myocardial infarction, rheumatic disease, and severe renal disease which did not met the criteria were excluded. To select more efficient variables, we utilised the Akaike Information Criterion (AIC) and performed stepwise selection through three repeated five-fold cross-validation steps 27 . Finally, when variables that were related to each other coexisted, we carefully modified our final list of variables, considering their applicability in the clinical field. For instance, benzodiazepine use was selected over anxiety disorder because use of benzodiazepine or possession of a medication could be easily identified by the interviewer, while the patient may be reluctant to reveal anxiety disorder to healthcare providers. Multivariable logistic regression with the final list of variables was performed for the entire development cohort. Then, the risk-score model was developed through the risk-score assignment for each variable, by multiplying the β coefficient of each variable by 10 and rounding it to the nearest integer. Individual patient risk-scores were calculated using this model. We also created a simple prediction model that could be easily and accurately identified in a primary care setting. For this purpose, information that the patient may be reluctant to relay to the clinician or may not recall correctly, such as substance use disorder, number of NIOA prescriptions, and number of ED visits at baseline, were excluded from the full model. Performance of the model was evaluated for discrimination and calibration in the validation cohort 28 . The c-statistic or area under the receiver operating characteristic curve was measured to quantify the discrimination performance. Calibration was evaluated qualitatively by constructing a calibration plot.
Considering the risk-score distribution of individuals and predicted probability in the development cohort, we classified participants into low-risk, intermediate-risk, and high-risk groups. The cut-off point for the intermediate-risk group was determined using the Youden index, which balances sensitivity and specificity 29 . The cut-off for the high-risk group was arbitrary determined by the score corresponding to the top 10% of all the scores obtained, as described previously 12 . Sensitivity, specificity, and positive-and negative-likelihood ratios were evaluated for these cut-off value. We also evaluated the distribution of severe cases of opioid overdose in each risk group.
We applied the developed prediction model with the strict outcome definition of limiting the active NIOA prescription window to 30 days before the index date, instead of 180 days, when identifying cases with outcome (B) and confirmed the performance in the total cohort. Sample size calculation. Sample size was calculated as described previously 30 . As we planned to match the case and control groups at a ratio of 1:4, the overall outcome proportion was set at 0.2. We targeted the mean absolute prediction error, R 2 Nagelkerke, and shrinkage factors as 0.05, 0.05, and 0.9, respectively. As we expected to develop a parsimonious model, the number of predictor parameters was set at 18. The event size required for model development was calculated to be 630 cases. Thus, the total of 690 cases in the development cohort provided adequate statistical power.

Statistical analysis.
For descriptive statistics, we used percentage or mean (standard deviation). The χ 2 test or Fisher's exact test was applied to compare categorical variables between groups, while t tests were used to compare continuous variables between groups. Logistic regression was performed to assess the association between the variables and outcomes. Statistical significance was determined at p-value less than 0.05. Statistical analyses were performed using SAS version 9.4 (SAS Institute, Cary, NC).
In the development cohort, compared to the control group, the patients in the case group were older (69.3 ± 14.6 vs. 58.2 ± 17.8, p < 0.001) and had higher CCI scores (4.4 ± 2.8 vs. 2.7 ± 2.6, p < 0.001), and the proportion of medical aid or NMS beneficiaries was higher (21.4% vs. 8.2%, p < 0.001) ( Table 1). Patients in the case group were more likely to be diagnosed with mood disorders, anxiety, schizophrenia, substance use disorder, cerebrovascular disease, dementia, pancreatitis, and respiratory disease. The proportion of patients prescribed each of the concurrent medications, such as benzodiazepine, was also higher in the case group ( Table 2).
Initiation of NIOA due to surgery was more frequent in the control group (23.7%) than those in the case group (17.7%). NIOA use patterns, such as prescription frequency, number of prescribers, and proportion of the patients prescribed ER/LA NIOA were significantly different between the groups. Compared to the control group, the patients in the case group were more frequently prescribed ER/LA opioids (≥ 1 agent; 23.4% vs. 45.8%), prescribed a higher daily dose (≥ 20 MMEs; 13.8% vs. 30.7%), had more prescribers (≥ 2 prescribers; 9.2% vs. 22.8%), and had a higher prescription frequency (≥ 3 times; 15.0% vs. 34.1%).
Development of risk-score model for opioid overdose. The results of univariable and multivariable logistic regression analyses that were used to develop the risk screening scores for opioid overdose are summarised in Table 3  were identified as factors markedly increasing the risk of opioid overdose. On the other hand, NIOA initiation due to surgical pain appeared to be a protective factor (aOR 0.62; 95% CI 0.49-0.80). Using the above-mentioned 13 variables, the screening score was developed, with a possible range of − 5 to 129. The c-statistic of the screening score in development cohort was 0.82 (95% CI 0.80-0.84), while in the validation cohort, it was 0.84 (95% CI 0.81-0.87) (Fig. 2). The calibration plot indicated that the model fit the observed data well, as the prediction line was close to the perfect agreement line (Fig. 3).

Discussion
Using a national claim database comprising virtually the entire Korean adult population who had started NIOA, we developed and internally validated a risk-score model as a screening tool that can classify patients at high risk of opioid overdose. The risk screening tool comprised 13 items, including several well-documented risk factors Table 1. Baseline characteristics of study participants according to outcomes in the development and validation cohorts. SD standard deviation, CCI Charlson comorbidity index, NMS National Meritorious Service, PVD peripheral vascular disease, CVD cerebrovascular disease. a Fisher's exact test. www.nature.com/scientificreports/ related to opioid overdose 26,31-34 . We found that older age; beneficiaries with medical aid or NMS medical care; the existence of baseline comorbidities, such as substance use disorder, cerebrovascular disease, and dementia; exposure to medication with sedative properties; having a high number of NIOA prescriptions; use of ER/LA NIOA; and an ED visit before NIOA initiation predicted an increased risk of opioid overdose. On the other hand, use of NIOA for pain due to surgery showed a protective effect. Compared to the previous studies in the USA, which reported the incidence of opioid overdose as 0.4-0.6% 12,35 , the observed incidence of opioid overdose events in the population that initiated NIOA in our study was very low (0.05%).

Control (N = 2760) Case (N = 690) p-value Control (N = 704) Case (N = 176) p-value
Predictors showing the strongest association were the number of NIOA prescriptions within the period from the cohort entry date to the index date, with ≥ 10 NIOA prescriptions resulting in more than six times the risk (aOR 6.07; 95% CI 3.79-9.72). While several previous studies found the number of opioid prescribers or pharmacies as risk factors for opioid overdose 13,26,36 , the number of opioid prescriptions, instead of the number of prescribers, was a predictor of opioid overdose in the present study. These results may be attributable to the health care system of Korea. Similar to previous studies, medical aid beneficiaries 32 , a prior history of substance use disorder 14,16,17,26,31,34 , ER/LA opioid prescription 14,34 , and exposure to medications acting on the central nervous system, such as benzodiazepines and gabapentinoids, increased the odds of opioid overdose 34,[37][38][39] .
Since tramadol is not regulated as a controlled substance in Korea, we did not consider tramadol as a NIOA. Thus, patients prescribed only tramadol were not included in the study. However, among the participants included, tramadol use was evaluated as an additional medication in this study. While previous studies have identified tramadol as a protective factor against serious opioid-related overdoses 34 , we found that exposure to tramadol was associated with an increased risk of opioid overdose. However, this result should be cautiously interpreted.    15,39,41 . Our results showed that the odds for opioid overdose increased with age. Older individuals are more sensitive to opioids due to pharmacokinetic factors, such as reduced renal and hepatic clearance, which induce prolonged effects of opioids, followed by respiratory depression 42 .
Previously developed models for prediction of opioid overdose have reported c-statistics of between 0.69 and 0.91 [12][13][14][15][16][17] . Our final model showed consistent, good discrimination and calibration with c-statistics of 0.84 (95% CI 0.81-0.87) in the validation cohort. While previous studies have developed prediction models with databases from specific populations (e.g., a veterans' health administration database) 12,16 , we utilised a national claims database that included nearly 100% of the Korean population. This implies generalizability of our findings to all patients prescribed NIOA in Korea. The high prediction performance obtained in the validation cohort provided evidence of the strong internal validity of the study. However, this tool needs to be used with caution, considering the low prevalence of opioid overdose, as a low prevalence results in a low positive-predictive value and a high-negative predictive value. Taking this into consideration, this tool should not be used as a standalone tool to make decisions about initiation or modification of NIOA treatment, but should be used to identify patients requiring more intensive monitoring with respect to opioid overdose. By focusing preferentially on patients in the intermediate-to high-risk group, almost 75% cases of opioid overdose can be captured, saving time and resources for evaluating patients unlikely to develop serious adverse effects.
Our prediction model is expected to be applicable in the clinical setting and in national-level surveillance, because of its readily identifiable and easily calculable nature. In particular, a simple model that utilises variables that can be identified through interviews and prescription details is expected to be useful in primary care. We identified several modifiable predictors, such as NIOA prescription patterns and non-opioid medication use patterns that can predict future opioid overdose. In this regard, this tool can help clinicians identify patients atrisk of opioid overdose, and to assist in decision-making about reducing the risk of preventable adverse effects when evaluating the risks and benefits of the treatment.
Our study had several limitations. Due to the nature of the claims database, we could not investigate some important sociodemographic factors, such as family history, smoking behaviour, and alcohol consumption. Second, an individual's actual NIOA intake could not be evaluated because only prescription records were available. Third, although we identified all patients eligible for our definition of opioid overdose, only a portion of patients without opioid overdose were included as control patients, because four control patients were assigned to each case. As a result, it was not possible to calculate positive-and negative-predictive values directly, which are subject to the prevalence of the outcome 43 . We also could not estimate the absolute incidence of opioid overdose in each risk group from our model. Finally, although our novel screening score had been internally validated, it should be externally validated in different data sources or in specific subgroups of patients to ensure generalizability in different settings 44 . In conclusion, we have newly developed and internally validated a model for predicting opioid overdose in opioid incident users in Korea, based on the national claim database. This tool can stratify patients according to their risk of opioid overdose, allowing healthcare providers to focus medical resources on a limited number of patients. Ideally, this tool can also be utilised as a national-level surveillance tool to identify patients at risk of opioid overdose. To ensure the clinical usefulness of the model, external validation and prospective evaluation of the model in a real-world setting are needed.