Introduction

Opioids are considered crucial for pain relief; however, they are associated with significant harms such as overdose1. In the USA, along with the rise in opioid prescription since 1999, opioid-related overdose admissions had quadrupled by 20102. During 1999‒2017, 702,568 drug-related deaths occurred, of which two-thirds were opioid-related3.

US regulatory authorities declared the opioid crisis a national public health emergency4. To encourage safe use of opioids, the Centers for Disease Control and Prevention published guidelines on safe opioid prescription practice in primary care settings5,6. Risk evaluation and mitigation strategies and prescription drug monitoring programs were implemented to encourage safe opioid use. Consequently, a decrease in opioid prescription was observed from 2012, and death by prescription opioids had decreased by 13.5% in 2017‒20187. However, overdose deaths involving opioids are still on the rise, and the proportion of drug-related overdose deaths due to illicit opioids has risen sharply since 20178. This phenomenon might be related to the unexpected interruption of opioid prescription due to the increased prescription burden of physicians. Thus, according to experts, prescription restriction policies should be applied with caution, and each patient’s situation should be considered9,10. For instance, patients inevitably in need of opioids, such as those with malignant pain, should receive opioid prescriptions. Therefore, to prevent opioid-related adverse outcomes, it is important to assess the individual patient risk rather than to regulate opioid prescriptions according to uniform standards.

In this context, several models for predicting opioid overdose have been reported. These included a risk index to identify patients with a risk of overdose or serious opioid-induced respiratory depression11 and a machine learning algorithm to predict opioid overdose12. However, most such studies were conducted in North America11,12,13,14,15,16,17. Considering that the healthcare system is key in prescription practice and healthcare utilisation18,19, and that ethnicity influences opioid-related negative outcomes20, these tools should be developed or validated in domestic populations.

Although Korea has had few issues regarding opioid overdose because opioid use is lower than in other countries, a significant increase in opioid prescription trends has been observed in recent years (347.5 prescriptions/1000 person in 2009 vs. 531.3 prescriptions/1000 person in 2019)21. Also, the prevalence of chronic use of opioid, a well-known risk factor for overdose, has sharply increased (strong opioids: 0.04% in 2002 to 0.24% in 2015)22. A multicentre study reported that 21% of patients on chronic opioid therapy for more than three months had instances of inappropriate or excessive opioid use23. To ameliorate the opioid-related problem due to opioid prescription increase, a program examining opioid prescriptions for physicians has recently been established in Korea24. However, there are no tools that can predict the risk of opioid overdose for an individual patient in the Korean population.

In this study, we aimed to develop and validate a predictive risk-score model for opioid overdose among Korean patients prescribed non-injectable opioid analgesic (NIOA). We used this model to stratify Korean patients prescribed NIOA into subgroups with similar risks for opioid overdose.

Methods

Study design and database

This study was a retrospective, nested case‒control study that used the National Health Insurance Claims Database of the Korean Health Insurance Review Service (HIRA) from 2016 to 2018. In Korea, it is compulsory for all citizens to enrol in the national health insurance system, and all medical claims are electronically submitted to the HIRA, which covers almost 98% of the entire population. Thus, these data are representative of the population25. The database contains detailed information on basic demographic characteristics, healthcare utilisation, medical procedures, prescription, and disease diagnosis, based on the International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10). This study was approved by the Yeungnam University Institutional Review Board (approval number: YU 2019-01-001). The informed consent from the participants was waived by the Yeungnam University Institutional Review Board because this study used anonymized data retrospectively, and there was no or minimal risk of harm to the participants. All methods were performed according to relevant guidelines and regulations.

Study participants and outcome definition

First, adult patients who initiated NIOA use during 2017‒2018 (N = 1,752,380) were defined as the NIOA cohort, and the date of the first prescription was the cohort entry date. We excluded patients with NIOA prescriptions in 2016. We defined NIOA as oral or transdermal opioid analgesics (ATC code: N02A and R05DA04), excluding tramadol, and excluded injectable formulations.

Among them, the case group with opioid overdose events was defined as follows: (A) those with occurrence of an opioid-related poisoning event, (B) those with occurrence of emergency department (ED) visits involving a naloxone injection and active NIOA prescriptions within 180 days before the event (those with naloxone injection for the first developed stroke or within the context of a surgical procedure were not included). When a patient had more than one event, we gave a higher priority to outcome (A) than to (B). When outcome (A) occurred, we set the index date as the date of occurrence; otherwise, the date of occurrence of outcome (B) was set as the index date. Additionally, we classified the cases as severe when central respiratory depression was concurrently confirmed (Supplementary Table 1). We excluded patients with diagnosis codes for (A) or naloxone injection before the entry date to identify a new event. Patients who had naloxone injection after the entry date, but who were not classified as the case group, were also excluded.

In a ratio of 1:4, control groups were selected among patients without any aforementioned event, using the exact-match method, based on cohort-entry date and follow-up duration (duration between the date of initial opioid prescription and the date of the last medical record).

Variables

The dependent variable for prediction was opioid overdose at the index date. Variables previously identified as risk factors or that were likely to be related to opioid overdose were selected as candidate predictors14,17,26.

The assessed variables were demographics (age, sex, and insurance status), baseline comorbidities (mental health disorders and other medical diagnoses), cause of pain at the initiation of NIOA, healthcare utilisation at baseline (hospitalisation period, number of ED visits), NIOA use pattern during the month prior to the index date (main ingredient, number of extended-release and long-acting [ER/LA] opioids, persistence of NIOA use, mean daily milligrams of morphine equivalents [MME]), other medications used during the month prior to the index date (anxiolytics, anticonvulsants, antidepressants, antipsychotics, benzodiazepines, gabapentinoids, muscle relaxants, non-opioid analgesics, naltrexone, other hypnotics, stimulants, and tramadol) (Supplementary Table 2), number of prescribers, and number of prescriptions for NIOA from cohort-entry date to index date. Individual mean daily MME was calculated as the sum of the total MMEs prescribed 1 month prior to the index date, divided by the number of days covered with NIOA prescriptions. Based on the continuity of the NIOA use 1 month before the index date, we classified the persistence of NIOA use as follows: (1) none: no active NIOA prescription for 2 months (− 60 to − 1 days), (2) new: new active NIOA prescription 1 month (− 30 to − 1 days), (3) past: active NIOA prescription ended before 1 month (− 60 to − 31 days), and (4) persistent: active NIOA prescription for 2 months (− 60 to − 1 days) prior to the event.

We assumed no missing values for the measured variables. Since we utilised a claim database, it was not possible to determine whether the absence of records, such as prescription or diagnosis records, was due to missing data. Therefore, we deemed the absence of record as the absence of a corresponding condition. We confirmed that there were no missing values for demographic factors, such as age, sex, and insurance status.

Derivation of risk-score model

First, we randomly split the data into the development cohort (80%) and validation cohort (20%), stratified by outcome. To select the variables to be included in the prediction model, the following process was performed. The frequencies of candidate predictors were evaluated, and items with a prevalence of less than 1% were excluded. The excluded items were moderate to severe liver disease, sleep apnoea, stimulant use, and naltrexone use. We examined the multicollinearity between variables using the variance inflation factor. Next, variables with α ≤ 0.1 in the univariable logistic regression analysis were selected as potential predictors of opioid overdose. During this process, sex, myocardial infarction, rheumatic disease, and severe renal disease which did not met the criteria were excluded. To select more efficient variables, we utilised the Akaike Information Criterion (AIC) and performed stepwise selection through three repeated five-fold cross-validation steps27. Finally, when variables that were related to each other coexisted, we carefully modified our final list of variables, considering their applicability in the clinical field. For instance, benzodiazepine use was selected over anxiety disorder because use of benzodiazepine or possession of a medication could be easily identified by the interviewer, while the patient may be reluctant to reveal anxiety disorder to healthcare providers.

Multivariable logistic regression with the final list of variables was performed for the entire development cohort. Then, the risk-score model was developed through the risk-score assignment for each variable, by multiplying the β coefficient of each variable by 10 and rounding it to the nearest integer. Individual patient risk-scores were calculated using this model. We also created a simple prediction model that could be easily and accurately identified in a primary care setting. For this purpose, information that the patient may be reluctant to relay to the clinician or may not recall correctly, such as substance use disorder, number of NIOA prescriptions, and number of ED visits at baseline, were excluded from the full model.

Performance of the model was evaluated for discrimination and calibration in the validation cohort28. The c-statistic or area under the receiver operating characteristic curve was measured to quantify the discrimination performance. Calibration was evaluated qualitatively by constructing a calibration plot.

Considering the risk-score distribution of individuals and predicted probability in the development cohort, we classified participants into low-risk, intermediate-risk, and high-risk groups. The cut-off point for the intermediate-risk group was determined using the Youden index, which balances sensitivity and specificity29. The cut-off for the high-risk group was arbitrary determined by the score corresponding to the top 10% of all the scores obtained, as described previously12. Sensitivity, specificity, and positive- and negative-likelihood ratios were evaluated for these cut-off value. We also evaluated the distribution of severe cases of opioid overdose in each risk group.

We applied the developed prediction model with the strict outcome definition of limiting the active NIOA prescription window to 30 days before the index date, instead of 180 days, when identifying cases with outcome (B) and confirmed the performance in the total cohort.

Sample size calculation

Sample size was calculated as described previously30. As we planned to match the case and control groups at a ratio of 1:4, the overall outcome proportion was set at 0.2. We targeted the mean absolute prediction error, R2Nagelkerke, and shrinkage factors as 0.05, 0.05, and 0.9, respectively. As we expected to develop a parsimonious model, the number of predictor parameters was set at 18. The event size required for model development was calculated to be 630 cases. Thus, the total of 690 cases in the development cohort provided adequate statistical power.

Statistical analysis

For descriptive statistics, we used percentage or mean (standard deviation). The χ2 test or Fisher’s exact test was applied to compare categorical variables between groups, while t tests were used to compare continuous variables between groups. Logistic regression was performed to assess the association between the variables and outcomes. Statistical significance was determined at p-value less than 0.05. Statistical analyses were performed using SAS version 9.4 (SAS Institute, Cary, NC).

Results

Clinical characteristics of the development cohort

From 1 January 2017 to 31 December 2018, we identified 866 case patients (0.05%) and 3464 matched control patients among the 1,752,380 patients initiated NIOA (Fig. 1). Patients in the development and validation cohorts had similar characteristics and outcome distributions (Supplementary Table 3). The mean ± standard deviation of age (development cohort: 60.5 ± 17.8 vs. validation cohort: 61.0 ± 18.3), Charlson comorbidity index (CCI) score (3.1 ± 2.7 vs. 3.0 ± 2.7), proportion of male patients (46.4% vs. 44.7%), beneficiaries with medical aid or national meritorious service (NMS) (10.8% vs. 12.0%), and patients with comorbidities, such as cancer (21.2% vs. 21.3%) and substance use disorder (1.3% vs. 1.4%), were not statistically significantly different between the development and validation cohorts.

Figure 1
figure 1

Patient-selection flow chart. NIOA non-injectable opioid analgesics.

Full size image

In the development cohort, compared to the control group, the patients in the case group were older (69.3 ± 14.6 vs. 58.2 ± 17.8, p < 0.001) and had higher CCI scores (4.4 ± 2.8 vs. 2.7 ± 2.6, p < 0.001), and the proportion of medical aid or NMS beneficiaries was higher (21.4% vs. 8.2%, p < 0.001) (Table 1). Patients in the case group were more likely to be diagnosed with mood disorders, anxiety, schizophrenia, substance use disorder, cerebrovascular disease, dementia, pancreatitis, and respiratory disease. The proportion of patients prescribed each of the concurrent medications, such as benzodiazepine, was also higher in the case group (Table 2).

Table 1 Baseline characteristics of study participants according to outcomes in the development and validation cohorts.
Full size table
Table 2 Medication use pattern of study participants according to outcomes in the development and validation cohorts.
Full size table

Initiation of NIOA due to surgery was more frequent in the control group (23.7%) than those in the case group (17.7%). NIOA use patterns, such as prescription frequency, number of prescribers, and proportion of the patients prescribed ER/LA NIOA were significantly different between the groups. Compared to the control group, the patients in the case group were more frequently prescribed ER/LA opioids (≥ 1 agent; 23.4% vs. 45.8%), prescribed a higher daily dose (≥ 20 MMEs; 13.8% vs. 30.7%), had more prescribers (≥ 2 prescribers; 9.2% vs. 22.8%), and had a higher prescription frequency (≥ 3 times; 15.0% vs. 34.1%).

Development of risk-score model for opioid overdose

The results of univariable and multivariable logistic regression analyses that were used to develop the risk screening scores for opioid overdose are summarised in Table 3 (results of univariable analysis of all candidate predictors are listed in Supplementary Table 4). Thirteen predictors were associated with the development of opioid overdose: age group, insurance status, cause of opioid prescription, three comorbidities (cerebrovascular disease, dementia, and substance use disorder), four classes of medication use (anticonvulsants, benzodiazepine, gabapentinoids, and tramadol), three types of NIOA use patterns (cause of initiations, ER/LA opioid use, and number of opioid prescriptions), and the number of ED visits at baseline. Among those variables, older age (≥ 75 years) (adjusted odds ratio [aOR] 3.47; 95% confidence interval [CI] 2.48–4.86), history of substance use disorder (aOR 4.67; 95% CI 2.34–9.34), exposure to anticonvulsants (aOR 5.00; 95% CI 3.34–7.47), and frequent opioid prescriptions (≥ 10) (aOR 6.07; 95% CI 3.79–9.72) were identified as factors markedly increasing the risk of opioid overdose. On the other hand, NIOA initiation due to surgical pain appeared to be a protective factor (aOR 0.62; 95% CI 0.49–0.80). Using the above-mentioned 13 variables, the screening score was developed, with a possible range of − 5 to 129.

Table 3 Development of the predictive risk-score model for opioid overdose.
Full size table

The c-statistic of the screening score in development cohort was 0.82 (95% CI 0.80–0.84), while in the validation cohort, it was 0.84 (95% CI 0.81–0.87) (Fig. 2). The calibration plot indicated that the model fit the observed data well, as the prediction line was close to the perfect agreement line (Fig. 3).

Figure 2
figure 2

Receiver operating characteristic curve of predictive risk-score model for opioid overdose. AUC area under the receiver operating characteristic curve, CI confidence interval.

Full size image
Figure 3
figure 3

Calibration plot for predictive risk-score model for opioid overdose. (a) Calibration plot in the development cohort. (b) Calibration plot in the validation cohort.

Full size image

When the full model was adjusted to a simple model after excluding variables that were difficult to identify in the primary care setting, including substance use disorder, number of NIOA prescriptions, and number of ED visits at baseline, the simple model showed good performance, although the performance on the ROC analysis was slightly reduced (c-statistic 0.81, 95% CI 0.77–0.84; ROC comparison, p < 0.001) (see Supplementary Figs. 1, 2, and Supplementary Table 5).

Risk stratification

Table 4 shows the results of stratification into low- (< 18 points), intermediate- (18–31 points), and high-risk (≥ 32 points) groups in the development and validation cohorts. In the validation cohort, 579 (65.8%), 214 (24.3%), and 87 (9.9%) patients were classified as low, intermediate, and high risk, respectively, of which 45 (7.8%), 74 (34.6%), and 57 (65.5%) eventually developed opioid overdose, respectively. In each risk group, there were 14 (2.4%), 22 (10.3%), and 21 (24.1%) severe cases. When using the intermediate-risk threshold, the sensitivity and specificity of the model for predicting opioid overdose was 74.4% (95% CI 67.3‒80.7%) and 75.9% (95% CI 72.5‒79.0%), respectively, and the positive- and negative-likelihood ratios were 3.08 (95% CI 2.63–3.61) and 0.34 (95% CI 0.26–0.44), respectively. When using the high-risk threshold, the sensitivity and specificity were 32.4% (95% CI 25.5‒39.8%) and 95.7% (95% CI 94.0‒97.1%), respectively, while the positive- and negative-likelihood ratios were 7.60 (95% CI 5.04–11.45) and 0.71 (95% CI 0.64–0.78), respectively. The odds ratios of the intermediate-risk group and high-risk group, as compared to the low-risk group, were 6.27 (95% CI 4.14–9.50) and 22.55 (95% CI 13.18–38.56), respectively. The results of the simple model are presented in Supplementary Table 6.

Table 4 Prediction performance stratified by risk.
Full size table

The application results of the full model or simple model with strict outcome definitions showed similar trends (Supplementary Figs. 3–6, and Supplementary Tables 7, 8).

Discussion

Using a national claim database comprising virtually the entire Korean adult population who had started NIOA, we developed and internally validated a risk-score model as a screening tool that can classify patients at high risk of opioid overdose. The risk screening tool comprised 13 items, including several well-documented risk factors related to opioid overdose26,31,32,33,34. We found that older age; beneficiaries with medical aid or NMS medical care; the existence of baseline comorbidities, such as substance use disorder, cerebrovascular disease, and dementia; exposure to medication with sedative properties; having a high number of NIOA prescriptions; use of ER/LA NIOA; and an ED visit before NIOA initiation predicted an increased risk of opioid overdose. On the other hand, use of NIOA for pain due to surgery showed a protective effect. Compared to the previous studies in the USA, which reported the incidence of opioid overdose as 0.4–0.6%12,35, the observed incidence of opioid overdose events in the population that initiated NIOA in our study was very low (0.05%).

Predictors showing the strongest association were the number of NIOA prescriptions within the period from the cohort entry date to the index date, with ≥ 10 NIOA prescriptions resulting in more than six times the risk (aOR 6.07; 95% CI 3.79‒9.72). While several previous studies found the number of opioid prescribers or pharmacies as risk factors for opioid overdose13,26,36, the number of opioid prescriptions, instead of the number of prescribers, was a predictor of opioid overdose in the present study. These results may be attributable to the health care system of Korea. Similar to previous studies, medical aid beneficiaries32, a prior history of substance use disorder14,16,17,26,31,34, ER/LA opioid prescription14,34, and exposure to medications acting on the central nervous system, such as benzodiazepines and gabapentinoids, increased the odds of opioid overdose34,37,38,39.

Since tramadol is not regulated as a controlled substance in Korea, we did not consider tramadol as a NIOA. Thus, patients prescribed only tramadol were not included in the study. However, among the participants included, tramadol use was evaluated as an additional medication in this study. While previous studies have identified tramadol as a protective factor against serious opioid-related overdoses34, we found that exposure to tramadol was associated with an increased risk of opioid overdose. However, this result should be cautiously interpreted.

The impact of age on opioid overdose is controversial. Hasegawa et al. showed that the risk of opioid overdose decreased in the elderly and increased in the middle-aged group compared to younger adults32, Bonhert et al. also found decreased risk with older age40, while Cho et al., Rose et al., and Khanna et al. reported increased risk with older age15,39,41. Our results showed that the odds for opioid overdose increased with age. Older individuals are more sensitive to opioids due to pharmacokinetic factors, such as reduced renal and hepatic clearance, which induce prolonged effects of opioids, followed by respiratory depression42.

Previously developed models for prediction of opioid overdose have reported c-statistics of between 0.69 and 0.9112,13,14,15,16,17. Our final model showed consistent, good discrimination and calibration with c-statistics of 0.84 (95% CI 0.81–0.87) in the validation cohort. While previous studies have developed prediction models with databases from specific populations (e.g., a veterans’ health administration database)12,16, we utilised a national claims database that included nearly 100% of the Korean population. This implies generalizability of our findings to all patients prescribed NIOA in Korea. The high prediction performance obtained in the validation cohort provided evidence of the strong internal validity of the study. However, this tool needs to be used with caution, considering the low prevalence of opioid overdose, as a low prevalence results in a low positive-predictive value and a high-negative predictive value. Taking this into consideration, this tool should not be used as a standalone tool to make decisions about initiation or modification of NIOA treatment, but should be used to identify patients requiring more intensive monitoring with respect to opioid overdose. By focusing preferentially on patients in the intermediate-to high-risk group, almost 75% cases of opioid overdose can be captured, saving time and resources for evaluating patients unlikely to develop serious adverse effects.

Our prediction model is expected to be applicable in the clinical setting and in national-level surveillance, because of its readily identifiable and easily calculable nature. In particular, a simple model that utilises variables that can be identified through interviews and prescription details is expected to be useful in primary care. We identified several modifiable predictors, such as NIOA prescription patterns and non-opioid medication use patterns that can predict future opioid overdose. In this regard, this tool can help clinicians identify patients at-risk of opioid overdose, and to assist in decision-making about reducing the risk of preventable adverse effects when evaluating the risks and benefits of the treatment.

Our study had several limitations. Due to the nature of the claims database, we could not investigate some important sociodemographic factors, such as family history, smoking behaviour, and alcohol consumption. Second, an individual’s actual NIOA intake could not be evaluated because only prescription records were available. Third, although we identified all patients eligible for our definition of opioid overdose, only a portion of patients without opioid overdose were included as control patients, because four control patients were assigned to each case. As a result, it was not possible to calculate positive- and negative-predictive values directly, which are subject to the prevalence of the outcome43. We also could not estimate the absolute incidence of opioid overdose in each risk group from our model. Finally, although our novel screening score had been internally validated, it should be externally validated in different data sources or in specific subgroups of patients to ensure generalizability in different settings44.

In conclusion, we have newly developed and internally validated a model for predicting opioid overdose in opioid incident users in Korea, based on the national claim database. This tool can stratify patients according to their risk of opioid overdose, allowing healthcare providers to focus medical resources on a limited number of patients. Ideally, this tool can also be utilised as a national-level surveillance tool to identify patients at risk of opioid overdose. To ensure the clinical usefulness of the model, external validation and prospective evaluation of the model in a real-world setting are needed.