Machine learning prediction of incidence of Alzheimer’s disease using large-scale administrative health data

Nationwide population-based cohort provides a new opportunity to build an automated risk prediction model based on individuals’ history of health and healthcare beyond existing risk prediction models. We tested the possibility of machine learning models to predict future incidence of Alzheimer’s disease (AD) using large-scale administrative health data. From the Korean National Health Insurance Service database between 2002 and 2010, we obtained de-identified health data in elders above 65 years (N = 40,736) containing 4,894 unique clinical features including ICD-10 codes, medication codes, laboratory values, history of personal and family illness and socio-demographics. To define incident AD we considered two operational definitions: “definite AD” with diagnostic codes and dementia medication (n = 614) and “probable AD” with only diagnosis (n = 2026). We trained and validated random forest, support vector machine and logistic regression to predict incident AD in 1, 2, 3, and 4 subsequent years. For predicting future incidence of AD in balanced samples (bootstrapping), the machine learning models showed reasonable performance in 1-year prediction with AUC of 0.775 and 0.759, based on “definite AD” and “probable AD” outcomes, respectively; in 2-year, 0.730 and 0.693; in 3-year, 0.677 and 0.644; in 4-year, 0.725 and 0.683. The results were similar when the entire (unbalanced) samples were used. Important clinical features selected in logistic regression included hemoglobin level, age and urine protein level. This study may shed a light on the utility of the data-driven machine learning model based on large-scale administrative health data in AD risk prediction, which may enable better selection of individuals at risk for AD in clinical trials or early detection in clinical settings.


INTRODUCTION
Screening individuals at risk for Alzheimer's disease (AD) based on medical health records in preclinical stages may lead to early detection of AD pathology and to better therapeutic strategies for delaying the onset of AD [1][2][3] . Current biomarkers of AD requires the collection of specimen (e.g., serum or fluid) or imaging data. On the other hand, the electronic healthcare data, such as health records in clinical settings, or administrative health data, does not require additional time or effort for data collection. Also, with the advent of digitalization the amounts of such data have exponentially increased 4 . Since it is ubiquitous, cost-effective and enormous, the digitalized healthcare database may be an invaluable resource for testing scalable predictive models for AD and other diseases alike. However, despite of its tremendous potential value, little is known about the extents to which the large-scale administrative health data is useful in AD risk prediction.
For AD risk prediction, prior models are typically based on predefined health profile variables, such as sociodemographic (age, sex, education), lifestyle (physical activity), midlife health risk factors (systolic blood pressure, BMI and total cholesterol level) 5,6 ; and cognitive profiles 7,8 . An important outstanding question is whether those simple predictive models based on the small sets of selected variables may sufficiently account for the heterogeneous etiologies of multi-factorial AD in clinical settings. Indeed, a meta-analysis study shows that multi-factor models best predict risk for dementia, whereas single-factor models do poorly 6 , suggesting accurate AD risk prediction requires a large feature space. Here we test the extents to which a data-driven machine approach harvests salient information from the large-scale healthcare data containing thousands of data of individuals' health trajectories and make an individual-specific prediction of AD risk.
Machine learning is an optimal choice of analytics for analyzing the large-scale administrative health data containing thousands of descriptors from hundreds of thousands of individuals. Studies show successful applications of machine learning to the largescale administrative data in predicting incident diseases other than AD (diabetes, metabolic syndrome, suicide death, opioid overdose or drug-resistant epilepsy, etc) [9][10][11][12][13] . Given the recent rapid growth of the machine learning technology, application of the AI technology to clinical predictive modeling is likely to have a deep impact on medicine [14][15][16] . But to our knowledge the datadriven predictive modeling based on nationwide populationbased administrative health data has yet to be tested in AD risk prediction.
In testing predictive models, it is important to use sufficiently large data representative of the population. The size of the data is important for the model performance (e.g., accuracy), while the representativeness is important for the model generalizability. In this study, we used the National Health Insurance Service-national sample cohort (NHIS-NSC) of one million people representative of the contemporary South Korean population within the Korean National Health Insurance Service database 17 . Using the largescale, thorough, longitudinal, administrative healthcare data (e.g., insurance claims and health check-ups) within this database, we constructed and validated data-driven machine learning models to predict future incidence of AD.

Sample characteristics
Of 40,736 individuals with age above 65 years in 2002, we identified 614 unique individuals with AD incidence using the definite AD outcome, 2026 with AD incidence using the probable AD definition, and 38,710 elders with no AD incidence (Fig. 1). The rate of AD in this cohort was 1.56% using the definite AD definition, and 4.97% using the probable AD definition. Demographic characteristics showed significant differences in age between both AD groups and non-AD groups and nonsignificant differences in income and sex (Table 1).

Model prediction
Classifiers were trained to predict 0, 1, 2, 3, and 4 subsequent-year incident AD. In balanced samples (bootstrapping with replacement), when using the definite AD definition (based on ICD-10 codes and dementia prescription), in predicting 0 year incidence of AD, random forest (RF) showed the best performance with accuracy 0.823 and AUC of 0.898 ( Fig. 2 and Table 2). When using the probable AD definition (based on ICD-10 codes), classification performance was slightly lower with accuracy of 0.788 and AUC of 0.850 (RF). Classification performance decreased in predicting future incident AD of later years: using the definite AD definition, accuracy/AUC of 0.713/0.775 ( Table 3).
Model prediction using important features only After identified the important features related to incident AD by logistic regression, classifiers were trained with top 20 important features only to predict 0, 1, 2, 3, and 4 subsequent-year incidence of AD. These models showed overall similar performance: in 0 and 1 subsequent-year prediction, the AUC was higher up to 11.5% in the all feature model, compared with the top 20 feature model; in 2, 3, 4 subsequent-year prediction, the differences in AUC were much smaller with the range of negative 5 to positive 1% ( Table 2,  Supplementary Table 3).

DISCUSSION
This study assessed the utility of the nationwide population-based administrative health data in predicting the future incidence of AD. Using machine learning, we predicted future incidence of AD with acceptable accuracy of 0.713 (in terms of AUC 0.781) in oneyear prediction. The high accuracy of our models based on large nationwide samples may lend a support to the potential utility of the administrative data-based predictive model in AD. Despite of the limitations inherent to the administrative health data, such as the inability to directly ascertain clinical phenotypes, this study demonstrates its potential utility in AD risk prediction, when combined with data-driven machine learning.
Our model performance with AUC of 0.898, 0.775, and 0.725 in predicting baseline, subsequent one-year, and four-year incident AD is relatively accurate compared with the literature. In all-cause dementia risk prediction based on genetic (ApoE) or neuropsychological evaluations, MRI, health indices (diabetes, hypertension, lifestyle), and demographic (age, sex, education) variables, prior models show accuracy ranging from 0.5 to 0.78 in AUC (reviewed in ref. 18 ). Of note, no direct comparisons of our results with those studies should be made because of the differences in the study 433 Receiver-Opertating Characteristics Fig. 2 Performance of machine learning models in predicting incident AD. Receiver-Operating Characteristic plots are shown for 0, 1, 2, 3, 4subsequent year prediction. Incident AD was defined based on ICD-10 AD codes and anti-dementia medication for AD, "Definite AD", or based on AD codes only, "Probable AD". In each year prediction, a best performing model was selected for plotting. Best performing models based on AUC.
b Subsequent years of incidence predicted = an year of incidence-the last year of health data (e.g., 3 year = an incidence in 2013-the health data used in the prediction up to 2010; 3 year future prediction).
design (e.g., predicting AD risk in 20 years later), populations (e.g., non-Asians), and analytical model (e.g., linear models). Nevertheless, it should be noted that compared with the prior studies primarily based on targetted variables obtained from elaborate neuropsychological, genetic testing, or brain imaging, our approach is solely based on the administrative health data. This has important implications for the practical utility, in that it can provide an early indication of AD risk to clinicians prior to any assessments or tests. Together with existing screening tools (e.g., MMSE), this may assist deciding when to seek a further clinical assessment to a given patient in an individual-specific manner.
Comparing the models based on the sampled, balanced set and on the entire, unbalanced set showed small-to-moderate differences in model performance. For example, based on the RF model in predicting 0-year definite AD, the AUC's are 0.887 and 0.898 in the unbalanced and balanced samples, respectively, showing a 1% increase. On the other hand, in predicting 4-year definite AD, the AUC's are 0.662 and 0.725 in the unbalanced and balanced samples, respectively, showing a 9.5% increase. These results show trivial-to-moderate differences in model performance between balanced and balanced samples. However, we should point that, if one uses an algorithm capable of processing the temporal information among the clinical features, such as recurrent neural networks 19 , then using the entire data for scalable learning is likely to be beneficial.
Comparing the model performance across years, the 3-year prediction is less accurate than the 4-year prediction. This seems counter-intuitive at first, but our data shows that the length of data is greater in 4-year prediction than in 3-year prediction (Supplementary Table 2). We suspect that this difference in data availability may be a cause of the expected performance increase in later year prediction. This might be also related to the irregularity of the NHIS-NSC dataset due to changes in healthcare policy.
Our model detected the interesting clinical features associated with incident AD. The data-driven selection of features is consistent with risk factors found in the literature. A decrease in hemoglobin level was selected as the feature most strongly associated with incident AD. Indeed, anemia is known as an important risk factor for dementia [20][21][22] . A study using National Health Insurance Service-National Health Screening Cohort (NHIS-HEALS), the NHIS health screening data in Korea, not only found that anemia was associated with dementia, but also revealed a dose-dependent relationship between anemia and dementia 23 . Likewise, our data-driven model shows the hemoglobin level as the most significant predictor. This finding has implications for public health because anemia is a modifiable factor. Given our finding and the consistent literature on the association between hemoglobin level and AD and other dementia, future research may investigate the biological pathway of anemia's contribution to AD pathology and cognitive decline.
We also discovered a positive association between urine protein level and incident AD. In the NHIS-NSC, protein in urine is typically measured using dip sticks. Though this is not a quantitative measure of urine protein, it is useful as a screening method for proteinuria 24,25 . Literature shows an association between albuminuria and dementia 26 . Our finding suggests the potential utility of a urine test as part of the routine health check-up for AD risk prediction.
Four medications were also associated with incident dementia within top ten features. We found that Zotepine, Eperisone hydrochloride had a positive association and Nicametate Citrate and Tolfenamic acid had a negative association with incident AD. It is interesting that patients prescribed tolfenamic acid showed lower incidence of AD. This drug used in Korea for pain control in conditioner such as rheumatoid arthritis. It is known to lower the gene expression of Amyloid precursor protein 1(APP1) and betasite APP cleaving enzyme 1(BACE1) by promoting the degradation of specificity protein 1(Sp1) [27][28][29] . As a potential modifier of tau protein, Tolfenamic acid is under investigation as a potential drug to prevent and modify the progression of AD 30 . The results of this study support the above experimental result and show that tolfenamic acid may be a potential anti-dementia medication.
Zotepine is an atypical antipsychotic drug with proven efficacy for treatment of schizophrenia. Our model showed the use of zotepine positively correlated with incident AD. There are two possible interpretations. Zotepine may have been used to treat behavioral and psychological symptoms of dementia (BPSD) before incident AD or diagnosis of AD 31 . Thus, the prescription of Zotepine may indicate early AD symptoms and, consequently, an increasing likelihood of incident AD. Alternatively, some studies indicate that individuals with schizophrenia may have an increased risk for the development of dementia 32 . Given this, it might be possible that incident AD is high in individuals with schizophrenic symptoms to whom Zotepine is prescribed. However, this alternative interpretation may be questionable considering that, in our model, the disease code of Schizophrenia has not been selected as an important feature. In either case, it should be noted that, though our results indicate a potential relationship between Zotepin and incident AD (likely reflecting the common practice in dementia), no causal relationship should be drawn.
Nicametate Citrate, a vasodilator, was also negatively associated with incident AD. This may be in line with the literature showing effects of vasodilators on increasing cognitive function and  reducing the risk of vascular dementia, although the exact mechanism remains unclear 33,34 . Further research is required. One of the limitations of this study is that diagnoses of AD in our database are not clinically ascertained. For example, there may be incorrect diagnoses or misdiagnoses of AD in the claim data. To mitigate this issue, we firstly confirmed the similar prediction results using two different definitions of incident AD, "probable AD" (based on AD disease codes) and "definite AD" (based on both AD disease codes and anti-dementia medication). Secondly, in South Korea, every elder with age 60 years old is required to have complementary dementia screening supported by the National Health Insurance Service at public healthcare centers, where individuals that high-risk for dementia get referred to physicians for further clinical examination. Such a system may help reduce false negative cases. Lastly, Korean health insurance system and policies support the reliability of the AD diagnoses. That is, the Health Insurance Review and Assessment Service of NHIS reviews and supervises the medical claims of AD medication.  Fig. 1). Thus, it is likely that individuals with records of receiving dementia medication meet strong diagnostic criteria. These aspects may alleviate potential validity issues of the AD diagnoses in the Korean administrative health data. Another limitation is that the features associated with incident AD do not indicate causality. Rather, this finding indicates a data-driven discovery from the large administrative data. This knowledge might be useful to generate new hypotheses, to confirm existing ones, or to compare relative importance in predicting incident AD considering large feature space. We believe this is a useful value of data-driven science.
In sum, this study lends support to a statistically meaningful detection of individuals with AD risk solely based on the administrative health data. Generalizability of our findings to independent data in other nations, ethnicities, and healthcare and insurance systems remains to be tested. If replicated, this study may further motivate the implementation of a system in clinical settings that could alarm a risk for AD, which may enable earlier and more accurate screening for subsequent clinical testing.

METHODS Datasets
NHIS-NSC cohort consist of randomly selected 1,025,340 participants comprising 2.2% of the total eligible Korean population in 2002, and followed for 11 years until 2013 unless participants' eligibility was disqualified due to death or emigration 17 . This database contains for each individual's features of services, diagnoses, and prescriptions associated with all the health care services provided by the NHIS. Clinical features include demographics and income levels divided by 10 levels based on subject's monthly salary from the Participant Insurance Eligibility database; disease and medication codes from the Healthcare Utilization database; and laboratory values, health profiles, and history of personal and family illness from the National Health Screening database (from bi-annual health check-up required for elders with age above 40). Of those samples, 40,736 elders were selected in this study, whose records exist in all the three databases (Participant Insurance Eligibility database, Healthcare Utilization database, and National Health Screening database).

Operational definition of AD
For an operational definition of AD, a study of Canadian EMR from" 3,404 adults shows sensitivity of 79% and specificity of 99% when they used an algorithm of "one hospitalization code OR three physician claims codes at least 30 days apart in a two year period OR a prescription filled for an AD-RD specific medication" 35 . In this study, to further improve the accuracy of an operational definition of AD, particularly sensitivity, we used the following algorithm to operationally define incident AD, herein labeled as "definite AD": ICD-10 codes of AD 36 (F00, F00.0, F00.1, F00.2, F00.9, G30, G30.0, G30.1, G30.8, G30.9) AND dementia medication prescribed with an AD diagnosis (e.g., donepezil, rivastigmine, galantamine, and memantine). Furthermore, we considered a broader definition of AD using only ICD-10 codes to minimize false negative cases (e.g., individuals with AD diagnose who did not take medication); this was labeled as "probable AD". Within each individual with either definition of incident AD, the data after the incidence was excluded. Based on these two operational definitions, the prevalence rates were 1.5% for definite AD and 4.9% for probable AD; the former was smaller than what is reported in a door-to-door visit study in Korean elders (age >65 years old), but the latter was similar to that 37 .

Data and preprocessing
We used the following variables from the NHIS-NSC data: 21 features including laboratory values, health profiles, history of family illness from the Health Screening database; 2 features including age and sex from the Participant Insurance Eligibility database; and 6412 features including ICD-10 codes and medication codes. Descriptions of data coding and exclusion criteria for all the features except for ICD-10 codes and medication codes are available in Supplementary Table 4.
Our data preprocessing steps are as follows. (i) Data alignment: We aligned the data to each individual's initial AD diagnosis (event-centric ordering). (ii) ICD-10 and medication coding: Since ICD- 10  , and the first 4 characters for the medication codes representing main ingredients. (iii) Rare disease or medication codes found less than five times in the entire data were excluded from the analysis (1179 disease and 362 medication codes). (iv) If a participant has no health screening data (laboratory values, health profiles, and history of personal and family illness from the National Health Screening database) during the last two years of the processed data (in Korea a biannual health screening is required for every elder), we excluded that participant from the analysis. This preprocessing procedure yielded 4894 unique variables used in the models (see Supplementary Table 2  For model training, validation, and testing, we used the randomly sampled balanced dataset, as well as the entire, unbalanced dataset. For the balanced dataset, we performed bootstrap sampling with replacement 10 times.

Machine learning analysis
We implemented three machine learning algorithms: random forest, support vector machine with linear kernel, and logistic regression. Model training, validation, and testing was done using nested stratified 5-fold cross validation with 5 iterations. Feature selection was done within train sets using the variance threshold method 38 . Hyper-parameters optimization was done within validation sets. The following hyper-parameters were tuned: for random forest, the minimum number of samples required at a leaf node and the number of trees in the forest; for support vector machine, regularization strength; for logistic regression, the inverse of regularization strength. In logistic regression, L2 regularization was used. Lastly, generalizability of model performance was assessed on the test sets. We measured the following model performance metrics in the test set: The area under the receiver operating characteristic curve (ROC), sensitivity and specificity.

Ethical approval
This study complies with the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline. The study with exemption of informed consent (for J.H. Park et al. retrospective, de-identified, publicly available data) was approved by the Institutional Review Board of National Health Insurance Service (NHIS) Ilsan Hospital, Gyeonggi-do, Korea (IRB number NHIMC 2018-12-006). All methods in this study were performed in accordance with the Declaration of Helsinki.

Reporting summary
Further information on research design is available in the Nature Research Reporting Summary linked to this article.

DATA AVAILABILITY
The data in this study is available upon request.