Development and internal validation of a model to predict type 2 diabetic complications after gestational diabetes

Gestational diabetes mellitus (GDM) increases the risk of early-onset type 2 diabetes, which further exacerbates the risk of developing diabetic complications such as kidney, circulatory, and neurological complications. Yet, existing models have solely focused on the prediction of type 2 diabetes, and not of its complications, which are arguably the most clinically relevant outcomes. Our aim was to develop a prediction model for type 2 diabetic complications in patients with GDM. Using provincial administrative data from Quebec, Canada, we developed a model to predict type 2 diabetic complications within 10 years among 90,143 women with GDM. The model was internally validated and assessed for discrimination, calibration, and risk stratification accuracy. The incidence of diabetic complications was 3.8 (95% confidence interval (CI) 3.4–4.3) per 10,000 person-years. The final prediction model included maternal age, socioeconomic deprivation, substance use disorder, gestational age at delivery, severe maternal morbidity, previous pregnancy complications, and hypertensive disorders of pregnancy. The model had good discrimination [area under the curve (AUROC) 0.72 (95% CI 0.69–0.74)] and calibration (slope ≥ 0.9) to predict diabetic complications. In the highest category of the risk stratification table, the positive likelihood ratio was 8.68 (95% CI 4.14–18.23), thereby showing a moderate ability to identify women at highest risk of developing type 2 diabetic complications. Our model predicts the risk of type 2 diabetic complications with moderate accuracy and, once externally validated, may prove to be a useful tool in the management of women after GDM.


Study population.
We conducted a retrospective cohort study of women who had hospital-based deliveries in Quebec, Canada from April 1989 until March 2016 (cohort entry); women were then followed until 2018 to identify outcomes 14 . The cohort was constructed from the Maintenance and Use of Data for the Study of Hospital Clientele registry, which comprises > 99% of deliveries in Quebec.
Individuals aged 18 to 45 years who had GDM in at least one pregnancy were included, with the cohort entry point (t 0 ) at the first GDM-affected pregnancy. GDM was defined as abnormal glucose tolerance of the mother, first identified during pregnancy, and identified using diagnostic codes from the 9th and 10th revisions of the International Classification of Diseases (ICD) ( Table S1). These codes have been previously validated and adequately capture GDM diagnoses with specificity of > 90% and positive predictive values of > 80% 15,16 . There are some variations in approaches for identifying GDM in different centres, that is, one-step vs two-step approaches; however, both approaches are endorsed by Diabetes Canada 17 .
Women who died in their first affected pregnancy and women with pre-existing diabetes, or its complications were excluded (Fig. 1).
Outcome. The primary outcome was hospitalization for type 2 diabetic complications within 10 years after delivery of the first pregnancy affected by GDM. Type 2 diabetic complications was defined as a diagnosis of type 2 diabetes with the development of one or more of the following complications: diabetic coma, acidosis, kidney, ophthalmic, neurological, circulatory, or other complications resulting from diabetes and identified using ICD-9 and 10 codes, previously validated in studies with specificity of 99% and positive predictive values of > 80% (Table S1).
The secondary outcome was type 2 diabetic complications occurring anytime (up to 29 years) after delivery of the first pregnancy affected by GDM.
Women were followed from cohort entry until any of the outcome occurrence, death, or the end of the study period (March 31, 2018).

Statistical analyses.
We developed Cox proportional hazards regression models to predict type 2 diabetic complications, according to the previously outlined steps 18,19 , and report the process using the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines (Table S2) 20 .
Candidate predictors, variable selection and coding. We considered demographic, reproductive, and clinical factors known to be associated with an increased risk of type 2 diabetes as potential predictor variables 5,21 . These factors included maternal age, substance use, morbid obesity, socioeconomic deprivation (measured using a composite score of neighbourhood income, education, and employment) 22 , pregnancy factors such as parity, and multifetal pregnancy, and pregnancy complications such as hypertensive disorders of pregnancy (HDP), severe maternal morbidity (SMM) 23 , stillbirth, preterm delivery, low birth weight, and admission into neonatal intensive care unit (NICU) or adult intensive care unit (ICU). Candidate predictors were measured at the time of the index delivery (cohort entry).
Clinical variables that had low incidence were combined with other similar variables (e.g., previous obstetric complications such as SMM, stillbirth, preterm delivery, low birthweight, NICU admission, or neonatal death were combined). Previous history of obstetric complications was further combined with parity as follows: previous obstetric complication (among multiparous women), no previous obstetric complication (among multiparous women) and no previous obstetric complication (among primiparous women). When collinearity (r > 0.5) existed between variables, the most clinically relevant variable was selected.
Continuous candidate predictor variables (e.g., maternal age) were modelled using restricted cubic splines with three knot locations 19 . We assessed interaction terms and retained predictors that were statistically significant (alpha = 0.10) 18 . The final model variables were selected using Least Absolute Selection and Shrinkage Operator (LASSO) regression 18 .

Model performance and internal validation.
Predictive performance of the model was assessed based on discriminatory, calibration, and risk stratification accuracy 18 . Discrimination was measured by the c-statistic, which is equivalent to the area under the receiver operating characteristic curve (AUROC) 19 . An AUROC of ≥ 0.7 was interpreted as good discrimination and 0.6 to < 0.7 modest, while 0.5 to 0.6 was considered poor and < 0.5 as not having any discriminative ability. Calibration performance was examined by plotting the mean observed www.nature.com/scientificreports/ events versus the mean predicted risks by decile. Calibration slopes were interpreted as good (slope > 0.7), poor (0.5 < slope ≤ 0.7) or non-informative (slope ≤ 0.5) 24 . Using a risk classification table, we examined the ability of the model to stratify the population into low-and high-risk categories. We divided the population into four risk groups, with the highest calculated risk group corresponding to the overall incidence rate of the outcome in the study population 25 . Likelihood ratios (LR) were computed to assess the classification accuracy within each group 26 . For clinical use, positive LRs (LR+) of > 5 or > 10 were interpreted as moderate or good "rule-in" tests, respectively, whereas negative LRs (LR−) of < 0.2 and < 0.1 were considered as moderate or good rule-out tests, respectively 24 .
The model was assessed for internal validity using the bootstrap method with 200 iterations and the overoptimism (i.e., degree to which a model is overfit) was reported 18 . Secondary analyses. Using the same final selected variables, we also developed a prediction model for type 2 diabetic complication up to 29 years after delivery and assessed the model discriminatory performance. www.nature.com/scientificreports/ Sample size. We estimated our sample size based on the rule of thumb of 10-20 events per degree of freedom 19 , to avoid model overfitting. With a total of 1025 events during follow-up, we had sufficient sample size to consider up to 50 degrees of freedom for candidate predictors. Analyses were conducted using R version 3.5.1 (The R Project for Statistical Computing).

Result
Cohort description. Among 1,229,263 women who delivered between 1989 and 2016 in Quebec, our final cohort included 90,143 (7.3%) individuals who met study inclusion criteria (Fig. 1). Among these individuals, the number of people admitted to hospital within 10 years of delivery with any diagnosis of type 2 diabetes, including both complicated and uncomplicated diabetes, was 1858 (2.06%) corresponding to an incidence rate of 25.0 (23.9-26.2) per 10,000 person-years. Within 10 years of delivery, there were 283 (0.3%) women who were hospitalized with type 2 diabetic complications [incidence rate 3.8 per 10,000 person-years (95% CI 3.4-4. 3)], ( Table 1). The median follow-up time was 6.2 years. The incidence of type 2 diabetic complications was higher for women who were younger than 25 years, obese, had substance use disorders, or were socioeconomically deprived at their first GDM-affected pregnancy, compared with those who did not have these characteristics (Table 1). Diabetic ketoacidosis [0.9 per 10,000 person-years (95% CI 0.7-1.12)], kidney complications [0.8 per 10,000 person-years (95% CI 0.7-1.1)], and neurological complications [0.8 per 10,000 person-years (95% CI 0.7-1.1)] were the specific diabetic complications with the highest incidence rates during the 10-year period ( Table 2).
Model performance for prediction of type 2 diabetic complications. The final models consisted of seven variables measured at the first GDM-affected pregnancy: maternal age, socioeconomic deprivation, substance use disorder, gestational age at delivery, SMM, previous pregnancy complications, and type of hypertensive disorder of pregnancy (Table 3; prediction model equation is provided in Table S4). www.nature.com/scientificreports/ The model had good discriminatory performance for the prediction of type 2 diabetic complications (AUROC 0.72, 95% CI 0.69-0.74) with a mean optimism of 0.0077 upon internal validation, indicating minimal overfitting (Fig. 2). Calibration performance was good with a slope of 0.99 and an intercept of − 0.31 (Fig. 3).
In the risk stratification table, the proportion of women with the primary outcome increased in each risk category, as the predicted risk of the outcome increased (Table 4). At the highest risk group (women with calculated risk of ≥ 0.03), 2.5% had the primary outcome whereas only 0.2% in the lowest risk group (women with calculated risk of < 0.006) had the outcome. The resulting LR+ in the highest risk group was 8.68 (95% CI 4.14-18.23) suggesting a moderate rule-in of type 2 diabetic complications in the top 0.3% of the women with the highest calculated predicted risk. However, the confidence interval was wide, due to low number of outcomes in this group (n = 7).
Secondary analyses (prediction of outcome up to 29 years after delivery). The incidence of the type 2 diabetic complications up to 29 years of follow-up was 9.0 (8.5-9.6) per 10,000 person-years, respectively (Table S3). Discriminatory performance decreased slightly, with AUROC of 0.63 (0.61-0.64).

Discussion
In this study comprising 90,143 women with GDM, we developed Cox-proportional hazard models to predict the risk of developing type 2 diabetic complications occurring within 10 years of obstetric delivery. The incidence of hospitalization for diabetic complications in this timeframe was low, reflecting the young age of the cohort, and possibly good clinical care. However, our developed models consisting of readily measured variables at the time of birth showed moderate discriminative and calibration abilities for the prediction of type 2 diabetic complications after GDM pregnancy. Although external validation of the model is required, the AUROC of > 0.7 and LR+ of 8.68 in the highest risk category of ≥ 0.03 suggests that the model may eventually be used to reliably identify women at high risk of developing complicated diabetes.
Previous work has established a strong etiologic link between GDM and a future diagnosis of type 2 diabetes mellitus 4,5 . In a systematic review of 28 studies evaluating the risk of type 2 diabetes risk after GDM, the cumulative incidence of type 2 diabetes varied depending on time since delivery and ethnicity, and ranged from 2.6% to over 70% in included studies 2 . Our outcome of interest was hospitalization with type 2 diabetic complications within 10 years of delivery, reflecting more serious outcomes related to diabetes. The systematic review by Kim et al. also revealed that the cumulative incidence of type 2 diabetes increased over the first 5 years after delivery and plateaued after 10 years, supporting the chosen timeframe for our model 2 .
Previous models have focused on predicting type 2 diabetes after GDM, irrespective of the presence of diabetic complications 12,13 . A genetic risk score developed by Kwak et al. using data from 395 women with GDM reported a C statistic of 0.775 for predicting type 2 diabetes 12 . Another clinical model which included three types of lipid was developed by Lappas et al. in a cohort of 104 women and had C statistic ranging from 0.756-0.865 13 . These models did not include diabetic complications, which arguably have a more significant impact on women's health, and would justify enhanced surveillance and early aggressive risk reduction therapy. Additionally, the small sample sizes used in these prior studies and the inclusion of variables not easily measured (e.g. genetic factors) lower the generalizability to routine clinical settings 18 .
Despite advances in our understanding of the association between pregnancy complications and long-term conditions, incorporation of pregnancy factors into existing risk scores has not improved net reclassification [27][28][29] . Possible reasons for the lack of success of previous models include their development in older populations and short length of follow-up, making them less suitable for women of reproductive age 30 . Developing prediction models among specific subpopulations, including postpartum women who have had a pregnancy complication, may circumvent some of these issues [12][13][14] . Despite the young age of individuals in our cohort, the predicted probability of type 2 diabetic complications at 10 years is substantial, indicating a robust model.
Late diagnosis of type 2 diabetes is often accompanied by development of diabetic complications, which is associated with healthcare expenditures three times greater than treatment of uncomplicated diabetes 6 . Clinical practice guidelines in Canada recommend performing a 75-g oral glucose tolerance test between 6 weeks and 6 months of delivery after GDM to identify type 2 diabetes early, with the goal of minimizing downstream sequelae 11 . Unfortunately, uptake of screening for type 2 diabetes after GDM remains low (< 50%), due to limited www.nature.com/scientificreports/ appreciation of risk, logistics of appointments and tests, and concerns of new mothers 11 . A shift in focus on screening strategies based on risk of more severe diabetes may allow for a greater mobilization of resources to a smaller segment of the population with GDM, particularly in lower resource settings. Our study shows that future type 2 diabetic complications can be predicted with moderate accuracy in women with GDM, using readily available clinical variables present in administrative datasets. The advantages of administrative data are that they are population-based records, and are larger than data from clinical settings, therefore our model can forecast future complications for health planning purposes 31 . The model requires external validation before ultimately being converted to an online risk calculator 18,32,33 8-10 to aid in counselling and triaging of resources in postpartum women with GDM. If the postpartum patient has a calculated high risk, the approach and followup would differ (e.g., endocrinology referral, early OGTT, intensive lifestyle approach) from that of a lower-risk patient (e.g., glycosylated hemoglobin, community follow-up, usual postpartum lifestyle recommendations). www.nature.com/scientificreports/ This individualized approach will help with the study, planning, and implementation of targeted preventive strategies at the population level. Our study has several strengths. First, we used a large population-based cohort of women from Quebec, which represents more than one-quarter of the Canadian population with a diverse multi-ethnic population. We had sufficient sample size for model development and minimal overfitting as shown with internal validation. Second, our data allowed us to test short-term as well as long-term risks of the outcomes. We chose 10 years for the primary prediction period as this represents the time when up to 20% of individuals develop type 2 diabetes 2 . Furthermore, we followed model development methods recommended by experts, including the use of LASSO for final variable selection, rather than forward and backward selections which are more prone to bias 18 . Finally, we ensured transparency in modelling and reporting by following the TRIPOD statement 20 .
As is the case with many large administrative datasets, a downfall is the reliance on ICD codes to define predictors. In particular, the dataset lacked information on clinical parameters such as blood pressure, body mass index, lipid levels or blood glucose measurements which are commonly used in type 2 diabetes clinical prediction models. These continuous variables could potentially improve model discrimination as they increase heterogeneity between women 20 . Other markers of severity of GDM such as insulin levels are predictive of type 2 diabetes   www.nature.com/scientificreports/ but were lacking in our dataset 2 . In addition, we lacked information on ethnicity and family history, which are known determinants of diabetes and its complications 2 . Our prediction models, therefore, need to be improved and externally validated before they can be considered for use in a clinical setting. Nonetheless, our study identified easily measured variables that contribute to the prediction of long-term type 2 diabetic complications after GDM. Our models should therefore be used as a foundation to improve upon risk classification after pregnancy. Finally, it is important to note that the diagnosis of GDM has evolved over time 34 . However, in un-shown analyses, restricting our population to deliveries prior to 2010, corresponding with the International Association of Diabetes and Pregnancy Study Groups (IADPSG) diagnostic recommendations publication 34 , did not change model performance.

Conclusion
Type 2 diabetic complications can be moderately predicted by our clinical model developed from administrative health data among individuals with a history of GDM. Future work is needed to improve the model performance and test the validity of the model in other datasets. After validation in external cohorts, women with GDM who are at high risk of developing type 2 diabetic complications can be earlier identified immediately after delivery. This will enable individualized planning and targeted approaches, such as endocrinology referral, early OGTT, and appropriate lifestyle modifications, to prevent future type 2 diabetic complications. Simple models to predict diabetic complications such as ours have the potential to help to develop a rational risk-based approach to screening and surveillance after GDM.

Data availability
The data that support the findings of this study are available from Institut de la statistique du Québec but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from Institut de la statistique du Québec upon reasonable request and with permission.