Despite the increasing adoption of insulin pumps and continuous glucose monitoring devices, most people with type 1 diabetes do not achieve their glycemic goals1. This could be related to a lack of expertise or inadequate time for clinicians to analyze complex sensor-augmented pump data. We tested whether frequent insulin dose adjustments guided by an automated artificial intelligence-based decision support system (AI-DSS) is as effective and safe as those guided by physicians in controlling glucose levels. ADVICE4U was a six-month, multicenter, multinational, parallel, randomized controlled, non-inferiority trial in 108 participants with type 1 diabetes, aged 10–21 years and using insulin pump therapy (ClinicalTrials.gov no. NCT03003806). Participants were randomized 1:1 to receive remote insulin dose adjustment every three weeks guided by either an AI-DSS, (AI-DSS arm, n = 54) or by physicians (physician arm, n = 54). The results for the primary efficacy measure—the percentage of time spent within the target glucose range (70–180 mg dl−1 (3.9–10.0 mmol l−1))—in the AI-DSS arm were statistically non-inferior to those in the physician arm (50.2 ± 11.1% versus 51.6 ± 11.3%, respectively, P < 1 × 10−7). The percentage of readings below 54 mg dl−1 (<3.0 mmol l−1) within the AI-DSS arm was statistically non-inferior to that in the physician arm (1.3 ± 1.4% versus 1.0 ± 0.9%, respectively, P < 0.0001). Three severe adverse events related to diabetes (two severe hypoglycemia, one diabetic ketoacidosis) were reported in the physician arm and none in the AI-DSS arm. In conclusion, use of an automated decision support tool for optimizing insulin pump settings was non-inferior to intensive insulin titration provided by physicians from specialized academic diabetes centers.
Intensive insulin therapy is currently the standard of care for individuals with type 1 diabetes and is needed to reduce the risk of diabetes-related complications2,3,4. However, despite the increasing adoption of insulin pumps and continuous glucose monitoring (CGM) technologies5, fewer than one-third of individuals with type 1 diabetes achieve their diabetes management goals1,6. Inaccurate insulin pump settings due to lack clinician knowledge or inadequate frequency of insulin dosage adjustments could, in part, explain why glycemic control continues to worsen in these patients7.
Frequent adjustment of insulin pump settings is particularly important in younger individuals so as to accommodate their rapidly changing insulin requirements8. In the landmark Diabetes Control and Complications Trial (DCCT), people with type 1 diabetes who were treated with intensive insulin therapy achieved an average 2% reduction in glycated hemoglobin by attending weekly in-person clinic visits until glycemic goals were met, followed by monthly visits and weekly telephone contacts2,9. These findings and others support that regular and frequent insulin dose adjustments are needed to improve glycemic control in a broad population of people with either type 1 diabetes or type 2 diabetes10,11,12. Nevertheless, current guidelines recommend that children, adolescents and young adult individuals with type 1 diabetes attend clinical visits every 3–4 months13,14. Current standard of care guidelines are thus inadequate. Additionally, patient adherence to this recommendation is often poor15. Obtaining pump and continuous glucose data then analyzing the data are also time-consuming tasks for physicians, presenting an additional barrier to the provision of frequent insulin pump adjustments16.
Given the mounting shortage of endocrinologists, especially in rural areas17, the current paradigm of clinical care lacks the ability to provide the timely insulin adjustments needed to improve the management of type 1 diabetes. Digital decision support systems can address these issues by facilitating timely and more frequent insulin dose adjustments, either in person or remotely, and thus improve glycemic control for individuals with diabetes.
We evaluated an automated artificial intelligence-based decision support system (AI-DSS; The DreaMed Advisor Pro; DreaMed Diabetes Ltd., Petah Tikva, Israel), which provides insulin therapy adjustment recommendations and tips for diabetes management to healthcare professionals managing individuals with type 1 diabetes using an insulin pump and CGM. Preliminary studies provided initial indications of the safety and clinical validity of using the AI-DSS device18. A survey of 26 physicians with experience in insulin pump dosing titration showed that the level of agreement among physicians regarding the direction of insulin recommendations (increase, decrease or no change) for 15 patients with type 1 diabetes was only around 41–46% for each of the pump parameters (basal insulin infusion rate, the insulin-to-carbohydrate ratio (CR) and the correction factor (CF)) and the level of disagreement was around 10–12% (ref. 18). In the same study, similar levels of agreement were found between the physicians and the AI-DSS. We hypothesized that frequent optimization of insulin pump therapy based on CGM readings using the AI-DSS would result in non-inferior glycemic control compared with dose adjustments performed by physicians from specialized academic diabetes centers.
From 20 November 2017 to 25 July 2019, 138 individuals were assessed for eligibility; 122 were randomized and 108 participants (AI-DSS, n = 54; physician, n = 54) were included in the intention-to-treat analysis (Fig. 1). The demographic and clinical characteristics of the randomized participants were similar between the two arms (Table 1). Sixty participants were randomized at US sites and 62 at European and Israeli sites. The disposition of randomized participants is presented in Fig. 1. The average (s.d.) number of CGM-valid days for analysis was 137 ± 35 days per participant in the ‘physician’ arm and 133 ± 32 days in the AI-DSS (‘advisor’) arm (P = 0.25, two-tailed paired t-test).
The percentage of time spent within target glucose range in the AI-DSS arm was statistically non-inferior to the physician arm (50.2 ± 11.1% versus 51.6 ± 11.3%, respectively) (95% lower confidence bound CI: −3.3%, P < 1 × 10−7). The percentage of readings below 54 mg dl−1 was statistically non-inferior between the AI-DSS and physician arms (1.3 ± 1.4% and 1.0 ± 0.9%, respectively. P < 0.0001).
A total of 20 diabetes-related study adverse events (AEs) (AI-DSS arm, n = 8; physician arm, n = 12) were reported. Among these, there were three severe AEs (two severe hypoglycemia, one diabetic ketoacidosis), which were reported in the physician arm. All AEs are presented in Table 2.
Both arms showed statistically significant reductions in mean glycated hemoglobin level from baseline to mid study (week 12; Fig. 2). A statistically significant reduction of 0.32% in mean glycated hemoglobin level from baseline to end of study (week 24) was observed in the AI-DSS arm (two-sided 95% CI of −0.55% to −0.08%, P = 0.008). The reduction from baseline to week 24 in the physician arm was 0.19% (two-sided 95% CI of −0.49% to 0.11%, P = 0.22; Fig. 2). However, the change in glycated hemoglobin level from baseline to the end of the study when comparing the two arms was found to be statistically not different (P = 0.49).
No between-arm differences were observed in the percentage of time above and below glucose target ranges (Extended Data Fig. 1). Mean total daily insulin and daily basal insulin doses were not statistically different. Daily bolus insulin doses among participants in the AI-DSS arm were higher than in the physician arm (29.6 ± 9.7 and 26.6 ± 8.3 units, respectively, P = 0.03).
Eight of the 13 AI-DSS physicians completed the satisfaction questionnaire at week 12 and all completed the questionnaire at week 24. At study end, the majority (n = 11, 85%) stated they were interested in continuing to use the AI-DSS as part of their routine practice. Responses to the Healthcare Professional Post-Intervention Survey are presented in Extended Data Fig. 4.
Overall, 74.9% of the planned insulin dose titration visits were performed in the AI-DSS arm and 87.7% in the physician arm. The main reason for not providing a dose titration was due to insufficient or missing glucose and/or insulin data. Descriptions of the numbers and technical issues related to missing insulin dose visits recommendations in both study arms are presented in Extended Data Fig. 5.
Override information was available for 50 of 54 participants in the AI-DSS arm (four participants did not receive any recommendations throughout the study; see Extended Data Fig. 5 for further details). A total of 12,864 comparison time points were available for each pump setting parameter. The percentage of overrides was 3.2% for the basal plan, 1.7% for the CF and 0.9% for the CR. The overall percentage of override was 1.9% for the combined pump settings parameters. The absolute relative differences in the magnitude of change made were 10 ± 6.3%, 6.2 ± 7.3% and 15.5 ± 4.7% for basal, CF and CR, respectively.
In this randomized, controlled, single-masked, non-inferiority trial, we found that the use of the AI-DSS for optimization of insulin pump settings was non-inferior in efficacy and safety to intensified clinical care provided by trained physicians. The AI-DSS was found to be safe and the percentage of readings in the hypoglycemia range below 54 mg dl−1 within the AI-DSS arm was statistically non-inferior to that in the physician arm.
Frequent insulin dose adjustments by healthcare professionals between in-person visits using downloaded data from devices have been shown to improve glycemic control12,19. The current study was not designed to evaluate the effect of dose titration; however, a significant reduction in glycated hemoglobin level was observed in both arms after 12 weeks, with a significant reduction using the AI-DSS at the end of the study.
There was a high level of satisfaction among the physicians who used the AI-DSS and who completed the Healthcare Professional Post-Intervention Survey. However, there were cases where physicians assigned to the AI-DSS arm decided to change the recommendations given by the AI-DSS. Because the percentage of these overrides was relatively small, their influence on the outcomes is likely to be negligible. That some physicians override the AI-DSS recommendations is not surprising. Insulin dose adjustments are not only influenced by science but also by different subjective attitudes—individual expert clinicians have been shown previously to provide different insulin dosing adjustments to the same given patient data18.
A key strength of our study was the use of a non-inferiority design to evaluate a new AI-DSS by comparison with a known treatment approach provided by physicians from academic centers who are experienced in the use of diabetes technologies. Another strength of our study was the randomization of study physicians, which helped to ensure equal levels of expertise in diabetes management between the study arms. Furthermore, participants were masked to their randomization arm, eliminating bias regarding the source of the given insulin recommendations (automated versus physician). Additionally, stratification of participants according to age and baseline glycated hemoglobin level facilitated a more robust comparison of closely matched study arms. Finally, use of a multicenter, multinational design further supports the generalizability of the study findings.
A potential limitation is the effort involved in frequent uploading and downloading of data as well as the minimum CGM data required for the scheduled, obligatory, insulin titrations. Nonetheless, the relatively low number of study withdrawals probably indicates that the benefits of frequent insulin titration and the ease with which the recommendations were communicated to the participants could outweigh the burden of data sharing. Moreover, several technological advancements already exist that potentially mitigate this limitation (for example, pre-calibrated CGM and connected devices). In addition, the remote nature of the data sharing and subsequent dose adjustments through online or mobile applications were probably less disruptive than frequent in-person clinic visits, which often require absences from school or work. Real-world studies are needed to fully explore the persistence of physicians and individuals with diabetes in using the AI-DSS in daily diabetes management. The study is also limited by the fact that it included a selected population of young people with type 1 diabetes who were already receiving their diabetes care at academic tertiary care institutions and who were willing to participate in the study by providing written informed consent and satisfying the inclusion and exclusion criteria. These individuals might not generally represent young people with type 1 diabetes, which could limit the ability to generalize these data to broader populations. An additional limitation of the study is that the AI-DSS is suitable for a specific population of people with diabetes who have the ability to upload the device’s diabetes-related data and who have regular access to the internet. The challenge of manual data uploading and downloading could be eliminated by real-time data transmission in the near future; however, an internet connection will still be required.
Given the increasing prevalence of diabetes20,21 and the mounting shortage of specialized endocrinologists22, the responsibility to manage individuals with diabetes may be increasingly taken on by primary care physicians. Because many of these physicians are inexperienced in insulin therapy and the use of CGM data, innovative and readily available tools will probably be welcome additions to support clinical decision making in the primary care setting. Decision support systems have the potential to deliver this assistance and to elevate the quality of diabetes care by creating a virtual expert diabetes clinic that facilitates more frequent insulin adjustments between clinic visits. These systems can be used with telemedicine approaches to deliver expert knowledge, reduce disease burden for patients and alleviate burdens on caregivers. They can also lower the costs associated with clinical visits (for example, lost work and/or school days) and overcome missed visits15, particularly among people who live in rural areas17 or during other circumstances in which access to face-to-face visits with physicians is limited. Moreover, the AI-DSS provides the ability to standardize care across health systems, ensuring that all people with type 1 diabetes would receive quality care, even if they live in remote areas or are managed by a care team with little experience with CGM interpretation or insulin pump therapy.
Our study shows that optimizing insulin pump settings with an AI-DSS at three-week intervals for six months achieved statistically similar levels of glucose control to those achieved by physicians with diabetes expertise. The AI-DSS used in our study provides opportunities for a new modality for intensive insulin management that can offer the frequently needed insulin adjustments necessary to improve glucose control in young people with type 1 diabetes.
Trial design and oversight
The ADVICE4U study was designed to evaluate whether frequent insulin dose titration using a new treatment modality—an automated decision support system (AI-DSS arm)—would be comparable in efficacy to current best practice, that is, titration by physicians from specialized academic diabetes centers (physician arm). Accordingly, a non-inferiority design was preferred over a superiority design23. The ADVICE4U study was a six-month, multicenter, multinational, single-blind, parallel (two-arm), randomized controlled, prospective, non-inferiority trial that evaluated the equivalence of frequent insulin dose adjustments at three-week intervals on the percentage of time in the target glucose range (70–180 mg dl−1, 3.9 to 10 mmol l−1) made using a decision support system (AI-DSS arm) compared with adjustments made by physicians from academic diabetes centers including certified endocrinologists, diabetologists and fellows in pediatric endocrinology (physician arm). The trial was conducted at seven clinical sites; four in the USA (University of Florida, Joslin Diabetes Center, Barbara Davis Center, Yale University School of Medicine), two in Europe (University Children’s Hospital Ljubljana Slovenia, Children’s Hospital AUF DER BULT, Germany) and one in Israel (Schneider Children’s Medical Center of Israel). A complete list of centers and center staff is provided in the Acknowledgments section.
The trial was conducted in compliance with the International Conference on Harmonisation/Good Clinical Practice (ICH/GCP) applicable regulatory requirements, in accordance with International Organization for Standardization (ISO) 14155:2012, the Declaration of Helsinki and Code of Federal Regulations title 21, parts 50, 54, 56 and 812. The study protocol was approved by the local institutional review boards at each participating center as follows: Joslin Diabetes Center, Committee of Human Studies; Barbara Davis Center for Childhood Diabetes, Colorado Multiple Institutional Review Board; Yale University School of Medicine, Yale University Human Investigation Committee; Diabetes Institute University of Florida, Institutional Review Board University of Florida; Schneider Children’s Medical Center of Israel, Institutional Review Board Rabin Medical Center; University Children’s Hospital Ljubljana Slovenia, national regulatory authorities—Republic of Slovenia National Medical Ethics Committee (JAZMP); Children’s Hospital AUF DER BULT, national regulatory authority—Bundesinstitut für Arzneimittel und Medizinprodukte (BfArM). Protocol version 1.5 (available in the Supplementary Information) was used for all sites. The German site was the last site to join the study as the IRB approval was delayed and used protocol version 1.6. According to a BfArM request, two changes were made to protocol version 1.5: the exclusion criteria included hypoglycemia unawareness and an additional term for premature study discontinuation: occurrence of three device/advice related SAEs during the study conduct (SAE, serious adverse event). All participants (or parents/legal guardians) provided written informed consent, and assent according to local requirements.
An external contract research organization coordinated the trial, performed site monitoring and ensured data collection and management. Safety aspects of the study were monitored by an independent data and safety monitoring board.
Children, adolescents and young adults who had had type 1 diabetes for at least one year and were treated with insulin pump therapy were eligible for participation in the trial. Inclusion criteria for participation were as follows: subjects with type 1 diabetes for at least a year since diagnosis; ages 10 to 21 years; glycated hemoglobin level from 7.0% (53 mmol mol−1) to 10.0% (86 mmol mol−1); use of insulin pump therapy for at least four months (OmniPod insulin pump (Insulet Corp.) or Medtronic pump: MiniMed 530G System (MMT-551, MMT-751), MiniMed Paradigm REAL-Time Revel (MMT-523, MMT-723, MMT-523K, MMT-723K) and MiniMed Paradigm (MMT-515, MMT-715, MMT-522, MMT-722, MMT-522K, MMT-722K), MiniMed Veo Insulin Pump (MMT-754, MMT-554) or any other Medtronic pump that is compatible with the diabetes management system (DMS) app used in this study); BMI-SDS below the 97th percentile for age; willingness to follow study instructions (that is, to measure at least two capillary blood glucose readings per day as needed for CGM calibration and use the bolus calculator feature of the pump); patients/parents were required to have a minimum level of computer skills and understanding of navigating the internet; willingness to use a Dexcom CGM device throughout the study duration; patients/parents have a smartphone (Apple or Android and Windows).
Exclusion criteria were an episode of diabetic ketoacidosis within the month before study entry; any significant diseases/conditions including psychiatric disorders and substance abuse that in the opinion of the investigator were likely to affect the subject’s ability to complete the study or compromise patient safety; current participation in any other interventional study; known or suspected allergy to trial products such as adhesives, tapes and needles; female subject who is pregnant or lactating or planning to become pregnant within the planned study duration; severe hypoglycemia within six months prior to enrollment (as defined by the ADA and Endocrine Society as follows: severe hypoglycemia is an event requiring assistance of another person (due to change in mental status) to actively administer carbohydrates, glucagon or take other corrective actions); current use of medications that are used to lower blood glucose, such as pramlintide, metformin and GLP-1 analogs, or beta blockers, glucocorticoids and other medications, which in the judgment of the investigator would be a contraindication to participation in the study (anticoagulant therapy such as plavix, low-molecular-weight heparin, coumadin, immunosuppressant therapy); subjects who have a relevant severe organ disorder (diabetic nephropathy, diabetic retinopathy, diabetic foot syndrome) or any secondary disease or complication of diabetes mellitus (for example, subjects who have unstable or rapidly progressive renal disease or are receiving dialysis; subjects who have active proliferative retinopathy or active gastroparesis); subjects who are suffering from an eating disorder.
Trial devices and interventions
The AI-DSS utilizes data from glucose monitoring (CGM readings and capillary blood glucose measurements), as well as insulin doses and carbohydrate intake obtained from the insulin pump data and the pump bolus calculator over at least 12 days of routine diabetes care. The AI-DSS uses artificial intelligence that detects and analyzes glucose patterns and insulin dosing events in a similar approach to that used by a healthcare provider based on expert knowledge, recommendations and data acquired from various clinical studies. Its recommendations include exact adjustments for insulin pump settings (basal rates, CRs and CFs) and it provides personalized diabetes management tips related to the way insulin is delivered (for example, missed boluses, timing of pre-meal bolus, overtreatment of hypoglycemia and so on), highlighting patterns related to an individual’s insulin dosing and delivery behaviors.
The AI-DSS comprises a cloud-based server and an algorithm module. The cloud-based server is a secured database and computing platform that hosts the data inputs from the DMS (Glooko Platform—mobile app and online software; Glooko Inc.). It is used to retrieve anonymous data from the DMS, temporarily store raw data, transmit these data to the algorithm for processing and result generation, and store this to the AI-DSS results. The result is then sent to the DMS, which generates a report for the healthcare provider, who can edit, approve and share the recommendation with the patient (see Extended Data Fig. 6 for the data flow).
The algorithm is designed to provide a comprehensive analysis of an individual’s diabetes data consisting of glucose levels, insulin delivery history and meal consumption, as reported through the insulin pump’s bolus calculator, to recommend adjustments to the patient-specific insulin pump settings (CR, CF and basal plans) as well as suggestions for personalized diabetes management tips. The system pulls the most recent 21 days of data from the DMS from the last date of data upload. Of the data pulled, AI-DSS requires at least 12 valid days to produce recommendations. A valid day should include at least 67% of CGM sensor readings according to the sensor’s sampling rate and at least one basal and one bolus record. In addition, it requires at least three records from the bolus calculator and a valid recent insulin pump settings plan (that is, the basal plan is set and the CR, CF, glucose targets and active insulin are set inside the pump bolus calculator). The AI-DSS uses the raw data input to detect dosing decision events by the patient using both the insulin pump and CGM glucose meter to characterize each insulin dosing event. In parallel, it detects patterns of hypoglycemia, euglycemia and hyperglycemia. The event-based analysis is then integrated with the patient’s glycemic patterns to produce a personalized recommendation. The recommended changes to the patient’s pump settings are limited to ensure safety: basal rate changes are limited to ±20% of the current hourly rate and no more than 50–150% of the hourly average basal rate calculated from the patient total daily insulin dose. The CR and CF changes are limited to ±30% of the current insulin pump settings.
The AI-DSS recommendations are created in a few seconds after the participant uploads the device’s data. The recommendation is sent to the physician for review and can be shared with the patient. An example of an AI-DSS recommendation report for the healthcare provider is presented in Extended Data Fig. 7.
Once the healthcare provider has approved a recommendation, it is shared with the patient through the AI-DSS section in the DMS app. The patients will receive a push notification to their phone as a reminder that they have a new recommendation. The patient can review the current recommendation (Extended Data Fig. 8) and agree or disagree with it. In the case of disagreement, the patient can document the reason and send feedback to the healthcare provider.
The DMS with integrated algorithm or physician recommendation report was used in both study arms for downloading the pump and CGM data, for a visual display of current insulin pump settings and for recording recommended insulin pump dose changes according to the designated study arm (allowing edits as needed), as well as uploading of the recommendations to share with participants/families. The DreaMed Advisor Pro algorithm version 01.07.xx was used during the study.
This was an investigator-initiated study and the AI-DSS was provided by DreaMed Diabetes Ltd for the study. Because the investigators did not have access to the device’s internal database, which is essential for error analysis, no such analysis was performed as part of the study. In addition, the AI-DSS software has been tested extensively as part of an FDA regulatory submission. DreaMed Diabetes Ltd reported no need to use the data collected in this study for further error analysis.
The trial consisted of a three-week run-in period in which participants continued their regular treatment and used the provided study CGM device. At the end of the run-in period, participants were randomized either to the AI-DSS or physician arms. Three weeks after randomization and every three weeks thereafter, all participants uploaded their insulin pump and CGM data using the DMS. Participants then received insulin pump settings recommendations either from the physician or AI-DSS according to their randomization, for a 24-week period. The number of scheduled contacts with study staff was identical for both study arms. The study design scheme and visit schedule are presented in Extended Data Figs. 9 and 10, respectively.
At the screening visit, baseline characteristics were collected, and all participants received a DexCom G5 CGM system (Dexcom) to use throughout the study. Participants continued to use their insulin pump (Medtronic MiniMed 530G, 640G/630G, Veo or Paradigm Realtime-Revel (all Medtronic Diabetes) or an OmniPod (Insulet Corporation)).
After a three-week run-in period, eligible participants were randomly assigned in a 1:1 ratio to either the AI-DSS or physician arms, using computer-generated permuted block randomization stratified by site, age (10–14 years, 15–17 years, 18–21 years) and glycated hemoglobin level at screening (7 to <8% (53 to <64 mmol mol−1), 8 to <9% (64 to <75 mmol mol−1) and 9–10% (75–86 mmol mol−1)) to achieve an equal distribution of participants in the two study arms. In this participants stratification there were thus nine participant strata. As the sample size calculation showed that 56 pairs of participants would give the study sufficient power and the study included seven centers, each site was assigned eight of the nine age-glycated hemoglobin strata and asked to recruit one pair of participants in each stratum (for example, each pair had one participant designated to the AI-DSS arm and the other to the physician arm). The decision as to which eight strata to allot to each center was done at random, with the constraint of keeping the number of pairs in each stratum as equal as possible. The purpose of using this kind of randomization was to ensure the inclusion of a wide range of ages with different levels of glycemic control in an equal fashion. On 3 September, 2018, open baskets for randomization were open for competitive recruitment. In addition, each clinical site included four physicians (except for one site that had two physicians) who were split into pairs, with each pair possessing similar levels of training and experience. Each pair of physicians was randomly assigned to review pump settings either by themselves (physician arm) or by using the AI-DSS (AI-DSS arm) for the study. It is important to emphasize that none of the physicians who actively participated in the study provided expert advice and/or guidance that informed the AI-DSS design. Additional information is presented in the study protocol.
Three weeks after randomization and every three weeks thereafter, for a total of seven times, participants uploaded their insulin pump and CGM data using the DMS. Extended Data Fig. 6 illustrates the flow of the data in the study. The DMS pulled the last 21 days of insulin pump and CGM data. The data were transferred to the AI-DSS; if the source of the data was a participant in the physician arm, a report form was created without insulin dosing recommendations for the physician to fill in electronically with their dosing recommendations. If the data came from a participant in the AI-DSS arm, the system verified that the data requirements included at least 12 valid days before producing recommendations. A valid day for the AI-DSS system is defined as a day with at least 67% of CGM data and at least one basal record and one bolus record. If the minimal data requirements were met, the AI-DSS produced insulin pump settings recommendations, which were then shown to the physician in the report form. Thus, at this point, in both arms, a report including the insulin pump and CGM data was available in the DMS for the designated physician to review. The AI-DSS arm physicians also received the automated insulin recommendations for review (for approval or revision by a designated physician). Physicians in the physician arm analyzed participant data and determined insulin pump adjustments based on their clinical judgment. For both arms, the recommended adjustments were then shared with the participant and/or family via the DMS, email and/or in a follow-up phone call in a similar manner, maintaining the masking of group assignment to participants. An example of the recommendations report as viewed in the DMS web platform for the physicians is presented in Extended Data Fig. 7 and the recommendations report for the participants as viewed through their smartphone is shown in Extended Data Fig. 8. Participants were instructed to enter the recommended pump settings and download the pump data within 24 h to confirm that the new settings had been correctly entered.
Glycated hemoglobin was measured four times throughout the study (at baseline, randomization, week 12 and week 24) using a point-of care standardized capillary glycated hemoglobin measurement (DCA 2000, DCA 2000+ or DCA Vantage, utilizing the same brand of reagents). Physicians participating in the AI-DSS arm were surveyed for their satisfaction at week 12 and week 24.
The primary efficacy outcome was the non-inferiority for percentage of time that the glucose level, as measured by the continuous glucose monitor, was in the target range of 70–180 mg dl−1 (3.9–10.0 mmol l−1) over the active treatment period between the two arms. The primary safety outcome was the percentage of time that the glucose level was less than 54 mg dl−1 (3.0 mmol l−1). Secondary outcomes were changes in glycated hemoglobin level from baseline to the end of study and AEs and SAEs. The exploratory glycemic measures were mean glucose, glycemic variability as measured by standard deviation (s.d.) and coefficient of variation (%CV), percentage of time that glucose level readings were <50 mg dl−1 (<2.8 mmol l−1), <70 mg dl−1 (<3.9 mmol l−1), >180 mg dl−1 (>10 mmol l−1) and >240 mg dl−1 (13.3 mmol l−1), insulin doses, as well as overrides of AI-DSS recommendations done by the physicians in the AI-DSS arm. Device satisfaction was evaluated using a 50-item investigator-developed questionnaire (Healthcare Professional Post-Intervention Survey) completed by the physicians who used the AI-DSS during the study (at 12 and 24 weeks). The psychosocial questionnaire completed by the healthcare providers during the study had good face validity, but its psychometric properties are unknown as it was developed in collaboration with participants, specifically for this study. Future research aims to validate the questionnaire.
We hypothesized that use of the AI-DSS would not be inferior to physician-guided recommendations in percentage of sensor readings within the target range of 70–180 mg dl−1 (3.9–10 mmol l−1). For the sample size calculation, data from the JDRF CGM randomized controlled study24 and the REPLACE-BG study25 were used to estimate the properties of the primary endpoint and to define the non-inferiority limit. The sample size was calculated using WINPEPI26 for non-inferiority analysis, with a margin of 7.5% between arms in a parallel study design with power of 90%, a one-sided significance level of 5% and s.d. of 13% with a correlation of 0.48 between the baseline and outcome time in range over the 24-week treatment period. The resulting analysis showed that 104 subjects were needed for the study. The sample size was set at 112 participants (16 patients at each site), given an estimated 5–10% dropout rate, with randomization in a 1:1 ratio between the two study arms.
The analyses complied with the intention-to-treat principle, including participants who were recruited according to the randomization rules and those pairs of participants for which follow-up data were available for both members of the pair. All principal analyses were based on paired comparisons, with pairs formed from participants within the same center. This methodology was used to eliminate inter-center differences. The relationship between the percentage of time in range and the treatment arm was examined by analysis of covariance (ANCOVA), adjusting for baseline percentage of time in range and number of valid days (a valid day is defined as a day with at least 67% of CGM values) used in the analysis. A one-sided hypothesis test and 95% confidence interval for the difference in treatment arms were computed to assess non-inferiority according to the defined margin. A randomization test was used to evaluate the non-inferiority between the treatment arms for the primary safety endpoint (percentage of time below 54 mg dl−1) with a non-inferiority limit of 2%. A two-sided paired t-test was used to evaluate the change in glycated hemoglobin level from baseline to 24 weeks within each arm. A two-sided, two-group t-test was used to evaluate the difference in the mean change of glycated hemoglobin level for the AI-DSS versus physician arms. For both AE count and SAE count, randomization tests were used to test the null hypothesis of no difference between physician and AI-DSS arms. Primary and secondary endpoints were also analyzed in a per protocol cohort that included pairs from AI-DSS and physician arms in which each participant received at least five of the seven scheduled recommendations.
The physician’s overrides analysis was computed as the percentage of change from the original AI-DSS recommendations. The recommended daily adjustments were divided into 48 half-hourly periods (half-hour period of available AI-DSS recommendation change) for basal, CR and CF pump setting parameters. There were a total of 144 comparison points (48 time periods and three pump setting parameters) per visit whenever AI-DSS recommendations were available. The recommendation comparison points were compared for the relative changes overall and by the pump setting parameters (that is basal, CR and CF plans).
Further information on research design is available in the Nature Research Reporting Summary linked to this Article.
Any requests for raw data (that is, glucose levels, insulin delivery, de-identified patient characteristics) will be reviewed by the NextDREAM consortium steering committee, which comprises the principal investigators of the different sites participating in this study. Applications for non-commercial use only will be considered and should be sent to the corresponding author (M.P.). Applications should outline how the specific use of the data would catalyze considerable advancement in the treatment and management of type 1 diabetes or improve care for those living with type 1 diabetes, including the specific purpose of developing therapies and technology that can be used by patients to help manage their disease and improve their health outcomes. Any data that can be shared will need approval from the NextDREAM consortium steering committee and a Material Transfer Agreement in place. All data shared will be de-identified.
The code cannot be made available due to proprietary reasons.
Miller, K. M. et al. Current state of type 1 diabetes treatment in the US: updated data from the T1D exchange clinic registry. Diabetes Care 38, 971–978 (2015).
The Diabetes Control and Complications Trial Research Group. The effect of intensive treatment of diabetes on the development and progression of long-term complications in insulin-dependent diabetes mellitus. N. Engl. J. Med. 329, 977–986 (1993).
Nathan, D. M. et al. Intensive diabetes treatment and cardiovascular disease in patients with type 1 diabetes. N. Engl. J. Med. 353, 2643–2653 (2005).
American Diabetes Association. Standards of medical care in diabetes—2019. Diabetes Care 42, S1–S193 (2019).
Foster N. C. et al. Marked increases in CGM use has not prevented increases in HbA1c levels in participants in the T1D exchange (T1DX) clinic network. Diabetes 67 (2018); https://doi.org/10.2337/db18-1689-P
Edelman, S. V., Argento, N. B., Pettus, J. & Hirsch, I. B. Clinical implications of real-time and intermittently scanned continuous glucose monitoring. Diabetes Care 41, 2265–2274 (2018).
Zisser, H. et al. Clinical performance of three bolus calculators in subjects with type 1 diabetes mellitus: a head-to-head-to-head comparison. Diabetes Technol. Ther. 12, 955–961 (2010).
Petitti, D. B. et al. Glycemic control in youth with diabetes: the SEARCH for Diabetes in Youth Study. J. Pediatr. 155, 668–672 (2009).
The DCCT Research Group. Diabetes Control and Complications Trial (DCCT): results of feasibility study. Diabetes Care 10, 1–19 (1987).
Davidson, M. B. How our current medical care system fails people with diabetes: lack of timely, appropriate clinical decisions. Diabetes Care 32, 370–372 (2009).
Bergenstal, R. M. et al. Automated insulin dosing guidance to optimise insulin management in patients with type 2 diabetes: a multicentre, randomised controlled trial. Lancet 393, 1138–1148 (2019).
Shalitin, S. et al. Using the Internet-based upload blood glucose monitoring and therapy management system in patients with type 1 diabetes. Acta Diabetol. 51, 247–256 (2014).
Pihoker, C. et al. ISPAD clinical practice consensus guidelines 2018: the delivery of ambulatory diabetes care to children and adolescents with diabetes. Pediatr. Diabetes 19, 84–104 (2018).
Chiang, J. L. et al. Type 1 diabetes in children and adolescents: a position statement by the American Diabetes Association. Diabetes Care 41, 2026–2044 (2018).
Markowitz, J. T., Volkening, L. K. & Laffel, L. M. B. Care utilization in a pediatric diabetes clinic: cancellations, parental attendance and mental health appointments. J. Pediatr. 164, 1384–1389 (2014).
Comellas, M. J. et al. Evaluation of a new digital automated glycemic pattern detection tool. Diabetes Technol. Ther. 19, 633–640 (2017).
Lu, H. et al. Population-based geographic access to endocrinologists in the United States, 2012. BMC Health Serv. Res. 15, 541 (2015).
Nimri, R. et al. Adjusting insulin doses in patients with type 1 diabetes who use insulin pump and continuous glucose monitoring: variations among countries and physicians. Diabetes Obes. Metab. 20, 2458–2466 (2018).
Beck, R. W. Downloading diabetes device data: empowering patients to download at home to achieve better outcomes. Diabetes Technol. Ther. 17, 536–537 (2015).
IDF Diabetes Atlas, 8th edn (International Diabetes Federation, accessed 3 December 2018); https://www.idf.org/e-library/epidemiology-research/diabetes-atlas/134-idf-diabetes-atlas-8th-edition.html
Patterson, C. C. et al. Trends and cyclical variation in the incidence of childhood type 1 diabetes in 26 European centres in the 25 year period 1989–2013: a multicentre prospective registration study. Diabetologia 62, 408–417 (2019).
Vigersky, R. A. et al. The clinical endocrinology workforce: current status and future projections of supply and demand. J. Clin. Endocrinol. Metab. 99, 3112–3121 (2014).
Hahn, S. Understanding noninferiority trials. Korean J. Pediatr. 55, 403–407 (2012).
Tamborlane, W. V. et al. Continuous glucose monitoring and intensive treatment of type 1 diabetes. N. Engl. J. Med. 359, 1464–1476 (2008).
Aleppo, G. et al. REPLACE-BG: a randomized trial comparing continuous glucose monitoring with and without routine blood glucose monitoring in adults with well-controlled type 1 diabetes. Diabetes Care 40, 538–545 (2017).
Abramson, J. H. WINPEPI updated: computer programs for epidemiologists, and their teaching potential. Epidemiol. Perspect. Innov. 8, 1 (2011).
We thank the following colleagues for assistance in the study: David Steinberg and Sigal Fleishman of Tel Aviv University for the statistical analysis; the Data and Safety Monitoring Board (DSMB) members: David Maahs, Larry Fox, Jan Bolinder, and the study statistician David Steinberg; Clinlogix, Contract Research Organization: Ksenia Keller, Oliver Hautz, Hiroko Griffin, Jean-Luc Baeriswyl, Christian Dierker, Daniel Rider, Andrew Milczarek, Jennifer M. Kratz, Vanessa Okonkwo; DreaMed Diabetes Ltd: Eran Atlas, Ido Muller, Eran Agmon, Daniel Koch, Yaron Matiash, Tomer Segall, Anat Cohen, Moshe Lev, Igal Shtudiner; Glooko, Inc: Dave Conn, Jeff Chang, Holly McGarraugh, Makenzie Wells, Shannon Gillum-Collins, A.-J. Boyer; Prof. Katharine Barnard, PhD CPsychol AFBPsS, Bournemouth University, Bournemouth, UK BHR Ltd, Portsmouth, UK for developing the ‘Healthcare Professionals Post-Intervention Survey’. We also thank Chris Parkin for his assistance in editing this manuscript. We thank the patients and their families for their participation. Funding for the study was provided by The Leona M. and Harry B. Helmsley Charitable Trust through DreaMed Diabetes Ltd grant no. 2016PG-T1D050. The trial was conducted in collaboration with DreaMed Diabetes Ltd, which provided the DreaMed Advisor Pro, offered technical support to the sites and generated the glucose and insulin data from Glooko system uploads for statistical analysis by an external statistician. Glooko provided the Glooko Platform for the sites, conducted training and offered technical support. Dexcom provided the CGM devices (Dexcom G5) free of charge. DexCom had no role in the trial conduct. Insulet supplied six discounted Omnipod pumps. The DreaMed Advisor Pro development was funded by DreaMed Diabetes Ltd in addition to support from a grant by The Leona M. and Harry B. Helmsley Charitable Trust.
R.N. reports receiving grants from Helmsley Charitable Trust, Dexcom and Insulet, personal fees and others from DreaMed Diabetes Ltd, personal fees from Novo Nordisk and Eli Lilly and grants from Medtronic. In addition, R.N. owns DreaMed Diabetes Ltd stock and has a patent licensed by DreaMed Diabetes Ltd. T.B. reports receiving grants and personal fees from Novo Nordisk, Medtronic and Abbott, personal fees from Eli Lilly, Sanofi, Indigo, Roche, AstraZeneca and Dexcom and grants from Glusense and Zealandpharma. In addition, T.B. owns DreaMed Diabetes Ltd stocks. L.M.L. reports receiving grants from Helmsley Charitable Trust and personal fees from Eli Lilly, Novo Nordisk, Sanofi, Convatec, Roche, Insulogic, Dexcom, Insulet, AstraZeneca, Boehringer Ingelheim and Janssen. S.A.W. reports receiving grants from Helmsley Charitable Trust, grants and personal fees from Medtronic and personal fees from Insulet, Tandem, Eli Lilly, Sanofi and Zealand. T.D. reports receiving grants and personal fees from AstraZeneca, DexCom, Boehringer, Novo Nordisk, Medtronic, Sanofi, Eli Lilly and Insulet. In addition, T.D. owns DreaMed Diabetes Ltd stocks. M.P. reports receiving grants from Helmsley Charitable Trust, Dexcom and Insulet, personal fees and others from DreaMed Diabetes Ltd, grants and personal fees from Medtronic and Novo Nordisk, grants from Roche, Eli Lilly and Sanof, grants from Lexicon and OPKO and personal fees from RSP Systems and Qulab Medical. In addition, M.P. owns DreaMed Diabetes Ltd stock and has a patent licensed by DreaMed Diabetes Ltd. R.H.S., D.S. and K.D. declare no competing interests.
Peer review information Jennifer Sargent was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
* Plus-minus values are means ±SD calculated over the entire 24-weeks study period. To convert the values for glucose to millimoles per liter, multiply by 0.05551. This category of SD is an average of the variability of sensor glucose measurements for each patient, rather than the variation in the mean glucose values among patients in the trial. ¥ Insulin information was recorded only for 52 patients per arm. § Two-sided ANCOVA adjusting for baseline level and number of valid days. Ϯ Randomization test.
Extended Data Fig. 2 Glycated hemoglobin levels at baseline, randomization, 12-weeks, and 24-weeks for per-protocol (n=30 per arm) cohort.
Values are means (markers) ± standard error (error bars). Black circles mark the AI-DSS arm and empty diamonds mark the physician arm. The results of the two-sided, paired t-test analysis are marked by asterisk symbol to show statistically significant difference between baseline and specific time point within AI-DSS arm and by pound symbol to show statistically significant difference between baseline and specific time point within physician arm, where one symbol denotes P<0.05 and two denotes P<0.01.
* Plus-minus values are means ±SD. To convert the values for glucose to millimoles per liter, multiply by 0.05551. ¥ One-sided test using ANCOVA model adjusted for baseline % of time in range and number of valid days used in the analysis. § Non-inferiority randomization test. ϮTwo-sided paired t-test. ΨTwo-sided ANCOVA adjusting for baseline level and number of valid days.
Extended Data Fig. 4 Healthcare professionals post-intervention survey responses: physicians who used the AI-DSS during the study (n=13).
The scale of the reply was: 5—Strongly Agree, 4—Somewhat Agree, 3—Do not Agree or Disagree, 2—Somewhat Disagree, 1—Strongly Disagree. The Healthcare Professionals Post-Intervention Survey is a 50 items questionnaire. The questionnaire comprises: (a) 28 items for pertaining to the physician’s experience with the Advisor Pro use and recommendations. Each item is score on a 5-point scale range from 5 (strongly agree) to 1 (strongly disagree) (Extended Data Fig. 4) (b) 22 items are questions that asses the physician’s view regarding integration of the Advisor Pro into daily routine practice (14 items are yes/no questions and 8 items are open questions). The questionnaire was developed by Prof Katharine Barnard PhD CP sychol AFBPsS and reviewed by the investigators (first and third authors).
Extended Data Fig. 5 Description of the number and technical issues related to insulin dose recommendations in both study arms - intent to treat cohort.
1 In both groups, there were cases in which insulin dosing recommendation was not shared with the participant via the DMS system but in another method. 2 The reasons for missing scheduled recommendations were: physician did not give a recommendation for the physician arm (supplement to the number of given recommendation) and lack of minimal requirements needed for creating advisor recommendation such as lack of valid days for analysis for the AI-DSS arm (supplement to the number of recommendation by the AI-DSS). 3 In 7 cases, recommendation was given by the physician and not the Advisor. In 5 of these cases the minimal data requirement of the AI-DSS was not meet and in the other 2 cases there was a suspected pump clock shifting that prevented the system from providing a recommendation. 4 Technical issues that caused missing recommendation include: any technical failure related to pump, continuous glucose monitoring, diabetes management system or temporary algorithm faults that were resolved during the study. 5 Insufficient insulin and glucose data according to the system requirements prevent formation of recommendation in both trial arms. 6 Pump clock shifting is a safety layer of the advisor that prevent the formation of a recommendation.
1. participants upload devices data to DMS 2. AI-DSS pulled data from DMS after DMS requested advice and validated data 3. AI-DSS detected the source of the data: in case the data came from participants in the physician arm no advice was created and in case the data came from participants in the AI-DSS arm advice was created 4. AI-DSS report was sent via DMS to the physician 5. Physician could review new pump settings recommendations within AI-DSS report page using the DMS platform 6. Physician shared new pump settings and management tips with participant through smartphone.
a, Once the participant uploaded data to the DMS and algorithm has enough data, a new recommendation will be sent and physician could enter into the patient page and review it. Current basal rates are displayed on the left, for the physician arm—a duplicate of the current settings was displayed on the right, clinicians could click EDIT and make changes if necessary and add their own comments for behavioral tips. For the AI-DSS arm—the right side displayed the AI-DSS recommended changes (if necessary) in bold. Comments may be added from the AI-DSS. b, Current carbohydrate ratio values are displayed on the left, for the physician arm - a duplicate of the current settings was displayed on the right, clinicians could click EDIT and make changes if necessary and add their own comments for behavioral tips. For the AI-DSS arm—the right side displayed the AI-DSS recommended changes (if necessary), in bold. Comments may be added from the AI-DSS. c, Current correction factor (Insulin Sensitivity Factor) are displayed on the right, for the physician arm - a duplicate of the current settings was displayed on the right, clinicians could click EDIT and make changes if necessary and add their own comments for behavioral tips. For the AI-DSS arm—the right side displayed the AI-DSS recommended changes (if necessary), in bold. Comments may be added from the AI-DSS. d, The Ambulatory Glucose Profile, Logbook and Daily reports are displayed under each of the previous settings screens for guidance. e, Once all settings screens are completed, a summary of the recommendation will appear. This is what the patients sees on their mobile app once the physician shares the recommendations with them.
Extended Data Fig. 8 AI-DSS insulin pump recommendations presented on patient’s smartphone: new basal plan, new carbohydrates ratio, new correction factors and diabetes management tips.
The recommendation report for the patient is presented in the app, three screen shots from the application. The data in the figure related to virtual patient, Mia Foster.
After a 3-week run-in period, participants were randomized to participate either at the IA-DSS or physician arm. Three weeks after randomization and every 3 weeks thereafter, all participants uploaded their insulin pump and continuous glucose monitoring data using the DMS. Participants then received insulin pump settings recommendations either from physician or AI-DSS according to their randomization for a 24-week period.
Procedures conducted at each visit.
About this article
Cite this article
Nimri, R., Battelino, T., Laffel, L.M. et al. Insulin dose optimization using an automated artificial intelligence-based decision support system in youths with type 1 diabetes. Nat Med 26, 1380–1384 (2020). https://doi.org/10.1038/s41591-020-1045-7
Clinical integration of machine learning for curative-intent radiation treatment of patients with prostate cancer
Nature Medicine (2021)
Acta Diabetologica (2021)
Diabetes Therapy (2021)