Introduction

Diabetic eye disease (DED) is a complication of diabetes that is the primary cause of blindness in working-age adults in the U.S.1,2. Early detection (‘screening’) and treatment can frequently prevent progression, but the majority of the 34 million people with diabetes in the US have a DED screening care gap, due to a lack of access, and education around the need for a diabetic eye exam3,4. This care gap is a major source of health disparity, with racial and ethnic minorities, and under-resourced communities, having worse outcomes, and a disproportionately higher prevalence of DED2,5,6,7,8,9.

While DED prevalence is lower in youth (defined as those aged <21 years) with diabetes, it affects approximately 4–9% of youth with type 1 diabetes (T1D) and 4-15% of youth with type 2 diabetes (T2D)10,11,12,13,14. The risk for DED increases with the duration of diabetes in T1D, and recent data from the TODAY2 follow-up study demonstrated a diabetic retinopathy prevalence rate of 49% at a mean diabetes duration of 12 years in youth onset T2D15. The American Diabetes Association (ADA) and American Academy of Ophthalmology (AAO) recommend DED screening in T1D within 3–5 years of diagnosis and age greater than 11 years, and in T2D at the time of diagnosis in youth16. However, only 35-72% of diabetic youth undergo recommended screening exams, with even higher care gap rates in minority and lower socioeconomic background youth10,17. Commonly reported barriers to screening include miscommunication regarding the need for a diabetic eye exam, time for an additional doctor’s visit, and transportation barriers17,18.

While the introduction of telemedicine over the last two decades has improved screening and facilitated early detection of diabetic eye disease19,20,21,22,23, the development of diagnostic autonomous artificial intelligence (AI) systems for diagnosing DED has ushered in the next chapter of DED screening24,25,26,27,28,29. Autonomous AI systems require a camera operator to obtain point-of-care fundus images, which are then interpreted by an AI algorithm to provide a diagnosis without human oversight. The regulatory approval of the first diagnostic autonomous AI system for diabetic eye exams was based on a pivotal trial against a prognostic standard, i.e., patient outcome24, showing its safety, efficacy, and lack of racial and ethnic bias for diagnosing DED in adults with diabetes with 87% sensitivity and 91% specificity24. In prior studies of youth with diabetes, we demonstrated that the diagnosability of autonomous AI in youth was 97.5%, with 85.7% sensitivity and 79.3% specificity in detecting DED, with no difference in diagnosability across demographic groups30. We have also shown that this system can be implemented in a multidisciplinary diabetes clinic and has the potential to increase DED screening rates in underserved youth, while also being cost-savings to patients and caregivers30,31.

While diagnostic accuracy has been a focus of study of diagnostic AI systems25,30,32, the effectiveness of autonomous AI to increase adherence and follow-up compared to traditional referral has not been evaluated in a rigorously designed randomized trial. We hypothesized that autonomous AI closes the diabetic eye exam care gap, and increases follow-up, compared to traditional eye care provider(ECP) referral in youth. To test this hypothesis, we designed ACCESS (AI for Childrens’ diabetiC Eye examS Study), a pre-registered, rigorously designed randomized control trial (RCT), to measure diabetic eye exam completion rates in a racially and ethnically diverse cohort of youth with T1D and T2D.

Results

Participant flow

One-hundred seventy candidates were determined to be eligible, and 164 participants completed informed consent and were randomized, 81 to the intervention and 83 to the control arm. The final allocation was not equal between the 2 groups due to recruitment completion in the middle of the permutated blocks. Six patients who were approached declined participation as described in the flowchart in Fig. 1.

Fig. 1
figure 1

Flow diagram of patient enrollment and randomization in the ACCESS study (Based on CONSORT guideline 2010 flow diagram).

Full size image

Baseline patient characteristics

Baseline characteristics were similar in the two groups (Table 1): mean age 15.2 years (SD 2.8), 58% female, 35% Black, 6% Hispanic, 47% Medicaid insurance, 34% with household income <$50,000, 38% with parental education of up to high school completion, 73% had T1D, the median duration of diabetes was 5.8 years, and mean HbA1c was 8.6%.

Table 1 ACCESS patient characteristics by randomization group
Full size table

Regarding disparities in ever having a prior diabetic eye exam, 79% of participants reported having had such an exam at some time in the past (but per inclusion criteria, not in the prior 6 months) and 35/164 (21%) did not (i.e., had a care gap). Those without a prior eye exam were more likely to be Black (p = <0.001), Hispanic (p = 0.02), have household income of less than $50,000 (p = 0.005), Medicaid insurance (p < 0.001), parental education of up to high school completion (p = 0.02), and a shorter duration of diabetes (p < 0.001) in univariate analysis. (Table 2) When controlling for these factors in multivariate analysis, only the duration of diabetes remained significantly associated with prior eye exam completion (OR 1.36; 95% CI: 1.0–1.8, p = 0.04), suggesting that racial, ethnic, and SES disparities were driven by differences in duration of diabetes in the respective populations. There were differences in diabetes type, race, parental education, and diabetes management characteristics between the two recruitment sites, but glycemic control (measured by hemoglobin A1c) was similar across sites. ([Supplementary Note 8 and Table S1]).

Table 2 Univariate analysis of participant characteristics based on prior diabetic eye exam completion
Full size table

Primary outcome

As shown in Table 3, in the intervention arm, 81/81 (100%) participants completed their diabetic eye exams, so the primary care gap closure rate was 100% (95%CI: 96%, 100%). All images were diagnostic for the AI system (output was either “DED present” or “DED absent”). In the control arm, 18/82 completed the diabetic eye exam within 6 months, so the primary care gap closure rate was 22% (95%CI: 14%, 32%).

Table 3 ACCESS primary outcome—completion of the diabetic eye exam (DED) (n = 163a)
Full size table

The difference of 78% (95% CI: 69%, 87%) in gap closure between control and intervention groups was statistically significant (p < 0.001). There were no statistically significant differences in care-gap closure rates by race, ethnicity, SES, or education. There were 3 participants in the control arm who did not return for clinic visits, were not able to be reached by phone or EHR-based messaging, and thus their eye exams were considered incomplete. Sensitivity analysis was performed excluding these 3 participants and also assuming they completed their eye exams, and the results were unchanged. [Supplementary Note 8; Tables S2 and S3].

Secondary outcomes

In the intervention arm, 25 participants received a “DED present” output, and received the referral intervention; of these, 16/25 completed an ECP visit within 6 months, so the follow-through completion rate was 64% (95%CI: 43%, 81%). In the control arm, 18 participants visited the ECP, for a follow-through completion rate of 22% (95%CI: 14%, 32%) and none had DED. The difference of 42% (95%CI: 21%, 63%) in follow-through completion rates between control and intervention groups was significant (p < 0.001). Further analysis comparing the 16 participants who completed follow-up after “DED present” to those that did not, showed that those who did not complete follow-up were more likely to be NH Black and have Medicaid insurance, but this was a small subgroup and not statistically significant.

A retina specialist review of the AI images (a level 4 reference standard) demonstrated an estimated sensitivity of 100% and specificity of 78.9%33.

Participants reported a high level of satisfaction with autonomous AI, 92.5% were satisfied with the length of time it took to complete the exam, and 96% were satisfied with the experience. Of those who were in the intervention arm, 85% reported they would choose to do the AI-based eye exam in the future, and only 57%, an ECP-based diabetic eye exam (Table 4).

Table 4 Survey results
Full size table

There were no adverse events during the study visits or fundus photography.

Discussion

The results of the ACCESS trial confirmed our hypothesis that autonomous AI increases diabetic eye exam completion rates, and closes this care gap in a racially and ethnically diverse population of youth with diabetes, compared to standard of care. This result held true despite augmenting the standard of care referral in the control arm with deliberate education for the patient and caregiver regarding the importance of diabetic eye exams. Additionally, the results indicate that autonomous AI improves the likelihood of receiving follow-through eye care for those patients identified as having DED at the point-of-care compared to control. Care gap closure rates for the diabetic eye exam are an important component of value-based care through MIPS and HEDIS quality metrics, and the results show that autonomous AI can contribute to meeting these historically hard-to-achieve metrics, especially in racial/ethnic minority and under-resourced youth34,35. To our knowledge and based on a PubMed search [Supplementary Note 4], the present study is the first RCT to evaluate the role of autonomous AI in closing a guideline-based care gap.

Autonomous AI allows real-time, point-of-care diagnosis, whether in the primary care, endocrine, or other outpatient setting, can be integrated into the diabetes care workflow, and was included in the ADA standards of care for DED screening in adults as of 202036. While teleretinal screening programs have improved care-gap closure rates over the last two decades19,20,22,23,37, care-gap closure in adults nationwide remains at only 15.3% in 20187, despite all these efforts. In designing this randomized control trial, we considered the alternative comparator of teleretinal screening but chose the ECP exam as it is considered the standard of care, and both ECP and the AI system have been validated against patient outcome (as required by FDA)33 and prognostic standards (Early Treatment of Diabetic Retinopathy Study (ETDRS and DRCR)), while teleretinal networks have not been validated against prognostic standards. Additionally, studies have shown large variability between telemedicine reading networks38,39,40, and the sustainability and scalability of telemedicine have been limited. The advantages of autonomous AI are its point-of-care procedure, immediate results, and its scalability as it does not require additional clinical experts, ophthalmic oversight, or highly skilled operators. However, future studies should compare autonomous AI to teleretinal screening programs.

Compared to adults, the prevalence of DED is low in children. Although screening for DED is recommended, the reported DED care-gap closure in pediatrics ranges from 35-70%10, and at the start of ACCESS, 79% of participants reported an eye exam at some time in the past. However, our previous studies demonstrated substantial racial/ethnic disparity, with non-white youth significantly less likely to have had a prior diabetic eye exam yet more likely to have DED17. The baseline data in this study confirmed these racial, ethnic, and socioeconomic gap closure disparities, similar to other studies demonstrating the wide range of disparities associated with social determinants of health in diabetic retinopathy screening9,41. However, the 100% care-gap closure rate by autonomous AI in this racially and ethnically diverse pediatric population could have the potential to reduce health disparities for vision loss from diabetes, furthering health equity, and deserves further study42,43.

The high satisfaction and acceptance rates for autonomous AI in ACCESS, suggest that this racially and socioeconomically diverse patient population is comfortable with a “computer” or autonomous AI diagnosing their disease. Importantly, the use of AI did not introduce health disparities into care-gap closure.

While this study is an important step towards increased DED screening in youth, to prevent vision loss, patients identified with DED need follow-through care to manage and treat the disease44. Follow-through completion rates remain as low as 5-30% after teleophthalmology screening with referable DED21,45,46, and educational interventions have been shown to help in some cases47. In this study, referral augmented with targeted education was insufficient, as shown by the 22% follow-through at 6 months in the control arm. However, in the intervention arm, follow-through was significantly higher at 64%, as participants with “DED present” results were immediately informed of the diagnosis and provided education. Other studies have also demonstrated that point-of-care diagnosis and immediate results may affect patient behavior in seeking follow-through care48,49. An RCT of assistive AI that provided point-of-care results demonstrated a 30% increase in follow-through compared to telemedicine screening with deferred results in 3–5 days49. Another study reported improvement in follow-up rates from 35% to 72% with point-of-care diagnosis of referable DED48.

Strengths of the present ACCESS study are its rigorous hypothesis-testing, pre-registered RCT design, as well as the scripted clinical education given to participants above the standard of care in the control arm, to maximize the likelihood of care gap closure in the control arm. The study sample was an adequate representation of youth with diabetes in the U.S. with respect to racial, ethnic, SES, education, and sex distribution, increasing the external validity of our results. Additionally, while this particular autonomous AI has been shown not to have racial bias, it is important to consider pigmentation of the retina that could introduce bias in an AI system for the diabetic eye exam43.

Limitations of the present study are that the autonomous AI used is not FDA-cleared for use in ages 21 and under, though we showed in a previous study that the risk of false negatives is low30. Some of the participants in this study were familiar with autonomous AI diabetic eye exams from our prior study and may have been more accepting of participation in the current study30. Additionally, although pharmacologic dilation is not necessary for the pediatric population to obtain sufficient fundus imaging, this may not be applicable to adults where real-world studies have demonstrated higher rates of insufficient images without pharmacologic dilation21,50.

In summary, the ACCESS trial shows that autonomous AI diabetic eye exams close more care gaps in youth with diabetes than the standard of care, and the availability of point-of-care AI diabetic eye exams may mitigate known screening disparities in racial/ethnic minority and under-resourced youth. Furthermore, sharing a diagnosis of referable disease at the point-of-care was associated with a higher rate of follow-through with eye care providers for management and treatment, potentially improving visual outcomes in this vulnerable population and advancing health equity in youth with diabetes.

Methods

Trial design

ACCESS is a hypothesis-driven, pre-registered, prospective parallel, RCT with a 1:1 allocation ratio that was conducted at the Johns Hopkins Pediatric Diabetes Center at two sites (Johns Hopkins Hospital and Mount Washington Pediatric Hospital) in Baltimore, Maryland, which serves a racially and ethnically diverse population. Participants were enrolled from November 24, 2021, through June 6, 2022, and follow-up was completed by Dec 6, 2022. The CONSORT requirements for RCT were followed51. ACCESS was pre-registered on ClinicalTrials.gov (NCT05131451). The study was approved by the Johns Hopkins IRB, the tenets of the Declaration of Helsinki were followed, and an independent Data Safety and Monitoring Board was established.

Participants

Youth with T1D (11-21 years) or T2D (8-21 years) were eligible for inclusion if they met the criteria for DED screening per American Diabetes Association (ADA) 2021 guidelines16, had no known DED, and had not had a diabetic eye exam within the last 6 months. Patients with maturity-onset diabetes of the young, cystic-fibrosis-related diabetes, known DED, or other pre-existing eye conditions (retinal disease, cataracts) were excluded from this study.

Interventions

Potential study participants were approached in the diabetes clinic to confirm eligibility and then recruited by a study coordinator with written informed consent. Consented participants were randomized to either the control arm or the intervention arm. At the time of enrollment, the study coordinator collected baseline data, as well as 3 phone numbers from the participant in order to facilitate follow-up.

Control arm: standard of care augmented with an educational intervention

In the control arm, participants were referred to an eye care provider (ECP: optometrist or ophthalmologist) through a deliberate educational process by the study coordinator, in the form of a scripted educational intervention, including a paper handout guide on how to get a diabetic eye exam [Supplementary Note 1]. The goal of this intervention was to minimize the effect of the most commonly reported barrier, i.e., communication and confusion around the necessity of the diabetic eye exam17. Diabetic eye exam completion was achieved with ECP eye exam documentation. If the participant could not be reached to determine completion (despite calls/voicemails to all 3 phone numbers and EHR-based secure messaging) or if the exam had not been completed by 6 months, it was considered not completed.

Intervention arm: autonomous AI

In the intervention arm participants underwent the 5–10 min autonomous AI system diabetic eye exam without pharmacologic dilation24. The autonomous AI system (IDx-DR, Digital Diagnostics, Coralville, Iowa, USA) for diagnosing diabetic eye disease (DED) was US FDA De Novo authorized (“FDA approval”) in 2018 for adults with diabetes15. The system diagnoses specific levels of diabetic retinopathy and diabetic macular edema (Early Treatment of Diabetic Retinopathy Study level 35 and higher, clinically significant macular edema, and or center-involved macular edema)32,33, referred to as “referable DED”34, that requires further management or treatment by an ophthalmologist or retina specialist. If the ETDRS level is 20 or lower and no macular edema is present, appropriate management is to retest in 12 months35. With this autonomous AI system, a medical diagnosis is made independently by the system without human oversight.

In this study, the participant’s eyes were not pharmacologically dilated, as pilot studies found that pharmacologic dilation is unnecessary in youth30. The autonomous AI system guided the operator to acquire two color fundus images determined to be of adequate quality using an image quality algorithm36, one each centered on the fovea and the optic nerve and guided the operator to retake any images of insufficient quality. This process requires approximately 10 min, after which the autonomous AI system reports one of the following within 60 s: “DED present, refer to a specialist”, “DED not present, test again in 12 months”, or “insufficient image quality”. The latter response occurs when the operator is unable to obtain images of adequate quality after 3 attempts.

If the autonomous AI output was “DED absent,” participants were informed the diabetic eye exam was normal, while if “DED present,” (ETDRS level 35 or higher, and/or clinically significant, and or/center-involved macular edema) they received a deliberate educational process by the study coordinator in the form of a scripted educational intervention for follow-up eye care [Supplementary Note 2]. In either case, the diabetic eye exam was considered complete. If the AI output was “insufficient quality” the participant was referred for eye care.

The IDx-DR autonomous AI is not labeled for youth <22 years, as currently no autonomous AI for DED has been cleared for a pediatric population. To ensure safety and that no cases of disease would be missed, all images were also overread by a board-certified retina specialist.

Follow-up procedures

Participants who completed the diabetic eye exam (intervention arm: after autonomous AI exam; control arm: after documented completion of eye exam) were asked to fill out a survey on acceptability and satisfaction with the screening method. [Supplementary Note 3] Participants received a $25 gift card and parking pass for participation in the study.

Randomization and masking

To prevent selection bias and ensure sample size balance between the groups and sites, stratified randomization (by site) was used, and participants were randomized in permutated block schedules of 4 and 6. Within each block, participants were randomized with a 1:1 allocation ratio to the control group and intervention group. This randomization sequence was created by a statistician unaffiliated with the study to ensure masking to the randomization scheme and was implemented by REDCAP’s randomization software based on the participant’s location52,53. After consent, the research coordinator entered the participant location and the randomization allocation was generated. All parties were masked to the allocation until the participant was randomized in the study, and then all parties were unmasked.

Outcomes

In order to test the primary hypothesis, the pre-specified primary outcome was defined as the proportion of participants who completed a documented diabetic eye exam in each arm (“primary care gap closure rate”). In the control arm, this is the proportion of patients who completed a documented diabetic eye exam with an ECP within 6 months of randomization; in the intervention arm, this is the proportion of participants who completed the autonomous AI exam at the study visit.

The pre-specified secondary outcome (“follow-through completion rate”) was defined as the proportion of participants who completed follow-through at the ECP in each arm. In the control arm, this was the proportion who completed the diabetic eye exam at ECP after referral, and in the intervention arm, the proportion who completed follow-up at the ECP after a “DED-present.” This proportion assumes that control-arm patients who arrive at ECP for screening remain at ECP for management or treatment when found to have DED. Both outcomes were stratified by race, ethnicity, SES, and education level, using univariate and multivariate analysis to determine any differential effect on these categories. There were no changes in trial design or outcome after trial commencement.

Data collection

Data were collected from the electronic health record, specifically age, date of birth, sex at birth, race, ethnicity, type of diabetes, date of diabetes diagnosis, medication use (insulin, metformin, GLP1 agonist, etc.), form of insulin administration, use of continuous glucose monitor (CGM) and CGM data, blood pressure, height, weight, body mass index (BMI), presence of other diabetes-related complications (hyperlipidemia, hypertension, microalbuminuria), abnormal thyroid function, past four hemoglobin A1C readings (if available), diabetic eye exam history, medical history, family medical history, health insurance, and zip code. Parental education status and household income were self-reported by participants using a paper/pencil form.

Sample size calculation

We assumed that a 20% difference in DED screening completion rates (care gap closure) would be clinically relevant. Based on our prior study30, where baseline screening rates before AI were 49%, we assumed that with the educational intervention, screening rates for usual care would be closer to 60% in this study, and demonstrating a difference of 20% would be clinically relevant for AI screening. We calculated that a sample size of 164 (n = 82, n = 82 AI) would provide 80% power with a 2-tailed type-1 error of 0.05. Since randomization and study visits occurred at the same time there was little risk of attrition and thus the sample size was not expanded to account for attrition.

Data governance

Although this was a low-risk clinical trial, an independent Data Safety and Monitoring Board was established to protect the interests of study participants and to preserve the integrity and credibility of the study data, based on pre-specified aims, thereby reducing any concerns that interim data could influence or bias the study results and interpretation. At the time of the DSMB meeting on 9/16/2022, all participants had already been enrolled in the trial, and the DSMB determined that there were no safety concerns and the study should continue to completion.

Statistical analysis

The primary outcome of care-gap closure between the randomization groups, and the secondary outcome of follow-through completion rate between the randomization groups were assessed by Pearson’s chi-squared tests. Characteristics of the completed vs non-completed participants in each arm were assessed by Pearson’s chi-squared tests, Wilcoxon rank-sum tests, and two sample t-tests, depending on the nature and distribution of each characteristic. A mixed multilevel multivariable logistic regression analysis, using the site as a nesting level to account for the clustering of observations, was performed in order to examine the relationship between demographic characteristics and the odds of having a previous diabetic eye exam amongst the entire study cohort, adjusting for known covariates associated with DED and site. All analyses were performed using Stata 15.1 (StataCorp, College Station, TX).

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.