Duration and accuracy of automated stroke CT workflow with AI-supported intracranial large vessel occlusion detection

The Automation Platform (AP) is a software platform to support the workflow of radiologists and includes a stroke CT package with integrated artificial intelligence (AI) based tools. The aim of this study was to evaluate the diagnostic performance of the AP for the detection of intracranial large vessel occlusions (LVO) on conventional CT angiography (CTA), and the duration of CT processing in a cohort of acute stroke patients. The diagnostic performance for intracranial LVO detection on CTA by the AP was evaluated in a retrospective cohort of 100 acute stroke patients and compared to the diagnostic performance of five radiologists with different levels of experience. The reference standard was set by an independent neuroradiologist, with access to the readings of the different radiologists, clinical data, and follow-up. The data processing time of the AP for ICH detection on non-contrast CT, LVO detection on CTA, and the processing of CTP maps was assessed in a subset 60 patients of the retrospective cohort. This was compared to 13 radiologists, who were prospectively timed for the processing and reading of 21 stroke CTs. The AP showed shorter processing time of CTA (mean 60 versus 395 s) and CTP (mean 196 versus 243–349 s) as compared to radiologists, but showed lower sensitivity for LVO detection (sensitivity 77% of the AP vs mean sensitivity 87% of radiologists). If the AP would have been used as a stand-alone system, 1 ICA occlusion, 2 M1 occlusions and 8 M2 occlusions would have been missed, which would be eligible for mechanical thrombectomy. In conclusion, the AP showed shorter processing time of CTA and CTP as compared with radiologists, which illustrates the potential of the AP to speed-up the diagnostic work-up. However, its performance for LVO detection was lower as compared with radiologists, especially for M2 vessel occlusions.

We measured the reading time of NCCT and CTA and the duration of CTP processing by 13 radiologists without the use of the AP in a separate, prospective cohort of 21 patients who presented with stroke-like symptoms.
All patients of the two study cohorts received the same stroke CT scanning protocol, including non-contrast brain CT, head-neck CTA and whole-brain CTP on a 320-row detector CT scanner (Aquilion One, Vision or Genesis edition, Canon Medical Systems).
The study was approved by the medical-ethical committee, and informed consent was waived because of the retrospective collection of the study data (IRB Oost-Nederland, Radboudumc, Nijmegen, The Netherlands). The study was performed in accordance with relevant guidelines and regulations. Clinical data and results of imaging analyses were anonymized and stored in a protected database Castor EDC (http:// www. casto redc. com), following Good Clinical Practice (GCP) and European Data Protection Directive guidelines.
The Automation Platform Stroke CT package. The Stroke CT package implemented in the Automation Platform (base version 1.3, Canon Medical Systems, Otawara, Japan) was evaluated. The embedded AI algorithm to detect LVOs is an integrated third party application called CINA LVO (version 1.0.3, Avicenna.ai) and its intended use is to detect distal internal carotid artery (ICA), M1 occlusions and proximal M2 occlusions. NCCTs were analyzed with CINA intracerebral hemorrhage (ICH) application (version 1.0.4, Avicenna.ai) also part of the Stroke CT package. The conventional and AP supported workflows are illustrated in the supplementary figure.
Assessment of large vessel occlusion detection. An LVO is defined as an occlusion of the distal internal carotid artery (ICA) and M1 or M2 segments of the middle cerebral artery (MCA) eligible for mechanical thrombectomy 4 . The CTAs were retrospectively analyzed on a PACS system (Impax 6.6.1.0 2015, Agfa Healthcare, Morsel, Belgium) for the presence or absence of a LVO by two neuroradiologists, one radiologist and two radiology residents. The readers were only provided the sidedness of the neurological deficit without further details and they did not have access to non-contrast CT or perfusion CT. The AP evaluated the same set of CTAs.
The reference standard was set by an independent neuroradiologist, with access to the results of the different radiologists, clinical data, and follow-up. Measures of diagnostic performance were calculated. A p-value below 0.05 was considered statistically significant. for the AP was timed in 60 cases of the retrospective cohort. The duration of the processing and reading of 13 different radiologists (4 neuroradiologists, 4 non-neuroradiologists and 5 radiology residents) in clinical practice was timed in 21 prospective stroke CTs. Out of the 21 scans, 8 were read by a neuroradiologist, 4 by a nonneuroradiologist and 9 by a radiology resident. An independent t-test was used to compare the four groups (AP, neuroradiologists, non-neuroradiologists and radiology residents) for the duration of processing the CTP. A p-value < 0.05 was considered statistically significant. Furthermore, the minimum and maximum duration of the different measurements were provided.

Results
Diagnostic performance for LVO detection. The

Duration of CT readings by AP and radiologists.
The duration of processing for the AP and three groups of radiologists are presented in Fig. 1. The evaluation time for intracranial hemorrhage detection on NCCT was comparable between the AP (mean: 65 s) and radiologists (mean: 69 s). The AP needed on average 60 s for the assessment of CTA reading for LVO detection, which was shorter compared to the groups of radiologists (mean: 395 s). Mean duration of CTP processing by the AP was 196 s, which was significantly shorter than non-neuroradiologists (mean: 349 s, p = 0.01) and radiology residents (mean 335 s, p < 0.001).

Discussion
Our study found that while the AP had a lower performance than radiologists in detecting LVOs on CTA, it was able to process CTP and detect LVOs in a shorter time than radiologists, highlighting the potential of automated workflow support in speeding up the diagnostic work, particularly for less experienced radiologists. If the AP would have been used as a stand-alone system, 1 ICA occlusion, 2 M1 occlusions and 8 M2 occlusions would have been missed, which would be eligible for mechanical thrombectomy. The low detection rate for M2 vessel occlusions is in line with previous studies showing low sensitivity for M2 vessel occlusion detection by the same and other AI-algorithms [4][5][6][7][8] . Most AI algorithms currently available are only intended for M1 and ICA occlusion detection, so it is important to ensure that end-users are aware of the intended purpose of the AI algorithm. In clinical practice, an AI-tool could especially add value by detecting smaller but treatable vessel occlusions, such as M2 vessel occlusions, which may be overlooked by radiologists. For now, the potential to improve healthcare outcomes and to save costs by using this AI-tool to reduce the number of missed LVOs, as discussed in an early health technology assessment study 2 , would probably not yet be realized. It should be noted that the availability of more detailed clinical information, non-contrast CT and the use of CTP maps helps to improve the detection of intracranial vessel occlusions 9,10 . Although the duration of NCCT evaluation for ICH detection was similar between AP and radiologists, the actual performance of ICH detection was not evaluated in our study as patients with intracranial hemorrhage were excluded from our cohort. A clear advantage of the AP was the automated processing of CTP maps, which saves time for radiologists, especially for those who were less experienced.
Our study has some limitations. The study group is rather small and stroke mimickers were excluded from the dataset, which probably resulted in a higher than expected incidence of LVOs as compared to daily clinical practice. It is however not expected that this would have a significant influence on the results of our study, because LVOs are not expected to be present in stroke mimickers. A more diverse and larger population size would be required to comprehensively evaluate the AP's performance. Additionally, the AP was evaluated as a stand-alone system, and its actual (added) value for radiologists in LVO detection was not assessed. It remains to be determined whether the diagnostic performance and confidence of less experienced readers actually improves by the use of an AI-tool. Incorrect results of the AI-based tool could impair the performance of radiologists with varying levels of expertise, which is called automation bias 11 . Furthermore, the duration of the AP for LVO and ICH assessment and CTP processing was retrospectively compared with the prospective timing of radiologists, as the AP was not yet implemented in clinical practice, which limited a direct comparison between the AP and radiologists. It should also be emphasized that the AP cannot be used as a stand-alone system, because of the moderate diagnostic performance but also because of its narrow target-use, and that radiologists should verify the AP's results, which takes time and results in a smaller time difference between the AP and radiologists. Finally, the CTP datasets' slice thickness was 0.5 mm, and data was sent to the AP in a non-enhanced DICOM format. Changing the slice thickness to 1.0 mm and transferring data in an enhanced DICOM format could decrease the duration of data transfer and processing in clinical practice.

Conclusion
The AP showed shorter processing time of CTA and CTP as compared with radiologists, which illustrates the potential of the AP to speed-up the diagnostic work-up. However, its performance for LVO detection was lower as compared with radiologists, especially for M2 vessel occlusions.

Data availability
All data generated and analyzed during this study are included in this published article and its supplementary information files.
Received: 12 May 2023; Accepted: 31 July 2023 Figure 1. Timelines of the duration of data processing for the Automation Platform and radiologists with different levels of experience (4 neuroradiologists, 4 non-neuroradiologists and 5 radiology residents). Each timeframe represents the mean duration of processing/reading, with the minimum and maximum times (range) provided in seconds. The durations were times manually. ICH: intracerebral hemorrhage, LVO: large vessel occlusion, ASPECTS: Alberta Stroke Program Early CT Score, CTP: CT perfusion.