A novel deep learning model for a computed tomography diagnosis of coronary plaque erosion

Patients with acute coronary syndromes caused by plaque erosion might be managed conservatively without stenting. Currently, the diagnosis of plaque erosion requires an invasive imaging procedure. We sought to develop a deep learning (DL) model that enables an accurate diagnosis of plaque erosion using coronary computed tomography angiography (CTA). A total of 532 CTA scans from 395 patients were used to develop a DL model: 426 CTA scans from 316 patients for training and internal validation, and 106 separate scans from 79 patients for validation. Momentum Distillation-enhanced Composite Transformer Attention (MD-CTA), a novel DL model that can effectively process the entire set of CTA scans to diagnose plaque erosion, was developed. The novel DL model, compared to the convolution neural network, showed significantly improved AUC (0.899 [0.841–0.957] vs. 0.724 [0.622–0.826]), sensitivity (87.1 [70.2–96.4] vs. 71.0 [52.0–85.8]), and specificity (85.3 [75.3–92.4] vs. 68.0 [56.2–78.3]), respectively, for the patient-level prediction. Similar results were obtained at the slice-level prediction AUC (0.897 [0.890–0.904] vs. 0.757 [0.744–0.770]), sensitivity (82.2 [79.8–84.3] vs. 68.9 [66.2–71.6]), and specificity (80.1 [79.1–81.0] vs. 67.3 [66.3–68.4]), respectively. This newly developed DL model enables an accurate CT diagnosis of plaque erosion, which might enable cardiologists to provide tailored therapy without invasive procedures. Clinical Trial Registration: http://www.clinicaltrials.gov, NCT04523194.


OCT analysis
OCT examination was performed using either a frequency-domain (C7/C8, OCT Intravascular Imaging System, St. Jude Medical, St. Paul, Minnesota) or a time-domain (M2/M3 Cardiology Imaging Systems, LightLab Imaging Inc., Westford, Massachusetts) OCT system.The images were analyzed by three independent investigators who were blinded to patients' data, using an offline review workstation (St.Jude Medical).Qualitative and quantitative analyses were performed using previously established criteria 14 by independent investigators blinded to the clinical, angiographic, and laboratory data.

Cross-correlation with CTA and OCT images
As previously reported, the matching of OCT and CTA images was performed using an offline algorithm (Matcher version 2.1 Leiden, the Netherlands) 15 .In the first step, the OCT images were mapped onto the CT image along the vessel centerline using anatomical landmarks.In the second step, the individual OCT images were translated and rotated to fit best on the CT image, using the vessel center and landmarks for orientation.The algorithm also corrected for deviations in the OCT pullback speed by using interpolation between landmarks.
Among 256 ACS patients, the diagnosis of plaque erosion (n = 113) or rupture (n = 143) on OCT was used as the ground truth and the site on the CTA image that matched the culprit plaque on the OCT image was determined to be the culprit lesion.In addition, 276 CTA scans without plaques detected by OCT and/or angiography and CTA images were chosen as the scans with no plaque.Thus, a total of 532 CTA scans were included in the final analysis (113 CTA scans with plaque erosion, 143 with plaque rupture, and 276 scans with no plaque [Supplementary Fig. S1B]).
For the development and validation of the deep learning model, CTA images in digital imaging and communications in medicine (DICOM) format and their corresponding labels were transferred to the Bio-Imaging, Signal Processing, and Learning laboratory at the Korea Advanced Institute of Science and Technology after anonymization.
Among 395 patients (532 CTA scans), the data were divided into non-overlapping patient subsets, training and cross-validation datasets containing 316 patients (426 CTA scans) for model development and tuning, and the test set containing 79 patients (106 CTA scans) for final performance evaluation (Supplementary Methods and Supplementary Figs.S1C and S2) The disease prevalence of the non-plaque erosion class is 33.3% in the training set and 37.3% in the test set.

Development and evaluation of the deep learning algorithm
As we aimed to develop a DL model that can discriminate between plaque erosion and other entities, we divided labels into two classes: plaque erosion and non-plaque erosion.In the non-plaque erosion class, plaque rupture, as well as the other images without significant lesions were included.
To make an accurate diagnosis, we had to take the entire collection of CTA images into consideration.Thus, we designed a vision transformer (ViT)-based model tailored to the data structure of CTA, dubbed the MD-CTA model.Unlike most contemporary medical AI models that lack the ability to incorporate the information of the entire volume, we utilized the transformer model ( 16) tailored for sequential data structure.Specifically, we simultaneously optimized the spatial transformer that extracts the information within a single slice and the sequence transformer that incorporates the extracted information of all slices to produce the final outcome.We trained the model using both the slice-level and patient-level annotations to enable the model to learn the location of the lesion of interest as well as the label classes.We also implemented the standard convolutional neural network (CNN) based model for comparison with the same design and settings as the proposed DL model (Supplementary Methods, Supplementary Table S1, and Supplementary Figs.S3, S4).We performed the internal five-fold cross-validation to get the best hyperparameter as well as evaluate the model performance.The model is visualized via the attention weights of the spatial and sequence transformers (Supplementary Methods).
A reader study was performed to evaluate the clinical utility of the DL model as an assisting tool as well as to compare the performances with experienced cardiologists.The definition of the experienced cardiologist is provided in Supplementary Methods.In the first round, the performance of the DL model for the test set was compared with that of experienced cardiologists.Then, in the second round, the prediction results by the DL model along with the corresponding CTA scans were provided to the readers to evaluate whether the diagnostic performances were improved with the model's assistance (Supplementary Methods).

Statistical analysis
Categorical data are presented as counts and percentages, and are compared using the chi-squared test or Fisher exact test, as appropriate.Continuous variables have been shown as mean ± SD or median (25th to 75th percentiles), as appropriate, depending on the normality of distribution.Per-lesion data were analyzed using the generalized estimating equations with a logit link for the binary variables to consider the potential clustering of multiple plaques in a single patient.Between-group differences in continuous variables were compared using the Student t-test or Mann-Whitney U test, as appropriate.A P value < 0.05 was considered statistically significant.
The model performance was evaluated with the area under the receiver-operating-characteristic curves (AUC), and the sensitivities, specificities, accuracy, positive predictive values (PPVs), and negative predictive values (NPVs) were calculated for the detailed analysis.To estimate the false alarms by the model, the falsepositive rate (FPR) and false-negative rate (FNR) were calculated.The 95% confidence intervals (Cis) were calculated by DeLong's method for AUC, and "exact" Clopper-Pearson confidence intervals for sensitivity, specificity, Vol:.( 1234567890

Study population
For model development and internal validation, we used a total of 532 CTA scans from 395 patients.Patients were randomly divided into non-overlapping subsets, training and cross-validation datasets for model development and tuning (426 scans from 316 patients), and the test set for final performance evaluation (containing 106 scans from 79 patients) (Supplementary Fig. S1C).
Detailed patient characteristics are summarized in Table 1.Other than a higher prevalence of diabetes mellitus in the training dataset than in the test dataset (109 [34.5%] vs. 18 [22.8%],p = 0.018), no differences were observed in patient characteristics, medications, and laboratory data between the two datasets.The subset of patients with ACS showed the same pattern (Diabetes: 78 [38.6%] vs. 11 [20.4%],p = 0.013) (Supplementary Table S2).When the location of the culprit lesion and underlying pathology were compared between training and test datasets, no significant difference was found between the two groups (Table 2).Among patients with ACS, there were no significant differences in qualitative and quantitative OCT analyses between training and test datasets (Supplementary Table S2).In addition, when minimum lumen area measured by OCT and area stenosis measured by CTA were calculated in patients with plaque erosion or rupture, there were no significant differences between the two groups (Supplementary Table S3).Slice-level prediction performances are provided in Fig. 1C,D and Table 4.In the five-fold cross-validation, the MD-CTA model provided the diagnostic performances with an AUC of 0.891 (0.887-0.895), sensitivity of 82.9 (81.7-84.0),and specificity of 80.0 (79.5-80.5),while the CNN model showed an AUC of 0.729 (0.722-0.737), sensitivity of 66.2 (64.8-67.6), and specificity of 66.8 (66.3-67.4).Likewise, in the test set validation, the MD-CTA model's AUC, sensitivity, specificity, and accuracy were 0.897 (0.890-0.904), 82.2 (79.8-84.3),and 80.1 (79.1-81.0),while those of the CNN model were 0.757 (0.744-0.770), 68.9 (66.2-71.6), and 67.3 (66.3-68.4),respectively.The NPVs for the slice level prediction were over 90.0% in both five-fold cross-validation and the test set validation, while the PPVs were relatively low, attributed to the imbalance between positives and negatives.
The model performances evaluated exclusively on the ACS patients are provided in the Supplementary Results, Supplementary Table S4.In the test set validation, the Cohen's kappa coefficient was 0.679 between the model and the ground truth labels.When the two key components, the composite transformer attention and the modalityspecific self-supervised pre-training, were not used together, the performance of the vanilla ViT model's performances were sub-optimal (Supplementary Results and Supplementary Table S5).The diagnostic performance of the same model for culprit lesion is presented in the Supplementary Result and Supplementary Table S6.

Analysis of the false estimates
Tables 3 and 4 show the results of the analysis of the false estimates.In the five-fold cross-validation, the FPR and FNR of the MD-CTA model were 13.4 (9.8-17.6)and 18.8 (12.0-27.2) for the patient-level diagnosis, and 20.0 (19.5-20.5)and 17.1 (16.0-18.3)for the slice-level diagnosis, which was lower than the CNN model.In the test set validation, the FPR and FNR of the MD-CTA model were 14.7 (7.6-24.7)and 12.9 (3.6-29.8)for the patient-level, and 19.9 (19.0-80.9)and 17.8 (15.7-20.2) for the slice-level diagnoses, providing lower false estimates than the CNN model.More detailed information on false estimates is provided in the Supplementary Results and Supplementary Fig. S5.

Model interpretability results
We visualized the attention of the slice-level and sequence-level transformer, which reflect the model's attention within the slice and between the slices, respectively.The relative importance estimated by the model has been normalized between 0 (low) and 1 (high), and this estimated relative importance is visualized in accordance with a scale bar, as depicted in Fig. 2. As provided in the representative cases in Fig. 2, Supplementary Fig. S6, and Supplementary Videos S1 and S2, the DL model paid attention accurately to the lesion location compared to the ground truth annotation at the patient level.Within a single frame, the suspected culprit lesion was well localized by the model attention, suggesting that the model can identify the clinically important features within the given frame.

Reader study comparing the model performance with the experienced cardiologists
In the first round of the reader study, the performances of the MD-CTA model were compared with the experienced cardiologists as shown in Table 5.The model outperformed the expert readers in all diagnostic performance metrics, and the superiority of the model was most prominent for the sensitivity (87.1% in DL model vs. www.nature.com/scientificreports/16.1% in reader 1, 12.9% in reader 2, and 16.1% in reader 3).The second round of the reader study was performed to evaluate whether the model can be used as an assisting tool to improve the diagnostic performance of the human reader.When given the model's prediction results for the probability and location of the plaque erosion, the diagnostic performances of the human readers markedly improved, especially for the sensitivity, increasing from 16.1% to 83.9% in reader 1, from 12.9% to 77.4% in reader 2, and from 16.1% to 77.4% in reader 3.

Discussion
To the best of our knowledge, this is the first report showing that the automated diagnosis of plaque erosion is possible with a non-invasive coronary CTA using a novel DL algorithm.To achieve this goal, we have developed the MD-CTA model, which is able to leverage composite transformer attentions to incorporate the information and relationships between the coronary CTA slices, emulating the reading process of a human expert, and we enhanced the model's performance with modality-specific self-supervised pre-training.The five-fold crossvalidation and the test set validation results have shown that the DL model can diagnose plaque erosion solely from the CTA images, attaining a clinically useful level of diagnostic performance.Our model outperformed the experienced cardiologists, and when used as an assisting tool, the diagnostic performances of cardiologists were markedly improved.
Recent efforts to apply artificial intelligence to coronary artery imaging such as OCT, CTA, and IVUS have been made.However, most works were devoted to dense prediction and quantification like plaque segmentation 16,17 , or primarily focused on easily discernible abnormalities, for instance, thin-cap fibroatheroma 18 ; only a minority of works have reported the DL application for the end-to-end diagnosis of specific findings.Although we have previously reported novel DL model for the diagnosis of plaque erosion, intravascular imaging with OCT was required 19 , thus, the use of the algorithm was restricted to the catheterization laboratory.
Plaque erosion, which is responsible for up to 50% of patients with non-ST-segment elevation (NSTE)-ACS 2 , is characterized by an intact fibrous cap, preserved vascular structure, and platelet-rich thrombus.Thrombus in plaque erosion is attributed to apoptosis or denudation of superficial endothelial cells as opposed to fibrous cap disruption and creation of a cavity inside a plaque in plaque rupture.Previous studies have demonstrated that ACS patients with plaque erosion have fewer cardiovascular risk factors, less atherosclerotic burden, and lower frequency of complex lesions, less multivessel coronary artery disease, and higher prevalence of close proximity to a bifurcation than those with plaque rupture [20][21][22] .In addition, on OCT images, patients with plaque erosion have smaller reference vessel diameter, lower prevalence of calcification and thrombus in culprit lesions 20 , and lower prevalence of macrophage accumulation, microvessels, and spotty calcium in non-culprit lesions 23 than those with plaque rupture.These findings might suggest that plaque erosion is associated with lower levels of pan-vascular vulnerability and exhibits rather subtle structural changes at the microscopic level 24 .If the aforementioned microscopic structural changes could be identified by using deep learning, plaque erosion can be diagnosed by these specific findings, rather be diagnosed by excluding plaque rupture, as it currently stands.Since patients with NSTE-ACS can usually be stabilized with medical therapy and preliminary data suggest that conservative management might be an option for ACS patients caused by plaque erosion 5,25 , we thought if we could make a diagnosis of plaque erosion by using CTA, this subset of patients might be able to be managed without invasive procedures (Fig. 3).The challenge with CTA is its capability to detect the subtle structural changes that occur in plaque erosion due to its lower resolution.We successfully surmounted this conundrum by leveraging the following approaches.First, we utilized a unique database comprised of paired coronary CTA and OCT images obtained simultaneously from the same subject.This approach enabled the model to learn from superior supervision regarding the presence and location of plaque erosion than CTA alone.As a result, the trained DL model could detect subtle changes in CT attenuation that might not be visible to the human eye.Had we built a model for diagnosing plaque erosion using the dataset lacking paired OCT images, the model's performance would have been restricted to learning only from the lesions detectable by human experts in CTA images.Secondly, we integrated a novel design of composite transformer attention along with a self-supervised learning method to endow the model with a comprehensive understanding of the structural features of the CTA volume.Our approach resulted in a remarkable improvement in diagnostic performance compared to conventional CNN models.Specifically, in slices that were ambiguous and perplexing, the proposed MD-CTA model utilized a composite of intra-slice and inter-slice attention to provide a more precise diagnosis.In recent years, there have been more than 800,000 patients with myocardial infarction in the United States per year 26 and NSTEMI has recently become the most frequent type of MI (NSTEMI increased from 52.8% in 2002 to 68.6% in 2011) 27 .In patients with NSTEMI, plaque erosion is the underlying pathology in up to 75% of cases 2 .Thus, the potential number of patients who might benefit from this new approach is enormous.

Study limitations
Our study has several limitations.First, this was a retrospective analysis of patients who underwent both CTA and OCT prior to PCI.Therefore, selection bias cannot be excluded because patients without significant stenosis on CTA might have been excluded from invasive procedures such as coronary angiography or OCT.Second, the test set validation was performed only for the randomly split subset from the single data source.We adopted this approach as it was not possible to conduct the external validation in another institution since multi-modality imaging data with an adequate number of patients were not available elsewhere.To alleviate concerns for the generalizability, we did not use any vendor-specific pre-or post-processing, and the raw Hounsfield Unit values were used as the input of the model after simple normalization between 0-1.Furthermore, we employed two methods to improve the generalizability, namely transfer learning from pre-trained models on general domain data and domain-specific self-supervised learning.Without these methods, the performance was found to be compromised, suggesting the possibility of overfitting.Third, instead of histological ground truth, the concurrent OCT images that have higher resolution were leveraged as the gold standard.This approach was adopted since it was impossible to obtain histological diagnosis in living patients.This approach has been widely adopted in developing the DL model for medical image analysis when histological validation is not feasible [28][29][30] .Fourth, less common ACS pathologies such as a calcified plaque, spontaneous coronary dissection, and intraplaque acksedge were excluded.Fifth, although the prevalence of disease in non-plaque erosion was not low (34.1%), the possibility of the falsely high sensitivity of the MD-CTA model could not be completely ruled out.Sixth the diagnostic accuracy of the model tends to be affected by the quality of the image, for instance, severely calcified plaque or severe luminal narrowing lowered the accuracy of the diagnosis.Of note, plaque erosion, compared to plaque rupture, in general has a larger lumen.Seventh, the performance of the model slightly decreased when evaluated only on ACS patients.Furthermore, while the model demonstrates excellent performance in the overall diagnosis of plaque erosion versus non-plaque erosion, it shows a reduced performance in differentiating between plaque erosion and rupture.Nonetheless, our MD-CTA model clearly outperforms the CNN model (Supplementary Table S7).This suggests that our model may be utilized in clinical applications for purposes such as a screening

Figure 1 .
Figure 1.Diagnostic accuracy of the deep learning (DL) models for patient-level (A,B) and slice-level (C,D) predictions.Diagnostic performance of the deep learning models at the patient level in the five-fold crossvalidation (A), in the test set validation (B), and at the slice level in the five-fold cross-validation (C), in the test set validation (D).AUC area under the curve, CNN convolutional neural network, MD-CTA momentum distillation-enhanced composite transformer attention.

Figure 2 .
Figure 2. Plaque rupture and plaque erosion as seen on OCT, CTA, and CTA enhanced by DL model.Representative images of each label are shown.(A) shows an OCT image of plaque rupture.Plaque rupture is characterized by the presence of fibrous cap discontinuity with a cavity formation (asterisks) within the plaque.(A) also shows the residual ruptured cap (red arrow).(B,C) show CTA images of the corresponding site.(B) shows the ruptured cap (yellow arrow) protruding into the vessel lumen at the same site observed by OCT.(C) shows that the DL model attends on the ruptured cap and cavity.(D) shows an OCT image of plaque erosion.Definite plaque erosion is characterized by the presence of attached thrombus (blue arrow) overlying an intact and visualized plaque.(E,F) show CTA images of the corresponding site.(E) shows a small lumen surrounded by plaque without a cavity.(F) shows that the DL model attends on the site of stenosis without evidence of a cavity.In panels (C) and (F), the visualized model attention represents the relative importance as determined by the DL model for each specific image, with values normalized to a range between 0 and 1 for the images under consideration.CTA computed tomography angiography, DL deep learning, OCT optical coherence tomography, RI relative importance.

Figure 3 .
Figure 3. Potential Future Approach for Evaluation and Management of Patients With ACS.Patients with STEMI would undergo emergency catheterization.If plaque rupture is confirmed, the culprit lesion would be treated with stenting.If OCT demonstrated plaque erosion with preserved lumen, antithrombotic therapy without stenting could be considered.Patients with NSTE-ACS would undergo noninvasive coronary CTA with DL model after stabilization.If there is high probability of plaque erosion and preserved lumen, antithrombotic therapy without stenting could be considered.CTA computed tomography angiography, DL deep learning; NSTE-ACS non-ST-segment elevation acute coronary syndromes, OCT optical coherence tomography, PCI percutaneous coronary intervention, STEMI ST-segment elevation myocardial infarction. )

Table 2 .
Coronary CTA scan lesion characteristics in the training versus test datasets.Values are n (%).CTA computed tomography angiography, LAD left anterior descending artery, LCX left circumflex artery, RCA right coronary artery.

Table 3 .
Performances of the deep learning models for patient-level diagnosis.AUC area under the curve, CI confidence interval, CNN convolutional neural network, FNR false-negative rate, FPR false-positive rate, MD-CTA momentum distillation-enhanced composite transformer attention, NPV negative predictive value, PPV positive predictive value.

Table 5 .
Results of the reader study to validate the clinical utility of the deep learning model.CI confidence interval, DL deep learning, FNR false-negative rate, FPR false-positive rate, NPV negative predictive value, PPV positive predictive value.