The feasibility of deep learning-based synthetic contrast-enhanced CT from nonenhanced CT in emergency department patients with acute abdominal pain

Our objective was to investigate the feasibility of deep learning-based synthetic contrast-enhanced CT (DL-SCE-CT) from nonenhanced CT (NECT) in patients who visited the emergency department (ED) with acute abdominal pain (AAP). We trained an algorithm generating DL-SCE-CT using NECT with paired precontrast/postcontrast images. For clinical application, 353 patients from three institutions who visited the ED with AAP were included. Six reviewers (experienced radiologists, ER1-3; training radiologists, TR1-3) made diagnostic and disposition decisions using NECT alone and then with NECT and DL-SCE-CT together. The radiologists’ confidence in decisions was graded using a 5-point scale. The diagnostic accuracy using DL-SCE-CT improved in three radiologists (50%, P = 0.023, 0.012, < 0.001, especially in 2/3 of TRs). The confidence of diagnosis and disposition improved significantly in five radiologists (83.3%, P < 0.001). Particularly, in subgroups with underlying malignancy and miscellaneous medical conditions (MMCs) and in CT-negative cases, more radiologists reported increased confidence in diagnosis (83.3% [5/6], 100.0% [6/6], and 83.3% [5/6], respectively) and disposition (66.7% [4/6], 83.3% [5/6] and 100% [6/6], respectively). In conclusion, DL-SCE-CT enhances the accuracy and confidence of diagnosis and disposition regarding patients with AAP in the ED, especially for less experienced radiologists, in CT-negative cases, and in certain disease subgroups with underlying malignancy and MMCs.

Since its introduction in the 1970s, the use of computed tomography (CT) has increased in various clinical settings, including emergency departments (EDs) 1 . In particular, CT-associated ED visits have increased dramatically in patients older than 65 years and in patients with acute abdominal pain (AAP) 1 . According to a previous study, in patients who visited the ED complaining of AAP, CT may make substantial contributions to diagnosis or disposition decisions and may confirm or exclude alternative diagnoses 2 . Because physician diagnostic accuracy and confidence increase with CT 2 , CT plays a critical role in the diagnosis and management of patients with AAP in the ED [1][2][3] .
Intravenous (IV) contrast agents are widely used in CT examination and are known to improve sensitivity and specificity in many indications 4 . However, the risk of adverse events, including allergies and nephropathy, must be considered before administering IV contrast agents 5,6 . In patients with advanced age and underlying

Results
Baseline demographics and clinical characteristics of the test dataset. The

Outcomes of clinical validation of deep learning-based synthetic CT Images.
Step 1: Review of diagnostic performance of image analysis with NECT alone. With NECT alone, the accuracy of diagnosis ranged from 69.4 to 81.5%, and the accuracy of the disposition decision ranged from 70.1 to 84.4%. The accuracy of both diagnosis and disposition decisions differed according to the dataset and reviewer expertise. The accuracy of diagnosis was superior for the experienced radiologists (76.5-81.5%) compared to that of the training radiologists (69.4-77.1%) and in selectively enrolled datasets with specific diagnoses (Dataset-A, 75.5-87.0%) compared to that in consecutively enrolled datasets (Dataset-B, 61.4-73.2%). Similarly, the accuracy of the disposition decision was better in ERs (76. 8 Step 2: Review of diagnostic performance of image analysis with the aid of DL-SCE-CT. Table 1 and Fig. 1 summarize the diagnostic performance of radiologists with or without the aid of DL-SCE-CT. Overall, with the aid of DL-SCE-CT, the accuracy of diagnosis increased from 69.4-81.0% to 70.5-84.7%. Of the six radiologists, three radiologists (50%, P = 0.023, 0.012, < 0.001) reported a significant increase in diagnostic accuracy with DL-SCE-CT. In particular, two-thirds of TRs experienced significant improvement in accuracy. The confidence of diagnosis (from 2.87-4.09 to 2.99-4.50) and disposition (from 3.83-4.19 to 4.11-4.53) also increased, with statistically significant increments observed by five of the six radiologists (83.3%, P < 0.001) (Figs. 2, 3). The accuracy of the disposition decision did not show a significant change for any radiologist. The diagnostic performance of each radiologist with or without the aid of DL-SCE-CT is shown in Supplementary Tables S1 to S6. www.nature.com/scientificreports/ Subgroup analysis according to disease category. The accuracy and confidence of diagnosis and disposition decisions were variably reported in different subgroups and are summarized in Table 2. Disease categories in which more than half of the radiologists experienced an increase in confidence of both diagnosis and disposition decisions included oncologic conditions (5/6: 83.3% and 4/6: 66.7%, respectively), MMCs (6/6: 100.0% and 5/6: 83.3%, respectively), and NSAP (5/6: 83.3% and 6/6: 100.0%, respectively). For acute pancreatitis, acute diverticulitis, biliary disease, and acute appendicitis, more than half of the radiologists showed no significant change in confidence in diagnosis and disposition decisions. The confidence of diagnosis and disposition decisions in the 1st session were generally lower in useful subgroups (oncologic conditions, MMC, and NSAP, 1.31-4.71 and 2.81-4.77, respectively) than in less useful subgroups (acute pancreatitis, acute diverticulitis, biliary disease, and acute appendicitis, 3.48-4.90 and 3.70-4.90, respectively). There was no particular subgroup in which DL-SCE-CT could significantly improve more than half of the radiologists' accuracy of diagnosis or disposition. The radiologists determined that the image quality of DL-SCE-CT was sufficient (with moderate limitations for clinical use but no substantial loss of information; mean score 3.33), and the artifact degree was moderate (with preserved diagnostic reliability; mean score 3.25). The image quality score ranged from 1.83 to 4.17, and the artifact degree score ranged from 2.00 to 4.33.

Discussion
In our study, DL-SCE-CT was feasible and helpful for patients visiting the ED with complaints of AAP, increasing the radiologists' accuracy of diagnosis and confidence level in diagnosis and disposition decisions. In particular, DL-SCE-CT was useful in cases with oncologic conditions, MMCs, and NSAP and for less experienced radiologists. The confidence of diagnosis and disposition decisions significantly increased in five of the radiologists (83.3%, P < 0.001). In addition, diagnostic accuracy was significantly improved in half of the radiologists (P = 0.023, 0.012, < 0.001). In particular, DL-SCE-CT was more helpful in training radiologists, improving the diagnostic accuracy of two-thirds of the radiologists (P = 0.023 and 0.012). Technically, the image quality of DL-SCE-CT was rated as sufficient with moderate limitations and without substantial loss of information, and the degree of artifact was rated as moderate with preserved diagnostic reliability. www.nature.com/scientificreports/ DL-SCE-CT was more helpful in oncologic conditions, MMCs and NSAP (helpful subgroups) but less useful in the acute pancreatitis, acute diverticulitis, biliary disease, and acute appendicitis subgroups (unhelpful subgroups). In diseases belonging to the unhelpful subgroups, diagnosis often depends on findings such as fat strandings in the organ-fat interface (e.g., peripancreatic, periappendiceal, or peridiverticular fat strandings) or radio-opaque stones (e.g., acute calculous cholecystitis or cholangitis), which are easily detected by NECT. In these less useful subgroups, radiologists showed high confidence in diagnosis and disposition decisions using NECT alone (3.48-4.91 and 3.23-4.95). In contrast, radiologists showed lower confidence for NECT evaluation of helpful subgroups (1.31-4.36 and 2.81-4.55). It is meaningful that DL-SCE-CT increases the confidence of radiologists in diseases that are difficult to diagnose using NECT alone.
Recently, deep-learning-based synthetic medical images have been an active area of research with broad applications in various medical disciplines. Attempts to increase patient safety using deep learning-based contrast dose reduction have been made. Some researchers have accomplished reductions in the gadolinium dose used for brain MRI using a deep learning method 10 , while other researchers synthesized fake contrast enhancement images for brain MRI and cardiac CT 11,12 . Technically, unlike other previous algorithms, our algorithm was able to minimize the misalignment issue caused by minute breathing and patient movement between the input data NECT image and the reference standard CECT image using a two-stage approach 27 . A detailed description of our algorithm is provided in the Methods section.
Moreover, consideration of clinical significance is necessary for such technology to be used in real-world situations. AAP is one of the most common reasons for visiting the ED, accounting for up to 7-10% of all ED visits 13,14 . Diseases causing AAP range from self-limiting to life-threatening conditions 14 , causing large medical and socioeconomic burdens 15,16 . In particular, the overall burden of AAP and difficulty of diagnosis increase www.nature.com/scientificreports/ with advancing age 14;16-18 . Thus, the number and proportion of CT-associated ED visits has rapidly increased in elderly patients with AAP 1-3;19-21 . However, liberal use of CT is accompanied by an increased risk of ionizing radiation exposure 22,23 and adverse effects due to IV contrast agents 5,6,8 . Thus, the results of our study are valuable in this situation by augmenting the diagnostic performance of NECT and radiologists' confidence in decision making, although more improvements are warranted in the future. Our study has several limitations. First, there is inherent selection bias owing to the retrospective nature of our study. A considerable portion (56.7%, 200/353) of our study population was selectively enrolled, increasing selection bias. Second, the improvement was greater in confidence than in the accuracy of radiologists' decisions. Although increased confidence is a meaningful benefit, DL-SCE-CT should improve radiologists' actual performance to improve the clinical outcomes in patients. We hope to improve clinical outcomes by further elaborating the quality of synthetic contrast enhancement. Third, the increments of confidence were present in both correct and incorrect cases, raising concern for increasing confidence of misdiagnosis or mistreatment.
Deep learning-based synthetic CT (DLSCT) might be developed and applied in various clinical settings. Synthetic contrast enhancement is only one of many possibilities. Various kinds of image augmentation could potentially improve patient outcomes. Appropriate clinical settings are necessary for the development of useful synthetic images. For elderly patients with decreased renal function, both an NECT-based synthetic enhancement method and a method that uses a small dose of contrast agent but mimics the use of a full dose (e.g. as used in brain MRI 10 ) could be developed. A combination with pre-existing contrast dose reduction technologies, such as dual energy CT, might be attempted. For particularly radiosensitive populations, such as pediatric patients, deep-learning-based denoising algorithms might be especially helpful [24][25][26] to facilitate ultralow-dose imaging. Therefore, further investigation of potentially useful DLSCTs is warranted. www.nature.com/scientificreports/ Conclusion. In conclusion, according to our preliminary study, DL-SCE-CT is feasible and is helpful for more accurate and confident diagnosis and disposition decisions regarding patients with AAP in the ED. In particular, DL-SCE-CT is useful in cases with oncologic conditions, MMCs, negative CT findings, and for less experienced radiologists.

Methods
This retrospective multicenter study was approved by the joint Institutional Review Board of Seoul National University Hospital, Seoul National University Bundang Hospital, and Boramae Medical Center. The Institutional Review Board granted a waiver of informed patient consent due to the retrospective nature of our study. All methods were performed in accordance with the relevant guidelines and regulations.

Study population and image dataset.
We trained the conversion model using a pre-existing algorithm that generates DL-SCE-CT from NECT 27 < 0.001, 0.013*, 0.625, 0.001 < 0.001, 0.709, 0.003, 0.001 Figure 4 summarizes the inclusion process of the study population.
For each case, one radiologist (S.W.K.) meticulously reviewed the CT images and electronic medical records, including the clinicopathologic data and laboratory test results, to determine the most appropriate diagnosis and disposition decisions (admission for medical treatment, admission for surgical treatment, or discharge).

CT techniques.
All CT examinations were conducted using MDCT scanners with 64-160 detector rows.
The acquisition parameters were as follows: tube voltage, 100-120 kVp; tube current, 50-300 mAs; slice thickness, 2.0-3.0 mm; reconstruction interval, 1.0-3.0 mm; pitch, 0.5-1.3; and rotation time, 0.33-0.75 s. Iodinated contrast agent (320 or 350 mg I/mL) was injected using an automatic power injector at a rate of 2.0-5.0 mL/s (total amount 1.6-2.0 mL/kg). The portal phase acquisition time was determined by either the bolus tracking method (beginning of portal phase scan 45-60 s after reaching threshold attenuation [100 HU] at the descending thoracic aorta or immediately after reaching threshold attenuation [100 HU] at the hepatic parenchyma) or fixed time delay (90 s after contrast injection).

Development of DL-SCE-CT image.
A conversion model that generates DL-SCE-CT from NECT was developed in a previous study using 23,923 paired NECT and CECT images from 327 CT examinations 27 . In our study, we adopted the same model with identical architecture and trained it using our training dataset (226 CT examinations with 35,414 paired NECT and CECT images). In contrast to other neural image syntheses, a major problem in synthetic contrast enhancement of abdominal CT is inevitable misalignment between NECT and CECT images owing to patients' breathing and involuntary movements during examinations. Our model was developed using a two-stage approach to overcome the misalignment issue using a conditional generative adversarial network (cGAN) and a deep convolutional neural network (CNN). In the first stage, a generator (G C→N ) that creates synthetic NECT from real CECT was trained adversarially with a discriminator (D) that distinguishes among synthetic NECT, real NECT and real CECT. This stage, which is an inverse of our target task, is technically much easier because NECT images are much less patient-specific than CECT images due to monotonic intensities. The resulting synthetic NECT, which is perfectly aligned with real CECT, is used in the second stage for training a generator (G N→C ), which creates synthetic CECT from synthetic NECT. Generators (G C→N and G N→C ) were trained using the SPADE architecture, one of the state-of-the-art methods in image-toimage translation. The second generator (G N→C ) was finally used to create our synthetic images in the test dataset (Datasets A and B). Figure 5 shows the development process.  www.nature.com/scientificreports/ TR3: K.J.; training radiologists with 4, 5, and 4 years of experience, respectively). In the first session, the reviewers were asked to report the diagnosis in subjective form and the disposition decision in three-option multiple choice format (admission for medical treatment, admission for surgical treatment, and discharge) based on NECT alone. Reviewers were aware that the images being reviewed were from patients who visited the ED with AAP, but no further clinical or laboratory data were provided. The reviewers' confidence in the diagnosis and disposition decision was rated on a 5-point scale (1: Not confident at all; 2: Slightly confident; 3: Somewhat confident; 4: Fairly confident; and 5: Completely confident).
Step 2: Image analysis with the aid of DL-SCE-CT. The second review session was initiated two weeks after the first session ended. In the second session, the reviewers were asked to report the diagnosis and disposition decision after reviewing both NECT and DL-SCE-CT with the 5-point scale confidence level. In addition, the reviewers were asked to report the image quality (1: poor image quality-image not usable; 2: restricted image quality-severe limitations for clinical use; clear loss of information; 3: sufficient image quality-moderate limitations for clinical use but no substantial loss of information; 4: good image quality-minimal limitations for clinical use; and 5: excellent image quality-no limitations for clinical use) and artifact degree (1: artifacts resulting in a nondiagnostic image; 2: severe artifacts resulting in limited diagnostic reliability; 3: moderate degree with preserved diagnostic reliability; 4: minimal degree with preserved diagnostic reliability; and 5: excellent without artifact) of DL-SCE-CT on a 5-point scale. The arithmetic means of scores rated by six radiologists were used as representative image quality and artifact degree scores for each case.
Statistical analysis. The accuracy of the diagnosis and disposition decisions were compared between the two review sessions using McNemar's test. Confidence in the diagnosis and disposition decisions were compared using the Wilcoxon test. For subgroup analysis, the whole study population was divided into ten subgroups according to the CT diagnosis: acute pancreatitis, acute diverticulitis, liver disease (e.g., acute hepatitis and liver abscess), biliary disease (e.g., acute cholecystitis and acute cholangitis), oncologic condition (e.g., malignant bowel obstruction, malignant biliary obstruction and ruptured HCC with hemoperitoneum), acute appendicitis, bowel obstruction, miscellaneous surgical conditions, miscellaneous medical conditions, and NSAP. The accuracy and confidence were separately evaluated in each subgroup. A commercially available software package (MedCalc Statistical Software, Version 19.2.1, MedCalc Software) was used for statistical analysis. A p value less than 0.05 was considered statistically significant.

Data availability
The datasets generated and/or analyzed during the current study are available from the corresponding author upon reasonable request. However, our Institutional Review Board prohibits open the data relate to patient's personal medical information and images.
Received: 29 May 2021; Accepted: 30 September 2021 Figure 5. Schematic diagram of the two-stage approach used for making the conversion model. In the first stage, the generator (G C→N ), which generates synthetic NECT from real CECT, is trained adversarially using a conditional generative adversarial network. In the second stage, another generator (G N→C ) that generates synthetic CECT from NECT is trained using a deep convolutional neural network. During the second stage of training, synthetic NECT, which is generated from and perfectly aligned with real CECT, is used as input data, resolving the misregistration issue between input data and ground truth (real CECT). NECT, nonenhanced CT; CECT, contrast-enhanced CT; L Adv , adversarial loss; L rec , reconstruction loss; G C→N , generator that generates synthetic NECT from real CECT; G N→C , generator that generates synthetic CECT from NECT. www.nature.com/scientificreports/ Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.