Diagnostic ability of artificial intelligence using deep learning analysis of cyst fluid in differentiating malignant from benign pancreatic cystic lesions

The diagnosis of pancreatic cystic lesions remains challenging. This study aimed to investigate the diagnostic ability of carcinoembryonic antigen (CEA), cytology, and artificial intelligence (AI) by deep learning using cyst fluid in differentiating malignant from benign cystic lesions. We retrospectively reviewed 85 patients who underwent pancreatic cyst fluid analysis of surgical specimens or endoscopic ultrasound-guided fine-needle aspiration specimens. AI using deep learning was used to construct a diagnostic algorithm. CEA, carbohydrate antigen 19-9, carbohydrate antigen 125, amylase in the cyst fluid, sex, cyst location, connection of the pancreatic duct and cyst, type of cyst, and cytology were keyed into the AI algorithm, and the malignant predictive value of the output was calculated. Area under receiver-operating characteristics curves for the diagnostic ability of malignant cystic lesions were 0.719 (CEA), 0.739 (cytology), and 0.966 (AI). In the diagnostic ability of malignant cystic lesions, sensitivity, specificity, and accuracy of AI were 95.7%, 91.9%, and 92.9%, respectively. AI sensitivity was higher than that of CEA (60.9%, p = 0.021) and cytology (47.8%, p = 0.001). AI accuracy was also higher than CEA (71.8%, p < 0.001) and cytology (85.9%, p = 0.210). AI may improve the diagnostic ability in differentiating malignant from benign pancreatic cystic lesions.

Comparison of diagnostic ability of cyst fluid analysis, cytology, and AI. Comparison results of the diagnostic abilities of AI, AI using only CEA, cyst fluid analysis (CEA), and cytology in differentiating malignant from benign pancreatic cystic lesions is shown in Table 3. ROC curve analysis demonstrated (Fig. 1b) that the AUC of the diagnostic ability in differentiating malignant from benign pancreatic cystic lesions were as follows: AI, 0.966 (p < 0.001); AI using only CEA, 0.956 (p < 0.001); CEA, 0.719 (p = 0.002); and cytology, 0.739 (p = 0.001). Among them, AI has the highest AUC. The sensitivity of AI (95.7%) was higher than that of AI using only CEA (91.3%, p = 1.000), CEA (60.9%, p = 0.021), and cytology (47.8%, p = 0.001). The accuracy of AI (92.9%) was also higher than those of AI using only CEA (89.4%, p = 0.375), CEA (71.8%, p < 0.001), and cytology (85.9%, p = 0.210). The univariate and multivariate analyses of malignant cystic lesion including AI, pancreatic cyst fluid, and clinical data are shown in Table 4. In the univariate analysis, CEA, connection of the pancreatic duct and cyst, and AI were statistically significant. Variables with p < 0.05 in the univariate analysis were selected for entry into the multivariate analysis. In the multivariate analysis, CEA (p = 0.036; odds ratio [

Discussion
Diagnosis of malignant pancreatic cystic lesions is necessary for determining the appropriate treatment strategy. The diagnostic ability of CEA using pancreatic cyst fluid for malignant cysts was poor, and a previous meta-analysis reported its diagnostic sensitivity of 63% and specificity of 63% 14 . Even in the present study, the diagnostic ability of CEA by cyst fluid analysis was poor. Several studies have noted elevated cyst fluid CEA in LEC 28 and EDC 29 . These studies reported CEA expression on the surface of squamous epithelial cells, indicating that squamous epithelial cells could produce CEA, resulting in the elevated cyst fluid CEA levels. In our study, two patients had EDC and two patients had LEC. These results increased the cut-off value of CEA in differentiating malignant from benign pancreatic cystic lesions. This is an important limitation of differentiating malignant from benign cystic lesions using cyst fluid CEA.
Although cytology had excellent specificity, it has a limited role because of its lack of sensitivity in previous studies [30][31][32] . In the present study, the sensitivity of cytology in differentiating malignant from benign cystic lesions was 47.8%. Thus, we constructed AI using deep learning algorithm for differentiating malignant from benign pancreatic cystic lesions based on the analysis of pancreatic cyst fluid and clinical data.
AI using deep learning is used for various fields such as skin cancer 22 , radiation oncology 23 , Helicobacter pylori infection 24 , colorectal polyp 25,26 , and histopathology 27 using image information. However, the application of AI using deep learning to analyse pancreatic cystic lesions is relatively new. We used a system of AI using deep learning to simulate the human brain nervous system. In this study, we constructed AI using deep learning with TensorFlow. This framework accepts sets of data and corresponding labels as input clinical data and constructs a neural network for diagnosis. In this study, AI using deep learning analysed pancreatic cyst fluid and clinical data. By using this deep learning method, AI learns the characteristics of malignant cystic lesions by combining cyst fluid analysis and clinical data, and AI can possibly exclude the bias generated by human judgment. Although it is difficult for clinicians to diagnose malignant pancreatic cystic lesions by cyst fluid analysis and clinical data, AI using deep learning achieved adequate diagnostic ability in differentiating malignant from benign cystic lesions compared to cyst fluid analysis such as CEA and cytology. AI and CEA were also significant factor in the multivariate analysis of malignant cystic lesion. Specifically, although it is generally a problem that cytology diagnosis has low sensitivity, AI using deep learning achieved high sensitivity (95.7%). AI could be a powerful tool for exclusion diagnosis of malignant pancreatic cystic lesions. Although AI using only CEA was slightly lower in diagnostic ability than AI, AI using only CEA has good diagnostic ability. If CA 19-9, CA 125, and amylase cannot be used in the cystic fluid analysis, AI using only CEA may be useful for diagnosis in differentiating malignant from benign pancreatic cystic lesions. AI will improve the diagnostic ability of pancreatic cystic lesions if introduced as a supporting system. In the future, an AI using deep learning-based diagnostic system will change the method of diagnosis of differentiating malignant from benign pancreatic cystic lesions.
This study has some limitations. First, it is a retrospective and single-centre study. Second, this study had a small sample size and many cases were excluded (53/138 cases). The study is as long as 20 years, and we excluded cases with incorrect data to maintain the accuracy of the study. The AI algorithm cannot analyse data when there is a missing item. Hence, by improving the system and AI performance through developing new algorithms, it is desirable to achieve the good diagnostic ability of AI even if some items are missing. In addition, Because of the small sample size, we used cross-validation method, and verification with external data was not done. Hence, the diagnostic ability of AI using external data should be confirmed in the future. Thrid, in this study, subjective factors such as connection of the main pancreatic duct and cyst and type of cyst (monolocular or multilocular) were adopted. It is desirable to create algorithms that can be analysed in the future using only objective data, such as tumour marker of the pancreatic cyst fluid, age, and sex. Fourth, 30.6% (26/85) of cases using pancreatic cyst fluid was collected by EUS-FNA in this study. Analysis and cytology by EUS-FNA for pancreatic cyst fluid are widely used worldwide. Previous studies reported that the frequency of peritoneal seeding in the EUS-FNA group and the no sampling group has no significant difference 33 . However, EUS-FNA for the diagnosis of mucinous cystic lesions is rarely performed in Japan because there was one report of fatal peritoneal tumour seeding following EUS-FNA in Japan 34 . Therefore, there are only a few cases of pancreatic cyst fluid collection by EUS-FNA, especially cases suspected of mucinous cystic lesion prior to fluid analysis. A study reported that the rate of obtaining adequate amount of specimen through EUS-FNA was low and may not have sufficient diagnostic ability 35 . The diagnostic ability of cytology between surgical specimens and EUS-FNA may differ; thus, interpretation of the results of cytology in this study requires caution. Further, cytology was considered an effective factor for the AI algorithm, so cytology was used for the AI algorithm in this study. If EUS-FNA cytology is positive, AI may not be needed. In addition, developing an AI algorithm that excludes cytology is expected in the future.
In conclusion, AI may improve the diagnostic ability of differentiating malignant from benign cystic lesions. Moreover, compared to pancreatic cyst fluid analysis such as CEA and cytology, AI is highly sensitive of differentiating malignant from benign cystic lesions and may be useful for exclusion of malignant pancreatic cystic lesions.

Methods
Patients. We retrospectively reviewed data of patients with pancreatic cystic lesions who underwent cyst fluid analysis and cytology with surgical specimens or EUS-FNA from January 1997 to December 2017 in Aichi Cancer Center hospital. Patients who underwent cyst fluid analysis (CEA, CA19-9, CA125, amylase, and cytology) were included in this study, and patients with missing data were excluded. The cysts were IPMN, MCN, SCN, PPC, EDC, and LEC. Clinical data, biochemical, and pathological results were collected. To analyse AI algorithm, sex, location of the cystic lesion, connection of the pancreatic duct and cyst, type of cyst (monolocular or multilocular) were extracted as characteristic factors of pancreatic cysts. In this study, image data were referenced to results of CT, magnetic resonance cholangiopancreatography (MRCP), and EUS. CT, MRCP, and EUS images were reviewed by an experienced pancreatic radiologist and endoscopist.
www.nature.com/scientificreports www.nature.com/scientificreports/ This study was approved by the institutional review board of Aichi Cancer Center Hospital (No.2018-1-022). In this retrospective observational study, only medical information without invasion to participants is used. Informed consent for all participants was acquired in the form of opt-out. Those who rejected were excluded. All research and examinations were performed in accordance with relevant guidelines and regulations. All authors had agreed to all the contents and were involved in the review and approval of the manuscript.
Pancreatic cyst fluid sampling method. Pancreatic cyst fluid was collected surgically or by EUS-FNA.
Surgically, a cyst of a resected specimen was punctured with a needle and manually aspirated, and cyst fluid was collected. EUS-FNA procedures were performed with the patient under conscious sedation using 35 mg of intravenous pethidine hydrochloride (Mitsubishi Tanabe Pharma, Osaka, Japan) and 5-10 mg of intravenous midazolam (Astellas, Tokyo, Japan). EUS-FNA was performed using GF-UC30P, GF-UC240P-AL5, GF-UCT240-AL5, and GF-UCT260-AL5 convex linear echoendoscope (Olympus Medical Systems, Tokyo, Japan) connected to an ultrasound scanning system (SSD-5500, Prosound SSD a-10; Hitachi Aloka Medical, Tokyo Japan or EU-ME2; Olympus Medical Systems, Tokyo, Japan). Different needle types (19-, 22-, or 25-gauge Echo Tip Ultra; Cook Medical, IN, USA, or NA-U200H; Olympus Medical Systems, Tokyo, Japan, or Expect; Boston Scientific, Tokyo, Japan) were employed. Needle sizes and types were chosen at the discretion of the endosonographer. We uniformly used negative suction with a 10-mL or 20-mL syringe during all FNA procedures.

Final diagnosis and definition of malignant and benign cystic lesions.
Histopathological examination of the surgical specimen was regarded as the final diagnosis in case of surgical resection. For patients who did not undergo surgery, they were clinically diagnosed by clinical data, image diagnosis, and follow-up for 1 year or more. In particular, diagnosis of benign lesions was made in patients in whom we collected pancreatic cyst fluid by EUS-FNA when there were no malignant findings on cytology of the EUS-FNA and imaging examinations (CT, MRCP, and EUS reviewed by radiologist and endoscopist) and a lack of progression for 1 year or more on follow-up CT, MRCP, and EUS. On the contrary, in patients in whom pancreatic cyst was collected by EUS-FNA, malignant lesions were defined as lesions when surgical resection at a later date was malignant or pancreatic cyst fluid cytology was malignant. Figure 2 summarises the routes by which the 85 patients were finally diagnosed.
For malignant and benign classifications of cystic lesions, high-grade dysplasia or invasive carcinoma (IPMN or MCN) was considered a malignancy, and low-or intermediate-grade dysplasia (IPMN or MCN), PPC, EDC, and LEC were considered benign lesions.
Pancreatic cyst fluid analysis and cytology. Pancreatic cyst fluid was taken for analysis of CEA, CA19-9, CA125, and amylase 10 . Cyst fluid concentrations of CEA, CA19-9, CA125, and amylase were measured by specific chemiluminescent immunoassays. CEA was analysed on ARCHITECT analyser (Abbot Japan, Tokyo, Japan). CA19-9 and CA125 were analysed on LUMIPULSE (FUJIREBIO Inc., Tokyo, Japan). Amylase was analysed on LABOSPECT 008 (Hitachi High-technologies Corporation, Tokyo, Japan). Pancreatic cyst fluid was also taken for cytology. Results of cytology were classified as benign, atypical, suspicious for malignancy, consistent with malignancy, and malignancy. Cytology was considered malignant when the report was suspicious for malignancy, consistent with malignancy, or malignancy. Benign and atypical in cytology were considered benign.
Artificial intelligence using deep learning algorithm. A neural network is a machine learning algorithm that provides mathematical result after entering various numerical values or information. In this study, to construct an AI-based diagnostic algorithm, deep learning, a multi-hidden layer of neural network, was used (Fig. 3a). TensorFlow version 1.5 (Google LCC, Mountain View, USA) was used for the deep learning analysis. During the training process of deep learning, labelled information such as clinical information was entered into the deep learning algorithm. Subsequently, output values were calculated. Data were labelled as follows: malignant cystic lesion was defined as 1 and benign as 0 according to the final diagnosis. In this study, the input layers www.nature.com/scientificreports www.nature.com/scientificreports/ were CEA, CA19-9, CA125, amylase in the cyst fluid, sex, cyst location, connection of the pancreatic duct and cyst, type of cyst (monolocular or multilocular), and cytology. The value of the input layer was normalised and keyed in the algorithm. The hidden layer has two layers, and each hidden layer has nine nodes. Tanh (hyperbolic tangent) was used for activation function of the hidden layer. The softmax function was used for the activation function of the output layer. The optimisation algorithm used to train the network weights was stochastic gradient descent. The output layer was defined as malignant predictive value calculated by AI using deep learning. The malignant predictive value is a continuous variable from 0 to 1. When the malignant predictive value is close to 1, the diagnosis of AI using deep learning was regarded as malignant cystic lesions. The accuracy validation method used fivefold cross validation: (learning/test sample ratio: 80%/20% × 5). In other words, 80% of all cases were randomly assigned to the learning sample and 20% to the test sample, and its diagnostic ability was measured five times. In addition, because none of the markers are routinely used or recommended in the fluid analysis other than CEA in clinical practice, we also created AI algorithm using only CEA and clinical data (sex, cyst location, connection of the pancreatic duct and cyst, type of cyst and cytology) except CA19-9 CA125 and amylase (Fig. 3b).
Endpoint. The primary endpoint was to investigate and compare the diagnostic ability of AI, AI using only CEA, cyst fluid analysis (CEA, CA19-9 CA125, and amylase) and cytology in differentiating malignant from benign pancreatic cystic lesions. The main outcomes were diagnostic sensitivity, specificity, PPV, NPV, and accuracy.
The secondary endpoint was factors of malignant pancreatic cystic lesion. Factors of malignant pancreatic cystic lesion were analysed using univariate and multivariate analyses. Variables employed for analyses were CEA, CA19-9 CA125, amylase, sex, location of the cystic lesion, connection of the pancreatic duct and cyst, type of cyst, cytology, and AI.

Statistics. Chi-squared test (or Fisher's exact test, if appropriate) was used for categorical variables and
Mann-Whitney U test used for continuous variables in the univariate analyses. McNemar test was used to calculate the sensitivity, specificity, and accuracy. A P value < 0.05 was considered significant. ROC curve analysis was performed to characterise the diagnostic ability of cyst fluid tumour markers, amylase, cytology, and AI in differentiating malignant from benign cysts. The AUC was calculated to define the cut-off value for the diagnosis of malignant cystic lesions. The value at the top left was considered the cut-off value by ROC. Logistic regression models for multivariate analysis were used. Variables with p < 0.05 in the univariate analysis were selected for entry into the multivariate analysis. These statistical analyses were performed using SPSS version 21 software (IBM, Armonk, NY, USA).