The detection and removal of precancerous polyps via colonoscopy is the gold standard for the prevention of colon cancer. However, the detection rate of adenomatous polyps can vary significantly among endoscopists. Here, we show that a machine-learning algorithm can detect polyps in clinical colonoscopies, in real time and with high sensitivity and specificity. We developed the deep-learning algorithm by using data from 1,290 patients, and validated it on newly collected 27,113 colonoscopy images from 1,138 patients with at least one detected polyp (per-image-sensitivity, 94.38%; per-image-specificity, 95.92%; area under the receiver operating characteristic curve, 0.984), on a public database of 612 polyp-containing images (per-image-sensitivity, 88.24%), on 138 colonoscopy videos with histologically confirmed polyps (per-image-sensitivity of 91.64%; per-polyp-sensitivity, 100%), and on 54 unaltered full-range colonoscopy videos without polyps (per-image-specificity, 95.40%). By using a multi-threaded processing system, the algorithm can process at least 25 frames per second with a latency of 76.80 ± 5.60 ms in real-time video analysis. The software may aid endoscopists while performing colonoscopies, and help assess differences in polyp and adenoma detection performance among endoscopists.
Subscribe to Journal
Get full journal access for 1 year
only $8.67 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
The authors declare that all data supporting the findings of this study are available within the paper and its supplementary information. Restrictions apply to the availability of the medical training/ validation data, which were used with permission for the current study, and so are not publicly available. Some data may be available from the authors upon reasonable request and with permission of the Sichuan Academy of Medical Sciences & Sichuan Provincial People’s Hospital.
Lieberman, D. A. et al. Guidelines for colonoscopy surveillance after screening and polypectomy: a consensus update by the US Multi-Society Task Force on Colorectal Cancer. Gastroenterology 143, 844–857 (2012).
Seeff, L. C. et al. How many endoscopies are performed for colorectal cancer screening? Results from CDC’s survey of endoscopic capacity. Gastroenterology 127, 1670–1677 (2004).
Zauber, A. G. et al. Colonoscopic polypectomy and long-term prevention of colorectal-cancer deaths. N. Engl. J. Med. 366, 687–696 (2012).
Edwards, B. K. et al. Annual report to the nation on the status of cancer, 1975-2006, featuring colorectal cancer trends and impact of interventions (risk factors, screening, and treatment) to reduce future rates. Cancer 116, 544–573 (2010).
Siegel, R., Naishadham, D. & Jemal, A. Cancer statistics, 2013. CA Cancer J. Clin. 63, 11–30 (2013).
Chinese Society of Gastroenterology. Chinese consensus: screening, early diagnosis and treatment, comprehensive prevention of large bowel cancer. Chin. J. Gastroenterology Hepatology 20, 979–995 (2011).
Brenner, H., Chang-Claude, J., Seiler, C. M., Rickert, A. & Hoffmeister, M. Protection from colorectal cancer after colonoscopy: a population-based, case-control study. Ann. Intern. Med. 154, 22–30 (2011).
Corley, D. A. et al. Adenoma detection rate and risk of colorectal cancer and death. N. Engl. J. Med. 370, 1298–1306 (2014).
Ahn, S. B. et al. The miss rate for colorectal adenoma determined by quality-adjusted, back-to-back colonoscopies. Gut Liver 6, 64–70 (2012).
Marr, D. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information (Henry Holt and Co. Inc., New York, 1982).
Rex, D. K. et al. Colonoscopic miss rates of adenomas determined by back-to-back colonoscopies. Gastroenterology 112, 24–28 (1997).
Heresbach, D. et al. Miss rate for colorectal neoplastic polyps: a prospective multicenter study of back-to-back video colonoscopies. Endoscopy 40, 284–290 (2008).
Leufkens, A. et al. Factors influencing the miss rate of polyps in a back-to-back colonoscopy study. Endoscopy 44, 470–475 (2012).
Mahmud, N., Cohen, J., Tsourides, K. & Berzin, T. M. Computer vision and augmented reality in gastrointestinal endoscopy. Gastroenterol. Rep. (Oxf.) 3, 179–184 (2015).
Aslanian, H. R. et al. Nurse observation during colonoscopy increases polyp detection: a randomized prospective study. Am. J. Gastroenterol. 108, 166–172 (2013).
Lee, C. K. et al. Participation by experienced endoscopy nurses increases the detection rate of colon polyps during a screening colonoscopy: a multicenter, prospective, randomized study. Gastrointest. Endosc. 74, 1094–1102 (2011).
Buchner, A. M. et al. Trainee participation is associated with increased small adenoma detection. Gastrointest. Endosc. 73, 1223–1231 (2011).
Mori, Y. et al. Computer-aided diagnosis for colonoscopy. Endoscopy 49, 813–819 (2017).
Wang, Y. et al. Part-based multiderivative edge cross-sectional profiles for polyp detection in colonoscopy. IEEE J. Biomed. Health Inform. 18, 1379–1389 (2014).
Brandao, P. et al. Towards a computed-aided diagnosis system in colonoscopy: automatic polyp segmentation using convolution neural networks. J. Medical Robotics Res. 3, 1840002 (2018).
Chen, J. H. & Asch, S. M. Machine learning and prediction in medicine-beyond the peak of inflated expectations. N. Engl. J. Med. 376, 2507–2509 (2017).
Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402–2410 (2016).
Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).
Misawa, M. et al. Artificial intelligence-assisted polyp detection for colonoscopy: initial experience. Gastroenterology 154, 2027–2029 (2018).
Gkolfakis, P. et al. New endoscopes and add-on devices to improve colonoscopy performance. World J. Gastroenterol. 23, 3784–3796 (2017).
Pogorelov, K. et al. Efficient disease detection in gastrointestinal videos–global features versus neural network. Multimedia Tools Appl. 76, 22493–22525 (2017).
Fernandez-Esparrach, G. et al. Exploring the clinical potential of an automatic colonic polyp detection method based on the creation of energy maps. Endoscopy 48, 837–842 (2016).
Wang, Y., Tavanapong, W., Wong, J., Oh, J. H. & de Groen, P. C. Polyp-Alert: near real-time feedback during colonoscopy. Comput. Methods Programs Biomed. 120, 164–179 (2015).
Kuo, S. M., Lee, B. H. & Tian, W. Real-Time Digital Signal Processing: Implementations and Applications 2nd edn, Ch. 1 (John Wiley & Sons Ltd, Chichester, 2013).
East, J. E. et al. Advanced endoscopic imaging: European Society of Gastrointestinal Endoscopy (ESGE) Technology Review. Endoscopy 48, 1029–1045 (2016).
Mori, Y. et al. Real-time use of artificial intelligence in identification of diminutive polyps during colonoscopy: a prospective study. Ann. Intern. Med. https://doi.org/10.7326/M18-0249 (2018).
Takemura, Y. et al. Computer-aided system for predicting the histology of colorectal tumors by using narrow-band imaging magnifying colonoscopy (with video). Gastrointest. Endosc. 75, 179–185 (2012).
Kominami, Y. et al. Computer-aided diagnosis of colorectal polyp histology by using a real-time image recognition system and narrow-band imaging magnifying colonoscopy. Gastrointest. Endosc. 83, 643–649 (2016).
Chen, P. J. et al. Accurate classification of diminutive colorectal polyps using computer-aided analysis. Gastroenterology 154, 568–575 (2018).
Byrne, M. F. et al. Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model. Gut https://doi.org/10.1136/gutjnl-2017-314547 (2017)
Yao, K. The endoscopic diagnosis of early gastric cancer. Ann. Gastroenterol. 26, 11–22 (2013).
Winawer, S. J. et al. Colorectal cancer screening: clinical guidelines and rationale. Gastroenterology 112, 594–642 (1997).
Badrinarayanan, V., Kendall, A. & Cipolla, R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Analysis Machine Intelligence 39, 2481–2495 (2017).
Dean, J. et al. Large scale distributed deep networks. Adv. Neural Inf. Processing Syst. 25, 1223–1231 (2012).
Bandos, A. I., Rockette, H. E., Song, T. & Gur, D. Area under the free-response ROC curve (FROC) and a related summary index. Biometrics 65, 247–256 (2009).
Bernal, J. et al. Comparative validation of polyp detection methods in video colonoscopy: results from the MICCAI 2015 endoscopic vision challenge. IEEE Trans. Med. Imaging 36, 1231–1249 (2017).
We thank the Machine Intelligence Laboratory of the University of Cambridge for developing SegNet and making it publicly available.
X.X., J.L., J.H. and X.Y. are employees of Shanghai Wision AI Co., Ltd. The automatic polyp detection system was developed by the company and the software was provided free of charge for the purposes of this study. All other authors declare no competing interests.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary figures and video captions.
Real-time visual assistance during colonoscopy on an adjacent monitor.
Additional video of real-time visual assistance during colonoscopy on an adjacent monitor.
Sample video from the simulated real-time video analysis.
Additional sample video from the simulated real-time video analysis.
Demonstration of simulated real-time video analysis on datasets C and D.
About this article
Cite this article
Wang, P., Xiao, X., Glissen Brown, J.R. et al. Development and validation of a deep-learning algorithm for the detection of polyps during colonoscopy. Nat Biomed Eng 2, 741–748 (2018). https://doi.org/10.1038/s41551-018-0301-3
The Impact of Artificial Intelligence in the Endoscopic Assessment of Premalignant and Malignant Esophageal Lesions: Present and Future
Depth Information-Based Automatic Annotation of Early Esophageal Cancers in Gastroscopic Images Using Deep Learning Techniques
IEEE Access (2020)
Prediction of Pulmonary to Systemic Flow Ratio in Patients With Congenital Heart Disease Using Deep Learning–Based Analysis of Chest Radiographs
JAMA Cardiology (2020)
Automatic Polyp Recognition in Colonoscopy Images Using Deep Learning and Two-Stage Pyramidal Feature Prediction
IEEE Transactions on Automation Science and Engineering (2020)