Article | Published:

Development and validation of a deep-learning algorithm for the detection of polyps during colonoscopy

Nature Biomedical Engineeringvolume 2pages741748 (2018) | Download Citation


The detection and removal of precancerous polyps via colonoscopy is the gold standard for the prevention of colon cancer. However, the detection rate of adenomatous polyps can vary significantly among endoscopists. Here, we show that a machine-learning algorithm can detect polyps in clinical colonoscopies, in real time and with high sensitivity and specificity. We developed the deep-learning algorithm by using data from 1,290 patients, and validated it on newly collected 27,113 colonoscopy images from 1,138 patients with at least one detected polyp (per-image-sensitivity, 94.38%; per-image-specificity, 95.92%; area under the receiver operating characteristic curve, 0.984), on a public database of 612 polyp-containing images (per-image-sensitivity, 88.24%), on 138 colonoscopy videos with histologically confirmed polyps (per-image-sensitivity of 91.64%; per-polyp-sensitivity, 100%), and on 54 unaltered full-range colonoscopy videos without polyps (per-image-specificity, 95.40%). By using a multi-threaded processing system, the algorithm can process at least 25 frames per second with a latency of 76.80 ± 5.60 ms in real-time video analysis. The software may aid endoscopists while performing colonoscopies, and help assess differences in polyp and adenoma detection performance among endoscopists.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.


All prices are NET prices.

Data availability

The authors declare that all data supporting the findings of this study are available within the paper and its supplementary information. Restrictions apply to the availability of the medical training/ validation data, which were used with permission for the current study, and so are not publicly available. Some data may be available from the authors upon reasonable request and with permission of the Sichuan Academy of Medical Sciences & Sichuan Provincial People’s Hospital.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.


  1. 1.

    Lieberman, D. A. et al. Guidelines for colonoscopy surveillance after screening and polypectomy: a consensus update by the US Multi-Society Task Force on Colorectal Cancer. Gastroenterology 143, 844–857 (2012).

  2. 2.

    Seeff, L. C. et al. How many endoscopies are performed for colorectal cancer screening? Results from CDC’s survey of endoscopic capacity. Gastroenterology 127, 1670–1677 (2004).

  3. 3.

    Zauber, A. G. et al. Colonoscopic polypectomy and long-term prevention of colorectal-cancer deaths. N. Engl. J. Med. 366, 687–696 (2012).

  4. 4.

    Edwards, B. K. et al. Annual report to the nation on the status of cancer, 1975-2006, featuring colorectal cancer trends and impact of interventions (risk factors, screening, and treatment) to reduce future rates. Cancer 116, 544–573 (2010).

  5. 5.

    Siegel, R., Naishadham, D. & Jemal, A. Cancer statistics, 2013. CA Cancer J. Clin. 63, 11–30 (2013).

  6. 6.

    Chinese Society of Gastroenterology. Chinese consensus: screening, early diagnosis and treatment, comprehensive prevention of large bowel cancer. Chin. J. Gastroenterology Hepatology 20, 979–995 (2011).

  7. 7.

    Brenner, H., Chang-Claude, J., Seiler, C. M., Rickert, A. & Hoffmeister, M. Protection from colorectal cancer after colonoscopy: a population-based, case-control study. Ann. Intern. Med. 154, 22–30 (2011).

  8. 8.

    Corley, D. A. et al. Adenoma detection rate and risk of colorectal cancer and death. N. Engl. J. Med. 370, 1298–1306 (2014).

  9. 9.

    Ahn, S. B. et al. The miss rate for colorectal adenoma determined by quality-adjusted, back-to-back colonoscopies. Gut Liver 6, 64–70 (2012).

  10. 10.

    Marr, D. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information (Henry Holt and Co. Inc., New York, 1982).

  11. 11.

    Rex, D. K. et al. Colonoscopic miss rates of adenomas determined by back-to-back colonoscopies. Gastroenterology 112, 24–28 (1997).

  12. 12.

    Heresbach, D. et al. Miss rate for colorectal neoplastic polyps: a prospective multicenter study of back-to-back video colonoscopies. Endoscopy 40, 284–290 (2008).

  13. 13.

    Leufkens, A. et al. Factors influencing the miss rate of polyps in a back-to-back colonoscopy study. Endoscopy 44, 470–475 (2012).

  14. 14.

    Mahmud, N., Cohen, J., Tsourides, K. & Berzin, T. M. Computer vision and augmented reality in gastrointestinal endoscopy. Gastroenterol. Rep. (Oxf.) 3, 179–184 (2015).

  15. 15.

    Aslanian, H. R. et al. Nurse observation during colonoscopy increases polyp detection: a randomized prospective study. Am. J. Gastroenterol. 108, 166–172 (2013).

  16. 16.

    Lee, C. K. et al. Participation by experienced endoscopy nurses increases the detection rate of colon polyps during a screening colonoscopy: a multicenter, prospective, randomized study. Gastrointest. Endosc. 74, 1094–1102 (2011).

  17. 17.

    Buchner, A. M. et al. Trainee participation is associated with increased small adenoma detection. Gastrointest. Endosc. 73, 1223–1231 (2011).

  18. 18.

    Mori, Y. et al. Computer-aided diagnosis for colonoscopy. Endoscopy 49, 813–819 (2017).

  19. 19.

    Wang, Y. et al. Part-based multiderivative edge cross-sectional profiles for polyp detection in colonoscopy. IEEE J. Biomed. Health Inform. 18, 1379–1389 (2014).

  20. 20.

    Brandao, P. et al. Towards a computed-aided diagnosis system in colonoscopy: automatic polyp segmentation using convolution neural networks. J. Medical Robotics Res. 3, 1840002 (2018).

  21. 21.

    Chen, J. H. & Asch, S. M. Machine learning and prediction in medicine-beyond the peak of inflated expectations. N. Engl. J. Med. 376, 2507–2509 (2017).

  22. 22.

    Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402–2410 (2016).

  23. 23.

    Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

  24. 24.

    Misawa, M. et al. Artificial intelligence-assisted polyp detection for colonoscopy: initial experience. Gastroenterology 154, 2027–2029 (2018).

  25. 25.

    Gkolfakis, P. et al. New endoscopes and add-on devices to improve colonoscopy performance. World J. Gastroenterol. 23, 3784–3796 (2017).

  26. 26.

    Pogorelov, K. et al. Efficient disease detection in gastrointestinal videos–global features versus neural network. Multimedia Tools Appl. 76, 22493–22525 (2017).

  27. 27.

    Fernandez-Esparrach, G. et al. Exploring the clinical potential of an automatic colonic polyp detection method based on the creation of energy maps. Endoscopy 48, 837–842 (2016).

  28. 28.

    Wang, Y., Tavanapong, W., Wong, J., Oh, J. H. & de Groen, P. C. Polyp-Alert: near real-time feedback during colonoscopy. Comput. Methods Programs Biomed. 120, 164–179 (2015).

  29. 29.

    Kuo, S. M., Lee, B. H. & Tian, W. Real-Time Digital Signal Processing: Implementations and Applications 2nd edn, Ch. 1 (John Wiley & Sons Ltd, Chichester, 2013).

  30. 30.

    East, J. E. et al. Advanced endoscopic imaging: European Society of Gastrointestinal Endoscopy (ESGE) Technology Review. Endoscopy 48, 1029–1045 (2016).

  31. 31.

    Mori, Y. et al. Real-time use of artificial intelligence in identification of diminutive polyps during colonoscopy: a prospective study. Ann. Intern. Med. (2018).

  32. 32.

    Takemura, Y. et al. Computer-aided system for predicting the histology of colorectal tumors by using narrow-band imaging magnifying colonoscopy (with video). Gastrointest. Endosc. 75, 179–185 (2012).

  33. 33.

    Kominami, Y. et al. Computer-aided diagnosis of colorectal polyp histology by using a real-time image recognition system and narrow-band imaging magnifying colonoscopy. Gastrointest. Endosc. 83, 643–649 (2016).

  34. 34.

    Chen, P. J. et al. Accurate classification of diminutive colorectal polyps using computer-aided analysis. Gastroenterology 154, 568–575 (2018).

  35. 35.

    Byrne, M. F. et al. Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model. Gut (2017)

  36. 36.

    Yao, K. The endoscopic diagnosis of early gastric cancer. Ann. Gastroenterol. 26, 11–22 (2013).

  37. 37.

    Winawer, S. J. et al. Colorectal cancer screening: clinical guidelines and rationale. Gastroenterology 112, 594–642 (1997).

  38. 38.

    Badrinarayanan, V., Kendall, A. & Cipolla, R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Analysis Machine Intelligence 39, 2481–2495 (2017).

  39. 39.

    Dean, J. et al. Large scale distributed deep networks. Adv. Neural Inf. Processing Syst. 25, 1223–1231 (2012).

  40. 40.

    Bandos, A. I., Rockette, H. E., Song, T. & Gur, D. Area under the free-response ROC curve (FROC) and a related summary index. Biometrics 65, 247–256 (2009).

  41. 41.

    Bernal, J. et al. Comparative validation of polyp detection methods in video colonoscopy: results from the MICCAI 2015 endoscopic vision challenge. IEEE Trans. Med. Imaging 36, 1231–1249 (2017).

Download references


We thank the Machine Intelligence Laboratory of the University of Cambridge for developing SegNet and making it publicly available.

Author information


  1. Sichuan Academy of Medical Sciences & Sichuan Provincial People’s Hospital, Chengdu, China

    • Pu Wang
    • , Mengtian Tu
    • , Fei Xiong
    • , Xiao Hu
    • , Peixi Liu
    • , Yan Song
    • , Di Zhang
    • , Xue Yang
    • , Liangping Li
    •  & Xiaogang Liu
  2. Shanghai Wision AI Co., Ltd, Shanghai, China

    • Xiao Xiao
    • , Jiong He
    • , Xin Yi
    •  & Jingjia Liu
  3. Beth Israel Deaconess Medical Center and Harvard Medical School, Center for Advanced Endoscopy, Boston , MA, USA

    • Jeremy R. Glissen Brown
    •  & Tyler M. Berzin


  1. Search for Pu Wang in:

  2. Search for Xiao Xiao in:

  3. Search for Jeremy R. Glissen Brown in:

  4. Search for Tyler M. Berzin in:

  5. Search for Mengtian Tu in:

  6. Search for Fei Xiong in:

  7. Search for Xiao Hu in:

  8. Search for Peixi Liu in:

  9. Search for Yan Song in:

  10. Search for Di Zhang in:

  11. Search for Xue Yang in:

  12. Search for Liangping Li in:

  13. Search for Jiong He in:

  14. Search for Xin Yi in:

  15. Search for Jingjia Liu in:

  16. Search for Xiaogang Liu in:


P.W. and X.L. contributed to study concept and design. P.W., X.L., L.L., M.T., F.X., X.H., P.L., Y.S., D.Z. and X.Yang. contributed to acquisition of data and statistical analysis. P.W., T.M.B. and J.R.G.B contributed to analysis, interpretation of data and drafting of the manuscript. X.X. contributed to algorithm development. J.L., J.H. and X.Yi. contributed to algorithm and software/hardware implementation. All authors read and approved the final manuscript.

Competing interests

X.X., J.L., J.H. and X.Y. are employees of Shanghai Wision AI Co., Ltd. The automatic polyp detection system was developed by the company and the software was provided free of charge for the purposes of this study. All other authors declare no competing interests.

Corresponding author

Correspondence to Xiaogang Liu.

Supplementary information

  1. Supplementary Information

    Supplementary figures and video captions.

  2. Reporting Summary

  3. Supplementary Video 1

    Real-time visual assistance during colonoscopy on an adjacent monitor.

  4. Supplementary Video 2

    Additional video of real-time visual assistance during colonoscopy on an adjacent monitor.

  5. Supplementary Video 3

    Sample video from the simulated real-time video analysis.

  6. Supplementary Video 4

    Additional sample video from the simulated real-time video analysis.

  7. Supplementary Video 5

    Demonstration of simulated real-time video analysis on datasets C and D.

About this article

Publication history





Further reading