Review Article | Published:

A new era: artificial intelligence and machine learning in prostate cancer

Nature Reviews Urology (2019) | Download Citation


Artificial intelligence (AI) — the ability of a machine to perform cognitive tasks to achieve a particular goal based on provided data — is revolutionizing and reshaping our health-care systems. The current availability of ever-increasing computational power, highly developed pattern recognition algorithms and advanced image processing software working at very high speeds has led to the emergence of computer-based systems that are trained to perform complex tasks in bioinformatics, medical imaging and medical robotics. Accessibility to ‘big data’ enables the ‘cognitive’ computer to scan billions of bits of unstructured information, extract the relevant information and recognize complex patterns with increasing confidence. Computer-based decision-support systems based on machine learning (ML) have the potential to revolutionize medicine by performing complex tasks that are currently assigned to specialists to improve diagnostic accuracy, increase efficiency of throughputs, improve clinical workflow, decrease human resource costs and improve treatment choices. These characteristics could be especially helpful in the management of prostate cancer, with growing applications in diagnostic imaging, surgical interventions, skills training and assessment, digital pathology and genomics. Medicine must adapt to this changing world, and urologists, oncologists, radiologists and pathologists, as high-volume users of imaging and pathology, need to understand this burgeoning science and acknowledge that the development of highly accurate AI-based decision-support applications of ML will require collaboration between data scientists, computer researchers and engineers.

Key points

  • Applications of machine learning (ML) to prostate cancer care are rapidly growing owing to the many technological platforms involved in its diagnosis, prognosis and treatment.

  • In diagnostic imaging, ML is applied to perform low-level image analysis tasks such as prostate segmentation and fusion of different modalities (for example MRI, CT and ultrasonography) and high-level inference and prediction tasks such as prostate cancer detection and characterization.

  • ML algorithms are able to enhance prostate cancer treatment by augmenting the surgeon’s display with information such as cancer localization during robotic procedures and other image-guided interventions and could be used towards autonomous manipulation of tools for assistance in the operating room.

  • Computer-assisted diagnosis of prostate cancer in histopathological slides could be achieved by ML in order to optimize accuracy, reproducibility and throughput and to further enhance health-care delivery by enabling the use of customized precision-care pathways.

  • ML methods are used to identify genes or groups of genes for which expression specificity to predict outcomes of prostate cancer is high and could be used for screening, developing diagnostic tools, determining optimal individualized treatment and producing targeted drug regimens.

  • Collaboration between urologists, data scientists, computer researchers and engineers is required to ensure that artificial intelligence (AI)-based decision-support applications are properly trained, operated and regulated.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.


All prices are NET prices.

Additional information

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.


  1. 1.

    Russell, S. J. & Norvig, P. Artificial Intelligence: A Modern Approach 3rd edn (Prentice Hall, 2009).

  2. 2.

    McGinnis, D. What is the fourth industrial revolution? Salesforce (2018).

  3. 3.

    Hodges, A. Beyond Turing’s machines. Science 336, 163–164 (2012).

  4. 4.

    Friedman, T. L. Thank You for Being Late: An Optimist’s Guide to Thriving in the Age of Accelerations 38–39 (Picador, 2017).

  5. 5.

    Darcy, A. M., Louie, A. K. & Roberts, L. W. Machine learning and the profession of medicine. JAMA 315, 551–552 (2016).

  6. 6.

    Duda, R. O., Hart, P. E. & Stork, D. G. Pattern Classification (John Wiley & Sons, 2001).

  7. 7.

    Bishop, C. M. Pattern Recognition and Machine Learning (Springer-Verlag New York, 2006).

  8. 8.

    Nelder, J. A. & Baker, R. J. in Encyclopedia of Statistical Sciences 2nd edn (eds Kotz, S., Read, C. B., Balakrishnan, N., Vidakovic, B. & Johnson, N. L.) (John Wiley & Sons, 2004).

  9. 9.

    Breiman, L. Random forests. Mach. Learn. 45, 5–32 (2001).

  10. 10.

    Khurd, P. et al. Computer-aided Gleason grading of prostate cancer histopathological images using texton forests. Proc. IEEE Int. Symp. Biomed. Imaging 2010, 636–639 (2010).

  11. 11.

    Doyle, S., Feldman, M. D., Shih, N., Tomaszewski, J. & Madabhushi, A. Cascaded discrimination of normal, abnormal, and confounder classes in histopathology: Gleason grading of prostate cancer. BMC Bioinformatics 13, 282 (2012).

  12. 12.

    Gorelick, L. et al. Prostate histopathology: learning tissue component histograms for cancer detection and classification. IEEE Trans Med. Imaging 32, 1804–1818 (2013).

  13. 13.

    Jolliffe, I. in Encyclopedia of Statistics in Behavioral Science Vol. 1 (eds Everitt, B. & Howell, D.) (John Wiley & Sons, 2005).

  14. 14.

    Goodfellow, I., Bengio, Y., Courville, A. & Bengio, Y. Deep Learning (MIT Press, 2016).

  15. 15.

    Seligson, D. B. et al. Global histone modification patterns predict risk of prostate cancer recurrence. Nature 435, 1262 (2005). This study uses unsupervised learning techniques to identify markers of recurrence of prostate cancer.

  16. 16.

    Thananjeyan, B. et al. in IEEE International Conference on Robotics and Automation (ICRA) 2371–2378 (IEEE, 2017).

  17. 17.

    Russ, J. C. The Image Processing Handbook 6th edn (CRC Press, 2016).

  18. 18.

    LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436 (2015). This paper provides an overview of DL and its many applications by three pioneers in the field.

  19. 19.

    Bishop, C. M. Neural Networks for Pattern Recognition (Oxford Univ. Press, 1995).

  20. 20.

    LeCun, Y. et al. Backpropagation applied to handwritten zip code recognition. Neural Comput. 1, 541–551 (1989).

  21. 21.

    Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017). This article reviews the major DL concepts pertinent to medical image analysis and summarizes over 300 contributions to the field.

  22. 22.

    Shen, D., Wu, G. & Suk, H. I. Deep learning in medical image analysis. Annu. Rev. Biomed. Eng. 19, 221–248 (2017).

  23. 23.

    Suzuki, K. Overview of deep learning in medical imaging. Radiol. Phys. Technol. 10, 257–273 (2017).

  24. 24.

    Alam, I. S. et al. Emerging intraoperative imaging modalities to improve surgical precision. Mol. Imaging Biol. 20, 705–715 (2018).

  25. 25.

    Angermueller, C., Pärnamaa, T., Parts, L. & Stegle, O. Deep learning for computational biology. Mol. Systems Biol. 12, 878 (2016).

  26. 26.

    Litjens, G. et al. Evaluation of prostate segmentation algorithms for MRI: the PROMISE12 challenge. Med. Image Anal. 18, 359–373 (2014).

  27. 27.

    Madabhushi, A. & Lee, G. Image analysis and machine learning in digital pathology: challenges and opportunities. Med. Image Anal. 33, 170–175 (2016). This paper reviews both handcrafted feature extraction and DL approaches for histopathological image analysis and discusses digital pathology as a bridge between radiology and genomics.

  28. 28.

    Nir, G. et al. Comparison of artificial intelligence techniques to evaluate performance of a classifier for automatic grading of prostate cancer from digitized histopathologic images. JAMA Netw. Open 2, e190442 (2019).

  29. 29.

    Karimi, D., Samei, G., Kesch, C., Nir, G. & Salcudean, S. E. Prostate segmentation in MRI using a convolutional neural network architecture and training strategy based on statistical shape models. Int. J. Comput. Assist. Radiol. Surg. 13, 1211–1219 (2018).

  30. 30.

    Cootes, T. F., Taylor, C. J., Cooper, D. H. & Graham, J. Active shape models-their training and application. Comput. Vis. Image Underst. 61, 38–59 (1995).

  31. 31.

    Milletari, F., Rothberg, A., Jia, J. & Sofka, M. in International Conference on Medical Image Computing and Computer-Assisted Intervention 161–168 (Springer, 2017).

  32. 32.

    Dice, L. R. Measures of the amount of ecologic association between species. Ecology 26, 297–302 (1945).

  33. 33.

    Sørensen, T. A method of establishing groups of equal amplitude in plant sociology based on similarity of species and its application to analyses of the vegetation on Danish commons. Biol. Skr. 5, 1–34 (1948).

  34. 34.

    Zeng, Q. et al. Prostate segmentation in transrectal ultrasound using magnetic resonance imaging priors. Int. J. Comput. Assist. Radiol. Surg. 13, 749–757 (2018).

  35. 35.

    Anas, E. M. A., Mousavi, P. & Abolmaesumi, P. A deep learning approach for real time prostate segmentation in freehand ultrasound guided biopsy. Med. Image Anal. 48, 107–116 (2018).

  36. 36.

    Hu, Y. et al. Weakly-supervised convolutional neural networks for multimodal image registration. Med. Image Anal. 49, 1–13 (2018).

  37. 37.

    Litjens, G., Debats, O., Barentsz, J., Karssemeijer, N. & Huisman, H. Computer-aided detection of prostate cancer in MRI. IEEE Trans. Med. Imaging 33, 1083–1092 (2014).

  38. 38.

    Moradi, M. et al. Multiparametric MRI maps for detection and grading of dominant prostate tumors. J. Magn. Reson. Imaging 35, 1403–1413 (2012).

  39. 39.

    Liu, S., Zheng, H., Feng, Y. & Li, W. Prostate cancer diagnosis using deep learning with 3D multiparametric MRI. Proc. SPIE Int. Soc. Opt. Eng. 10134, 1013428 (2017).

  40. 40.

    Karimi, D. & Ruan, D. in International Conference on Medical Image Computing and Computer-Assisted Intervention 391–398 (Springer, 2017).

  41. 41.

    Lemaître, G. et al. Computer-aided detection and diagnosis for prostate cancer based on mono and multi-parametric MRI: a review. Comput. Biol. Med. 60, 8–31 (2015).

  42. 42.

    Shiradkar, R. et al. Radiomics based targeted radiotherapy planning (Rad-TRaP): a computational framework for prostate cancer treatment planning with MRI. Radiat. Oncol. 11, 148 (2016).

  43. 43.

    Boussion, N., Valeri, A., Malhaire, J. & Visvikis, D. Predicting the number of seeds in LDR prostate brachytherapy using machine learning and 320 patients [abstract PO-0897]. Radiother. Oncol. 127, S477–S478 (2018).

  44. 44.

    Kalan, S. et al. History of robotic surgery. J. Robot. Surg. 4, 141–147 (2010).

  45. 45.

    Kassahun, Y. et al. Surgical robotics beyond enhanced dexterity instrumentation: a survey of machine learning techniques and their role in intelligent and autonomous surgical actions. Int. J. Comput. Assist. Radiol. Surg. 11, 553–568 (2016). This article discusses current and future ML applications in surgical robotics.

  46. 46.

    Yip, M. & Das, N. Robot autonomy for surgery. Preprint at arXiv (2017).

  47. 47.

    Yang, G. Z. et al. Medical robotics—regulatory, ethical, and legal considerations for increasing levels of autonomy. Sci. Robot 2, 8638 (2017).

  48. 48.

    Ji, J. J., Krishnan, S., Patel, V., Fer, D. & Goldberg, K. in 2018 IEEE 14th International Conference on Automation Science and Engineering (CASE) 35–42 (IEEE, 2018).

  49. 49.

    Mohareri, O. et al. Intraoperative registered transrectal ultrasound guidance for robot-assisted laparoscopic radical prostatectomy. J. Urol. 193, 302–312 (2015).

  50. 50.

    Mohareri, O. et al. in International Conference on Medical Image Computing and Computer-Assisted Intervention 497–504 (Springer, 2015).

  51. 51.

    Samei, G. et al. Real-time FEM-based registration of 3D to 2.5 D transrectal ultrasound images. IEEE Trans. Med. Imaging 37, 1877–1886 (2018).

  52. 52.

    Teber, D. et al. Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. Eur. Urol. 56, 332–338 (2009).

  53. 53.

    Goldenberg, M. & Lee, J. Y. Surgical education, simulation and simulators-updating the concept of validity. Curr. Urol. Rep. 19, 52 (2018).

  54. 54.

    Sarikaya, D., Corso, J. & Guru, K. Detection and localization of robotic tools in robot-assisted surgery videos using deep neural networks for region proposal and detection. IEEE Trans. Med. Imaging 36, 1542–1549 (2017).

  55. 55.

    Hung, A. J., Chen, J. & Gill, I. S. Automated performance metrics and machine learning algorithms to measure surgeon performance and anticipate clinical outcomes in robotic surgery. JAMA Surg. 153, 770–771 (2018).

  56. 56.

    Pantanowitz, L. Digital images and the future of digital pathology. J. Pathol. Inform. 1, 15 (2010).

  57. 57.

    Huisman, A., Looijen, A., van den Brink, S. M. & van Diest, P. J. Creation of a fully digital pathology slide archive by high-volume tissue slide scanning. Hum. Pathol. 41, 751–757 (2010).

  58. 58.

    Thompson, D. et al. Multimegapixel images in histopathology. Anal. Quant. Cytol. Histol 23, 169–177 (2001).

  59. 59.

    Arevalo, J. et al. Histopathology image representation for automatic analysis: a state-of-the-art review. Revista Med. 22, 79–91 (2014).

  60. 60.

    Mosquera-Lopez, C., Agaian, S., Velez-Hoyos, A. & Thompson, I. Computer aided prostate cancer diagnosis from digitized histopathology: a review on texture-based systems. IEEE Rev. Biomed. Eng. 8, 98–113 (2015).

  61. 61.

    Bhargava, R. & Madabhushi, A. Emerging themes in image informatics and molecular analysis for digital pathology. Ann. Rev. Biomed. Eng. 18, 387–412 (2016).

  62. 62.

    Jothi, J. A. & Rajam, M. A. A survey on automated cancer diagnosis from histopathology images. Artif. Intell. Rev. 48, 31–81 (2017).

  63. 63.

    Chen, P. H. C. et al. An augmented reality microscope for real-time automated detection of cancer. Google (2018).

  64. 64.

    Montironi, R., Cheng, L., Lopez-Beltran, A. & Scarpelli, M. Quantitative image analysis on histologic virtual slides for prostate pathology diagnosis, response to chemopreventive agents, and prognosis. Eur. Urol. Focus 3, 467–469 (2017).

  65. 65.

    Diamond, J., Anderson, N. H., Bartels, P. H., Montironi, R. & Hamilton, P. W. The use of morphological characteristics and texture analysis in the identification of tissue composition in prostatic neoplasia. Hum. Pathol. 35, 1121–1131 (2004).

  66. 66.

    Janowczyk, A. & Madabhushi, A. Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases. J. Pathol. Inform. 7, 29 (2016).

  67. 67.

    Litjens, G. et al. Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis. Sci. Rep. 6, 26286 (2016).

  68. 68.

    Källén, H., Molin, J., Heyden, A., Lundström, C. & Åström, K. in 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI) 1163–1167 (IEEE, 2016).

  69. 69.

    Rezaeilouyeh, H., Mollahosseini, A. & Mahoor, M. H. Microscopic medical image classification framework via deep learning and shearlet transform. J. Med. Imaging (Bellingham) 3, 044501 (2016).

  70. 70.

    Zhou, N., Fedorov, A., Fennessy, F., Kikinis, R. & Gao, Y. Large scale digital prostate pathology image analysis combining feature extraction and deep neural network. Preprint at arXiv 02678 (2017).

  71. 71.

    Bejnordi, B. E. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318, 2199–2210 (2017).

  72. 72.

    Allsbrook, W. C. Jr. et al. Interobserver reproducibility of Gleason grading of prostatic carcinoma: urologic pathologists. Hum. Pathol. 32, 74–80 (2001).

  73. 73.

    Nir, G. et al. Automatic grading of prostate cancer in digitized histopathology images: learning from multiple experts. Med. Image Anal. 50, 167–180 (2018).

  74. 74.

    Kononen, J. et al. Tissue microarrays for high-throughput molecular profiling of tumor specimens. Nat. Med. 4, 844–847 (1998).

  75. 75.

    Radford, A., Metz, L. & Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial networks. Preprint at arXiv (2015).

  76. 76.

    Hou, L. et al. Unsupervised histopathology image synthesis. Preprint at arXiv (2017).

  77. 77.

    Senaras, C., Sahiner, B., Tozbikian, G., Lozanski, G. & Gurcan, M. N. Creating synthetic digital slides using conditional generative adversarial networks: application to Ki67 staining. Proc. SPIE Int. Soc. Opt. Eng. 10581, 1058103 (2018).

  78. 78.

    Ross, A. E., D’amico, A. V. & Freedland, S. J. Which, when and why? Rational use of tissue-based molecular testing in localized prostate cancer. Prostate Cancer Prostatic Dis. 19, 1–16 (2016).

  79. 79.

    Libbrecht, M. W. & Noble, W. S. Machine learning applications in genetics and genomics. Nat. Rev. Genet. 16, 321–332 (2015). This paper provides an overview of ML applications for the analysis of genome sequencing data sets, including the annotation of sequence elements and epigenetic, proteomic or metabolomic data.

  80. 80.

    Karnes, R. J. et al. Validation of a genomic classifier that predicts metastasis following radical prostatectomy in an at risk patient population. J. Urol. 190, 2047–2053 (2013).

  81. 81.

    Nguyen, P. L. et al. Ability of a genomic classifier to predict metastasis and prostate cancer-specific mortality after radiation or surgery based on needle biopsy specimens. Eur. Urol. 72, 845–852 (2017).

  82. 82.

    Lee, S. et al. Machine learning on a genome-wide association study to predict late genitourinary toxicity after prostate radiation therapy. Int. J. Radiat. Oncol. Biol. Phys. 101, 128–135 (2018).

  83. 83.

    Lee, G. et al. Supervised multi-view canonical correlation analysis (sMVCCA): integrating histologic and proteomic features for predicting recurrent prostate cancer. IEEE Trans. Med. Imaging 34, 284–297 (2015).

  84. 84.

    Mobadersany, P. et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl Acad. Sci. USA 115, E2970–E2979 (2018).

  85. 85.

    Ren, J., Karagoz, K., Gatza, M., Foran, D. J. & Qi, X. Differentiation among prostate cancer patients with Gleason score of 7 using histopathology image and genomic data. Proc. SPIE Int. Soc. Opt. Eng. 10579, 1057904 (2018).

  86. 86.

    Rubin, M. A. & Demichelis, F. The genomics of prostate cancer: emerging understanding with technologic advances. Mod. Pathol. 31, S1–S11 (2018).

  87. 87.

    Quotes. “Computers are incredibly fast, accurate, and stupid. Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination”. (2019).

  88. 88.

    Kristiansen, G. Markers of clinical utility in the differential diagnosis and prognosis of prostate cancer. Mod. Pathol. 31, S143–S155 (2018).

Download references


This work was funded by a Prostate Cancer Canada grant (D2016-1352), by the Canadian Institutes of Health Research (MOP-142439) and by the C. A. Laszlo Chair of S.E.S. G.N. is a recipient of a Prostate Cancer Canada Postdoctoral Research Fellowship Award (PDF2016-1338).

Reviewer information

Nature Reviews Urology thanks O. Vermesh and Q. Wang for their contribution to the peer review of this work.

Author information


  1. Department of Urologic Sciences and the Vancouver Prostate Centre, University of British Columbia, Vancouver, British Columbia, Canada

    • S. Larry Goldenberg
    • , Guy Nir
    •  & Septimiu E. Salcudean
  2. Department of Electrical and Computer Engineering, University of British Columbia, Vancouver, British Columbia, Canada

    • Guy Nir
    •  & Septimiu E. Salcudean


  1. Search for S. Larry Goldenberg in:

  2. Search for Guy Nir in:

  3. Search for Septimiu E. Salcudean in:


All authors researched data for the article, made substantial contributions to discussions of content and reviewed and edited the manuscript before submission. S.L.G. and G.N. wrote the manuscript.

Competing interests

The authors declare no competing interests.

Corresponding author

Correspondence to S. Larry Goldenberg.



A classifier in machine learning refers to the placing of a new observation into the appropriate category among those categories that were based on trained data sets of known observations.

Ground truth

Labels or annotations that were determined by an expert, considered to be the correct targets and used in training, testing and evaluating machine learning algorithms.

Naive Bayes classification

A supervised learning method that is based on a probabilistic approach and relies on Bayes’ theorem. The learning process involves parameter estimation of the probability distributions based on the data. For a new sample, the class with the maximum probability would be predicted on the basis of the probability of its features.

Support vector machines

(SVMs). Supervised learning methods for classification that learn the optimal ‘separation’ between the features of each class. The predicted class of a new sample would be based on the ‘region’ in feature space that the sample features occupy.

Random forests

A supervised learning method for classification that is based on decision trees. It consists of multiple trees, each with a random subset of the features, and tries to optimize the split values of the branches.

k-Means clustering

An unsupervised learning method for clustering. The algorithm iteratively assigns each data sample into one of k classes on the basis of the ‘distances’ between features.

Principal component analysis

An unsupervised learning method for dimensionality reduction.


Types of neural networks that are trained to encode an input into a lower dimensionality such that the reverse decoder can reconstruct the encoded sample as similar as possible to the original input. These models can be used for unsupervised learning of the most descriptive features of the data, for example, for dimensionality reduction.

Artificial neural networks

A collection of units that are connected to each other (typically) as layers and inspired by biological neural networks in the brain. Each unit, also referred to as an ‘artificial neuron’, has an output that is a function of the weighted sum of its multiple inputs and is ‘activated’ if that sum is higher than a threshold (bias). Given the data as the input to the first layer, the weights and biases of the neurons are optimized to match the output of the last layer with some target (for example, minimizing a classification error).

Convolutional neural network

(CNN). A type of artificial neural network in which a neuron in a layer is connected to a few adjacent neurons from the previous layer, and the next neuron in that layer is connected to the next adjacent neurons from the previous layer and so on. Such a network architecture is commonly used in learning vision tasks such as image classification.

‘Leave-patient-out’ validation

An approach, also known as a k-fold cross-validation, to evaluate the performance of a classifier by training it on all data except the samples of one or more patient or patients and then testing it on the left outpatient or outpatients. The process can be repeated over all patients, each time with a different (subset of) patient or patients left out, and averaging the results.

Statistical shape modelling

A representation of a set of shapes by modelling their geometry with a typically small number of parameters that control their main modes of variation and are derived using statistical methods.

Sørensen–Dice similarity coefficient

A value that measures the similarity between two sets, with a value of zero when the two sets are unique and a value of one when they completely overlap. If the sets are pixels within two shapes, the Sørensen–Dice similarity coefficient can measure the amount of overlap of the shapes.

Recurrent neural network

(RNN). A type of artificial neural network in which neurons are connected to other neurons at previous time steps and can, therefore, learn temporal patterns in sequential data.

Dynamic contrast-enhanced

(DCE). A modality of MRI that measures parameters of tissue perfusion (ktrans) in the presence of a contrast agent.

Diffusion tensor imaging

(DTI). An MRI technique that maps the diffusion of water molecules in the tissue.

Generative adversarial networks

(GANs). Types of artificial neural networks in which there are two paths of layers: one that generates samples from a random input and another that tests the similarity of the generated samples to real samples. The generative network is, therefore, trained to generate samples that mimic the real samples. Such networks are used to generate synthetic images that are visually similar to real images of their class.


When a machine learning model is trained to perform well on a limited data set and performs worse when it is applied to new data, it is said to be overfitted to those data. In order to avoid overfitting, a data set that is large enough to represent the real-world diversity should be used for training and the training should be stopped before fully converging.

About this article

Publication history