Recent advances in computer vision and machine learning underpin a collection of algorithms with an impressive ability to decipher the content of images. These deep learning algorithms are being applied to biological images and are transforming the analysis and interpretation of imaging data. These advances are positioned to render difficult analyses routine and to enable researchers to carry out new, previously impossible experiments. Here we review the intersection between deep learning and cellular image analysis and provide an overview of both the mathematical mechanics and the programming frameworks of deep learning that are pertinent to life scientists. We survey the field’s progress in four key applications: image classification, image segmentation, object tracking, and augmented microscopy. Last, we relay our labs’ experience with three key aspects of implementing deep learning in the laboratory: annotating training data, selecting and training a range of neural network architectures, and deploying solutions. We also highlight existing datasets and implementations for each surveyed application.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
Discover Artificial Intelligence Open Access 30 January 2023
Scientific Reports Open Access 14 December 2022
Acta Neuropathologica Communications Open Access 04 November 2022
Subscribe to Nature+
Get immediate online access to Nature and 55 other Nature journal
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
Links to the data referred to in this Review can be found in Table 2.
Grimm, J. B. et al. A general method to fine-tune fluorophores for live-cell and in vivo imaging. Nat. Methods 14, 987–994 (2017).
Liu, H. et al. Visualizing long-term single-molecule dynamics in vivo by stochastic protein labeling. Proc. Natl. Acad. Sci. USA 115, 343–348 (2018).
Regot, S., Hughey, J. J., Bajar, B. T., Carrasco, S. & Covert, M. W. High-sensitivity measurements of multiple kinase activities in live single cells. Cell 157, 1724–1734 (2014).
Sampattavanich, S. et al. Encoding growth factor identity in the temporal dynamics of FOXO3 under the combinatorial control of ERK and AKT kinases. Cell Syst. 6, 664–678 (2018).
Megason, S. G. In toto imaging of embryogenesis with confocal time-lapse microscopy. Methods Mol. Biol. 546, 317–332 (2009).
Udan, R. S., Piazza, V. G., Hsu, C. W., Hadjantonakis, A.-K. & Dickinson, M. E. Quantitative imaging of cell dynamics in mouse embryos using light-sheet microscopy. Development 141, 4406–4414 (2014).
Chen, B.-C. et al. Lattice light-sheet microscopy: imaging molecules to embryos at high spatiotemporal resolution. Science 346, 1257998 (2014).
Royer, L. A. et al. Adaptive light-sheet microscopy for long-term, high-resolution imaging in living organisms. Nat. Biotechnol. 34, 1267–1278 (2016).
McDole, K. et al. In toto imaging and reconstruction of post-implantation mouse development at the single-cell level. Cell 175, 859–876 (2018).
Shah, S., Lubeck, E., Zhou, W. & Cai, L. seqFISH accurately detects transcripts in single cells and reveals robust spatial organization in the hippocampus. Neuron 94, 752–758 (2017).
Keren, L. et al. A structured tumor-immune microenvironment in triple negative breast cancer revealed by multiplexed ion beam imaging. Cell 174, 1373–1387 (2018).
Lin, J.-R. et al. Highly multiplexed immunofluorescence imaging of human tissues and tumors using t-CyCIF and conventional optical microscopes. eLife 7, e31657 (2018).
Caicedo, J. C. et al. Data-analysis strategies for image-based cell profiling. Nat. Methods 14, 849–863 (2017).
van der Walt, S., Colbert, S. C. & Varoquaux, G. The NumPy array: a structure for efficient numerical computation. Comput. Sci. Eng. 13, 22–30 (2011).
Jones, E. et al. SciPy: open source scientific tools for Python. https://www.scipy.org/ (2001).
McKinney, W. Data structures for statistical computing in Python. In Proc. 9th Python in Science Conference (eds. van der Walt, S. & Millman, J.) 51–56 (SciPy, 2010).
van der Walt, S. et al. scikit-image: image processing in Python. PeerJ 2, e453 (2014).
Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).
Hunter, J. D. Matplotlib: a 2D graphics environment. Comput. Sci. Eng. 9, 90–95 (2007).
Kluyver, T. et al. Jupyter Notebooks—a publishing format for reproducible computational workflows. In Positioning and Power in Academic Publishing: Players, Agents and Agendas (eds. Loizides, F. & Schmidt, B.) 87–90 (IOS Press, 2016).
Stylianidou, S., Brennan, C., Nissen, S. B., Kuwada, N. J. & Wiggins, P. A. SuperSegger: robust image segmentation, analysis and lineage tracking of bacterial cells. Mol. Microbiol. 102, 690–700 (2016).
Paintdakhi, A. et al. Oufti: an integrated software package for high-accuracy, high-throughput quantitative microscopy analysis. Mol. Microbiol. 99, 767–777 (2016).
Ursell, T. et al. Rapid, precise quantification of bacterial cellular dimensions across a genomic-scale knockout library. BMC Biol. 15, 17 (2017).
Carpenter, A. E. et al. CellProfiler: image analysis software for identifying and quantifying cell phenotypes. Genome Biol. 7, R100 (2006).
McQuin, C. et al. CellProfiler 3.0: next-generation image processing for biology. PLoS Biol. 16, e2005970 (2018).
Sommer, C., Straehle, C., Köthe, U. & Hamprecht, F. A. Ilastik: interactive learning and segmentation toolkit. In 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro (Wright, S., Pan, X. & Liebling, M.) 230–233 (IEEE, 2011).
Belevich, I., Joensuu, M., Kumar, D., Vihinen, H. & Jokitalo, E. Microscopy Image Browser: a platform for segmentation and analysis of multidimensional datasets. PLoS Biol. 14, e1002340 (2016).
Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676–682 (2012).
Allan, C. et al. OMERO: flexible, model-driven data management for experimental biology. Nat. Methods 9, 245–253 (2012).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Krizhevsky, A., Sutskever, I. & Hinton, G.E. ImageNet classification with deep convolutional neural networks. In Proc. 25th International Conference on Neural Information Processing Systems (eds. Pereira, F. et al.) 1090–1098 (Curran Associates, 2012).
Carrasquilla, J. & Melko, R. G. Machine learning phases of matter. Nat. Phys. 13, 431–434 (2017).
Nguyen, T. Q. et al. Topology classification with deep learning to improve real-time event selection at the LHC. Preprint available at https://arxiv.org/abs/1807.00083 (2018).
Castelvecchi, D. Artificial intelligence called in to tackle LHC data deluge. Nature 528, 18–19 (2015).
Ramsundar, B. et al. Massively multitask networks for drug discovery. Preprint available at http://arxiv.org/abs/1502.02072 (2015).
Feinberg, E. N. et al. Spatial graph convolutions for drug discovery. Preprint available at http://arxiv.org/abs/1803.04465 (2018).
Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559–1567 (2018).
Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).
Poplin, R. et al. A universal SNP and small-indel variant caller using deep neural networks. Nat. Biotechnol. 36, 983–987 (2018).
Zhou, J. et al. Deep learning sequence-based ab initio prediction of variant effects on expression and disease risk. Nat. Genet. 50, 1171–1179 (2018).
Alipanahi, B., Delong, A., Weirauch, M. T. & Frey, B. J. Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning. Nat. Biotechnol. 33, 831–838 (2015).
Angermueller, C., Pärnamaa, T., Parts, L. & Stegle, O. Deep learning for computational biology. Mol. Syst. Biol. 12, 878 (2016).
Falk, T. et al. U-Net: deep learning for cell counting, detection, and morphometry. Nat. Methods 16, 67–70 (2019).
Karpathy, A. Software 2.0. Medium https://medium.com/@karpathy/software-2-0-a64152b37c35 (2017).
Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017).
Xing, F., Xie, F., Su, H., Liu, F. & Yang, L. Deep learning in microscopy image analysis: a survey. IEEE Trans. Neural Netw. Learn. Syst. 29, 4550–4568 (2018).
Smith, K. et al. Phenotypic image analysis software tools for exploring and understanding big image data from cell-based assays. Cell Syst. 6, 636–653 (2018).
Van Valen, D. A. et al. Deep learning automates the quantitative analysis of individual cells in live-cell imaging experiments. PLOS Comput. Biol. 12, e1005177 (2016).
Cireşan, D. C., Meier, U., Gambardella, L. M. & Schmidhuber, J. Deep, big, simple neural nets for handwritten digit recognition. Neural Comput. 22, 3207–3220 (2010).
Zhang, W. et al. Deep model based transfer and multi-task learning for biological image analysis. IEEE Trans. Big Data https://doi.org/10.1109/TBDATA.2016.2573280 (2016).
Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? In Proc. 27th International Conference on Neural Information Processing Systems (eds. Ghahramani, Z. et al.) 3320–3328 (MIT Press, 2014).
Caicedo, J.C. et al. Evaluation of deep learning strategies for nucleus segmentation in fluorescence images. Preprint available at https://www.biorxiv.org/content/early/2018/06/16/335216 (2018).
Newby, J. M., Schaefer, A. M., Lee, P. T., Forest, M. G. & Lai, S. K. Convolutional neural networks automate detection for tracking of submicron-scale particles in 2D and 3D. Proc. Natl. Acad. Sci. USA 115, 9026–9031 (2018).
Sadanandan, S. K., Ranefall, P., Le Guyader, S. & Wählby, C. Automated training of deep convolutional neural networks for cell segmentation. Sci. Rep. 7, 7860 (2017).
Chen, J. et al. The Allen Cell Structure Segmenter: a new open source toolkit for segmenting 3D intracellular structures in fluorescence microscopy images. Preprint available at https://www.biorxiv.org/content/early/2018/12/08/491035 (2018).
Hughes, A. J. et al. Quanti.us: a tool for rapid, flexible, crowd-based annotation of images. Nat. Methods 15, 587–590 (2018).
Sullivan, D. P. et al. Deep learning is combined with massive-scale citizen science to improve large-scale image classification. Nat. Biotechnol. 36, 820–828 (2018).
Abadi, M. et al. TensorFlow: a system for large-scale machine learning. In Proc. 12th USENIX Conference on Operating Systems Design and Implementation (eds. Keeton, K. & Roscoe, T.) 265–283 (USENIX Association, 2016).
Chollet, F. Keras. GitHub https://github.com/keras-team/keras (2015).
Paszke, A. et al. Automatic differentiation in PyTorch. Oral presentation at NIPS 2017 Workshop on Automatic Differentiation, Long Beach, CA, USA, 9 December 2017.
Chen, T. et al. MXNet: a flexible and efficient machine learning library for heterogeneous distributed systems. Preprint available at http://arxiv.org/abs/1512.01274 (2015).
Seide, F. & Agarwal, A. CNTK: Microsoft’s open-source deep-learning toolkit. In Proc. 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (eds. Krishnapuram, B. et al.) 2135 (ACM, 2016).
Bergstra, J. et al. Theano: deep learning on GPUs with Python. Paper presented at Big Learning 2011: NIPS 2011 Workshop on Algorithms, Systems, and Tools for Learning at Scale, Sierra Nevada, Spain, 16–17 December 2011.
Jia, Y. et al. Caffe: convolutional architecture for fast feature embedding. In Proc. 22nd ACM International Conference on Multimedia (eds. Hua, K. A. et al.) 675–678 (ACM, 2014).
Jouppi, N. P. et al. In-datacenter performance analysis of a tensor processing unit. In Proc. 44th Annual International Symposium on Computer Architecture (eds. Moshovos, A. et al.) 1–12 (ACM, 2017).
Owens, J. D. et al. GPU computing. Proc. IEEE 96, 879–899 (2008).
Chetlur, S. et al. cuDNN: efficient primitives for deep learning. Preprint available at http://arxiv.org/abs/1410.0759 (2014).
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. 29th IEEE Conference on Computer Vision and Pattern Recognition (eds. Agapito, L. et al.) 770–778 (IEEE, 2016).
Huang, G., Liu, Z., van der Maaten, L. & Weinberger, K. Q. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (eds. Liu, Y. et al.) 2261–2269 (IEEE, 2017).
Pelt, D. M. & Sethian, J. A. A mixed-scale dense convolutional neural network for image analysis. Proc. Natl. Acad. Sci. USA 115, 254–259 (2018).
Bishop, C. M. Pattern Recognition and Machine Learning (Information Science and Statistics) (Springer-Verlag, 2006).
Ebrahimi, M. S. & Abadi, H. K. Study of residual networks for image recognition. Preprint available at http://arxiv.org/abs/1805.00325 (2018).
Richardson, L. & Ruby, S. RESTful Web Services (O’Reilly Media, 2007).
Merkel, D. Docker: lightweight Linux containers for consistent development and deployment. Linux J. 2014, 2 (2014).
Haberl, M. G. et al. CDeep3M-Plug-and-Play cloud-based deep learning for image segmentation. Nat. Methods 15, 677–680 (2018).
Pawlowski, N., Caicedo, J. C., Singh, S., Carpenter, A. E. & Storkey, A. Automating morphological profiling with generic deep convolutional networks. Preprint available at https://www.biorxiv.org/content/early/2016/11/02/085118 (2016).
Godinez, W. J., Hossain, I., Lazic, S. E., Davies, J. W. & Zhang, X. A multi-scale convolutional neural network for phenotyping high-content cellular images. Bioinformatics 33, 2010–2019 (2017).
Kandaswamy, C., Silva, L. M., Alexandre, L. A. & Santos, J. M. High-content analysis of breast cancer using single-cell deep transfer learning. J. Biomol. Screen. 21, 252–259 (2016).
Sommer, C., Hoefler, R., Samwer, M. & Gerlich, D. W. A deep learning and novelty detection framework for rapid phenotyping in high-content screening. Mol. Biol. Cell 28, 3428–3436 (2017).
Simm, J. et al. Repurposing high-throughput image assays enables biological activity prediction for drug discovery. Cell Chem. Biol. 25, 611–618 (2018).
Buggenthin, F. et al. Prospective identification of hematopoietic lineage choice by deep learning. Nat. Methods 14, 403–406 (2017).
Kraus, O. Z., Ba, J. L. & Frey, B. J. Classifying and segmenting microscopy images with deep multiple instance learning. Bioinformatics 32, i52–i59 (2016).
Kraus, O. Z. et al. Automated analysis of high-content microscopy data with deep learning. Mol. Syst. Biol. 13, 924 (2017).
Pärnamaa, T. & Parts, L. Accurate classification of protein subcellular localization from high-throughput microscopy images using deep learning. G3 (Bethesda) 7, 1385–1392 (2017).
Nitta, N. et al. Intelligent image-activated cell sorting. Cell 175, 266–276 (2018).
Ronneberger, O., Fischer, P. & Brox, T. U-Net: convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015 (eds. Navab, N. et al.) 234–241 (Springer, 2015).
Bai, M. & Urtasun, R. Deep watershed transform for instance segmentation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (eds. Liu, Y. et al.) 2858–2866 (IEEE, 2017).
Wang, W. et al. Learn to segment single cells with deep distance estimator and deep cell detector. Preprint available at https://arxiv.org/abs/1803.10829 (2018).
Ren, S., He, K., Girshick, R. & Sun, J. Faster R-CNN: towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems 28 (eds. Cortes, C. et al.) 91–99 (Curran Associates, 2015).
Lin, T., Goyal, P., Girshick, R., He, K. & Dollar, P. Focal loss for dense object detection. In 2017 IEEE International Conference on Computer Vision (ICCV) (eds. Ikeuchi, K. et al.) 2999–3007 (IEEE, 2018).
He, K., Gkioxari, G., Dollar, P. & Girshick, R. Mask R-CNN. In 2017 IEEE International Conference on Computer Vision (ICCV) (eds. Ikeuchi, K. et al.) 2980–2988 (IEEE, 2018).
Johnson, J. W. Adapting Mask-RCNN for automatic nucleus segmentation. Preprint available at http://arxiv.org/abs/1805.00500 (2018).
Tsai, H.-F., Gajda, J., Sloan, T. F. W., Rares, A. & Shen, A. Q. Usiigaci: instance-aware cell tracking in stain-free phase contrast microscopy enabled by machine learning. Preprint available at https://www.biorxiv.org/content/early/2019/01/18/524041 (2019).
Hollandi, R. et al. A deep learning framework for nucleus segmentation using image style transfer. Preprint available at https://www.biorxiv.org/content/10.1101/580605v1 (2019).
De Brabandere, B., Neven, D. & Van Gool, L. Semantic instance segmentation with a discriminative loss function. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (eds. Liu, Y. et al.) 478–480 (IEEE, 2017).
Payer, C., Štern, D., Neff, T., Bischof, H. & Urschler, M. Instance segmentation and tracking with cosine embeddings and recurrent hourglass networks. In Medical Image Computing and Computer Assisted Intervention—MICCAI 2018 (eds. Frangi, A. F. et al.) 3–11 (Springer, 2018).
Zhu, J.-Y., Park, T., Isola, P. & Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. Preprint available at http://arxiv.org/abs/1703.10593 (2017).
Haering, M., Grosshans, J., Wolf, F. & Eule, S. Automated segmentation of epithelial tissue using cycle-consistent generative adversarial networks. Preprint available at https://www.biorxiv.org/content/early/2018/04/30/311373 (2018).
Mahmood, F. et al. Deep adversarial training for multi-organ nuclei segmentation in histopathology images. Preprint available at http://arxiv.org/abs/1810.00236 (2018).
Tokuoka, Y. et al. Convolutional neural network-based instance segmentation algorithm to acquire quantitative criteria of early mouse development. Preprint available at https://www.biorxiv.org/content/early/2018/06/01/324186 (2018).
Januszewski, M. et al. High-precision automated reconstruction of neurons with flood-filling networks. Nat. Methods 15, 605–610 (2018).
Li, P. H. et al. Automated reconstruction of a serial-section EM Drosophila brain with flood-filling networks and local realignment. Preprint at https://www.biorxiv.org/content/10.1101/605634v1 (2019).
Booz Allen Hamilton. 2018 Data Science Bowl. Kaggle https://www.kaggle.com/c/data-science-bowl-2018 (2018).
Facchetti, G., Knapp, B., Flor-Parra, I., Chang, F. & Howard, M. Reprogramming Cdr2-dependent geometry-based cell size control in fission yeast. Curr. Biol. 29, 350–358 (2019).
Khoshdeli, M., Winkelmaier, G. & Parvin, B. Fusion of encoder-decoder deep networks improves delineation of multiple nuclear phenotypes. BMC Bioinforma. 19, 294 (2018).
Kumar, N. et al. A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE Trans. Med. Imaging 36, 1550–1560 (2017).
Regev, A. et al. The Human Cell Atlas. eLife 6, e27041 (2017).
Rozenblatt-Rosen, O., Stubbington, M. J. T., Regev, A. & Teichmann, S. A. The Human Cell Atlas: from vision to reality. Nature 550, 451–453 (2017).
Purvis, J. E. & Lahav, G. Encoding and decoding cellular information through signaling dynamics. Cell 152, 945–956 (2013).
Kimmel, J. C., Chang, A. Y., Brack, A. S. & Marshall, W. F. Inferring cell state by quantitative motility analysis reveals a dynamic state system and broken detailed balance. PLoS Comput. Biol. 14, e1005927 (2018).
Wang, P. et al. Robust growth of Escherichia coli. Curr. Biol. 20, 1099–1103 (2010).
Dow, J. A., Lackie, J. M. & Crocket, K. V. A simple microcomputer-based system for real-time analysis of cell behaviour. J. Cell Sci. 87, 171–182 (1987).
Levine, M. D., Youssef, Y. M., Noble, P. B. & Boyarsky, A. The quantification of blood cell motion by a method of automatic digital picture processing. IEEE Trans. Pattern Anal. Mach. Intell. PAMI-2, 444–450 (1980).
Smal, I., Niessen, W. & Meijering, E. Bayesian tracking for fluorescence microscopic imaging. In 3rd IEEE International Symposium on Biomedical Imaging: Macro to Nano, 2006 (eds. Kovačević, J. et al.) 550–553 (IEEE, 2006).
Godinez, W. et al. Tracking of virus particles in time-lapse fluorescence microscopy image sequences. In 2007 4th IEEE International Symposium on Biomedical Imaging: From Nano to Macro (eds. Fessler, J. et al.) 256–259 (IEEE, 2007).
Ngoc, S. N., Briquet-Laugier, F., Boulin, C. & Olivo, J.-C. Adaptive detection for tracking moving biological objects in video microscopy sequences. In Proc. International Conference on Image Processing (eds. Chang, S.-F. et al.) 484–487 (IEEE, 1997).
Kachouie, N. N. & Fieguth, P. W. Extended-Hungarian-JPDA: exact single-frame stem cell tracking. IEEE Trans. Biomed. Eng. 54, 2011–2019 (2007).
Meijering, E., Dzyubachyk, O., Smal, I. & van Cappellen, W. A. Tracking in cell and developmental biology. Semin. Cell Dev. Biol. 20, 894–902 (2009).
Jaqaman, K. et al. Robust single-particle tracking in live-cell time-lapse sequences. Nat. Methods 5, 695–702 (2008).
Tinevez, J.-Y. et al. TrackMate: an open and extensible platform for single-particle tracking. Methods 115, 80–90 (2017).
Cooper, S., Barr, A. R., Glen, R. & Bakal, C. NucliTrack: an integrated nuclei tracking application. Bioinformatics 33, 3320–3322 (2017).
Magnusson, K. E. G., Jalden, J., Gilbert, P. M. & Blau, H. M. Global linking of cell tracks using the Viterbi algorithm. IEEE Trans. Med. Imaging 34, 911–929 (2015).
Amat, F. et al. Fast, accurate reconstruction of cell lineages from large-scale fluorescence microscopy data. Nat. Methods 11, 951–958 (2014).
Akram, S. U., Kannala, J., Eklund, L. & Heikkilä, J. Cell tracking via proposal generation and selection. Preprint available at https://arxiv.org/abs/1705.03386 (2017).
Cireşan, D. C., Giusti, A., Gambardella, L. M. & Schmidhuber, J. Mitosis detection in breast cancer histology images with deep neural networks. In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2013 (eds. Mori, K. et al.) 411–418 (Springer, 2013).
Nie, W.-Z., Li, W.-H., Liu, A.-A., Hao, T. & Su, Y.-T. 3D convolutional networks-based mitotic event detection in time-lapse phase contrast microscopy image sequences of stem cell populations. In 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (eds. Agapito, L. et al.) 55–62 (IEEE, 2016).
Mao, Y. & Yin, Z. A hierarchical convolutional neural network for mitosis detection in phase-contrast microscopy images. In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2016 (eds. Ourselin, S. et al.) 685–692 (Springer, 2016).
Mathis, A. et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 21, 1281–1289 (2018).
Pereira, T. D. et al. Fast animal pose estimation using deep neural networks. Nat. Methods 16, 117–125 (2019).
Romero-Ferrero, F., Bergomi, M. G., Hinz, R. C., Heras, F. J. H. & de Polavieja, G. G. idtracker.ai: tracking all individuals in small or large collectives of unmarked animals. Nat. Methods 16, 179–182 (2019).
Gordon, D., Farhadi, A. & Fox, D. Re3 : real-time recurrent regression networks for visual tracking of generic objects. IEEE Robot. Autom. Lett. 3, 788–795 (2018).
Cui, Z., Xiao, S., Feng, J. & Yan, S. Recurrently target-attending tracking. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (eds. Agapito, L. et al.) 1449–1458 (IEEE, 2016).
Wang, Y., Mao, H. & Yi, Z. Stem cell motion-tracking by using deep neural networks with multi-output. Neural Comput. Appl. https://doi.org/10.1007/s00521-017-3291-2 (2017).
Sadeghian, A., Alahi, A. & Savarese, S. Tracking the untrackable: learning to track multiple cues with long-term dependencies. In 2017 IEEE International Conference on Computer Vision (eds. Ikeuchi, K. et al.) 300–311 (IEEE, 2017).
Zhang, D., Maei, H., Wang, X. & Wang, Y.-F. Deep reinforcement learning for visual object tracking in videos. Preprint available at http://arxiv.org/abs/1701.08936 (2017).
Wen, C. et al. Deep-learning-based flexible pipeline for segmenting and tracking cells in 3D image time series for whole brain imaging. Preprint available at https://www.biorxiv.org/content/early/2018/08/06/385567 (2018).
Sullivan, D. P. & Lundberg, E. Seeing more: a future of augmented microscopy. Cell 173, 546–548 (2018).
Ounkomol, C. et al. Three dimensional cross-modal image inference: label-free methods for subcellular structure prediction. Preprint available at https://www.biorxiv.org/content/10.1101/216606v4 (2017).
Christiansen, E. M. et al. In silico labeling: predicting fluorescent labels in unlabeled images. Cell 173, 792–803 (2018).
Johnson, G. R., Donovan-Maiye, R. M. & Maleckar, M. M. Building a 3D integrated cell. Preprint available at https://www.biorxiv.org/content/early/2017/12/21/238378 (2017).
Osokin, A., Chessel, A., Salas, R. E. C. & Vaggi, F. GANs for biological image synthesis. In 2017 IEEE International Conference on Computer Vision (eds. Ikeuchi, K. et al.) 2252–2261 (IEEE, 2017).
Ounkomol, C., Seshamani, S., Maleckar, M. M., Collman, F. & Johnson, G. R. Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy. Nat. Methods 15, 917–920 (2018).
Johnson, G., Donovan-Maiye, R., Ounkomol, C. & Maleckar, M. M. Studying stem cell organization using “label-free” methods and a novel generative adversarial model. Biophys. J. 114, 43a (2018).
Stumpe, M. & Mermel, C. An augmented reality microscope for cancer detection. Google AI Blog https://ai.googleblog.com/2018/04/an-augmented-reality-microscope.html (2018).
Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Preprint available at https://www.preprints.org/manuscript/201812.0137/v1 (2018).
Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Preprint available at https://www.biorxiv.org/content/early/2018/07/03/236463 (2018).
Wang, H. et al. Deep learning achieves super-resolution in fluorescence microscopy. Preprint available at https://www.biorxiv.org/content/early/2018/04/27/309641 (2018).
Rivenson, Y. et al. Deep learning microscopy. Optica 4, 1437–1443 (2017).
Angelo, M. et al. Multiplexed ion beam imaging of human breast tumors. Nat. Med. 20, 436–442 (2014).
Acuna, D., Ling, H., Kar, A. & Fidler, S. Efficient interactive annotation of segmentation datasets with Polygon-RNN++. Preprint available at http://arxiv.org/abs/1803.09693 (2018).
Zoph, B. & Le, Q.V. Neural architecture search with reinforcement learning. Preprint available at http://arxiv.org/abs/1611.01578 (2016).
Zoph, B., Vasudevan, V., Shlens, J. & Le, Q.V. Learning transferable architectures for scalable image recognition. In Proc. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (eds. Forsyth, D. et al.) 8697–8710 (IEEE, 2018).
Jackson, A. S., Bulat, A., Argyriou, V. & Tzimiropoulos, G. Large pose 3D face reconstruction from a single image via direct volumetric CNN regression. Preprint available at http://arxiv.org/abs/1703.07834 (2017).
Nair, V. & Hinton, G. E. Rectified linear units improve restricted Boltzmann machines. In Proc. 27th International Conference on Machine Learning (eds. Fürnkranz, J. & Joachims, T.) 807–814 (Omnipress, 2010).
Li, H., Zhao, R. & Wang, X. Highly efficient forward and backward propagation of convolutional neural networks for pixelwise classification. Preprint available at http://arxiv.org/abs/1412.4526 (2014).
Chollet, F. Xception: deep learning with depthwise separable convolutions. In Proc. 30th IEEE Conference on Computer Vision and Pattern Recognition (eds. Liu, Y. et al.) 1800–1807 (IEEE, 2017).
Howard, A. G. et al. MobileNets: efficient convolutional neural networks for mobile vision applications. Preprint available at https://arxiv.org/abs/1704.04861v1 (2017).
Lin, T. et al. Feature pyramid networks for object detection. In Proc. 30th IEEE Conference on Computer Vision and Pattern Recognition (eds. Liu, Y. et al.) 936–944 (IEEE, 2017).
Ioffe, S. & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. Preprint available at http://arxiv.org/abs/1502.03167 (2015).
Santurkar, S., Tsipras, D., Ilyas, A. & Madry, A. How does batch normalization help optimization? (No, it is not about internal covariate shift). Preprint available at http://arxiv.org/abs/1805.11604 (2018).
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014).
Li, X., Chen, S., Hu, X. & Yang, J. Understanding the disharmony between dropout and batch normalization by variance shift. Preprint available at http://arxiv.org/abs/1801.05134 (2018).
Bannon, D. et al. DeepCell 2.0: automated cloud deployment of deep learning models for large-scale cellular image analysis. Preprint available at https://www.biorxiv.org/content/early/2018/12/22/505032 (2018).
Thul, P. J. et al. A subcellular map of the human proteome. Science 356, eaal3321 (2017).
Ljosa, V., Sokolnicki, K. L. & Carpenter, A. E. Annotated high-throughput microscopy image sets for validation. Nat. Methods 9, 637 (2012).
Maška, M. et al. A benchmark for comparison of cell tracking algorithms. Bioinformatics 30, 1609–1617 (2014).
He, K., Zhang, X., Ren, S. & Sun, J. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In Proc. 2015 IEEE International Conference on Computer Vision (eds. Bajcsy, R. et al.) 1026–1034 (IEEE, 2015).
Polyak, B. T. Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4, 1–17 (1964).
Nesterov, Y. E. A method for solving the convex programming problem with convergence rate O (1/k2). Dokl. Akad. Nauk SSSR 269, 543–547 (1983).
Sutskever, I., Martens, J., Dahl, G. & Hinton, G. On the importance of initialization and momentum in deep learning. Proc. Mach. Learn. Res. 28, 1139–1147 (2013).
Tieleman, T. & Hinton, G. Neural Networks for Machine Learning lecture 6.5—rmsprop: divide the gradient by a running average of its recent magnitude. Coursera https://www.coursera.org/learn/neural-networks (2012).
Duchi, J., Hazan, E. & Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12, 2121–2159 (2011).
Zeiler, M. D. ADADELTA: an adaptive learning rate method. Preprint available at http://arxiv.org/abs/1212.5701 (2012).
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint available at http://arxiv.org/abs/1412.6980 (2014).
Wilson, A. C., Roelofs, R., Stern, M., Srebro, N. & Recht, B. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems 30 (eds. Guyon, I. et al.) 4148–4158 (Curran Associates, Inc., 2017).
Keskar, N. S. & Socher, R. Improving generalization performance by switching from Adam to SGD. Preprint available at http://arxiv.org/abs/1712.07628 (2017).
Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by back-propagating errors. Nature 323, 533–536 (1986).
Sjoberg, J. & Ljung, L. Overtraining, regularization and searching for a minimum, with application to neural networks. Int. J. Control 62, 1391–1407 (1995).
Ting, K. M. Confusion matrix. In Encyclopedia of Machine Learning and Data Mining (eds. Sammut, C. & Webb, G. I.) 260–260 (Springer, 2017).
Bajcsy, P. et al. Survey statistics of automated segmentations applied to optical imaging of mammalian cells. BMC Bioinforma. 16, 330 (2015).
Sokolova, M. & Lapalme, G. A systematic analysis of performance measures for classification tasks. Inf. Process. Manag. 45, 427–437 (2009).
Everingham, M., Van Gool, L., Williams, C. K., Winn, J. & Zisserman, A. The Pascal Visual Object Classes (voc) challenge. Int. J. Comput. Vis. 88, 303–338 (2010).
Kotila, M. Hyperparameter Optimization for Keras Models (Autonomio, 2018).
We thank A. Anandkumar, M. Angelo, L. Cai, S. Cooper, M. Elowitz, K.C. Huang, G. Johnson, A. Karpathy, L. Keren, A. Raj, T. Vora, and R. Wollman for helpful discussions and comments. This work was supported by several funding sources, including the Allen Discovery Center (award supporting W.G.; award supporting T.K., M.C., and D.V.V.), the Burroughs Wellcome Fund Postdoctoral Enrichment Program, a Figure Eight AI for Everyone award, and the NIH (subaward U24CA224309-01 to D.V.V.).
The authors declare no competing interests.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Moen, E., Bannon, D., Kudo, T. et al. Deep learning for cellular image analysis. Nat Methods 16, 1233–1246 (2019). https://doi.org/10.1038/s41592-019-0403-1
This article is cited by
Nature Nanotechnology (2023)
Nature Protocols (2023)
Discover Artificial Intelligence (2023)
Organoids revealed: morphological analysis of the profound next generation in-vitro model with artificial intelligence
Bio-Design and Manufacturing (2023)
Acta Neuropathologica Communications (2022)