Abstract

We present a combined report on the results of three editions of the Cell Tracking Challenge, an ongoing initiative aimed at promoting the development and objective evaluation of cell segmentation and tracking algorithms. With 21 participating algorithms and a data repository consisting of 13 data sets from various microscopy modalities, the challenge displays today's state-of-the-art methodology in the field. We analyzed the challenge results using performance measures for segmentation and tracking that rank all participating methods. We also analyzed the performance of all of the algorithms in terms of biological measures and practical usability. Although some methods scored high in all technical aspects, none obtained fully correct solutions. We found that methods that either take prior information into account using learning strategies or analyze cells in a global spatiotemporal video context performed better than other methods under the segmentation and tracking scenarios included in the challenge.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    , & Cell migration in development and disease. Dev. Cell 2, 153–158 (2002).

  2. 2.

    Microscopic imaging techniques for drug discovery. Nat. Rev. Drug Discov. 7, 54–67 (2008).

  3. 3.

    & Digital image processing and analysis. in Video Microscopy (ed. Inoué, S.) 327–392 (Springer Sciences, 1986).

  4. 4.

    , & NIH Image to ImageJ: 25 years of image analysis. Nat. Methods 9, 671–675 (2012).

  5. 5.

    Cell segmentation: 50 years down the road. IEEE Signal Process. Mag. 29, 140–145 (2012).

  6. 6.

    et al. Signal processing challenges in quantitative 3-D cell morphology: more than meets the eye. IEEE Signal Process. Mag. 32, 30–40 (2015).

  7. 7.

    et al. On the digital trail of mobile cells. IEEE Signal Process. Mag. 23, 54–62 (2006).

  8. 8.

    , , , & Automatic stem cell detection in microscopic whole mouse cryo-imaging. IEEE Trans. Med. Imaging 35, 819–829 (2016).

  9. 9.

    , , , & Automatic signal classification in fluorescence in situ hybridization images. Cytometry 43, 87–93 (2001).

  10. 10.

    , & Automated segmentation, classification, and tracking of cancer cell nuclei in time-lapse microscopy. IEEE Trans. Biomed. Eng. 53, 762–766 (2006).

  11. 11.

    et al. PhagoSight: an open-source MATLAB package for the analysis of fluorescent neutrophil and macrophage migration in a zebrafish model. PLoS One 8, e72636 (2013).

  12. 12.

    , , , & Combining intensity, edge and shape information for 2D and 3D segmentation of cell nuclei in tissue sections. J. Microsc. 215, 67–76 (2004).

  13. 13.

    , & Wavelet-based circular hough-transform and its application in embryo development analysis. in Proc. of the International Conference on Computer Vision Theory and Applications 669–674 (Science and Technology Publications, 2013).

  14. 14.

    , , , & Network flow integer programming to track elliptical cells in time-lapse sequences. IEEE Trans. Med. Imaging 36, 942–951 (2017).

  15. 15.

    et al. Applying watershed algorithms to the segmentation of clustered nuclei. Cytometry 28, 289–297 (1997).

  16. 16.

    et al. Segmentation of confocal microscope images of cell nuclei in thick tissue sections. J. Microsc 193, 212–226 (1999).

  17. 17.

    et al. Quantitative 3D analysis of complex single border cell behaviors in coordinated collective cell migration. Nat. Commun. 8, 14905 (2017).

  18. 18.

    , & U-net: convolutional networks for biomedical image segmentation. in Proc. MICCAI 2015 LNCS 9351, 234–241 (Spring, Cham, 2015).

  19. 19.

    et al. Graphical model for joint segmentation and tracking of multiple dividing cells. Bioinformatics 31, 948–956 (2015).

  20. 20.

    , , , & Segmentation and tracking of migrating cells in videomicroscopy with parametric active contours: a tool for cell-based drug testing. IEEE Trans. Med. Imaging 21, 1212–1221 (2002).

  21. 21.

    , , , & 3-D active meshes: fast discrete deformable models for cell tracking in 3-D time-lapse microscopy. IEEE Trans. Image Process. 20, 1925–1937 (2011).

  22. 22.

    et al. Segmentation and shape tracking of whole fluorescent cells based on the Chan-Vese model. IEEE Trans. Med. Imaging 32, 995–1006 (2013).

  23. 23.

    , , & Segmentation of nuclei and cells using membrane related protein markers. J. Microsc. 201, 404–415 (2001).

  24. 24.

    , , , & Advanced level-set-based cell tracking in time-lapse fluorescence microscopy. IEEE Trans. Med. Imaging 29, 852–867 (2010).

  25. 25.

    et al. Segmenting and tracking fluorescent cells in dynamic 3-D microscopy with coupled active surfaces. IEEE Trans. Image Process. 14, 1396–1410 (2005).

  26. 26.

    & Cell segmentation and tracking in phase contrast images using graph cut with asymmetric boundary costs. In Proc. 2015 IEEE Int. Symp. Biomed. Imaging (ISBI) 1120–1123 (2015).

  27. 27.

    et al. Automatic analysis of dividing cells in live cell movies to detect mitotic delays and correlate phenotypes in time. Genome Res. 19, 2113–2124 (2009).

  28. 28.

    , & Reliable cell tracking by global data association. in Proc. 2011 IEEE Int. Symp. Biomed. Imaging (ISBI) 1004–1010 (2011).

  29. 29.

    , , & Global linking of cell tracks using the Viterbi algorithm. IEEE Trans. Med. Imaging 34, 911–929 (2015).

  30. 30.

    et al. A benchmark for comparison of cell tracking algorithms. Bioinformatics 30, 1609–1617 (2014).

  31. 31.

    & MitoGen: A framework for generating 3D synthetic time-lapse sequences of cell populations in fluorescence microscopy. IEEE Trans. Med. Imaging 36, 310–321 (2017).

  32. 32.

    et al. Automated analysis of embryonic gene expression with cellular resolution in C. elegans. Nat. Methods 5, 703–709 (2008).

  33. 33.

    et al. Fast, accurate reconstruction of cell lineages from large-scale fluorescence microscopy data. Nat. Methods 11, 951–958 (2014).

  34. 34.

    et al. Objective comparison of particle tracking methods. Nat. Methods 11, 281–289 (2014).

  35. 35.

    et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676–682 (2012).

  36. 36.

    & Handbook of Image Processing Operators (New York, Wiley, 1996).

  37. 37.

    & 3D characterization and analysis of particle shape using X-ray microtomography (XMT). Powder Technol. 154, 61–69 (2005).

  38. 38.

    et al. Cell tracking accuracy measurement based on comparison of acyclic oriented graphs. PLoS One 10, e0144959 (2015).

  39. 39.

    et al. Cell population tracking and lineage construction with spatiotemporal context. Med. Image Anal. 12, 546–566 (2008).

  40. 40.

    et al. Flow-based cytometric analysis of cell cycle via simulated cell populations. PLOS Comput. Biol. 6, e1000741 (2010).

Download references

Acknowledgements

We acknowledge the work of A. Urbiola, C. Ederra, T. España, S. Venkatesan, D.M.W. Balak, P. Karas, T. Bolcková, M. Štreitová, M. Charousová and L. Zátopková, who manually annotated the data sets to create the ground truths used to evaluate the performance of the algorithms. We also would like to thank F. Prósper (CIMA-University of Navarra), E. Bártová (Institute of Biophysics, Academy of Sciences of the Czech Republic), J. Essers (Erasmus University Medical Center), the Mitocheck consortium, A. Rouzaut (CIMA-University of Navarra), R. Kamm (Massachussets Institute of Technology), the Waterston Lab (The George Washington University), P. Keller (Howard Hughes Medical Institute), S. Kumar (University of California at Berkeley), G. van Cappellen (Erasmus University Medical Center) and T. Becker (Fraunhofer Institution for Marine Biology), who provided the data sets used in the three challenge editions. Finally, we thank R. Stoklasa for technical support. The participants would like to acknowledge the contributions of M. Schiegg, D. Stöckel, J. Crowe, M. Temerinac-Ott and P. Fischer. This work was funded by Spanish Ministry of Economy MINECO grants DPI2012-38090-C03-02 (C.O.-d.-S.) and DPI2015-64221-C2-2 (C.O.-d.-S.), TEC2013-48552-C2-1-R (A.M.B.), TEC2015-73064-EXP (A.M.B), and TEC2016-78052-R (A.M.B.); Netherlands Organization for Scientific Research (NWO) grants 612.001.018 (M.R. and E.M.) and 639.021.128 (I.S.); Dutch Technology Foundation (STW) grant 10443 (I.S. and E.M.); Czech Science Foundation (GACR) grant P302/12/G157 (M.K. and Pavel Matula); the Czech Ministry of Education, Youth and Sports grant LTC17016 in the frame of EU COST NEUBIAS project (M.M., Pavel Matula, Petr Matula, D.S. and M.K.); Helmholtz Association (J.S. and R.M.) and DFG grant MI 1315/4-1 (J.S. and R.M.); the Excellence Initiative of the German Federal and State Governments EXC 294 (O.R., T.B. and R.B.); the Swiss Commission for Technology and Innovation, CTI project 16997 (Ö.D. and L.M.); the BMBF, projects ENGINE (NGFN+), RNA-Code (e:Bio) and de.NBI, as well as the DFG, SFB 1129 and RTG 1653 (N.H. and K.R.); the HGS MathComp Graduate School, the SFB 1129 for integrative analysis of pathogen replication and spread, the RTG 1653 for probabilistic graphical models, and the CellNetworks Excellence Cluster/EcTop (C.H., S.W. and F.H.); the Baxter Foundation and US National Institutes of Health grant AG020961 (H.M.B.) and the Swedish Research Council VR Grant 2015-04026 (K.M. and J.J.); and the BMBF, project de.NBI, grant 031L0102 (V.U. and F.J.).

Author information

Author notes

    • Vladimír Ulman
    • , Olaf Ronneberger
    • , Nathalie Harder
    • , Pengdong Xiao
    •  & Yuexiang Li

    Present address: Max Planck Institute of Molecular Cell Biology and Genetics, Dresden, Germany (V.U.); DeepMind, London, UK (O.R.); Definiens AG, Munich, Germany (N.H.); National Heart Research Institute Singapore (NHRIS), National Heart Centre Singapore (NHCS), Singapore (P.X.); and Computer Vision Institute, College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China (Y.L.).

    • Vladimír Ulman
    •  & Martin Maška

    These authors contributed equally to this work.

Affiliations

  1. Centre for Biomedical Image Analysis, Masaryk University, Brno, Czech Republic.

    • Vladimír Ulman
    • , Martin Maška
    • , Pavel Matula
    • , Petr Matula
    • , David Svoboda
    •  & Michal Kozubek
  2. ACCESS Linnaeus Centre, KTH Royal Institute of Technology, Stockholm, Sweden.

    • Klas E G Magnusson
    •  & Joakim Jaldén
  3. Computer Science Department and BIOSS Centre for Biological Signaling Studies University of Freiburg, Frieburg, Germany.

    • Olaf Ronneberger
    • , Robert Bensch
    •  & Thomas Brox
  4. Heidelberg Collaboratory for Image Processing, IWR, University of Heidelberg, Heidelberg, Germany.

    • Carsten Haubold
    • , Steffen Wolf
    •  & Fred A Hamprecht
  5. Biomedical Computer Vision Group, Department of Bioinformatics and Functional Genomics, BIOQUANT, IPMB, University of Heidelberg and DKFZ, Heidelberg, Germany.

    • Nathalie Harder
    •  & Karl Rohr
  6. Biomedical Imaging Group Rotterdam, Departments of Medical Informatics and Radiology, Erasmus University Medical Center Rotterdam, Rotterdam, the Netherlands.

    • Miroslav Radojevic
    • , Ihor Smal
    •  & Erik Meijering
  7. Baxter Laboratory for Stem Cell Biology, Department of Microbiology and Immunology, and Institute for Stem Cell Biology and Regenerative Medicine, Stanford University School of Medicine, Stanford, California, USA.

    • Helen M Blau
  8. Division of Image Processing, Department of Radiology, Leiden University Medical Center, Leiden, the Netherlands.

    • Oleh Dzyubachyk
    •  & Boudewijn Lelieveldt
  9. Intelligent Systems Department, Delft University of Technology, Delft, the Netherlands.

    • Boudewijn Lelieveldt
  10. Institute of Molecular and Cell Biology, A*Star, Singapore.

    • Pengdong Xiao
  11. Department of Engineering, University of Nottingham, Nottingham, UK.

    • Yuexiang Li
  12. Faculty of Engineering, University of Nottingham, Ningbo, China.

    • Siu-Yeung Cho
  13. BioImage Analysis Unit, Institut Pasteur, Paris, France.

    • Alexandre C Dufour
    •  & Jean-Christophe Olivo-Marin
  14. Research Centre in Biomedical Engineering, School of Mathematics, Computer Science and Engineering, City University of London, London, UK.

    • Constantino C Reyes-Aldasoro
    •  & Jose A Solis-Lemus
  15. Group for Automated Image and Data Analysis, Institute for Applied Computer Science, Karlsruhe Institute of Technology, Eggenstein-Leopoldshafen, Germany.

    • Johannes Stegmaier
    •  & Ralf Mikut
  16. i3S - Instituto de Investigação e Inovação em Saúde, Universidade do Porto, Porto, Portugal.

    • Tiago Esteves
    •  & Pedro Quelhas
  17. Facultade de Engenharia, Universidade do Porto, Porto, Portugal.

    • Tiago Esteves
  18. S3IT, University of Zurich, Zurich, Switzerland.

    • Ömer Demirel
    •  & Lars Malmström
  19. Max Planck Institute of Molecular Cell Biology and Genetics, Dresden, Germany.

    • Florian Jug
    •  & Pavel Tomancak
  20. Bioengineering and Aerospace Engineering Department, Universidad Carlos III de Madrid, Getafe, Spain.

    • Arrate Muñoz-Barrutia
  21. Instituto de Investigación Sanitaria Gregorio Marañon, Madrid, Spain.

    • Arrate Muñoz-Barrutia
  22. CIBERONC, IDISNA and Program of Solid Tumors and Biomarkers, Center for Applied Medical Research, University of Navarra, Pamplona, Spain.

    • Carlos Ortiz-de-Solorzano
  23. Bioengineering Department, TECNUN School of Engineering, University of Navarra, San Sebastián, Spain.

    • Carlos Ortiz-de-Solorzano

Authors

  1. Search for Vladimír Ulman in:

  2. Search for Martin Maška in:

  3. Search for Klas E G Magnusson in:

  4. Search for Olaf Ronneberger in:

  5. Search for Carsten Haubold in:

  6. Search for Nathalie Harder in:

  7. Search for Pavel Matula in:

  8. Search for Petr Matula in:

  9. Search for David Svoboda in:

  10. Search for Miroslav Radojevic in:

  11. Search for Ihor Smal in:

  12. Search for Karl Rohr in:

  13. Search for Joakim Jaldén in:

  14. Search for Helen M Blau in:

  15. Search for Oleh Dzyubachyk in:

  16. Search for Boudewijn Lelieveldt in:

  17. Search for Pengdong Xiao in:

  18. Search for Yuexiang Li in:

  19. Search for Siu-Yeung Cho in:

  20. Search for Alexandre C Dufour in:

  21. Search for Jean-Christophe Olivo-Marin in:

  22. Search for Constantino C Reyes-Aldasoro in:

  23. Search for Jose A Solis-Lemus in:

  24. Search for Robert Bensch in:

  25. Search for Thomas Brox in:

  26. Search for Johannes Stegmaier in:

  27. Search for Ralf Mikut in:

  28. Search for Steffen Wolf in:

  29. Search for Fred A Hamprecht in:

  30. Search for Tiago Esteves in:

  31. Search for Pedro Quelhas in:

  32. Search for Ömer Demirel in:

  33. Search for Lars Malmström in:

  34. Search for Florian Jug in:

  35. Search for Pavel Tomancak in:

  36. Search for Erik Meijering in:

  37. Search for Arrate Muñoz-Barrutia in:

  38. Search for Michal Kozubek in:

  39. Search for Carlos Ortiz-de-Solorzano in:

Contributions

V.U. actively participated in the organization and management of the CTC challenges by handling submissions, producing synthetic data sets, evaluating the submitted results and globally analyzing the participant's contributions, and creating annotations for data set evaluation. V.U. contributed to the writing of the manuscript and produced the tables and plot results, as well as the Fiji plugin with the evaluation suite. M.M. actively participated in the organization and management of the CTC challenges by handling and evaluating submissions, providing evaluation and annotation software, supervising annotations, and creating consensual ground truths for the evaluation of the submitted results. M.M. contributed to the writing of the manuscript and was a challenge participant. K.E.G.M., O.R. and C.H. were top ranked challenge participants and contributed to the writing of the manuscript. N.H. was a top ranked challenge participant. Pavel Matula actively participated in the organization of the CTC challenges by leading the development of a suitable tracking measure and assessing the behavior of various measures on challenge data sets. Petr Matula, M.R. and I.S. actively participated in the organization of the CTC challenges by preparing data and supervising data annotation. D.S. actively participated in the organization of the CTC challenges by leading the development of synthetic data generator and creation of suitable collection of synthetic time-lapse sequences with absolute ground truth. K.R., J.J., H.M.B., O.D., B.L., P.X., Y.L., S.-Y.C., A.C.D., J.-.C.O.-M., C.C.R.-A., J.A.S.-L., R.B., T.B., J.S., R.M., S.W., F.A.H., T.E., P.Q., Ö.D. and L.M. were challenge participants. F.J. contributed to the revision of the manuscript and supported V.U. with the related data processing. P.T., E.M., A.M.-B. and M.K. were challenge organizers and contributed to the revision of the manuscript. C.O.-d.-S. was a challenge organizer, coordinated the work of the committee that organized the challenges and wrote the manuscript with input from all of the authors.

Competing interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to Carlos Ortiz-de-Solorzano.

Integrated supplementary information

Supplementary information

PDF files

  1. 1.

    Supplementary Text and Figures

    Supplementary Figures 1–14, Supplementary Tables 1–4 and Supplementary Notes 1–3

  2. 2.

    Life Sciences Reporting Summary

  3. 3.

    Supplementary Data 1

    Configuration file used for creating the synthetic competition videos

Excel files

  1. 1.

    Supplementary Data 2

    Parameter configurations used by the competing cell tracking methods across all applicable videos

  2. 2.

    Supplementary Data 3

    Technical measures calculated for all competing cell tracking methods across all applicable competition videos

  3. 3.

    Supplementary Data 4

    Biologically-oriented measures calculated for all competing cell tracking methods across all applicable competition videos

  4. 4.

    Supplementary Data 5

    Usability measures calculated for all competing cell tracking methods across s all applicable videos in the training and competition datasets

Videos

  1. 1.

    Dataset DIC-C2DH-HeLa

    Rendering of a representative fragment of datasets DIC-C2DH-HeLa

  2. 2.

    Dataset Fluo-C2DL-MSC

    Rendering of a representative fragment of datasets Fluo-C2DL-MSC

  3. 3.

    Dataset Fluo-C3DH-H157

    Rendering of a representative fragment of datasets Fluo-C3DH-H157

  4. 4.

    Dataset Fluo-C3DL-MDA321

    Rendering of a representative fragment of datasets Fluo-C3DL-MDA321

  5. 5.

    Dataset Fluo-N2DH-GOWT1

    Rendering of a representative fragment of datasets Fluo-N2DH-GOWT1

  6. 6.

    Dataset Fluo-N2DL-HeLa

    Rendering of a representative fragment of datasets Fluo-N2DL-HeLa

  7. 7.

    Dataset Fluo-N3DH-CE

    Rendering of a representative fragment of datasets Fluo-N3DH-CE

  8. 8.

    Dataset Fluo-N3DH-CHO

    Rendering of a representative fragment of datasets Fluo-N3DH-CHO

  9. 9.

    Dataset Fluo-N3DL-DRO

    Rendering of a representative fragment of datasets Fluo-N3DL-DRO

  10. 10.

    Dataset PhC-C2DH-U373

    Rendering of a representative fragment of datasets PhC-C2DH-U373

  11. 11.

    Dataset PhC-C2DL-PSC

    Rendering of a representative fragment of datasets PhC-C2DL-PSC

  12. 12.

    Dataset Fluo-N2DH-SIM+

    Rendering of a representative fragment of datasets Fluo-N2DH-SIM+

  13. 13.

    Dataset Fluo-N3DH-SIM+

    Rendering of a representative fragment of datasets Fluo-N3DH-SIM+

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nmeth.4473

Further reading