Analysis | Published:

Comparing the performance of biomedical clustering methods

Nature Methods volume 12, pages 10331038 (2015) | Download Citation

Abstract

Identifying groups of similar objects is a popular first step in biomedical data analysis, but it is error-prone and impossible to perform manually. Many computational methods have been developed to tackle this problem. Here we assessed 13 well-known methods using 24 data sets ranging from gene expression to protein domains. Performance was judged on the basis of 13 common cluster validity indices. We developed a clustering analysis platform, ClustEval (http://clusteval.mpi-inf.mpg.de), to promote streamlined evaluation, comparison and reproducibility of clustering results in the future. This allowed us to objectively evaluate the performance of all tools on all data sets with up to 1,000 different parameter sets each, resulting in a total of more than 4 million calculated cluster validity indices. We observed that there was no universal best performer, but on the basis of this wide-ranging comparison we were able to develop a short guideline for biomedical clustering tasks. ClustEval allows biomedical researchers to pick the appropriate tool for their data type and allows method developers to compare their tool to the state of the art.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    & Evaluation of clustering algorithms for protein-protein interaction networks. BMC Bioinformatics 7, 488 (2006).

  2. 2.

    , , & Large scale clustering of protein sequences with FORCE—a layout based heuristic for weighted cluster editing. BMC Bioinformatics 8, 396 (2007).

  3. 3.

    Developments in automatic text retrieval. Science 253, 974–980 (1991).

  4. 4.

    Word sense disambiguation: a survey. ACM Comput. Surv. 41, 10:11–10:69 (2009).

  5. 5.

    et al. Integrated genomic analysis identifies clinically relevant subtypes of glioblastoma characterized by abnormalities in PDGFRA, IDH1, EGFR, and NF1. Cancer Cell 17, 98–110 (2010).

  6. 6.

    et al. Meta-analysis of gene expression profiles in breast cancer: toward a unified understanding of breast cancer subtyping and prognosis signatures. Breast Cancer Res. 10, R65 (2008).

  7. 7.

    et al. Comprehensive cluster analysis with Transitivity Clustering. Nat. Protoc. 6, 285–295 (2011).

  8. 8.

    et al. Density parameter estimation for finding clusters of homologous proteins–tracing actinobacterial pathogenicity lifestyles. Bioinformatics 29, 215–222 (2013).

  9. 9.

    , & Protein complex prediction via cost-based clustering. Bioinformatics 20, 3013–3020 (2004).

  10. 10.

    , & Detecting overlapping protein complexes in protein-protein interaction networks. Nat. Methods 9, 471–472 (2012).

  11. 11.

    & Measuring the influence of individual data points in a cluster analysis. Journal of Classification 13, 315–335 (1996).

  12. 12.

    & Clustering algorithms in biomedical research: a review. IEEE Rev. Biomed. Eng. 3, 120–154 (2010).

  13. 13.

    , , & A roadmap of clustering algorithms: finding a match for a biomedical application. Brief. Bioinform. 10, 297–314 (2009).

  14. 14.

    How many clusters are best? - An experiment. Pattern Recognit. 20, 645–663 (1987).

  15. 15.

    , & Data clustering: a review. ACM Comput. Surv. 31, 264–323 (1999).

  16. 16.

    , , , & Online transitivity clustering of biological data with missing values. Proc. German Conference on Bioinformatics (eds. Böcker, S., Hufsky, F., Scheubert, K., Schleicher, J. & Schuster, S.) 57–68 (Schloss Dagstuhl–Leibniz-Zentrum für Informatik, 2012).

  17. 17.

    , & Clustering methods for microarray gene expression data. OMICS 10, 507–531 (2006).

  18. 18.

    & Unsupervised pattern recognition: an introduction to the whys and wherefores of clustering microarray data. Brief. Bioinform. 6, 331–343 (2005).

  19. 19.

    How does gene expression clustering work? Nat. Biotechnol. 23, 1499–1501 (2005).

  20. 20.

    , , & Techniques for clustering gene expression data. Comput. Biol. Med. 38, 283–293 (2008).

  21. 21.

    , , & Evaluation and comparison of gene clustering methods in microarray analysis. Bioinformatics 22, 2405–2412 (2006).

  22. 22.

    & Clustering by passing messages between data points. Science 315, 972–976 (2007).

  23. 23.

    & Clustering by fast search and find of density peaks. Science 344, 1492–1496 (2014).

  24. 24.

    , , & A density-based algorithm for discovering clusters in large spatial databases with noise. Kdd 96, 226–231 (1996).

  25. 25.

    , , , & cluster: cluster analysis basics and extensions. R package version 2.0.1 (2015).

  26. 26.

    R Core Team. A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, 2012).

  27. 27.

    A Cluster Algorithm for Graphs Technical Report INS-R0010 (National Research Institute for Mathematics and Computer Science in the Netherlands, 2000).

  28. 28.

    & An automated method for finding molecular complexes in large protein interaction networks. BMC Bioinformatics 4, 2 (2003).

  29. 29.

    & Self- and super-organizing maps in R: the kohonen package. J. Stat. Softw. 21, 1–19 (2007).

  30. 30.

    , , & kernlab–an S4 package for kernel methods in R. J. Stat. Softw. 11, 1–20 (2004).

  31. 31.

    et al. Partitioning biological data with transitivity clustering. Nat. Methods 7, 419–420 (2010).

  32. 32.

    , , & Consensus clustering—a resampling-based method for class discovery and visualization of gene expression microarray data. Machine Learning 52, 91–118 (2003).

  33. 33.

    Towards the Identification of Cancer Subtypes by Integrative Clustering of Molecular Data M.S. thesis, Universität des Saarlandes (2012).

  34. 34.

    et al. The MIPS mammalian protein-protein interaction database. Bioinformatics 21, 832–834 (2005).

  35. 35.

    , & The ASTRAL compendium for protein structure and sequence analysis. Nucleic Acids Res. 28, 254–256 (2000).

  36. 36.

    , , & A gold standard set of mechanistically diverse enzyme superfamilies. Genome Biol. 7, R8 (2006).

  37. 37.

    , & MAMMOTH (matching molecular models obtained from theory): an automated method for model comparison. Protein Sci. 11, 2606–2621 (2002).

  38. 38.

    An information flow model for conflict and fission in small groups. J. Anthropol. Res. 33, 452–473 (1977).

  39. 39.

    & Robust path-based spectral clustering. Pattern Recognit. 41, 191–203 (2008).

  40. 40.

    & Iterative shrinking method for clustering problems. Pattern Recognit. 39, 761–775 (2006).

  41. 41.

    & FLAME, a novel fuzzy clustering method for the analysis of DNA microarray data. BMC Bioinformatics 8, 3 (2007).

  42. 42.

    , & Clustering aggregation. ACM Trans. Knowl. Discov. Data 1, 4–es (2007).

  43. 43.

    , & A maximum variance cluster algorithm. IEEE Trans. Pattern Anal. Mach. Intell. 24, 1273–1280 (2002).

  44. 44.

    Graph-theoretical methods for detecting and describing gestalt clusters. IEEE Trans. Comput. C-20, 68–86 (1971).

  45. 45.

    & mlbench: Machine Learning Benchmark Problems R package version 2.1-1. (CRAN R Project, 2010).

  46. 46.

    WordNet: a lexical database for English. Commun. ACM 38, 39–41 (1995).

  47. 47.

    & A cluster separation measure. IEEE Trans. Pattern Anal. Mach. Intell. 1, 224–227 (1979).

  48. 48.

    Well-separated clusters and optimal fuzzy partitions. Cybern. Syst. 4, 95–104 (1974).

  49. 49.

    Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, 53–65 (1987).

  50. 50.

    Evaluation: from precision, recall and F-factor to ROC, informedness, markedness and correlation. Journal of Machine Learning Technologies 2, 1–24 (2007).

  51. 51.

    & Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. Series B Stat. Methodol. 57, 289–300 (1995).

  52. 52.

    , & The Elements of Statistical Learning (Springer, 2009).

  53. 53.

    & A method for comparing two hierarchical clusterings. J. Am. Stat. Assoc. 78, 553–569 (1983).

  54. 54.

    Etude comparative de la distribution florale dans une portion des Alpes et du Jura (Corbaz, 1901).

  55. 55.

    Objective criteria for the evaluation of clustering methods. J. Am. Stat. Assoc. 66, 846–850 (1971).

  56. 56.

    & V-Measure: a conditional entropy-based external cluster evaluation measure. In Proc. 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) (ed. Eisner, J.) 410–420 (Association for Computational Linguistics, 2007).

  57. 57.

    & A K-means clustering algorithm. J. R. Stat. Soc. Ser. C Appl. Stat. 28, 100–108 (1979).

  58. 58.

    , , & Density-based clustering in spatial databases: the algorithm GDBSCAN and its applications. Data Min. Knowl. Discov. 2, 169–194 (1998).

  59. 59.

    & New index for clustering tendency and its application to chemical problems. J. Chem. Inf. Comput. Sci. 30, 36–41 (1990).

  60. 60.

    , & Computational cluster validation in post-genomic data analysis. Bioinformatics 21, 3201–3212 (2005).

Download references

Acknowledgements

C.W. is supported by the SDU2020 funding initiative at the University of Southern Denmark. R.R. was partially supported by the International Max Planck Research School on Computer Science and the Saarland University Graduate School for Computer Science. J.B. is grateful for financial support from the Cluster of Excellence for Multimodal Computing and Interaction (MMCI).

Author information

Author notes

    • Jan Baumbach
    •  & Richard Röttger

    These authors jointly supervised this work.

Affiliations

  1. Department of Mathematics and Computer Science, University of Southern Denmark, Odense, Denmark.

    • Christian Wiwie
    • , Jan Baumbach
    •  & Richard Röttger
  2. Computational Systems Biology, Max Planck Institute for Informatics, Saarbrücken, Germany.

    • Jan Baumbach
  3. Cluster of Excellence for Multimodal Computing and Interaction, Saarland University, Saarbrücken, Germany.

    • Jan Baumbach

Authors

  1. Search for Christian Wiwie in:

  2. Search for Jan Baumbach in:

  3. Search for Richard Röttger in:

Contributions

C.W. implemented ClustEval and performed the study. J.B. and R.R. jointly directed this work and designed the study. All authors contributed equally to the manuscript.

Competing interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to Jan Baumbach.

Integrated supplementary information

Supplementary information

PDF files

  1. 1.

    Supplementary Text and Figures

    Supplementary Figures 1–2 and Supplementary Note

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nmeth.3583

Further reading