Abstract
The detection of circular RNA molecules (circRNAs) is typically based on short-read RNA sequencing data processed using computational tools. Numerous such tools have been developed, but a systematic comparison with orthogonal validation is missing. Here, we set up a circRNA detection tool benchmarking study, in which 16 tools detected more than 315,000 unique circRNAs in three deeply sequenced human cell types. Next, 1,516 predicted circRNAs were validated using three orthogonal methods. Generally, tool-specific precision is high and similar (median of 98.8%, 96.3% and 95.5% for qPCR, RNase R and amplicon sequencing, respectively) whereas the sensitivity and number of predicted circRNAs (ranging from 1,372 to 58,032) are the most significant differentiators. Of note, precision values are lower when evaluating low-abundance circRNAs. We also show that the tools can be used complementarily to increase detection sensitivity. Finally, we offer recommendations for future circRNA detection and validation.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$259.00 per year
only $21.58 per issue
Rent or buy this article
Prices vary by article type
from$1.95
to$39.95
Prices may be subject to local taxes which are calculated during checkout
Data availability
We anticipate that this study will serve as a future resource for the circRNA community. The information on all predicted circRNAs (n = 315,312), including the large extensively validated circRNA set (n = 1,516), along with the validation results are available in the GitHub repository (https://github.com/OncoRNALab/circRNA_benchmarking) and as Supplementary Tables. The set of circRNAs previously described in databases (Circ2Disease, circad, CircAtlas, circbank, circBase, CIRCpediav2, CircR2disease, CircRiC, circRNADb, CSCD, exoRBase, MiOncoCirc and TSCD) is also included in the GitHub repository. All databases were accessed in the context of a previous study30. Raw FASTQ files are stored in the Sequence Read Archive (PRJNA789110: SRX13414572 (untreated HLF), SRX13414573 (untreated NCI-H23), SRX13414574 (untreated SW480), SRX13414575 (RNase R-treated HLF), SRX13414576 (RNase R-treated NCI-H23), SRX13414577 (RNase R-treated SW480)). Source data are provided with this paper.
Code availability
All of the scripts used to compute the metrics described in the study and generate the figures are available at https://github.com/OncoRNALab/circRNA_benchmarking.
References
Kristensen, L. S. et al. The biogenesis, biology and characterization of circular RNAs. Nat. Rev. Genet. 20, 675–691 (2019).
Hulstaert, E. et al. Charting extracellular transcriptomes in the Human Biofluid RNA Atlas. Cell Rep. 33, 108552 (2020).
Wang, S. et al. Circular RNAs in body fluids as cancer biomarkers: the new frontier of liquid biopsies. Mol. Cancer 20, 13. (2021).
Vromman, M. et al. Validation of circular RNAs using RT-qPCR after effective removal of linear RNAs by ribonuclease R. Curr. Protoc. 1, e181 (2021).
Yu, C. Y., Liu, H. J., Hung, L. Y., Kuo, H. C. & Chuang, T. J. Is an observed non-co-linear RNA product spliced in trans, in cis or just in vitro?. Nucleic Acids Res. 42, 9410–9423 (2014).
Szabo, L. & Salzman, J. Detecting circular RNAs: bioinformatic and experimental challenges. Nat. Rev. Genet. 17, 679–692 (2016).
Nielsen, A. F. et al. Best practice standards for circular RNA research. Nat. Methods 19, 1208–1220 (2022).
Dodbele, S., Mutlu, N. & Wilusz, J. E. Best practices to ensure robust investigation of circular RNAs: pitfalls and tips. EMBO Rep. 22, e52072 (2021).
Jakobi, T. & Dieterich, C. Computational approaches for circular RNA analysis. Wiley Interdiscip. Rev. RNA 10, e1528 (2019).
Hansen, T. B., Venø, M. T., Damgaard, C. K. & Kjems, J. Comparison of circular RNA prediction tools. Nucleic Acids Res. 44, e58 (2015).
Gaffo, E., Buratin, A., Dal Molin, A. & Bortoluzzi, S. Sensitive, reliable and robust circRNA detection from RNA-seq with CirComPara2. Brief. Bioinform. 23, bbab418 (2022).
Jeck, W. R. & Sharpless, N. E. Detecting and characterizing circular RNAs. Nat. Biotechnol. 32, 453–461 (2014).
Nguyen, D. T. et al. Circall: fast and accurate methodology for discovery of circular RNAs from paired-end RNA-sequencing data. BMC Bioinformatics 22, 495 (2021).
Zeng, X., Lin, W., Guo, M. & Zou, Q. A comprehensive overview and evaluation of circular RNA detection tools. PLoS Comput. Biol. 13, e1005420 (2017).
Szabo, L. et al. Statistically based splicing detection reveals neural enrichment and tissue-specific induction of circular RNA during human fetal development. Genome Biol. 16, 126 (2015).
Song, X. et al. Circular RNA profile in gliomas revealed by identification tool UROBORUS. Nucleic Acids Res. 44, e87 (2016).
Ma, X. K. et al. CIRCexplorer3: a CLEAR pipeline for direct comparison of circular and linear RNA expression. Genomics Proteomics Bioinformatics 17, 511–521 (2019).
Westholm, J. O. et al. Genome-wide analysis of Drosophila circular RNAs reveals their structural and sequence properties and age-dependent neural accumulation. Cell Rep. 9, 1966–1980 (2014).
Ye, C.-Y. et al. Full-length sequence assembly reveals circular RNAs with diverse non-GT/AG splicing signals in rice. RNA Biol. 14, 1055–1063 (2017).
Feng, J. et al. Genome-wide identification of cancer-specific alternative splicing in circRNA. Mol. Cancer 18, 35 (2019).
Jakobi, T., Uvarovskii, A. & Dieterich, C. Circtools: a one-stop software solution for circular RNA research. Bioinformatics 35, 2326–2328 (2019).
Gao, Y., Zhang, J. & Zhao, F. Circular RNA identification based on multiple seed matching. Brief. Bioinform. 19, 803–810 (2018).
Zhang, J., Chen, S., Yang, J. & Zhao, F. Accurate quantification of circular RNAs identifies extensive circular isoform switching events. Nat. Commun. 11, 90 (2020).
Memczak, S. et al. Circular RNAs are a large class of animal RNAs with regulatory potency. Nature 495, 333–338 (2013).
Chuang, T. J. et al. NCLscan: accurate identification of non-co-linear transcripts (fusion, trans-splicing and circular RNA) with a good balance between sensitivity and precision. Nucleic Acids Res. 44, e29 (2016).
Chen, C. Y. & Chuang, T. J. NCLcomparator: systematically post-screening non-co-linear transcripts (circular, trans-spliced, or fusion RNAs) identified from various detectors. BMC Bioinformatics 20, 3 (2019).
Izuogu, O. G. et al. Analysis of human ES cell differentiation establishes that the dominant isoforms of the lncRNAs RMST and FIRRE are circular. BMC Genomics 19, 276 (2018).
Li, M. et al. Quantifying circular RNA expression from RNA-seq data using model-based framework. Bioinformatics 33, 2131–2139 (2017).
Hoffmann, S. et al. A multi-split mapping algorithm for circular RNA, splicing, trans-splicing and fusion detection. Genome Biol. 15, R34 (2014).
Vromman, M., Vandesompele, J. & Volders, P.-J. Closing the circle: current state and perspectives of circular RNA databases. Brief. Bioinform. 22, 288–297 (2021).
Vromman, M., Anckaert, J., Vandesompele, J. & Volders, P.-J. CIRCprimerXL: convenient and high-throughput PCR primer design for circular RNA quantification. Front. Bioinform. 2, 834655 (2022).
Zhang, J. et al. Comprehensive profiling of circular RNAs with nanopore sequencing and CIRI-long. Nat. Biotechnol. 39, 836–845 (2021).
Rahimi, K., Venø, M. T., Dupont, D. M. & Kjems, J. Nanopore sequencing of brain-derived full-length circRNAs reveals circRNA-specific exon usage, intron retention and microexons. Nat. Commun. 12, 4825 (2021).
Xin, R. et al. isoCirc catalogs full-length circular RNA isoforms in human transcriptomes. Nat. Commun. 12, 266 (2021).
Liu, Z. et al. circFL-seq reveals full-length circular RNAs with rolling circular reverse transcription and nanopore sequencing. Elife 10, e69457 (2021).
Wang, K. et al. MapSplice: accurate mapping of RNA-seq reads for splice junction discovery. Nucleic Acids Res. 38, e178 (2010).
Stefanov, S. R. & Meyer, I. M. CYCLeR: a novel tool for the full isoform assembly and quantification of circRNAs. Nucleic Acids Res. 51, e10 (2023).
R Core Team. R: A Language and Environment for Statistical Computing (2019); https://www.R-project.org/
RStudio Team. RStudio: Integrated Development for R (2020); http://www.rstudio.com/
van Rossum, G. and Drake, F. L. Python 3 Reference Manual (Createspace, 2009).
Langmead, B. & Salzberg, S. L. Fast gapped-read alignment with Bowtie 2. Nat. Methods 9, 357–359 (2012).
Danecek, P. et al. Twelve years of SAMtools and BCFtools. Gigascience 10, giab008 (2021).
Quinlan, A. R. & Hall, I. M. BEDTools: a flexible suite of utilities for comparing genomic features. Bioinformatics 26, 841–842 (2010).
Cunningham, F. et al. Ensembl 2022. Nucleic Acids Res. 50, D988–D995 (2022).
Acknowledgements
The authors thank S. Lefever for his contribution to primer design in the early stage of this project. The computational resources (Stevin Supercomputer Infrastructure) and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by Ghent University, FWO and the Flemish Government – department EWI. This work was supported by the Foundation Against Cancer grant STK F/2018/1,267 (M.V., J.V., P.-J.V.), the Standup Against Cancer grant STIVLK2018000601 (J.V., P.-J.V.), two Concerted Research Action of Ghent University grants BOF16/GOA/023 (J.V.), BOF/24J/2021/244 (M.V., J.V., P.-J.V.), the Research Foundation – Flanders FWO grant 1253321N (P.-J.V.), the Fondazione AIRC per la Ricerca sul Cancro Investigator Grant 2017 20052 (S.B.), the Italian Ministry of Education, Universities, and Research PRIN 2017 grant 2017PPS2X4_003 (S.B.), the EU within the MUR PNRR ‘National Center for Gene Therapy and Drugs based on RNA Technology’, Project no. CN00000041 CN3 RNA (S.B.), the ‘HPC, big data and quantum computing’ CN1 HPC (S.B.), the Department of Molecular Medicine of the University of Padova (S.B.), the National Health Research Institutes Taiwan grant NHRI-EX110-11011B1 (T.-J.C.), the German Science Foundation grant DI 1501/13-1 (C.D.), the Wellcome Trust grant WT108749/Z/15/Z (P.F.), the Fondazione Umberto Veronesi Fellowship 2020 (E.G.), the German Federal Ministry of Education and Research grant BMBF 031L0106D (S.H.), the NIH grant R01-NS083833 (E.C.L.), the MSK Core Grant P30-CA008748 (E.C.L.), the German Federal Ministry of Education and Research grant BMBF 031L0164C (P.S.), the Knut and Alice Wallenberg Foundation as part of the National Bioinformatics Infrastructure Sweden at SciLifeLab (J.W.), the National Natural Science Foundation of China (NSFC) grant 31925011 (L.Y.), the Ministry of Science and Technology of China (MoST) grant 2021YFA1300503 (L.Y.), and the National Science Foundation of China (31871589) (C.-Y.Y.).
Author information
Authors and Affiliations
Contributions
Validation co-author group: M.V., methodology, software, validation, formal analysis, investigation, data curation, writing – original draft, visualization, funding acquisition; J.A., software; O.T., methodology; J.N., E.V.E., K.V., N.Y., investigation; J.V., P.-J.V., conceptualization, methodology, writing – review and editing, supervision, project administration, funding acquisition. CircRNA prediction co-author group: S.B., A.B., C.-Y.C., Q.C., T.-J.C., R.D., C.D., X.D., P.F., E.G., W.G., C.H., S.H., O.I., M.S.J., T.J., E.C.L., J.S., M.S.-K., P.S., G.W., J.W., L.Y., C.-Y.Y., G.-H.Y., J.Z., F.Z., formal analysis and writing – review and editing.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Methods thanks Eduardo Andrés-León and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Rita Strack, in collaboration with the Nature Methods team. Peer reviewer reports are available.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Supplementary Information
Supplementary Notes, Data 1–13, Discussion 1, Figs. 1–40 and References.
Supplementary Tables 1–10
Workbook with multiple tabs for each Supplementary Table and their descriptions.
Source data
Source Data Fig. 1, 2, 3, 4
Statistical source data.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Vromman, M., Anckaert, J., Bortoluzzi, S. et al. Large-scale benchmarking of circRNA detection tools reveals large differences in sensitivity but not in precision. Nat Methods 20, 1159–1169 (2023). https://doi.org/10.1038/s41592-023-01944-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41592-023-01944-6
