Abstract
When fields lack consensus standard methods and accessible ground truths, reproducibility can be more of an ideal than a reality. Such has been the case for functional neuroimaging, where there exists a sprawling space of tools and processing pipelines. We provide a critical evaluation of the impact of differences across five independently developed minimal preprocessing pipelines for functional magnetic resonance imaging. We show that, even when handling identical data, interpipeline agreement was only moderate, critically shedding light on a factor that limits cross-study reproducibility. We show that low interpipeline agreement can go unrecognized until the reliability of the underlying data is high, which is increasingly the case as the field progresses. Crucially we show that, when interpipeline agreement is compromised, so too is the consistency of insights from brain-wide association studies. We highlight the importance of comparing analytic configurations, because both widely discussed and commonly overlooked decisions can lead to marked variation.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The data supporting this study’s findings are publicly available. We used the HNU test–retest dataset (https://fcon_1000.projects.nitrc.org/indi/CoRR/html/hnu_1.html) made available via the Consortium for Reliability and Reproducibility and Healthy Brain Network dataset (https://fcon_1000.projects.nitrc.org/indi/cmi_healthy_brain_network/).
Code availability
All software created and used in this project is publicly available. The C-PAC pipeline is released under a BSD 3-clause licence and is available on GitHub at https://github.com/FCP-INDI/C-PAC/releases/tag/v1.8.5; the ABCD–BIDS pipeline is released under a BSD 3-clause licence and is available on GitHub at https://github.com/DCAN-Labs/abcd-hcp-pipeline/releases/tag/v0.0.3; the CCS pipeline is available on GitHub at https://github.com/zuoxinian/CCS; the fMRIPrep–LTS pipeline is released under Apache Licence 2.0 and is available on GitHub at https://github.com/nipreps/fmriprep/releases/tag/20.2.1. All templates were accessed through TemplateFlow45. All analysis software, including experiments and figure generation, is available on GitHub at https://github.com/XinhuiLi/PipelineAgreement and on Zenodo at https://zenodo.org/badge/latestdoi/415936717 (ref. 72). The preprocessed functional connectivity data can be found on OSF at https://osf.io/kgpu2/. In the completion of this work we used the following versions of critically relevant software: Matlab (2014a), AFNI (21.1.00), FSL (6.0), ANTs (2.3.3.dev168-g29bdf), FreeSurfer (6.0.0), SPM (12) and C-PAC (1.8.5).
References
Shehzad, Z. et al. The resting brain: unconstrained yet reliable. Cereb. Cortex 19, 2209–2229 (2009).
Zuo, X.-N. et al. The oscillating brain: complex and reliable. Neuroimage 49, 1432–1445 (2010).
Bennett, C. M. & Miller, M. B. How reliable are the results from functional magnetic resonance imaging? Ann. N. Y. Acad. Sci. 1191, 133–155 (2010).
Zuo, X.-N., Xu, T. & Milham, M. P. Harnessing reliability for neuroscience research. Nat. Hum. Behav. 3, 768–771 (2019).
Kraemer, H. C. The reliability of clinical diagnoses: state of the art. Annu. Rev. Clin. Psychol. 10, 111–130 (2014).
Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013).
Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2, e124 (2005).
Noble, S., Scheinost, D. & Constable, R. T. A decade of test-retest reliability of functional connectivity: a systematic review and meta-analysis. Neuroimage 203, 116157 (2019).
Zuo, X.-N. & Xing, X.-X. Test-retest reliabilities of resting-state FMRI measurements in human brain functional connectomics: a systems neuroscience perspective. Neurosci. Biobehav. Rev. 45, 100–118 (2014).
Cho, J. W., Korchmaros, A., Vogelstein, J. T., Milham, M. P. & Xu, T. Impact of concatenating fMRI data on reliability for functional connectomics. Neuroimage 226, 117549 (2021).
Lynch, C. J. et al. Rapid precision functional mapping of individuals using multi-echo fMRI. Cell Rep. 33, 108540 (2020).
Nikolaidis, A. et al. Bagging improves reproducibility of functional parcellation of the human brain. Neuroimage 214, 116678 (2020).
Yoo, K. et al. Multivariate approaches improve the reliability and validity of functional connectivity and prediction of individual behaviors. Neuroimage 197, 212–223 (2019).
Elliott, M. L. et al. What is the test-retest reliability of common task-functional MRI measures? New empirical evidence and a meta-analysis. Psychol. Sci. 31, 792–806 (2020).
Palumbo, L. et al. Evaluation of the intra- and inter-method agreement of brain MRI segmentation software packages: a comparison between SPM12 and FreeSurfer v6.0. Phys. Med. 64, 261–272 (2019).
Oakes, T. R. et al. Comparison of fMRI motion correction software tools. Neuroimage 28, 529–543 (2005).
Klein, A. et al. Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. Neuroimage 46, 786–802 (2009).
Dickie, E., Hodge, S., Craddock, R., Poline, J.-B. & Kennedy, D. Tools matter: comparison of two surface analysis tools applied to the ABIDE dataset. Res. Ideas Outcomes 3, e13726 (2017).
Bhagwat, N. et al. Understanding the impact of preprocessing pipelines on neuroimaging cortical surface analyses. Gigascience 10, giaa155 (2021).
Carp, J. On the plurality of (methodological) worlds: estimating the analytic flexibility of FMRI experiments. Front. Neurosci. 6, 149 (2012).
Pauli, R. et al. Exploring fMRI results space: 31 variants of an fMRI analysis in AFNI, FSL, and SPM. Front. Neuroinform. 10, 24 (2016).
Bowring, A., Maumet, C. & Nichols, T. E. Exploring the impact of analysis software on task fMRI results. Hum. Brain Mapp. 40, 3362–3384 (2019).
Bowring, A., Nichols, T. E. & Maumet, C. Isolating the sources of pipeline-variability in group-level task-fMRI results. Hum. Brain Mapp. 43, 1112–1128 (2021).
Botvinik-Nezer, R. et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582, 84–88 (2020).
Feczko, E., Conan, G., Marek, S. & Tervo-Clemmens, B. Adolescent brain cognitive development (ABCD) community MRI collection and utilities. Preprint at bioRxiv https://doi.org/10.1101/2021.07.09.451638 (2021).
Xu, T., Yang, Z., Jiang, L., Xing, X.-X. & Zuo, X.-N. A connectome computation system for discovery science of brain. Sci. Bull. (Beijing) 60, 86–95 (2015).
Craddock, C. et al. Towards automated analysis of connectomes: the configurable pipeline for the analysis of connectomes (C-PAC). Front. Neuroinform. 42, 10-3389 (2013).
Chao-Gan, Y. & Yu-Feng, Z. DPARSF: a MATLAB toolbox for ‘pipeline’ data analysis of resting-State fMRI. Front. Syst. Neurosci. 4, 13 (2010).
Esteban, O. et al. fMRIPrep: a robust preprocessing pipeline for functional MRI. Nat. Methods 16, 111–116 (2019).
Murphy, K. & Fox, M. D. Towards a consensus regarding global signal regression for resting state functional connectivity MRI. Neuroimage 154, 169–173 (2017).
Zuo, X.-N. et al. An open science resource for establishing reliability and reproducibility in functional connectomics. Sci. Data 1, 140049 (2014).
Shou, H. et al. Quantifying the reliability of image replication studies: the image intraclass correlation coefficient (I2C2). Cogn. Affect. Behav. Neurosci. 13, 714–724 (2013).
Bridgeford, E. W. et al. Eliminating accidental deviations to minimize generalization error and maximize replicability: applications in connectomics and genomics. PLoS Comput. Biol. 17, e1009279 (2021).
Schaefer, A. et al. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI. Cereb. Cortex 28, 3095–3114 (2018).
Glasser, M. F. et al. The minimal preprocessing pipelines for the Human Connectome Project. Neuroimage 80, 105–124 (2013).
Greve, D. N. & Fischl, B. Accurate and robust brain image alignment using boundary-based registration. Neuroimage 48, 63–72 (2009).
Alexander, L. M. et al. An open resource for transdiagnostic research in pediatric mental health and learning disorders. Sci. Data 4, 170181 (2017).
Koo, T. K. & Li, M. Y. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J. Chiropr. Med. 15, 155–163 (2016).
Dong, Y., Ifrim, G., Mladenić, D., Saunders, C. & Van Hoecke, S. Machine learning and knowledge discovery in databases. Applied data science and demo track. In Proc. European Conference, ECML PKDD 2020 Part V (eds Dong, Y. et al.) 3–18 (Springer Nature, 2021).
Birn, R. M. et al. The effect of scan length on the reliability of resting-state fMRI connectivity estimates. Neuroimage 83, 550–558 (2013).
Liu, T. T., Nalci, A. & Falahpour, M. The global signal in fMRI: nuisance or information? Neuroimage 150, 213–229 (2017).
Ciric, R. et al. Benchmarking of participant-level confound regression strategies for the control of motion artifact in studies of functional connectivity. Neuroimage 154, 174–187 (2017).
Buades, A., Coll, B. & Morel, J.-M. Non-local means denoising. IPOL J. 1, 208–212 (2011).
Tustison, N. J. et al. N4ITK: improved N3 bias correction. IEEE Trans. Med. Imaging 29, 1310–1320 (2010).
Ciric, R. et al. TemplateFlow: FAIR-sharing of multi-scale, multi-species brain models. Nat. Methods 19, 1568–1571 (2022).
Fonov, V. S., Evans, A. C., McKinstry, R. C., Almli, C. R. & Collins, D. L. Unbiased nonlinear average age-appropriate brain templates from birth to adulthood. Neuroimage 47, S102 (2009).
Grabner, G. et al. Symmetric atlasing and model based segmentation: an application to the hippocampus in older adults. Med. Image Comput. Comput. Assist. Interv. 9, 58–66 (2006).
Mazziotta, J. et al. A probabilistic atlas and reference system for the human brain: International Consortium for Brain Mapping (ICBM). Philos. Trans. R. Soc. Lond. B Biol. Sci. 356, 1293–1322 (2001).
Wu, J. et al. Accurate nonlinear mapping between MNI volumetric and FreeSurfer surface coordinate systems. Hum. Brain Mapp. 39, 3793–3808 (2018).
Uddin, L. Q. Mixed signals: on separating brain signal from noise. Trends Cogn. Sci. 21, 405–406 (2017).
Murphy, K., Birn, R. M., Handwerker, D. A., Jones, T. B. & Bandettini, P. A. The impact of global signal regression on resting state correlations: are anti-correlated networks introduced? Neuroimage 44, 893–905 (2009).
Xu, H. et al. Impact of global signal regression on characterizing dynamic functional connectivity and brain states. Neuroimage 173, 127–145 (2018).
Gordon, E. M. et al. Precision functional mapping of individual human brains. Neuron 95, 791–807 (2017).
Di Martino, A. et al. Enhancing studies of the connectome in autism using the autism brain imaging data exchange II. Sci. Data 4, 170010 (2017).
Casey, B. J. et al. The Adolescent Brain Cognitive Development (ABCD) study: imaging acquisition across 21 sites. Dev. Cogn. Neurosci. 32, 43–54 (2018).
Richie-Halford, A. et al. Author correction: an analysis-ready and quality controlled resource for pediatric brain white-matter research. Sci. Data 10, 247 (2023).
Doshi, J. et al. MUSE: Multi-atlas region segmentation utilizing ensembles of registration algorithms and parameters, and locally optimal atlas selection. NeuroImage 127, 186–195 (2016).
Wu, D. et al. Resource atlases for multi-atlas brain segmentations with multiple ontology levels based on T1-weighted MRI. Neuroimage 125, 120–130 (2016).
Kiar, G., Chatelain, Y., Salari, A., Evans, A. C. & Glatard, T. Data augmentation through Monte Carlo arithmetic leads to more generalizable classification in connectomics. Neurons Behav. Data Anal. Theory 1, 1–20 (2021).
Kiar, G. et al. Numerical uncertainty in analytical pipelines lead to impactful variability in brain networks. PLoS ONE 16, e0250755 (2021).
Mehta, K. et al. XCP-D: a robust pipeline for the post-processing of fMRI data. Preprint at bioRxiv 10.1101/2023.11.20.567926 (2023).
Bujang, M. & Baharum, N. A simplified guide to determination of sample size requirements for estimating the value of intraclass correlation coefficient: a review. Arch. Orofac. Sci. 12, 1–11 (2017).
Smith, S. M. et al. Advances in functional and structural MR image analysis and implementation as FSL. Neuroimage 23, S208–S219 (2004).
Jenkinson, M., Bannister, P., Brady, M. & Smith, S. Improved optimization for the robust and accurate linear registration and motion correction of brain images. Neuroimage 17, 825–841 (2002).
Cox, R. W. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput. Biomed. Res. 29, 162–173 (1996).
Zhang, Y., Brady, M. & Smith, S. Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm. IEEE Trans. Med. Imaging 20, 45–57 (2001).
Avants, B. B., Tustison, N. & Song, G. Advanced normalization tools (ANTS). Insight J. 2, 1–35 (2009).
Fischl, B. FreeSurfer. Neuroimage 62, 774–781 (2012).
Jenkinson, M. & Smith, S. A global optimisation method for robust affine registration of brain images. Med. Image Anal. 5, 143–156 (2001).
Berger, V. W. & Zhou, Y. Kolmogorov–Smirnov test: overview https://doi.org/10.1002/9781118445112.stat06558 (2014).
Nachar, N. The Mann-Whitney U: a test for assessing whether two independent samples come from the same distribution. Tutor. Quant. Methods Psychol. 4, 13–20 (2008).
Li, X. & Clucas, J. XinhuiLi/PipelineHarmonization: Pipeline Harmonization Version 0.0.0 Beta. Zenodo https://doi.org/10.5281/zenodo.5733801 (2021).
Acknowledgements
This work was supported in part by gifts from J. P. Healey, P. Green and R. Cowen to the Child Mind Institute. In addition, grant awards were received from the NIH BRAIN Initiative to M.P.M. and C.C. (no. R24 MH11480602), to G.K. and M.P.M. (no. RF1MH130859) and to R.A.P., O.E., M.P.M. and T.S. (no. RF1MH121867); and from NIMH to T.S. and M.P.M. (no. R01MH120482). O.E. received support from SNSF Ambizione project no. 185872. C.-G.Y. received support from the National Natural Science Foundation of China (grant nos. 82122035, 81671774 and 81630031).
Author information
Authors and Affiliations
Contributions
G.K. and M.P.M. conceptualized the study design, supervised work, wrote the first draft of the manuscript and oversaw all revisions. X.L., N.B.E., L.A. and G.K. wrote the preprocessing, data analysis and computational model scripts. J.C. reviewed the code. A.S.H. and S.G. provided technical support. X.L., N.B.E., G.K. and M.P.M. wrote the first draft of the manuscript. All authors reviewed the manuscript and contributed to the discussion and results.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Human Behaviour thanks Anders Eklund and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Li, X., Bianchini Esper, N., Ai, L. et al. Moving beyond processing- and analysis-related variation in resting-state functional brain imaging. Nat Hum Behav (2024). https://doi.org/10.1038/s41562-024-01942-4
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41562-024-01942-4