Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Moving beyond processing- and analysis-related variation in resting-state functional brain imaging

Abstract

When fields lack consensus standard methods and accessible ground truths, reproducibility can be more of an ideal than a reality. Such has been the case for functional neuroimaging, where there exists a sprawling space of tools and processing pipelines. We provide a critical evaluation of the impact of differences across five independently developed minimal preprocessing pipelines for functional magnetic resonance imaging. We show that, even when handling identical data, interpipeline agreement was only moderate, critically shedding light on a factor that limits cross-study reproducibility. We show that low interpipeline agreement can go unrecognized until the reliability of the underlying data is high, which is increasingly the case as the field progresses. Crucially we show that, when interpipeline agreement is compromised, so too is the consistency of insights from brain-wide association studies. We highlight the importance of comparing analytic configurations, because both widely discussed and commonly overlooked decisions can lead to marked variation.

This is a preview of subscription content, access via your institution

Access options

Buy this article

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Interpipeline agreement for minimal preprocessing in five fMRI preprocessing packages.
Fig. 2: Minimal preprocessing comparisons of C-PAC harmonized pipelines.
Fig. 3: Impact of interpipeline agreement on result consistency.
Fig. 4: Impact of scan duration and GSR on minimal preprocessing results from C-PAC harmonized pipelines.
Fig. 5: Pairwise identification of sources of variation across harmonized pipelines.
Fig. 6: Impact of MNI152 template version and write-out resolution on functional connectomes.

Similar content being viewed by others

Data availability

The data supporting this study’s findings are publicly available. We used the HNU test–retest dataset (https://fcon_1000.projects.nitrc.org/indi/CoRR/html/hnu_1.html) made available via the Consortium for Reliability and Reproducibility and Healthy Brain Network dataset (https://fcon_1000.projects.nitrc.org/indi/cmi_healthy_brain_network/).

Code availability

All software created and used in this project is publicly available. The C-PAC pipeline is released under a BSD 3-clause licence and is available on GitHub at https://github.com/FCP-INDI/C-PAC/releases/tag/v1.8.5; the ABCD–BIDS pipeline is released under a BSD 3-clause licence and is available on GitHub at https://github.com/DCAN-Labs/abcd-hcp-pipeline/releases/tag/v0.0.3; the CCS pipeline is available on GitHub at https://github.com/zuoxinian/CCS; the fMRIPrep–LTS pipeline is released under Apache Licence 2.0 and is available on GitHub at https://github.com/nipreps/fmriprep/releases/tag/20.2.1. All templates were accessed through TemplateFlow45. All analysis software, including experiments and figure generation, is available on GitHub at https://github.com/XinhuiLi/PipelineAgreement and on Zenodo at https://zenodo.org/badge/latestdoi/415936717 (ref. 72). The preprocessed functional connectivity data can be found on OSF at https://osf.io/kgpu2/. In the completion of this work we used the following versions of critically relevant software: Matlab (2014a), AFNI (21.1.00), FSL (6.0), ANTs (2.3.3.dev168-g29bdf), FreeSurfer (6.0.0), SPM (12) and C-PAC (1.8.5).

References

  1. Shehzad, Z. et al. The resting brain: unconstrained yet reliable. Cereb. Cortex 19, 2209–2229 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  2. Zuo, X.-N. et al. The oscillating brain: complex and reliable. Neuroimage 49, 1432–1445 (2010).

    Article  PubMed  Google Scholar 

  3. Bennett, C. M. & Miller, M. B. How reliable are the results from functional magnetic resonance imaging? Ann. N. Y. Acad. Sci. 1191, 133–155 (2010).

    Article  PubMed  Google Scholar 

  4. Zuo, X.-N., Xu, T. & Milham, M. P. Harnessing reliability for neuroscience research. Nat. Hum. Behav. 3, 768–771 (2019).

    Article  PubMed  Google Scholar 

  5. Kraemer, H. C. The reliability of clinical diagnoses: state of the art. Annu. Rev. Clin. Psychol. 10, 111–130 (2014).

    Article  PubMed  Google Scholar 

  6. Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013).

    Article  CAS  PubMed  Google Scholar 

  7. Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2, e124 (2005).

    Article  PubMed  PubMed Central  Google Scholar 

  8. Noble, S., Scheinost, D. & Constable, R. T. A decade of test-retest reliability of functional connectivity: a systematic review and meta-analysis. Neuroimage 203, 116157 (2019).

    Article  PubMed  Google Scholar 

  9. Zuo, X.-N. & Xing, X.-X. Test-retest reliabilities of resting-state FMRI measurements in human brain functional connectomics: a systems neuroscience perspective. Neurosci. Biobehav. Rev. 45, 100–118 (2014).

    Article  PubMed  Google Scholar 

  10. Cho, J. W., Korchmaros, A., Vogelstein, J. T., Milham, M. P. & Xu, T. Impact of concatenating fMRI data on reliability for functional connectomics. Neuroimage 226, 117549 (2021).

    Article  PubMed  Google Scholar 

  11. Lynch, C. J. et al. Rapid precision functional mapping of individuals using multi-echo fMRI. Cell Rep. 33, 108540 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Nikolaidis, A. et al. Bagging improves reproducibility of functional parcellation of the human brain. Neuroimage 214, 116678 (2020).

    Article  PubMed  Google Scholar 

  13. Yoo, K. et al. Multivariate approaches improve the reliability and validity of functional connectivity and prediction of individual behaviors. Neuroimage 197, 212–223 (2019).

    Article  PubMed  Google Scholar 

  14. Elliott, M. L. et al. What is the test-retest reliability of common task-functional MRI measures? New empirical evidence and a meta-analysis. Psychol. Sci. 31, 792–806 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  15. Palumbo, L. et al. Evaluation of the intra- and inter-method agreement of brain MRI segmentation software packages: a comparison between SPM12 and FreeSurfer v6.0. Phys. Med. 64, 261–272 (2019).

    Article  CAS  PubMed  Google Scholar 

  16. Oakes, T. R. et al. Comparison of fMRI motion correction software tools. Neuroimage 28, 529–543 (2005).

    Article  CAS  PubMed  Google Scholar 

  17. Klein, A. et al. Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. Neuroimage 46, 786–802 (2009).

    Article  PubMed  Google Scholar 

  18. Dickie, E., Hodge, S., Craddock, R., Poline, J.-B. & Kennedy, D. Tools matter: comparison of two surface analysis tools applied to the ABIDE dataset. Res. Ideas Outcomes 3, e13726 (2017).

    Article  Google Scholar 

  19. Bhagwat, N. et al. Understanding the impact of preprocessing pipelines on neuroimaging cortical surface analyses. Gigascience 10, giaa155 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  20. Carp, J. On the plurality of (methodological) worlds: estimating the analytic flexibility of FMRI experiments. Front. Neurosci. 6, 149 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  21. Pauli, R. et al. Exploring fMRI results space: 31 variants of an fMRI analysis in AFNI, FSL, and SPM. Front. Neuroinform. 10, 24 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  22. Bowring, A., Maumet, C. & Nichols, T. E. Exploring the impact of analysis software on task fMRI results. Hum. Brain Mapp. 40, 3362–3384 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  23. Bowring, A., Nichols, T. E. & Maumet, C. Isolating the sources of pipeline-variability in group-level task-fMRI results. Hum. Brain Mapp. 43, 1112–1128 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  24. Botvinik-Nezer, R. et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582, 84–88 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Feczko, E., Conan, G., Marek, S. & Tervo-Clemmens, B. Adolescent brain cognitive development (ABCD) community MRI collection and utilities. Preprint at bioRxiv https://doi.org/10.1101/2021.07.09.451638 (2021).

  26. Xu, T., Yang, Z., Jiang, L., Xing, X.-X. & Zuo, X.-N. A connectome computation system for discovery science of brain. Sci. Bull. (Beijing) 60, 86–95 (2015).

    Article  Google Scholar 

  27. Craddock, C. et al. Towards automated analysis of connectomes: the configurable pipeline for the analysis of connectomes (C-PAC). Front. Neuroinform. 42, 10-3389 (2013).

    Google Scholar 

  28. Chao-Gan, Y. & Yu-Feng, Z. DPARSF: a MATLAB toolbox for ‘pipeline’ data analysis of resting-State fMRI. Front. Syst. Neurosci. 4, 13 (2010).

    PubMed  PubMed Central  Google Scholar 

  29. Esteban, O. et al. fMRIPrep: a robust preprocessing pipeline for functional MRI. Nat. Methods 16, 111–116 (2019).

    Article  CAS  PubMed  Google Scholar 

  30. Murphy, K. & Fox, M. D. Towards a consensus regarding global signal regression for resting state functional connectivity MRI. Neuroimage 154, 169–173 (2017).

    Article  PubMed  Google Scholar 

  31. Zuo, X.-N. et al. An open science resource for establishing reliability and reproducibility in functional connectomics. Sci. Data 1, 140049 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. Shou, H. et al. Quantifying the reliability of image replication studies: the image intraclass correlation coefficient (I2C2). Cogn. Affect. Behav. Neurosci. 13, 714–724 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. Bridgeford, E. W. et al. Eliminating accidental deviations to minimize generalization error and maximize replicability: applications in connectomics and genomics. PLoS Comput. Biol. 17, e1009279 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. Schaefer, A. et al. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI. Cereb. Cortex 28, 3095–3114 (2018).

    Article  PubMed  Google Scholar 

  35. Glasser, M. F. et al. The minimal preprocessing pipelines for the Human Connectome Project. Neuroimage 80, 105–124 (2013).

    Article  PubMed  Google Scholar 

  36. Greve, D. N. & Fischl, B. Accurate and robust brain image alignment using boundary-based registration. Neuroimage 48, 63–72 (2009).

    Article  PubMed  Google Scholar 

  37. Alexander, L. M. et al. An open resource for transdiagnostic research in pediatric mental health and learning disorders. Sci. Data 4, 170181 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  38. Koo, T. K. & Li, M. Y. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J. Chiropr. Med. 15, 155–163 (2016).

  39. Dong, Y., Ifrim, G., Mladenić, D., Saunders, C. & Van Hoecke, S. Machine learning and knowledge discovery in databases. Applied data science and demo track. In Proc. European Conference, ECML PKDD 2020 Part V (eds Dong, Y. et al.) 3–18 (Springer Nature, 2021).

  40. Birn, R. M. et al. The effect of scan length on the reliability of resting-state fMRI connectivity estimates. Neuroimage 83, 550–558 (2013).

    Article  PubMed  Google Scholar 

  41. Liu, T. T., Nalci, A. & Falahpour, M. The global signal in fMRI: nuisance or information? Neuroimage 150, 213–229 (2017).

    Article  PubMed  Google Scholar 

  42. Ciric, R. et al. Benchmarking of participant-level confound regression strategies for the control of motion artifact in studies of functional connectivity. Neuroimage 154, 174–187 (2017).

    Article  PubMed  Google Scholar 

  43. Buades, A., Coll, B. & Morel, J.-M. Non-local means denoising. IPOL J. 1, 208–212 (2011).

    Article  Google Scholar 

  44. Tustison, N. J. et al. N4ITK: improved N3 bias correction. IEEE Trans. Med. Imaging 29, 1310–1320 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  45. Ciric, R. et al. TemplateFlow: FAIR-sharing of multi-scale, multi-species brain models. Nat. Methods 19, 1568–1571 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  46. Fonov, V. S., Evans, A. C., McKinstry, R. C., Almli, C. R. & Collins, D. L. Unbiased nonlinear average age-appropriate brain templates from birth to adulthood. Neuroimage 47, S102 (2009).

    Article  Google Scholar 

  47. Grabner, G. et al. Symmetric atlasing and model based segmentation: an application to the hippocampus in older adults. Med. Image Comput. Comput. Assist. Interv. 9, 58–66 (2006).

    PubMed  Google Scholar 

  48. Mazziotta, J. et al. A probabilistic atlas and reference system for the human brain: International Consortium for Brain Mapping (ICBM). Philos. Trans. R. Soc. Lond. B Biol. Sci. 356, 1293–1322 (2001).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  49. Wu, J. et al. Accurate nonlinear mapping between MNI volumetric and FreeSurfer surface coordinate systems. Hum. Brain Mapp. 39, 3793–3808 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  50. Uddin, L. Q. Mixed signals: on separating brain signal from noise. Trends Cogn. Sci. 21, 405–406 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  51. Murphy, K., Birn, R. M., Handwerker, D. A., Jones, T. B. & Bandettini, P. A. The impact of global signal regression on resting state correlations: are anti-correlated networks introduced? Neuroimage 44, 893–905 (2009).

    Article  PubMed  Google Scholar 

  52. Xu, H. et al. Impact of global signal regression on characterizing dynamic functional connectivity and brain states. Neuroimage 173, 127–145 (2018).

    Article  PubMed  Google Scholar 

  53. Gordon, E. M. et al. Precision functional mapping of individual human brains. Neuron 95, 791–807 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  54. Di Martino, A. et al. Enhancing studies of the connectome in autism using the autism brain imaging data exchange II. Sci. Data 4, 170010 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  55. Casey, B. J. et al. The Adolescent Brain Cognitive Development (ABCD) study: imaging acquisition across 21 sites. Dev. Cogn. Neurosci. 32, 43–54 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  56. Richie-Halford, A. et al. Author correction: an analysis-ready and quality controlled resource for pediatric brain white-matter research. Sci. Data 10, 247 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  57. Doshi, J. et al. MUSE: Multi-atlas region segmentation utilizing ensembles of registration algorithms and parameters, and locally optimal atlas selection. NeuroImage 127, 186–195 (2016).

  58. Wu, D. et al. Resource atlases for multi-atlas brain segmentations with multiple ontology levels based on T1-weighted MRI. Neuroimage 125, 120–130 (2016).

    Article  PubMed  Google Scholar 

  59. Kiar, G., Chatelain, Y., Salari, A., Evans, A. C. & Glatard, T. Data augmentation through Monte Carlo arithmetic leads to more generalizable classification in connectomics. Neurons Behav. Data Anal. Theory 1, 1–20 (2021).

    Google Scholar 

  60. Kiar, G. et al. Numerical uncertainty in analytical pipelines lead to impactful variability in brain networks. PLoS ONE 16, e0250755 (2021).

  61. Mehta, K. et al. XCP-D: a robust pipeline for the post-processing of fMRI data. Preprint at bioRxiv 10.1101/2023.11.20.567926 (2023).

  62. Bujang, M. & Baharum, N. A simplified guide to determination of sample size requirements for estimating the value of intraclass correlation coefficient: a review. Arch. Orofac. Sci. 12, 1–11 (2017).

    Google Scholar 

  63. Smith, S. M. et al. Advances in functional and structural MR image analysis and implementation as FSL. Neuroimage 23, S208–S219 (2004).

    Article  PubMed  Google Scholar 

  64. Jenkinson, M., Bannister, P., Brady, M. & Smith, S. Improved optimization for the robust and accurate linear registration and motion correction of brain images. Neuroimage 17, 825–841 (2002).

    Article  PubMed  Google Scholar 

  65. Cox, R. W. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput. Biomed. Res. 29, 162–173 (1996).

    Article  CAS  PubMed  Google Scholar 

  66. Zhang, Y., Brady, M. & Smith, S. Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm. IEEE Trans. Med. Imaging 20, 45–57 (2001).

  67. Avants, B. B., Tustison, N. & Song, G. Advanced normalization tools (ANTS). Insight J. 2, 1–35 (2009).

    Google Scholar 

  68. Fischl, B. FreeSurfer. Neuroimage 62, 774–781 (2012).

    Article  PubMed  Google Scholar 

  69. Jenkinson, M. & Smith, S. A global optimisation method for robust affine registration of brain images. Med. Image Anal. 5, 143–156 (2001).

    Article  CAS  PubMed  Google Scholar 

  70. Berger, V. W. & Zhou, Y. Kolmogorov–Smirnov test: overview https://doi.org/10.1002/9781118445112.stat06558 (2014).

  71. Nachar, N. The Mann-Whitney U: a test for assessing whether two independent samples come from the same distribution. Tutor. Quant. Methods Psychol. 4, 13–20 (2008).

    Article  Google Scholar 

  72. Li, X. & Clucas, J. XinhuiLi/PipelineHarmonization: Pipeline Harmonization Version 0.0.0 Beta. Zenodo https://doi.org/10.5281/zenodo.5733801 (2021).

Download references

Acknowledgements

This work was supported in part by gifts from J. P. Healey, P. Green and R. Cowen to the Child Mind Institute. In addition, grant awards were received from the NIH BRAIN Initiative to M.P.M. and C.C. (no. R24 MH11480602), to G.K. and M.P.M. (no. RF1MH130859) and to R.A.P., O.E., M.P.M. and T.S. (no. RF1MH121867); and from NIMH to T.S. and M.P.M. (no. R01MH120482). O.E. received support from SNSF Ambizione project no. 185872. C.-G.Y. received support from the National Natural Science Foundation of China (grant nos. 82122035, 81671774 and 81630031).

Author information

Authors and Affiliations

Authors

Contributions

G.K. and M.P.M. conceptualized the study design, supervised work, wrote the first draft of the manuscript and oversaw all revisions. X.L., N.B.E., L.A. and G.K. wrote the preprocessing, data analysis and computational model scripts. J.C. reviewed the code. A.S.H. and S.G. provided technical support. X.L., N.B.E., G.K. and M.P.M. wrote the first draft of the manuscript. All authors reviewed the manuscript and contributed to the discussion and results.

Corresponding author

Correspondence to Michael P. Milham.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Human Behaviour thanks Anders Eklund and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, X., Bianchini Esper, N., Ai, L. et al. Moving beyond processing- and analysis-related variation in resting-state functional brain imaging. Nat Hum Behav (2024). https://doi.org/10.1038/s41562-024-01942-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s41562-024-01942-4

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing