Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Brief Communication
  • Published:

ReX: an integrative tool for quantifying and optimizing measurement reliability for the study of individual differences

Nature Methods volume 20, pages 1025–1028 (2023)Cite this article

Subjects

Abstract

Characterizing multifaceted individual differences in brain function using neuroimaging is central to biomarker discovery in neuroscience. We provide an integrative toolbox, Reliability eXplorer (ReX), to facilitate the examination of individual variation and reliability as well as the effective direction for optimization of measuring individual differences in biomarker discovery. We also illustrate gradient flows, a two-dimensional field map-based approach to identifying and representing the most effective direction for optimization when measuring individual differences, which is implemented in ReX.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Theoretical individual variation field map in ReX and its applications.
Fig. 2: GFM in ReX and its application example.

Similar content being viewed by others

Data availability

Data used in application examples are available from public repositories. HCP data are available on ConnectomeDB (https://www.humanconnectome.org/study/hcp-young-adult)10. Self-regulation data are available on GitHub (https://github.com/IanEisenberg/Self_Regulation_Ontology)11. HNU data are available from the Consortium for Reliability and Reproducibility (https://fcon_1000.projects.nitrc.org/indi/CoRR/html/index.html)19. Application data and code are available on GitHub (https://github.com/TingsterX/Reliability_Explorer/tree/main/application_examples). Source data are provided with this paper.

Code availability

ReX is implemented using multiple R packages (lme4, dplyr, ggplot2, scales, stats, reshape2, shinybusy, colorspace, RColorBrewer). The toolbox is available under a GNU version 3 license on GitHub (https://github.com/tingsterx/reliability_explorer), with a web-based R–Shiny application on Docker Hub (tingsterx:reliability_explorer) and shinyapps.io: https://tingsterx.shinyapps.io/ReliabilityExplorer. Docker images of the command line version (tingsterx:rex) used in this paper are available on Docker Hub.

References

  1. Seghier, M. L. & Price, C. J. Interpreting and utilising intersubject variability in brain function. Trends Cogn. Sci. 22, 517–530 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  2. Dubois, J. & Adolphs, R. Building a science of individual differences from fMRI. Trends Cogn. Sci. 20, 425–443 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  3. Barch, D. M. et al. Function in the human connectome: task-fMRI and individual differences in behavior. Neuroimage 80, 169–189 (2013).

    Article  PubMed  Google Scholar 

  4. Finn, E. S. et al. Can brain state be manipulated to emphasize individual differences in functional connectivity? NeuroImage 160, 140–151 (2017).

  5. Lebreton, M., Bavard, S., Daunizeau, J. & Palminteri, S. Assessing inter-individual differences with task-related functional neuroimaging. Nat. Hum. Behav. 3, 897–905 (2019).

    Article  PubMed  Google Scholar 

  6. Van Horn, J. D., Grafton, S. T. & Miller, M. B. Individual variability in brain activity: a nuisance or an opportunity? Brain Imaging Behav. 2, 327–334 (2008).

    Article  PubMed  PubMed Central  Google Scholar 

  7. Palminteri, S. & Chevallier, C. Can we infer inter-individual differences in risk-taking from behavioral tasks? Front. Psychol. 9, 2307 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  8. Genon, S., Eickhoff, S. B. & Kharabian, S. Linking interindividual variability in brain structure to behaviour. Nat. Rev. Neurosci. 23, 307–318 (2022).

    Article  CAS  PubMed  Google Scholar 

  9. Hsu, S., Poldrack, R., Ram, N. & Wagner, A. D. Observed correlations from cross-sectional individual differences research reflect both between-person and within-person correlations. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/zq37h (2022).

  10. Van Essen, D. C. et al. The WU-Minn Human Connectome Project: an overview. NeuroImage 80, 62–79 (2013).

    Article  PubMed  Google Scholar 

  11. Enkavi, A. Z. et al. Large-scale analysis of test–retest reliabilities of self-regulation measures. Proc. Natl Acad. Sci. USA 116, 5472–5477 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Chen, G. et al. Intraclass correlation: improved modeling approaches and applications for neuroimaging. Hum. Brain Mapp. 39, 1187–1206 (2018).

    Article  PubMed  Google Scholar 

  13. Koo, T. K. & Li, M. Y. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J. Chiropr. Med. 15, 155–163 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  14. Xu, M., Reiss, P. T. & Cribben, I. Generalized reliability based on distances. Biometrics 77, 258–270 (2021).

    Article  PubMed  Google Scholar 

  15. Shou, H. et al. Quantifying the reliability of image replication studies: the image intraclass correlation coefficient (I2C2). Cogn. Affect. Behav. Neurosci. 13, 714–724 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  16. Bridgeford, E. W. et al. Eliminating accidental deviations to minimize generalization error and maximize replicability: applications in connectomics and genomics. PLoS Comput. Biol. 17, e1009279 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Finn, E. S. et al. Functional connectome fingerprinting: identifying individuals using patterns of brain connectivity. Nat. Neurosci. 18, 1664–1671 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Zuo, X.-N., Xu, T. & Milham, M. P. Harnessing reliability for neuroscience research. Nat. Hum. Behav. 3, 768–771 (2019).

    Article  PubMed  Google Scholar 

  19. Cho, J. W., Korchmaros, A., Vogelstein, J. T., Milham, M. P. & Xu, T. Impact of concatenating fMRI data on reliability for functional connectomics. NeuroImage 226, 117549 (2021).

    Article  PubMed  Google Scholar 

  20. Noble, S., Scheinost, D. & Constable, R. T. A guide to the measurement and interpretation of fMRI test–retest reliability. Curr. Opin. Behav. Sci. 40, 27–32 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  21. Steyer, R., Smelser, N. J. & Jena, D. Classical (psychometric) test theory. In International Encyclopedia of the Social & Behavioral Sciences Vol. 3, 1955–1962 (2001).

    Google Scholar 

  22. Kline, T. J. B. Psychological Testing: a Practical Approach to Design and Evaluation (SAGE, 2005).

  23. Noble, S., Scheinost, D. & Constable, R. T. A decade of test–retest reliability of functional connectivity: a systematic review and meta-analysis. NeuroImage 203, 116157 (2019).

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

We thank X. Li for organizing the preprocessed HNU data from different pipelines. This work is supported by gifts from J.P. Healey, P. Green and R. Cowen to the Child Mind Institute and National Institutes of Health funding (RF1MH128696 to T.X., R24MH114806 and 5R01MH124045 to M.P.M.). Additional grant support for J.T.V. comes from R01MH120482 (to T.D. Satterthwaite, M.P.M.), and he has funding from Microsoft Research.

Author information

Authors and Affiliations

  1. Department of Brain Development, Child Mind Institute, New York, NY, USA

    Ting Xu, Gregory Kiar, Jae Wook Cho, Aki Nikolaidis & Michael P. Milham

  2. Johns Hopkins University, Baltimore, MD, USA

    Eric W. Bridgeford & Joshua T. Vogelstein

  3. Nathan Kline Institute for Psychiatric Research, Orangeburg, NY, USA

    Michael P. Milham

Authors
  1. Ting Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gregory Kiar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jae Wook Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eric W. Bridgeford
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aki Nikolaidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Joshua T. Vogelstein
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Michael P. Milham
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

T.X. conceptualized and developed the software. T.X. and J.W.C. prepared the data. T.X. wrote the original draft with input from M.P.M., G.K. and J.T.V. All authors reviewed, edited and approved the manuscript.

Corresponding author

Correspondence to Ting Xu.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Methods thanks Ye Tian and the other, anonymous, reviewers for their contribution to the peer review of this work. Primary Handling Editor: Nina Vogt, in collaboration with the Nature Methods team. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Results for Application 4 to compare the impact of global signal regression in multiple fMRI preprocessing pipelines at the parcel level.

a) The within- and between-individual variance of GSR and No-GSR results from four pipelines. b) The change of within- and between-individual variance comparing GSR versus No-GSR results of the fMRIprep pipeline. c) The normalized change of the within- and between-individual variance comparing GSR versus No-GSR results of the fMRIprep pipeline.

Source data

Supplementary information

Supplementary Information

Supplementary Note and Figs. 1–5

Reporting Summary

Peer Review File

Supplementary Video 1

The theoretical relationship between reliability and validity. Validity is determined by the proportion of variation for the trait of interest to the total variation of the observed score. If there is a signal but it is not related to the trait (that is, contaminator relative to the trait), validity is lower than reliability (https://github.com/TingsterX/Reliability_Explorer/blob/main/reliability_and_validity/reliability_and_validity.md).

Source data

Source Data Fig. 1

Reliability of the demo data calculated in ReX.

Source Data Fig. 2

Reliability of the National Institutes of Health Toolbox and self-regulation measures.

Source Data Extended Data Fig. 1

Reliability (dbICC) of functional connectivity at the parcel level for multiple functional magnetic resonance imaging preprocessing pipelines.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, T., Kiar, G., Cho, J.W. et al. ReX: an integrative tool for quantifying and optimizing measurement reliability for the study of individual differences. Nat Methods 20, 1025–1028 (2023). https://doi.org/10.1038/s41592-023-01901-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41592-023-01901-3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics