Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Harnessing reliability for neuroscience research

Nature Human Behaviour volume 3, pages 768–771 (2019)Cite this article

Subjects

Neuroscientists are amassing the large-scale datasets needed to study individual differences and identify biomarkers. However, measurement reliability within individual samples is often suboptimal, thereby requiring unnecessarily large samples. We focus our comment on reliability in neuroimaging and provide examples of how the reliability can be increased.

The neuroimaging community has made significant strides towards collecting large-scale neuroimaging datasets, which—until the past decade—had seemed out of reach. Between initiatives focused on the aggregation and open sharing of previously collected datasets and de novo data generation initiatives tasked with the creation of community resources, tens of thousands of datasets are now available online. These span a range of developmental statuses and disorders, and many more will soon be available. Such open data are allowing researchers to increase the scale of their studies, to apply various learning strategies (for example, artificial intelligence) with ambitions of brain-based biomarker discovery and to address questions regarding the reproducibility of findings, all at a pace that is unprecedented in imaging. However, based on the findings of recent works1,2,3, few of the datasets generated to date contain enough data per subject to achieve highly reliable measures of brain connectivity. Although our examination of this critical deficiency focuses on the field of neuroimaging, the implications of our argument and the statistical principles discussed are broadly applicable.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Interactions between reliability, sample and effect size for a given statistical power.
Fig. 2: Reliability and validity of measurements of individual differences.

Similar content being viewed by others

Code availability

All code employed in this effort can be found on GitHub at https://github.com/TingsterX/power__reliability_sample_size

References

  1. Laumann, T. O. et al. Neuron 87, 657–670 (2015).

    Article  CAS  Google Scholar 

  2. O’Connor, D. et al. Gigascience 6, 1–14 (2017).

    Article  Google Scholar 

  3. Xu, T., Opitz, A., Craddock, C., Zuo, X. N. & Milham, M. P. Cereb. Cortex 26, 4192–4211 (2016).

    Article  Google Scholar 

  4. Zuo, X. N. & Xing, X. X. Neurosci. Biobehav. Rev. 45, 100–118 (2014).

    Article  Google Scholar 

  5. Kanyongo, G. Y., Brook, G. P., Kyei-Blankson, L. & Gocmen, G. J. Mod. Appl. Stat. Methods 6, Article 9 (2007).

    Article  Google Scholar 

  6. Poldrack, R. A. et al. Nat. Rev. Neurosci. 18, 115–126 (2017).

    Article  CAS  Google Scholar 

  7. Bennett, C. M. & Miller, M. B. Ann. NY Acad. Sci. 1191, 133–155 (2010).

    Article  Google Scholar 

  8. Hedge, C., Powell, G. & Sumner, P. Behav. Res. Methods 50, 1166–1186 (2018).

    Article  Google Scholar 

  9. Button, K. S. et al. Nat. Rev. Neurosci. 14, 365–376 (2013).

    Article  CAS  Google Scholar 

  10. Zuo, X.N., Biswal, B.B. & Poldrack, R.A. Front. Neurosci. 13, 117 (2019).

    Article  Google Scholar 

  11. Tomasi, D. G., Shokri-Kojori, E. & Volkow, N. D. Cereb. Cortex 27, 4153–4165 (2017).

    PubMed  Google Scholar 

  12. Elliott, M. L. et al. Neuroimage 189, 516–532 (2019).

    Article  Google Scholar 

  13. Nichols, T. E. et al. Nat. Neurosci. 20, 299–303 (2017).

    Article  CAS  Google Scholar 

  14. Koo, T. K. & Li, M. Y. J. Chiropr. Med. 15, 155–163 (2016).

    Article  Google Scholar 

  15. Kraemer, H. C. Annu. Rev. Clin. Psychol. 10, 111–130 (2014).

    Article  Google Scholar 

  16. Yan, C. G. et al. Neuroimage 76, 183–201 (2013).

    Article  Google Scholar 

Download references

Acknowledgements

We thank X. Castellanos, A. Franco, H. Kimball, A. Nikolaidis and X.-X. Xing for their helpful comments in the preparation of this commentary, as well as D. Klein for his guidance and encouragement of our focus on issues of reliability over the years, X. Castellanos for his support along the way, and all the contributors from CoRR and R3BRAIN for their enthusiasm on open neuroscience and data sharing. The two consortia are supported in part by the National Basic Research (973) Program (2015CB351702).

Author information

Authors and Affiliations

  1. Department of Psychology, University of Chinese Academy of Sciences (UCAS), Beijing, Beijing, China

    Xi-Nian Zuo

  2. Key Laboratory of Brain and Education, Nanning Normal University, Nanning, Guangxi, China

    Xi-Nian Zuo

  3. Institute for Brain Research and Rehabilitation, South China Normal University, Guangzhou, China

    Xi-Nian Zuo

  4. CAS Key Laboratory of Behavioral Sciences, Research Center for Lifespan Development of Mind and Brain (CLIMB) and Magnetic Resonance Imaging Research Center, CAS Institute of Psychology, Beijing, Beijing, China

    Xi-Nian Zuo & Ting Xu

  5. Center for Developing Brain, Child Mind Institute, New York, NY, USA

    Ting Xu & Michael Peter Milham

  6. Center for Biomedical Imaging and Neuromodulation, Nathan S. Kline Institute for Psychiatric Research, Orangeburg, NY, USA

    Michael Peter Milham

Authors
  1. Xi-Nian Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ting Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael Peter Milham
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xi-Nian Zuo.

Ethics declarations

Competing interests

The authors declare no competing interests.

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zuo, XN., Xu, T. & Milham, M.P. Harnessing reliability for neuroscience research. Nat Hum Behav 3, 768–771 (2019). https://doi.org/10.1038/s41562-019-0655-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-019-0655-x

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing