Comment | Published:

Harnessing reliability for neuroscience research

Neuroscientists are amassing the large-scale datasets needed to study individual differences and identify biomarkers. However, measurement reliability within individual samples is often suboptimal, thereby requiring unnecessarily large samples. We focus our comment on reliability in neuroimaging and provide examples of how the reliability can be increased.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.


All prices are NET prices.

Code availability

All code employed in this effort can be found on GitHub at

Ethics declarations

Competing interests

The authors declare no competing interests.


  1. 1.

    Laumann, T. O. et al. Neuron 87, 657–670 (2015).

  2. 2.

    O’Connor, D. et al. Gigascience 6, 1–14 (2017).

  3. 3.

    Xu, T., Opitz, A., Craddock, C., Zuo, X. N. & Milham, M. P. Cereb. Cortex 26, 4192–4211 (2016).

  4. 4.

    Zuo, X. N. & Xing, X. X. Neurosci. Biobehav. Rev. 45, 100–118 (2014).

  5. 5.

    Kanyongo, G. Y., Brook, G. P., Kyei-Blankson, L. & Gocmen, G. J. Mod. Appl. Stat. Methods 6, Article 9 (2007).

  6. 6.

    Poldrack, R. A. et al. Nat. Rev. Neurosci. 18, 115–126 (2017).

  7. 7.

    Bennett, C. M. & Miller, M. B. Ann. NY Acad. Sci. 1191, 133–155 (2010).

  8. 8.

    Hedge, C., Powell, G. & Sumner, P. Behav. Res. Methods 50, 1166–1186 (2018).

  9. 9.

    Button, K. S. et al. Nat. Rev. Neurosci. 14, 365–376 (2013).

  10. 10.

    Zuo, X.N., Biswal, B.B. & Poldrack, R.A. Front. Neurosci. 13, 117 (2019).

  11. 11.

    Tomasi, D. G., Shokri-Kojori, E. & Volkow, N. D. Cereb. Cortex 27, 4153–4165 (2017).

  12. 12.

    Elliott, M. L. et al. Neuroimage 189, 516–532 (2019).

  13. 13.

    Nichols, T. E. et al. Nat. Neurosci. 20, 299–303 (2017).

  14. 14.

    Koo, T. K. & Li, M. Y. J. Chiropr. Med. 15, 155–163 (2016).

  15. 15.

    Kraemer, H. C. Annu. Rev. Clin. Psychol. 10, 111–130 (2014).

  16. 16.

    Yan, C. G. et al. Neuroimage 76, 183–201 (2013).

Download references


We thank X. Castellanos, A. Franco, H. Kimball, A. Nikolaidis and X.-X. Xing for their helpful comments in the preparation of this commentary, as well as D. Klein for his guidance and encouragement of our focus on issues of reliability over the years, X. Castellanos for his support along the way, and all the contributors from CoRR and R3BRAIN for their enthusiasm on open neuroscience and data sharing. The two consortia are supported in part by the National Basic Research (973) Program (2015CB351702).

Author information

Competing interests

The authors declare no competing interests.

Correspondence to Xi-Nian Zuo.

Rights and permissions

Reprints and Permissions

About this article

Fig. 1: Interactions between reliability, sample and effect size for a given statistical power.
Fig. 2: Reliability and validity of measurements of individual differences.