Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Reporting checklists in neuroimaging: promoting transparency, replicability, and reproducibility

Abstract

Neuroimaging plays a crucial role in understanding brain structure and function, but the lack of transparency, reproducibility, and reliability of findings is a significant obstacle for the field. To address these challenges, there are ongoing efforts to develop reporting checklists for neuroimaging studies to improve the reporting of fundamental aspects of study design and execution. In this review, we first define what we mean by a neuroimaging reporting checklist and then discuss how a reporting checklist can be developed and implemented. We consider the core values that should inform checklist design, including transparency, repeatability, data sharing, diversity, and supporting innovations. We then share experiences with currently available neuroimaging checklists. We review the motivation for creating checklists and whether checklists achieve their intended objectives, before proposing a development cycle for neuroimaging reporting checklists and describing each implementation step. We emphasize the importance of reporting checklists in enhancing the quality of data repositories and consortia, how they can support education and best practices, and how emerging computational methods, like artificial intelligence, can help checklist development and adherence. We also highlight the role that funding agencies and global collaborations can play in supporting the adoption of neuroimaging reporting checklists. We hope this review will encourage better adherence to available checklists and promote the development of new ones, and ultimately increase the quality, transparency, and reproducibility of neuroimaging research.

This is a preview of subscription content, access via your institution

Access options

Buy this article

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Addiction Cue-Reactivity Initiative (ACRI) fMRI Drug Cue Reactivity (FDCR) checklist development process and outcomes.
Fig. 2: Development cycle of reporting checklists.
Fig. 3: Checklist development process and research process coverage in sample neuroimaging checklists.
Fig. 4: Reporting status of the Addiction Cue-Reactivity Initiative (ACRI) fMRI Drug Cue Reactivity (FDCR) checklist.

Similar content being viewed by others

References

  1. Carp J. The secret lives of experiments: methods reporting in the fMRI literature. Neuroimage. 2012;63:289–300.

    Article  PubMed  Google Scholar 

  2. Gorgolewski KJ, Poldrack RA. A practical guide for improving transparency and reproducibility in neuroimaging research. PLOS Biol. 2016;14:e1002506.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Poldrack RA, Baker CI, Durnez J, Gorgolewski KJ, Matthews PM, Munafò MR, et al. Scanning the horizon: towards transparent and reproducible neuroimaging research. Nat Rev Neurosci. 2017;18:115–26.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Fusar-Poli P, Radua J, Frascarelli M, Mechelli A, Borgwardt S, Di Fabio F, et al. Evidence of reporting biases in voxel-based morphometry (VBM) studies of psychiatric and neurological disorders: reporting biases in VBM Studies of Psychiatric and Neurological Disorders. Hum Brain Mapp. 2014;35:3052–65.

    Article  PubMed  Google Scholar 

  5. David SP, Naudet F, Laude J, Radua J, Fusar-Poli P, Chu I, et al. Potential reporting bias in neuroimaging studies of sex differences. Sci Rep. 2018;8:6082.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Etkin A. A reckoning and research agenda for neuroimaging in psychiatry. AJP. 2019;176:507–11.

    Article  Google Scholar 

  7. Robbins KA, Touryan J, Mullen T, Kothe C, Bigdely-Shamlo N. How sensitive are EEG results to preprocessing methods: a benchmarking study. IEEE Trans Neural Syst Rehabil Eng. 2020;28:1081–90.

    Article  PubMed  Google Scholar 

  8. Gentili C, Cecchetti L, Handjaras G, Lettieri G, Cristea IA. The case for preregistering all region of interest (ROI) analyses in neuroimaging research. Eur J Neurosci. 2021;53:357–61.

    Article  CAS  PubMed  Google Scholar 

  9. Pernet C, Garrido MI, Gramfort A, Maurits N, Michel CM, Pang E, et al. Issues and recommendations from the OHBM COBIDAS MEEG committee for reproducible EEG and MEG research. Nat Neurosci. 2020;23:1473–83.

    Article  CAS  PubMed  Google Scholar 

  10. Carp J. Better living through transparency: improving the reproducibility of fMRI results through comprehensive methods reporting. Cogn Affect Behav Neurosci. 2013;13:660–6.

    Article  PubMed  Google Scholar 

  11. Klapwijk ET, van den Bos W, Tamnes CK, Raschle NM, Mills KL. Opportunities for increased reproducibility and replicability of developmental neuroimaging. Dev Cogn Neurosci. 2021;47:100902.

    Article  PubMed  Google Scholar 

  12. Hupalo S, Jordan CJ, Bowen T, Mahar J, Yepez E, Kunath L, et al. NPP’s approach toward improving rigor and transparency in clinical trials research. Neuropsychopharmacology. 2023;48:429–31.

    Article  PubMed  Google Scholar 

  13. Nichols TE, Das S, Eickhoff SB, Evans AC, Glatard T, Hanke M, et al. Best practices in data analysis and sharing in neuroimaging using MRI. Nat Neurosci. 2017;20:299–303.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Bossuyt, Reitsma PM, Bruns DE JB, Gatsonis CA, Glasziou PP, Irwig L, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. Clin Chem. 2015;61:1446–52.

    Article  CAS  PubMed  Google Scholar 

  15. Ekhtiari H, Zare-Bidoky M, Sangchooli A, Janes AC, Kaufman MJ, Oliver JA, et al. A methodological checklist for fMRI drug cue reactivity studies: development and expert consensus. Nat Protoc. 2022;17:567–95.

  16. Ekhtiari H, Ghobadi-Azbari P, Thielscher A, Antal A, Li LM, Shereen AD, et al. A checklist for assessing the methodological quality of concurrent tES-fMRI studies (ContES checklist): a consensus study and statement. Nat Protoc. 2022;17:596–617.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Kousta S, Pastrana E, Swaminathan S. Three approaches to support reproducible research. Sci Editor. 2020;42:77–82.

    Google Scholar 

  18. The NPQIP Collaborative group, Study steering committee, Macleod M, Sena E, Howells D, Macleod M, et al. Did a change in Nature Journals’ editorial policy for life sciences research improve reporting? BMJ Open Sci [Internet]. 2019 Feb [cited 2024 Mar 27]; 3. Available from: http://access.portico.org/stable?au=phzq8gmxdp1.

  19. Feng X, Park DS, Walker C, Peterson AT, Merow C, Papeş M. A checklist for maximizing reproducibility of ecological niche models. Nat Ecol Evol. 2019;3:1382–95.

    Article  PubMed  Google Scholar 

  20. de Jong Y, van der Willik EM, Milders J, Voorend CGN, Morton RL, Dekker FW, et al. A meta-review demonstrates improved reporting quality of qualitative reviews following the publication of COREQ- and ENTREQ-checklists, regardless of modest uptake. BMC Med Res Methodol. 2021;21:184.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Uddin MN, Figley TD, Kornelsen J, Mazerolle EL, Helmick CA, O’Grady CB, et al. The comorbidity and cognition in multiple sclerosis (CCOMS) neuroimaging protocol: Study rationale, MRI acquisition, and minimal image processing pipelines. Front Neuroimaging [Internet]. 2022 Aug [cited 2024 Mar 10];1. Available from: https://www.frontiersin.org/articles/10.3389/fnimg.2022.970385.

  22. Appelbaum M, Cooper H, Kline RB, Mayo-Wilson E, Nezu AM, Rao SM. Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. Am Psychol. 2018;73:3–25.

    Article  PubMed  Google Scholar 

  23. Köhler T, González-Morales MG, Banks GC, O’Boyle EH, Allen JA, Sinha R, et al. Supporting robust, rigorous, and reliable reviewing as the cornerstone of our profession: introducing a competency framework for peer review. Ind Organ Psychol. 2020;13:1–27.

    Article  Google Scholar 

  24. Higgins JPT, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Gorgolewski KJ, Auer T, Calhoun VD, Craddock RC, Das S, Duff EP, et al. The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Sci Data. 2016;3:160044.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Rid A, Schmidt H. The 2008 Declaration of Helsinki - first among equals in research ethics? J Law Med Ethics. 2010;38:143–8.

    Article  PubMed  Google Scholar 

  27. Buch ER, Santarnecchi E, Antal A, Born J, Celnik PA, Classen J, et al. Effects of tDCS on motor learning and memory formation: a consensus and critical position paper. Clin Neurophysiol. 2017;128:589–603.

    Article  PubMed  Google Scholar 

  28. Choi I, Kreis R. Advanced methodology for in vivo magnetic resonance spectroscopy. NMR Biomed. 2021;34:e4504.

    Article  PubMed  Google Scholar 

  29. Marek S, Tervo-Clemmens B, Calabro FJ, Montez DF, Kay BP, Hatoum AS, et al. Reproducible brain-wide association studies require thousands of individuals. Nature. 2022;603:654–60.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Elyounssi S, Kunitoki K, Clauss JA, Laurent E, Kane K, Hughes DE, et al. Uncovering and mitigating bias in large, automated MRI analyses of brain development. bioRxiv. 2023 Jan;2023.02.28.530498.

  31. Allen K, Geimer JL, Popp E. Context matters: developing peer reviewers to advance science and practice. Ind Organ Psychol. 2020;13:57–60.

    Article  Google Scholar 

  32. Nieminen P. Ten points for high-quality statistical reporting and data presentation. Appl Sci. 2020;10:3885.

    Article  CAS  Google Scholar 

  33. Eby LT, Shockley KM, Bauer TN, Edwards B, Homan AC, Johnson R, et al. Methodological checklists for improving research quality and reporting consistency. Ind Organ Psychol. 2020;13:76–83.

    Article  Google Scholar 

  34. Garcia-Costa D, Squazzoni F, Mehmani B, Grimaldo F. Measuring the developmental function of peer review: a multi-dimensional, cross-disciplinary analysis of peer review reports from 740 academic journals. PeerJ. 2022;10:e13539.

    Article  PubMed  PubMed Central  Google Scholar 

  35. ALBA Network. Alba Network. [cited 2024 Mar 22]. ALBA Declaration on Equity and Inclusion. 2024 Available from: https://www.alba.network/declaration.

  36. Tzovara A, Amarreh I, Borghesani V, Chakravarty MM, DuPre E, Grefkes C, et al. Embracing diversity and inclusivity in an academic setting: Insights from the Organization for Human Brain Mapping. NeuroImage. 2021;229:117742.

    Article  PubMed  Google Scholar 

  37. Silver JK. Is a lack of diversity among clinical practice guideline authors contributing to health inequalities for patients? BMJ. 2023;381:p1035.

    Article  Google Scholar 

  38. Synnot A, Hill S, Jauré A, Merner B, Hill K, Bates P, et al. Broadening the diversity of consumers engaged in guidelines: a scoping review. BMJ Open. 2022;12:e058326.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Poldrack RA, Fletcher PC, Henson RN, Worsley KJ, Brett M, Nichols TE. Guidelines for reporting an fMRI study. Neuroimage. 2008;40:409–14.

    Article  PubMed  Google Scholar 

  40. Nichols TE, Das S, Eickhoff SB, Evans AC, Glatard T, Hanke M, et al. Best practices in data analysis and sharing in neuroimaging using MRI [Internet]. Neuroscience; 2016 May [cited 2024 Mar 18]. Available from: http://biorxiv.org/lookup/doi/10.1101/054262.

  41. Pernet CR, Garrido M, Gramfort A, Maurits N, Michel C, Pang E, et al. Best practices in data analysis and sharing in neuroimaging using MEEG [Internet]. Open Science Framework; 2018 Aug [cited 2024 Mar 21]. Available from: https://osf.io/a8dhx.

  42. Uddin LQ, Betzel RF, Cohen JR, Damoiseaux JS, De Brigard F, Eickhoff S, et al. Controversies and progress on standardization of large-scale brain network nomenclature [Internet]. Open Science Framework; 2022 Mar [cited 2024 Mar 21]. Available from: https://osf.io/25za6.

  43. Voets N et al. COBIDAS Clinical fMRI for language mapping. [cited 2024 Mar 21]. COBIDAS Clinical fMRI for language mapping. 2023 Available from: https://cobidasclinicalfmriforlanguagemapping.wordpress.com/.

  44. Oz G, Alger JR, Barker PB, Bartha R, Bizzi A, Boesch C, et al. Clinical proton MR spectroscopy in central nervous system disorders. Radiology. 2014;270:658–79.

    Article  PubMed  Google Scholar 

  45. Wilson M, Andronesi O, Barker PB, Bartha R, Bizzi A, Bolan PJ, et al. Methodological consensus on clinical proton MRS of the brain: review and recommendations. Magn Reson Med. 2019;82:527–50.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Peek AL, Rebbeck T, Puts NAJ, Watson J, Aguila MER, Leaver AM. Brain GABA and glutamate levels across pain conditions: a systematic literature review and meta-analysis of 1H-MRS studies using the MRS-Q quality assessment tool. NeuroImage. 2020;210:116532.

    Article  CAS  PubMed  Google Scholar 

  47. Öngür D. Making progress with magnetic resonance spectroscopy. JAMA Psychiatry. 2013;70:1265.

    Article  PubMed  Google Scholar 

  48. Lin A, Andronesi O, Bogner W, Choi I, Coello E, Cudalbu C, et al. Minimum Reporting Standards for in vivo Magnetic Resonance Spectroscopy (MRSinMRS): experts’ consensus recommendations. NMR Biomed. 2021;34:e4484.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Addiction Cue-Reactivity Initiative (ACRI) Network. Parameter Space and Potential for Biomarker Development in 25 Years of fMRI Drug Cue Reactivity: A Systematic Review. JAMA Psychiatry [Internet]. 2024 Feb [cited 2024 Feb 13]; Available from: https://doi.org/10.1001/jamapsychiatry.2023.5483.

  50. Knudsen GM, Ganz M, Appelhoff S, Boellaard R, Bormans G, Carson RE, et al. Guidelines for the content and format of PET brain data in publications and archives: a consensus paper. J Cereb Blood Flow Metab. 2020;40:1576–85.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7:e1000217.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Waggoner J, Carline JD, Durning SJ. Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research. Acad Med. 2016;91:663–8.

    Article  PubMed  Google Scholar 

  53. Gratton C, Nelson SM, Gordon EM. Brain-behavior correlations: two paths toward reliability. Neuron. 2022;110:1446–9.

    Article  CAS  PubMed  Google Scholar 

  54. Kragel PA, Han X, Kraynak TE, Gianaros PJ, Wager TD. Functional MRI can be highly reliable, but it depends on what you measure: a commentary on Elliott et al. (2020). Psychol Sci. 2021;32:622–6.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Banks GC, Rogelberg SG, Woznyj HM, Landis RS, Rupp DE. Editorial: evidence on questionable research practices: the good, the bad, and the ugly. J Bus Psychol. 2016;31:323–38.

    Article  Google Scholar 

  56. Ganz M, Poldrack RA. Data sharing in neuroimaging: experiences from the BIDS project. Nat Rev Neurosci. 2023;24:729–30.

    Article  CAS  PubMed  Google Scholar 

  57. Li X, Guo N, Li Q. Functional neuroimaging in the new era of big data. Genomics Proteom Bioinforma. 2019;17:393–401.

    Article  Google Scholar 

  58. Webb-Vargas Y, Chen S, Fisher A, Mejia A, Xu Y, Crainiceanu C, et al. Big data and neuroimaging. Stat Biosci. 2017;9:543–58.

    Article  PubMed  PubMed Central  Google Scholar 

  59. Sudlow C, Gallacher J, Allen N, Beral V, Burton P, Danesh J, et al. UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med. 2015;12:e1001779.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Casey BJ, Cannonier T, Conley MI, Cohen AO, Barch DM, Heitzeg MM, et al. The Adolescent Brain Cognitive Development (ABCD) study: Imaging acquisition across 21 sites. Dev Cogn Neurosci. 2018;32:43–54.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  61. Markiewicz CJ, Gorgolewski KJ, Feingold F, Blair R, Halchenko YO, Miller E, et al. OpenNeuro: An open resource for sharing of neuroimaging data. bioRxiv. 2021.

  62. Wilkinson MD, Dumontier M, Aalbersberg IJJ, Appleton G, Axton M, Baak A, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016;3:160018.

    Article  PubMed  PubMed Central  Google Scholar 

  63. Saunders JB, Aasland OG, Babor TF, De La Fuente JR, Grant M. Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO collaborative project on early detection of persons with harmful alcohol consumption-II. Addiction. 1993;88:791–804.

    Article  CAS  PubMed  Google Scholar 

  64. Grasby KL, Jahanshad N, Painter JN, Colodro-Conde L, Bralten J, Hibar DP, et al. The genetic architecture of the human cerebral cortex. Science. 2020;367:eaay6690.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  65. Thompson PM, Jahanshad N, Ching CRK, Salminen LE, Thomopoulos SI, Bright J, et al. ENIGMA and global neuroscience: a decade of large-scale studies of the brain in health and disease across more than 40 countries. Transl Psychiatry. 2020;10:1–28.

    Article  Google Scholar 

  66. Mackey S, Kan KJ, Chaarani B, Alia-Klein N, Batalla A, Brooks S. et al. Chapter 10 - Genetic imaging consortium for addiction medicine: From neuroimaging to genes. In: Ekhtiari H, Paulus MP, editors. Progress in Brain Research [Internet]. Elsevier; 2016. p. 203–23. https://www.sciencedirect.com/science/article/pii/S0079612315001326 [cited 2021 Aug 14](Neuroscience for Addiction Medicine: From Prevention to Rehabilitation - Methods and Interventions; vol. 224).

  67. Mackey S, Allgaier N, Chaarani B, Spechler P, Orr C, Bunn J, et al. Mega-analysis of gray matter volume in substance dependence: general and substance-specific regional effects. AJP. 2019;176:119–28.

    Article  Google Scholar 

  68. Cao Z, McCabe M, Callas P, Cupertino RB, Ottino-González J, Murphy A, et al. Recalibrating single-study effect sizes using hierarchical Bayesian models. Front Neuroimaging [Internet]. 2023 Dec [cited 2024 Mar 22];2. Available from: https://www.frontiersin.org/articles/10.3389/fnimg.2023.1138193.

  69. Smit DJA, Andreassen OA, Boomsma DI, Burwell SJ, Chorlian DB, de Geus EJC, et al. Large-scale collaboration in ENIGMA-EEG: a perspective on the meta-analytic approach to link neurological and psychiatric liability genes to electrophysiological brain activity. Brain Behav. 2021;11:e02188.

    Article  PubMed  PubMed Central  Google Scholar 

  70. Caeyenberghs K, Imms P, Irimia A, Monti MM, Esopenko C, de Souza NL, et al. ENIGMA’s simple seven: Recommendations to enhance the reproducibility of resting-state fMRI in traumatic brain injury. NeuroImage: Clin. 2024;42:103585.

    Article  PubMed  Google Scholar 

  71. Agha RA, Fowler AJ, Limb C, Whitehurst K, Coe R, Sagoo H, et al. Impact of the mandatory implementation of reporting guidelines on reporting quality in a surgical journal: A before and after study. Int J Surg. 2016;30:169–72.

    Article  PubMed  Google Scholar 

  72. Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T, et al. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Methodology Review Group, editor. Cochrane Database of Systematic Reviews [Internet]. 2012 Nov [cited 2024 Mar 31]; 2013. Available from: https://doi.org/10.1002/14651858.MR000030.pub2.

  73. Vilaró M, Cortés J, Selva-O’Callaghan A, Urrutia A, Ribera JM, Cardellach F, et al. Adherence to reporting guidelines increases the number of citations: the argument for including a methodologist in the editorial process and peer-review. BMC Med Res Methodol. 2019;19:112.

    Article  PubMed  PubMed Central  Google Scholar 

  74. Shamseer L, Hopewell S, Altman DG, Moher D, Schulz KF. Update on the endorsement of CONSORT by high impact factor journals: a survey of journal “Instructions to Authors” in 2014. Trials. 2016;17:301.

    Article  PubMed  PubMed Central  Google Scholar 

  75. Taylor R, Kardas M, Cucurull G, Scialom T, Hartshorn A, Saravia E, et al. Galactica: A Large Language Model for Science [Internet]. arXiv; 2022 [cited 2024 Mar 22]. Available from: http://arxiv.org/abs/2211.09085.

  76. Liu R, Shah NB ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing [Internet]. arXiv; 2023 [cited 2024 Mar 22]. Available from: http://arxiv.org/abs/2306.00622.

  77. Schulz KF, Altman DG, Moher D. the CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMC Med. 2010;8:18.

    Article  PubMed  PubMed Central  Google Scholar 

  78. Hamilton CM, Strader LC, Pratt JG, Maiese D, Hendershot T, Kwok RK, et al. The PhenX Toolkit: get the most from your measures. Am J Epidemiol. 2011;174:253–60.

    Article  PubMed  PubMed Central  Google Scholar 

  79. Ekhtiari H, Zare-Bidoky M, Sangchooli A, Janes AC, Kaufman MJ, Oliver JA, et al. A methodological checklist for fMRI drug cue reactivity studies: development and expert consensus. Nat Protoc. 2022;17:567–95.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  80. Duarte RV, Bresnahan R, Copley S, Eldabe S, Thomson S, North RB, et al. Reporting guidelines for clinical trial protocols and reports of implantable neurostimulation devices: protocol for the SPIRIT-iNeurostim and CONSORT-iNeurostim extensions. Neuromodulation Technol Neural Interface. 2022;25:1045–9.

    Article  Google Scholar 

  81. Simera I, Moher D, Hirst A, Hoey J, Schulz KF, Altman DG. Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network. BMC Med. 2010;8:24.

    Article  PubMed  PubMed Central  Google Scholar 

  82. Sarafoglou A, Hoogeveen S, Matzke D, Wagenmakers EJ. Teaching good research practices: protocol of a research master course. Psychol Learn Teach. 2020;19:46–59.

    Article  Google Scholar 

  83. Kohrs FE, Auer S, Bannach-Brown A, Fiedler S, Haven TL, Heise V, et al. Eleven strategies for making reproducible research and open science training the norm at research institutions. Zaidi M, editor. eLife. 2023;12:e89736.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  84. Pownall M, Azevedo F, König LM, Slack HR, Evans TR, Flack Z, et al. Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes. R Soc Open Sci. 2023;10:221255.

    Article  PubMed  PubMed Central  Google Scholar 

  85. van Viegen T, Akrami A, Bonnen K, DeWitt E, Hyafil A, Ledmyr H, et al. Neuromatch Academy: teaching computational neuroscience with global accessibility. Trends Cogn Sci. 2021;25:535–8.

    Article  Google Scholar 

  86. Moher D, Altman DG, Schulz KF, Simera I. How to Develop a Reporting Guideline. In: Moher D, Altman DG, Schulz KF, Simera I, Wager E, editors. Guidelines for Reporting Health Research: A User’s Manual [Internet]. 1st ed. Wiley; 2014. p. 14–21. https://onlinelibrary.wiley.com/doi/10.1002/9781118715598.ch2.

    Chapter  Google Scholar 

  87. Bastow R, Leonelli S. Sustainable digital infrastructure. EMBO Rep. 2010;11:730–4.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  88. Zakaria S, Grant J, Luff J. Fundamental challenges in assessing the impact of research infrastructure. Health Res Policy Sys. 2021;19:119.

    Article  Google Scholar 

  89. Barker M, Katz DS. Overview of research software funding landscape. 2022 Feb [cited 2024 Mar 22]; Available from: https://zenodo.org/records/6102487.

  90. RFA-MH-22-145: BRAIN Initiative: Standards to Define Experiments Related to the BRAIN Initiative (R01 Clinical Trial Not Allowed) [Internet]. [cited 2024 Mar 26]. Available from: https://grants.nih.gov/grants/guide/rfa-files/RFA-MH-22-145.html.

  91. Backhausen LL, Herting MM, Tamnes CK, Vetter NC. Best practices in structural neuroimaging of neurodevelopmental disorders. Neuropsychol Rev. 2022;32:400–18.

    Article  PubMed  Google Scholar 

  92. Wachinger C, Rieckmann A, Pölsterl S. Alzheimer’s Disease Neuroimaging Initiative. Detect and correct bias in multi-site neuroimaging datasets. Med Image Anal. 2021;67:101879.

    Article  PubMed  Google Scholar 

  93. Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T, et al. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev. 2012;11:MR000030.

    PubMed  Google Scholar 

  94. Altman DG, Simera I, Hoey J, Moher D, Schulz K. EQUATOR: reporting guidelines for health research. Lancet. 2008;371:1149–50.

    Article  PubMed  Google Scholar 

  95. Ros T, Enriquez-Geppert S, Zotev V, Young KD, Wood G, Whitfield-Gabrieli S, et al. Consensus on the reporting and experimental design of clinical and cognitive-behavioural neurofeedback studies (CRED-nf checklist). Brain. 2020;143:1674–85.

    Article  PubMed  PubMed Central  Google Scholar 

  96. Davis KD, Flor H, Greely HT, Iannetti GD, Mackey S, Ploner M, et al. Brain imaging tests for chronic pain: medical, legal and ethical issues and recommendations. Nat Rev Neurol. 2017;13:624–38.

    Article  PubMed  Google Scholar 

  97. Cisek P. Making decisions through a distributed consensus. Curr Opin Neurobiol. 2012;22:927–36.

    Article  CAS  PubMed  Google Scholar 

  98. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6:e1000097.

    Article  PubMed  PubMed Central  Google Scholar 

  99. Jorm AF. Using the Delphi expert consensus method in mental health research. Aust N Z J Psychiatry. 2015;49:887–97.

    Article  PubMed  Google Scholar 

  100. Eickhoff S, Nichols TE, van Horn JD, Turner JA. Sharing the wealth: neuroimaging data repositories. Neuroimage. 2016;124:1065.

    Article  PubMed  Google Scholar 

  101. Petersen RC, Aisen PS, Beckett LA, Donohue MC, Gamst AC, Harvey DJ, et al. Alzheimer’s disease Neuroimaging Initiative (ADNI) clinical characterization. Neurology. 2010;74:201–9.

    Article  PubMed  PubMed Central  Google Scholar 

  102. Poline JB, Breeze JL, Ghosh S, Gorgolewski K, Halchenko YO, Hanke M, et al. Data sharing in neuroimaging research. Front Neuroinform. 2012;6:9.

    Article  PubMed  PubMed Central  Google Scholar 

  103. van Essen DC, Ugurbil K. The future of the human connectome. Neuroimage. 2012;62:1299–310.

    Article  PubMed  Google Scholar 

  104. Zidane YJT, Olsson NOE. Defining project efficiency, effectiveness and efficacy. Int J Manag Proj Bus. 2017;10:621–41.

    Article  Google Scholar 

  105. Roy A, Colpitts J, Becker K, Brewer J, van Lutterveld R. Improving efficiency in neuroimaging research through application of Lean principles. PloS One. 2018;13:e0205232.

    Article  PubMed  PubMed Central  Google Scholar 

  106. Shapiro L, Staroswiecki E, Gold G. Magnetic resonance imaging of the knee: optimizing 3 Tesla imaging. Semin Roentgenol. 2010;45:238–49.

    Article  PubMed  PubMed Central  Google Scholar 

  107. Khodyakov D, Mikesell L, Schraiber R, Booth M, Bromley E. On using ethical principles of community-engaged research in translational science. Transl Res. 2016;171:52–62.

    Article  PubMed  Google Scholar 

  108. Puri KS, Suresh KR, Gogtay NJ, Thatte UM. Declaration of Helsinki, 2008: implications for stakeholders in research. J Postgrad Med. 2009;55:131–4.

    Article  CAS  PubMed  Google Scholar 

  109. Rotstein HG, Santamaria F. Development of theoretical frameworks in neuroscience: a pressing need in a sea of data. arXiv preprint arXiv:220909953. 2022.

  110. Poline JB, Kennedy DN, Sommer FT, Ascoli GA, van Essen DC, Ferguson AR, et al. Is neuroscience FAIR? A call for collaborative standardisation of neuroscience data. Neuroinformatics. 2022;20:507–12.

    Article  PubMed  PubMed Central  Google Scholar 

  111. Poldrack RA, Whitaker K, Kennedy D. Introduction to the special issue on reproducibility in neuroimaging. NeuroImage. 2020;218:116357.

  112. Goldfarb MG, Brown DR. Diversifying participation: The rarity of reporting racial demographics in neuroimaging research. NeuroImage. 2022;254:119122.

  113. Schwab S, Janiaud P, Dayan M, Amrhein V, Panczak R, Palagi PM, et al. Ten simple rules for good research practice. PLoS Comput Biol. 2022;18:e1010139.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  114. Am Smeets P, Dagher A, Hare TA, Kullmann S, van der Laan LN, Poldrack RA, et al. Good practice in food-related neuroimaging. Am J Clin Nutr. 2019;109:491–503.

    Article  PubMed  Google Scholar 

  115. Nakayama T. What are “clinical practice guidelines”? J Neurol. 2007;254:2–7.

    Article  Google Scholar 

  116. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

    Article  PubMed  PubMed Central  Google Scholar 

  117. Pomponio R, Erus G, Habes M, Doshi J, Srinivasan D, Mamourian E, et al. Harmonization of large MRI datasets for the analysis of brain imaging patterns throughout the lifespan. Neuroimage. 2020;208:116450.

    Article  PubMed  Google Scholar 

  118. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Lancet. 1999;354:1896–900.

    Article  CAS  PubMed  Google Scholar 

  119. Matshabane OP. Promoting diversity and inclusion in neuroscience and neuroethics. EBioMedicine. 2021;67:103359.

    Article  PubMed  PubMed Central  Google Scholar 

  120. Noble S, Scheinost D, Constable RT. A decade of test-retest reliability of functional connectivity: a systematic review and meta-analysis. Neuroimage. 2019;203:116157.

    Article  PubMed  Google Scholar 

  121. Strickland JC. Guide to research techniques in neuroscience. J Undergrad Neurosci Educ. 2014;13:R1.

    PubMed Central  Google Scholar 

  122. Gross J, Baillet S, Barnes GR, Henson RN, Hillebrand A, Jensen O, et al. Good practice for conducting and reporting MEG research. Neuroimage. 2013;65:349–63.

    Article  PubMed  Google Scholar 

  123. Shekari M, Verwer EE, Yaqub M, Daamen M, Buckley C, Frisoni GB, et al. Harmonization of brain PET images in multi-center PET studies using Hoffman phantom scan. EJNMMI Phys. 2023;10:68.

    Article  PubMed  PubMed Central  Google Scholar 

  124. Sullivan JA. The multiplicity of experimental protocols: a challenge to reductionist and non-reductionist models of the unity of neuroscience. Synthese. 2009;167:511–39.

    Article  Google Scholar 

  125. Lu H, Kashani AH, Arfanakis K, Caprihan A, DeCarli C, Gold BT, et al. MarkVCID cerebral small vessel consortium: II. Neuroimaging protocols. Alzheimer’s Dement. 2021;17:716–25.

    Article  CAS  Google Scholar 

  126. Murphy A, Weerakkody Y. MRI protocols. In: Radiopaedia.org. Radiopaedia.org; 2005.

  127. O’Boyle EH, Götz M, Questionable research practices. Jussim, LJ, Krosnick, JA, and Stevens, ST Research integrity: Best practices for the social and behavioral sciences. 2022;260–94.

  128. Xie Y, Wang K, Kong Y. Prevalence of research misconduct and questionable research practices: A systematic review and meta-analysis. Sci Eng Ethics. 2021;27:41.

    Article  PubMed  Google Scholar 

  129. Siritzky EM, Cox PH, Nadler SM, Grady JN, Kravitz DJ, Mitroff SR. Standard experimental paradigm designs and data exclusion practices in cognitive psychology can inadvertently introduce systematic “shadow” biases in participant samples. Cogn Res: Princ Implic. 2023;8:66.

    Article  PubMed  Google Scholar 

  130. Barch DM, Yarkoni T. Introduction to the special issue on reliability and replication in cognitive and affective neuroscience research. Cogn Affect Behav Neurosci. 2013;13:687–9.

    Article  PubMed  Google Scholar 

  131. Plichta MM, Schwarz AJ, Grimm O, Morgen K, Mier D, Haddad L, et al. Test–retest reliability of evoked BOLD signals from a cognitive–emotive fMRI test battery. Neuroimage. 2012;60:1746–58.

    Article  PubMed  Google Scholar 

  132. Elliott ML, Knodt AR, Ireland D, Morris ML, Poulton R, Ramrakha S, et al. What is the test-retest reliability of common task-functional MRI measures? New empirical evidence and a meta-analysis. Psychol Sci. 2020;31:792–806.

    Article  PubMed  PubMed Central  Google Scholar 

  133. Rudeck J, Vogl S, Banneke S, Schönfelder G, Lewejohann L. Repeatability analysis improves the reliability of behavioral data. PloS One. 2020;15:e0230900.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  134. Sörös P, Wölk L, Bantel C, Bräuer A, Klawonn F, Witt K. Replicability, repeatability, and long-term reproducibility of cerebellar morphometry. Cerebellum. 2021;20:439–53.

    Article  PubMed  PubMed Central  Google Scholar 

  135. Miłkowski M, Hensel WM, Hohol M. Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail. J Comput Neurosci. 2018;45:163–72.

    Article  PubMed  PubMed Central  Google Scholar 

  136. Dienlin T, Johannes N, Bowman ND, Masur PK, Engesser S, Kümpel AS, et al. An agenda for open science in communication. J Commun. 2021;71:1–26.

    Article  Google Scholar 

  137. Kenall A, Edmunds S, Goodman L, Bal L, Flintoft L, Shanahan DR, et al. Better reporting for better research: a checklist for reproducibility. Gigascience. 2015;4:32.

    Article  PubMed  PubMed Central  Google Scholar 

  138. Weissgerber TL, Garovic VD, Winham SJ, Milic NM, Prager EM. Transparent reporting for reproducible science. J Neurosci Res. 2016;94:859.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  139. Heßler N, Rottmann M, Ziegler A. Empirical analysis of the text structure of original research articles in medical journals. PloS One. 2020;15:e0240288.

    Article  PubMed  PubMed Central  Google Scholar 

  140. Botvinik-Nezer R, Wager TD. Reproducibility in neuroimaging analysis: challenges and solutions. Biol Psychiatry Cogn Neurosci Neuroimaging. 2023;8:780–8.

    PubMed  Google Scholar 

  141. Glatard T, Lewis LB, Ferreira da Silva R, Adalat R, Beck N, Lepage C, et al. Reproducibility of neuroimaging analyses across operating systems. Front Neuroinformatics. 2015;9:12.

    Article  Google Scholar 

  142. Valkenburg G, Dix G, Tijdink J, de Rijcke S. Expanding research integrity: a cultural-practice perspective. Sci Eng Ethics. 2021;27:10.

    Article  PubMed  PubMed Central  Google Scholar 

  143. Beauvais MJS, Knoppers BM, Illes J. A marathon, not a sprint–neuroimaging, Open Science and ethics. Neuroimage. 2021;236:118041.

    Article  PubMed  Google Scholar 

  144. Graham M, Hallowell N, Savulescu J. A just standard: the ethical management of incidental findings in brain imaging research. J Law Med Ethics. 2021;49:269–81.

    Article  PubMed  PubMed Central  Google Scholar 

  145. Tedersoo L, Küngas R, Oras E, Köster K, Eenmaa H, Leijen Ä, et al. Data sharing practices and data availability upon request differ across scientific disciplines. Sci Data. 2021;8:192.

    Article  PubMed  PubMed Central  Google Scholar 

  146. Ciric R, Thompson WH, Lorenz R, Goncalves M, MacNicol EE, Markiewicz CJ, et al. TemplateFlow: FAIR-sharing of multi-scale, multi-species brain models. Nat Methods. 2022;19:1568–71.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  147. Hedge C, Powell G, Sumner P. The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behav Res Methods. 2018;50:1166–86.

    Article  PubMed  Google Scholar 

  148. Helwegen K, Libedinsky I, van den Heuvel MP. Statistical power in network neuroscience. Trends Cogn Sci. 2023;27:282–301.

    Article  PubMed  Google Scholar 

  149. Esteban O, Markiewicz CJ, Blair RW, Moodie CA, Isik AI, Erramuzpe A, et al. fMRIPrep: a robust preprocessing pipeline for functional MRI. Nat Methods. 2019;16:111–6.

    Article  CAS  PubMed  Google Scholar 

  150. Loss CM, Melleu FF, Domingues K, Lino-de-Oliveira C, Viola GG. Combining animal welfare with experimental rigor to improve reproducibility in behavioral neuroscience. Front Behav Neurosci. 2021;15:763428.

    Article  PubMed  PubMed Central  Google Scholar 

  151. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proc Natl Acad Sci USA. 2018;115:2600–6.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  152. Abrams MB, Bjaalie JG, Das S, Egan GF, Ghosh SS, Goscinski WJ, et al. A standards organization for open and FAIR neuroscience: the international neuroinformatics coordinating facility. Neuroinformatics. 2022;20:25–36.

    Article  PubMed  Google Scholar 

  153. Barnes J, Conrad K, Demont-Heinrich C, Graziano M, Kowalski D, Neufeld J, et al. Understanding generalizability and transferability. Writing@ CSU. 2012.

  154. Schleim S. Real neurolaw in the Netherlands: the role of the developing brain in the new adolescent criminal law. Front Psychol. 2020;11:549375.

    Article  Google Scholar 

  155. Bradley SH, DeVito NJ, Lloyd KE, Richards GC, Rombey T, Wayant C, et al. Reducing bias and improving transparency in medical research: a critical overview of the problems, progress and suggested next steps. J R Soc Med. 2020;113:433–43.

    Article  PubMed  PubMed Central  Google Scholar 

  156. James S, Rao SV, Granger CB. Registry-based randomized clinical trials—a new clinical trial paradigm. Nat Rev Cardiol. 2015;12:312–6.

    Article  PubMed  Google Scholar 

  157. Zarin DA, Tse T, Williams RJ, Califf RM, Ide NC. The ClinicalTrials. gov results database—update and key issues. N Engl J Med. 2011;364:852–60.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  158. Namiot ED, Smirnovová D, Sokolov AV, Chubarev VN, Tarasov VV, Schiöth HB. The international clinical trials registry platform (ICTRP): data integrity and the trends in clinical trials, diseases, and drugs. Front Pharmacol. 2023;14:1228148.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  159. Andrade C. Internal, external, and ecological validity in research design, conduct, and evaluation. Indian J Psychol Med. 2018;40:498–9.

    Article  PubMed  PubMed Central  Google Scholar 

  160. Wrightson JJ. CONSORT_GPT [Internet]. [cited 2024 Apr 13]. Available from: https://chat.openai.com/g/g-jOiNJ3mhR-consort-gpt?utm_source=gptshunter.com.

  161. Ekhtiari H, Soleimani G, Kuplicki R, Yeh H, Cha Y, Paulus M. Transcranial direct current stimulation to modulate fMRI drug cue reactivity in methamphetamine users: a randomized clinical trial. Hum Brain Mapp. 2022;43:5340–57.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Funding

HE is supported by funds from Laureate Institute for Brain Research and Medical Discovery Team on Addiction and Brain and Behavior Foundation (NARSAD Young Investigator Award 27305). JK is supported by the Department of Veterans Affairs (National Center for PTSD), NIAAA (Center for the Translational Neuroscience of Alcohol (2P50AA012870-23), and National Center for Translational Science Clinical and Translational Science Award (2UL1TR001863-06). He has stock, options, or has received consultation fees from the following companies Aptinyx, Biogen, Biohaven Pharmaceuticals, Bionomics, Boehringer Ingelheim, Cartego Therapeutics, Damona Pharmaceuticals, Epiodyne, Epivario, Freedom Biosciences, Janssen Research and Development, Jazz Pharmaceuticals, Neumora Therapeutics, Otsuka America, Response Therapeutics, Rest Therapeutics, Spring Care, Sumitomi America, Terran Biosciencews, Tetricus, inc. He is an inventor on patents licensed by Yale University to Biohaven Pharmaceuticals, Freedom Biosciences, Janssen Pharmaceuticals, and Novartis Pharmaceuticals. DO has received R01MH114982 funding and also an honorarium from Boehringer-Ingelheim in the past 12 months. CRP is supported by the Novo Nordisk Fonden NNF20OC0063277. MPP is partly supported by The William K. Warren Foundation, the National Institute of General Medical Sciences Center (Grant 2 P20 GM121312), and the National Institute on Drug Abuse (U01DA050989). He advises Spring Care, Inc., receives royalties from an article on methamphetamine in UpToDate, and has a compensated consulting agreement with Boehringer Ingelheim International GmbH. Other authors declare no conflicts of interest.

Author information

Authors and Affiliations

Authors

Contributions

All authors contributed to the conception of the manuscript first through two panels held at the American College of Neuropsychopharmacology Annual Meeting (ACNP2023) and then through online rounds of discussions. HE, MZB, AS, AV, DMC, HG, TEN, CRP, PMT, and MPP contributed to drafting the first version of the manuscript. HE supervised the process of receiving and implementing the comments. All authors contributed to the manuscript revision, read, and approved the submitted version.

Corresponding author

Correspondence to Hamed Ekhtiari.

Ethics declarations

Competing interests

TPG is the co-editor of the NeuroPschychoPharmacology. JK also serves as the editor of Biological Psychiatry. Other authors declare no conflicts of interest.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ekhtiari, H., Zare-Bidoky, M., Sangchooli, A. et al. Reporting checklists in neuroimaging: promoting transparency, replicability, and reproducibility. Neuropsychopharmacol. (2024). https://doi.org/10.1038/s41386-024-01973-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s41386-024-01973-5

Search

Quick links