Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Exploring the multiverse: the impact of researchers’ analytic decisions on relationships between depression and inflammatory markers

Neuropsychopharmacology volume 48, pages 1465–1474 (2023)Cite this article

Subjects

Abstract

In recent years, a replication crisis in psychiatry has led to a growing focus on the impact of researchers’ analytic decisions on the results from studies. Multiverse analyses involve examining results across a wide array of possible analytic decisions (e.g., log-transforming variables, number of covariates, or treatment of outliers) and identifying if study results are robust to researchers’ analytic decisions. Studies have begun to use multiverse analysis for well-studied relationships that have some heterogeneity in results/conclusions across studies.We examine the well-studied relationship between peripheral inflammatory markers (PIMs; e.g., white blood cell count (WBC) and C-reactive protein (CRP)) and depression severity in the large NHANES dataset (n = 25,962). Specification curve analyses tested the impact of 9 common analytic decisions (comprising of 58,000+ possible combinations) on the association of PIMs and depression severity. Relationships of PIMs and total depression severity are robust to analytic decisions (based on tests of inference jointly examining effect sizes and p-values). However, moderate/large differences are noted in effect sizes based on analytic decisions and the majority of analyses do not result in significant findings, with the percentage of analyses with statistically significant results being 46.1% for WBC and 43.8% for CRP. For associations of PIMs with specific symptoms of depression, some associations (e.g., sleep, appetite) in males (but not females) were robust to analytic decisions. We discuss how multiverse analyses can be used to guide research and also the need for authors, reviewers, and editors to incorporate multiverse analyses to enhance replicability of research findings.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Associations of covariate choice with ORs and POA.
Fig. 2: Odds ratios and JTI by symptom – raw analyses.

Similar content being viewed by others

References

  1. Baker M. Reproducibility crisis. Nature. 2016;533:353–66.

    Google Scholar 

  2. Steegen S, Tuerlinckx F, Gelman A, Vanpaemel W. Increasing transparency through a multiverse analysis. Perspect Psychological Sci. 2016;11:702–12.

    Article  Google Scholar 

  3. Simonsohn U, Simmons JP, Nelson LD. Specification curve: Descriptive and inferential statistics on all reasonable specifications. Available at SSRN 2694998. 2019.

  4. Lonsdorf T, Gerlicher A, Klingelhöfer-Jens M, Krypotos A-M. Multiverse analyses in fear conditioning research. Behaviour Research and Therapy. 2021;153:104072.

    Article  Google Scholar 

  5. Botvinik-Nezer R, Holzmeister F, Camerer CF, Dreber A, Huber J, Johannesson M, et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature. 2020;582:84–8.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Majd M, Saunders EF, Engeland CG. Inflammation and the dimensions of depression: a review. Front Neuroendocrinol. 2020;56:100800.

    Article  PubMed  Google Scholar 

  7. Horn SR, Long MM, Nelson BW, Allen NB, Fisher PA, Byrne ML. Replication and reproducibility issues in the relationship between C-reactive protein and depression: a systematic review and focused meta-analysis. Brain Behav Immun. 2018;73:85–114.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Yuan N, Chen Y, Xia Y, Dai J, Liu C. Inflammation-related biomarkers in major psychiatric disorders: a cross-disorder assessment of reproducibility and specificity in 43 meta-analyses. Transl Psychiatry. 2019;9:1–13.

    Article  Google Scholar 

  9. Moriarity DP, Horn SR, Kautz MM, Haslbeck JM, Alloy LB. How handling extreme C-reactive protein (CRP) values and regularization influences CRP and depression criteria associations in network analyses. Brain Behav Immun. 2021;91:393–403.

    Article  CAS  PubMed  Google Scholar 

  10. White J, Kivimäki M, Jokela M, Batty GD. Association of inflammation with specific symptoms of depression in a general population of older people: the English Longitudinal Study of Ageing. Brain Behav Immun. 2017;61:27–30.

    Article  PubMed  Google Scholar 

  11. Michal M, Wiltink J, Kirschner Y, Wild PS, Münzel T, Ojeda FM, et al. Differential associations of depressive symptom dimensions with cardio-vascular disease in the community: results from the Gutenberg health study. PLoS One. 2013;8:e72014.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Gialluisi A, Di Castelnuovo A, Bracone F, De Curtis A, Cerletti C, Donati MB, et al. Associations between systemic inflammation and somatic depressive symptoms: findings from the Moli‐sani study. Depress Anxiety. 2020;37:935–43.

    Article  CAS  PubMed  Google Scholar 

  13. O’Connor M-F, Bower JE, Cho HJ, Creswell JD, Dimitrov S, Hamby ME, et al. To assess, to control, to exclude: effects of biobehavioral factors on circulating inflammatory markers. Brain Behav Immun. 2009;23:887–97.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Rengasamy M, Da Costa E, Silva SA, Spada M, Price RB. Does the moderator matter? Identification of multiple moderators of the association between peripheral inflammatory markers and depression severity in a large racially diverse community cohort. Neuropsychopharmacology. 2022;47:1693–701.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. NHANES, National Center for Health Statistics. National Health and Nutrition Examination Survey. 2021. Centers for Disease Control and Prevention website, NHANES, National Center for Health Statistics; 2021. https://www.cdc.gov/nchs/nhanes/index.htm.

  16. Milaneschi Y, Kappelmann N, Ye Z, Lamers F, Moser S, Jones PB, et al. Association of inflammation with depression and anxiety: evidence for symptom-specificity and potential causality from UK Biobank and NESDA cohorts. Molecular Psychiatry. 2021;26:7393–402.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Moriarty AS, Gilbody S, McMillan D, Manea L. Screening and case finding for major depressive disorder using the Patient Health Questionnaire (PHQ-9): a meta-analysis. Gen Hospital Psychiatry. 2015;37:567–76.

    Article  Google Scholar 

  18. Riniolo TC, Porges SW. Evaluating group distributional characteristics: Why psychophysiologists should be interested in qualitative departures from the normal distribution. Psychophysiology. 2000;37:21–8.

    Article  CAS  PubMed  Google Scholar 

  19. Knief U, Forstmeier W. Violating the normality assumption may be the lesser of two evils. Behav Res Methods. 2021;53:2576–90.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Changyong F, Hongyue W, Naiji L, Tian C, Hua H, Ying L. Log-transformation and its implications for data analysis. Shanghai Arch Psychiatry. 2014;26:105.

    Google Scholar 

  21. Wright L, Head JA, Jivraj S. How robust is the association between youth unemployment and later mental health? An analysis of longitudinal data from English schoolchildren. Occup Environ Med. 2021;78:618–20.

    Article  PubMed  Google Scholar 

  22. Harder JA. The multiverse of methods: extending the multiverse analysis to address data-collection decisions. Perspect Psychological Sci. 2020;15:1158–77.

    Article  Google Scholar 

  23. Moore MJ, Demeyere N. Multiverse to inform neurological research: an example using recovery outcome of neglect. J Neurol. 2022;269:233–42.

    Article  PubMed  Google Scholar 

  24. Schäfer T, Schwarz MA. The meaningfulness of effect sizes in psychological research: differences between sub-disciplines and the impact of potential biases. Front Psychol. 2019;10:813.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Szucs D, Ioannidis JP. Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biol. 2017;15:e2000797.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Modecki KL, Low‐Choy S, Uink BN, Vernon L, Correia H, Andrews K. Tuning into the real effect of smartphone use on parenting: a multiverse analysis. J Child Psychol Psychiatry. 2020;61:855–65.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Barendse ME, Byrne ML, Flournoy JC, McNeilly EA, Guazzelli Williamson V, Barrett A-MY, et al. Multimethod assessment of pubertal timing and associations with internalizing psychopathology in early adolescent girls. J Abnormal Psychol. 2021;131:14.

    Google Scholar 

  28. Mac Giollabhui N, Ng TH, Ellman LM, Alloy LB. The longitudinal associations of inflammatory biomarkers and depression revisited: systematic review, meta-analysis, and meta-regression. Mol Psychiatry. 2020;26:3302–14.

    Article  Google Scholar 

  29. Schisterman EF, Cole SR, Platt RW. Overadjustment bias and unnecessary adjustment in epidemiologic studies. Epidemiology. 2009;20:488.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Lewis MW, et al. Multiverse analyses of fear acquisition and extinction retention in posttraumatic stress disorder. Psychophysiology. 2023:e14265.

  31. Bloom PA, VanTieghem M, Gabard‐Durnam L, Gee DG, Flannery J, Caldera C, et al. Age‐related change in task‐evoked amygdala—prefrontal circuitry: a multiverse approach with an accelerated longitudinal cohort aged 4–22 years. Hum Brain Mapp. 2022;43:3221–44.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Black L, Panayiotou M, Humphrey N. Internalizing symptoms, well-being, and correlates in adolescence: a multiverse exploration via cross-lagged panel network models. Dev Psychopathol. 2022;34:1477–91.

    Article  PubMed  Google Scholar 

  33. El Bahri M, Wang X, Biaggi T, Falissard B, Naudet F, Barry C. A multiverse analysis of meta-analyses assessing acupuncture efficacy for smoking cessation evidenced vibration of effects. J Clin Epidemiol. 2022;152:140–50.

    Article  PubMed  Google Scholar 

  34. Olsson-Collentine A, van Aert R, Bakker M, Wicherts J. Meta-analyzing the multiverse: A peek under the hood of selective reporting. Psychol Methods. 2023. https://psycnet.apa.org/fulltext/2023-71132-001.html.

  35. Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Sci. 2011;22:1359–66.

    Article  Google Scholar 

  36. Fanelli D. Do pressures to publish increase scientists’ bias? An empirical support from US States Data. PloS One. 2010;5:e10271.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Okura Y, Urban LH, Mahoney DW, Jacobsen SJ, Rodeheffer RJ. Agreement between self-report questionnaires and medical record data was substantial for diabetes, hypertension, myocardial infarction and stroke but not for heart failure. J Clin Epidemiol. 2004;57:1096–103.

    Article  PubMed  Google Scholar 

  38. Moriarity DP, Kautz MM, Mac Giollabhui N, Klugman J, Coe CL, Ellman LM, et al. Bidirectional associations between inflammatory biomarkers and depressive symptoms in adolescents: Potential causal relationships. Clin Psychological Sci. 2020;8:690–703.

    Article  Google Scholar 

  39. Xue Y, Liu G, Geng Q. Associations of cardiovascular disease and depression with memory related disease: a Chinese national prospective cohort study. J Affect Disord. 2020;266:187–93.

    Article  PubMed  Google Scholar 

  40. Bondy E, Norton SA, Voss M, Marks RB, Boudreaux MJ, Treadway MT, et al. Inflammation is associated with future depressive symptoms among older adults. Brain Behav Immun Health. 2021;13:100226.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Manfro PH, Anselmi L, Barros F, Gonçalves H, Murray J, Oliveira IO, et al. Youth depression and inflammation: Cross-sectional network analyses of C-Reactive protein, interleukin-6 and symptoms in a population-based sample. J Psychiatric Res. 2022;150:197–201.

    Article  Google Scholar 

  42. Frank P, Jokela M, Batty GD, Cadar D, Steptoe A, Kivimäki M. Association Between Systemic Inflammation and Individual Symptoms of Depression: A Pooled Analysis of 15 Population-Based Cohort Studies. Am J Psychiatry. 2021;178:1107–18.

    Article  PubMed  Google Scholar 

  43. Pitharouli MC, Hagenaars SP, Glanville KP, Coleman JR, Hotopf M, Lewis CM, et al. Elevated C-reactive protein in patients with depression, independent of genetic, health, and psychosocial factors: results from the UK Biobank. Am J Psychiatry. 2021;178:522–29.

    Article  PubMed  Google Scholar 

  44. Lee S, Oh SS, Jang S-I, Park E-C. Sex difference in the association between high-sensitivity C-reactive protein and depression: the 2016 Korea National Health and Nutrition Examination Survey. Sci Rep. 2019;9:1–10.

    Google Scholar 

  45. Manea L, Gilbody S, McMillan D. Optimal cut-off score for diagnosing depression with the Patient Health Questionnaire (PHQ-9): a meta-analysis. CMAJ. 2012;184:E191–6.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Tracy M, Morgenstern H, Zivin K, Aiello AE, Galea S. Traumatic event exposure and depression severity over time: results from a prospective cohort study in an urban area. Soc Psychiatry Psychiatr Epidemiol. 2014;49:1769–82.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Khandaker GM, Zuber V, Rees J, Carvalho L, Mason AM, Foley CN, et al. Shared mechanisms between coronary heart disease and depression: findings from a large UK general population-based cohort. Mol Psychiatry. 2020;25:1477–86.

    Article  PubMed  Google Scholar 

  48. Linkas J, Ahmed LA, Csifcsak G, Emaus N, Furberg A-S, Grimnes G, et al. Are pro-inflammatory markers associated with psychological distress in a cross-sectional study of healthy adolescents 15–17 years of age? The Fit Futures study. BMC Psychol. 2022;10:1–13.

    Google Scholar 

  49. Cong X, Tracy M, Edmunds LS, Hosler AS, Appleton AA. The relationship between inflammatory dietary pattern in childhood and depression in early adulthood. Brain Behav Immun Health. 2020;2:100017.

    Article  PubMed  Google Scholar 

  50. Burrows K, Stewart JL, Kuplicki R, Figueroa-Hall L, Spechler PA, Zheng H, et al. Elevated peripheral inflammation is associated with attenuated striatal reward anticipation in major depressive disorder. Brain Behav Immun. 2021;93:214–25.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  51. Stewart JC, Rand KL, Muldoon MF, Kamarck TW. A prospective evaluation of the directionality of the depression–inflammation relationship. Brain Behav Immun. 2009;23:936–44.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  52. Raison CL, Rutherford RE, Woolwine BJ, Shuo C, Schettler P, Drake DF, et al. A randomized controlled trial of the tumor necrosis factor antagonist infliximab for treatment-resistant depression: the role of baseline inflammatory biomarkers. JAMA Psychiatry. 2013;70:31–41.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  53. Lamers F, Milaneschi Y, Smit JH, Schoevers RA, Wittenberg G, Penninx BW. Longitudinal Association Between Depression and Inflammatory Markers: Results From the Netherlands Study of Depression and Anxiety. Biol Psychiatry. 2019;85:829–37.

    Article  PubMed  Google Scholar 

  54. Tayefi M, Shafiee M, Kazemi-Bajestani SMR, Esmaeili H, Darroudi S, Khakpouri S, et al. Depression and anxiety both associate with serum level of hs-CRP: a gender-stratified analysis in a population-based study. Psychoneuroendocrinology. 2017;81:63–9.

    Article  CAS  PubMed  Google Scholar 

  55. Bai Y-M, Chiou W-F, Su T-P, Li C-T, Chen M-H. Pro-inflammatory cytokine associated with somatic and pain symptoms in depression. J Affect Disord. 2014;155:28–34.

    Article  CAS  PubMed  Google Scholar 

  56. Schmidt FM, Schröder T, Kirkby KC, Sander C, Suslow T, Holdt LM, et al. Pro-and anti-inflammatory cytokines, but not CRP, are inversely correlated with severity and symptoms of major depression. Psychiatry Res. 2016;239:85–91.

    Article  CAS  PubMed  Google Scholar 

  57. Birur B, Amrock EM, Shelton RC, Li L. Sex differences in the peripheral immune system in patients with depression. Front Psychiatry. 2017;8:108.

    Article  PubMed  PubMed Central  Google Scholar 

  58. Köhler-Forsberg O, Buttenschøn HN, Tansey KE, Maier W, Hauser J, Dernovsek MZ, et al. Association between C-reactive protein (CRP) with depression symptom severity and specific depressive symptoms in major depression. Brain Behav Immun. 2017;62:344–50.

    Article  PubMed  Google Scholar 

Download references

Funding

This research was supported by funding from the Ruth L. Kirschstein National Research Service Award Institutional Research Training Grants sponsored by the National Institutes of Health (Grant No. NIH T32 MH018951: Brent).

Author information

Authors and Affiliations

  1. Department of Psychiatry, University of Pittsburgh, Pittsburgh, PA, USA

    Manivel Rengasamy, Thomas Kraynak & Rebecca Price

  2. Semel Institute for Neuroscience and Human Behavior, University of California Los Angeles, Los Angeles, CA, USA

    Daniel Moriarity

  3. Department of Genetics, Stanford University, Stanford, CA, USA

    Daniel Moriarity

  4. Department of Psychology, University of Pittsburgh, Pittsburgh, PA, USA

    Rebecca Price

  5. Department of Epidemiology, University of Pittsburgh, Pittsburgh, PA, USA

    Thomas Kraynak

  6. Center for Addiction Medicine, Harvard Medical School, Boston, MA, USA

    Brenden Tervo-Clemmens

Authors
  1. Manivel Rengasamy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Moriarity
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas Kraynak
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Brenden Tervo-Clemmens
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rebecca Price
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Author MR contributed to all aspects of the manuscript, including conceptualization, study design, data curation, data analysis, data interpretation and writing. Authors DM, TK, BT, and RP contributed to conceptualization, data interpretation and writing.

Corresponding author

Correspondence to Manivel Rengasamy.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rengasamy, M., Moriarity, D., Kraynak, T. et al. Exploring the multiverse: the impact of researchers’ analytic decisions on relationships between depression and inflammatory markers. Neuropsychopharmacol. 48, 1465–1474 (2023). https://doi.org/10.1038/s41386-023-01621-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41386-023-01621-4

Search

Advanced search

Quick links