Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Challenges and recommendations on the conduct of systematic reviews of observational epidemiologic studies in environmental and occupational health

Journal of Exposure Science & Environmental Epidemiology volume 31, pages 21–30 (2021)Cite this article

Abstract

Systematic reviews are powerful tools for drawing causal inference for evidence-based decision-making. Published systematic reviews and meta-analyses of environmental and occupational epidemiology studies have increased dramatically in recent years; however, the quality and utility of published reviews are variable. Most methodologies were adapted from clinical epidemiology and have not been adequately modified to evaluate and integrate evidence from observational epidemiology studies assessing environmental and occupational hazards, especially in evaluating the quality of exposure assessments. Although many reviews conduct a systematic and transparent assessment for the potential for bias, they are often deficient in subsequently integrating across a body of evidence. A cohesive review considers the impact of the direction and magnitude of potential biases on the results, systematically evaluates important scientific issues such as study sensitivity and effect modifiers, identifies how different studies complement each other, and assesses other potential sources of heterogeneity. Given these challenges of conducting informative systematic reviews of observational studies, we provide a series of specific recommendations based on practical examples for cohesive evidence integration to reach an overall conclusion on a body of evidence to better support policy making in public health.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Elements of a well-conducted systematic review.

References

  1. IARC. IARC monographs on the identification of carcinogenic hazards to humans: preamble. Lyon: International Agency for Research on Cancer; 2019.

    Google Scholar 

  2. NTP. Handbook for preparing the report on carcinogens monographs. Research Triangle Park, NC: National Toxicology Program; 2015.

    Google Scholar 

  3. Radke EG, Glenn B, Galizia A, Persad A, Nachman R, Bateson T, et al. Development of outcome-specific criteria for study evaluation in systematic reviews of epidemiology studies. Environ Int. 2019;130:104884.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Guyatt GH, Oxman AD, Schünemann HJ, Tugwell P, Knottnerus A. GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol. 2011;64:380–2.

    Article  PubMed  Google Scholar 

  5. Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Woodruff TJ, Sutton P. The Navigation Guide systematic review methodology: a rigorous and transparent method for translating environmental health science into better health outcomes. Environ Health Perspect. 2014;122:1007–14.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Cooper GS, Lunn RM, Ågerstrand M, Glenn BS, Kraft AD, Luke AM, et al. Study sensitivity: evaluating the ability to detect effects in systematic reviews of chemical exposures. Environ Int. 2016;92-93:605–10.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Howard J, Piacentino J, MacMahon K, Schulte P. Using systematic review in occupational safety and health. Am J Ind Med. 2017;60:921–9.

    Article  PubMed  Google Scholar 

  9. Lash TL, Fox MP, MacLehose RF, Maldonado G, McCandless LC, Greenland S. Good practices for quantitative bias analysis. Int J Epidemiol. 2014;43:1969–85.

    Article  PubMed  Google Scholar 

  10. Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.

    Article  PubMed  PubMed Central  Google Scholar 

  11. NTP. NTP monographon the systematic review of traffic-related air pollution and hypertensive disorders of pregnancy. Research Triangle Park, NC: National Toxicology Program; 2019. Report No.: NTP Monograph 7.

  12. Hamra GB, Laden F, Cohen AJ, Raaschou-Nielsen O, Brauer M, Loomis D. Lung cancer and exposure to nitrogen dioxide and traffic: a systematic review and meta-analysis. Environ Health Perspect. 2015;123:1107–12.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Armstrong BG. Effect of measurement error on epidemiological studies of environmental and occupational exposures. Occup Environ Med. 1998;55:651–6.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. NTP. Draft report on carcinogens monograph on night shift work and light at night. Peer Review Draft. Research Triangle Park, NC: National Toxicology Program; 2018.

  15. Radke E, Glenn B, Braun J, Cooper G. Phthalate exposure and female reproductive and developmental outcomes: a systematic review of the human epidemiological evidence. Environ Int. 2019;130:104580.

    Article  CAS  PubMed  Google Scholar 

  16. Johns LE, Cooper GS, Galizia A, Meeker JD. Exposure assessment issues in epidemiology studies of phthalates. Environ Int. 2015;85:27–39.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Herbison P, Hay-Smith J, Gillespie WJ. Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol. 2006;59:1249–56.

    Article  PubMed  Google Scholar 

  18. Savitz DA, Wellenius GA, Trikalinos TA. The problem with mechanistic risk of bias assessments in evidence synthesis of observational studies and a practical alternative: assess the impact of specific sources of potential bias. Am J Epidemiol. 2019;188:1581–5.

    Article  PubMed  Google Scholar 

  19. Ioannidis JPA. Meta-analyses in environmental and occupational health. Occup Environ Med. 2018;75:443–5.

    Article  PubMed  Google Scholar 

  20. National Research Council. Review of EPA's Integrated Risk Information System (IRIS) process. Washington, DC: National Academies Press; 2014.

    Google Scholar 

  21. Eze IC, Hemkens LG, Bucher HC, Hoffmann B, Schindler C, Künzli N, et al. Association between ambient air pollution and diabetes mellitus in Europe and North America: systematic review and meta-analysis. Environ Health Perspect. 2015;123:381–9.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa, Canada: The Ottawa Hospital; 2009. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp.

  23. Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol. 2010;25:603–5.

    Article  PubMed  Google Scholar 

  24. Stang A, Jonas S, Poole C. Case study in major quotation errors: a critical commentary on the Newcastle-Ottawa scale. Eur J Epidemiol. 2018;33:1025–31.

    Article  CAS  PubMed  Google Scholar 

  25. Yu L, Wang B, Cheng M, Yang M, Gan S, Fan L, et al. Association between indoor formaldehyde exposure and asthma: a systematic review and meta-analysis of observational studies. Indoor Air. 2020. ePub, ahead of print.

  26. Moola S, Munn Z, Tufanaru C, Aromataris E, Sears K, Sfetcu R, et al. Chapter 7: Systematic reviews of etiology and risk. In: Aromataris E, Munn Z, editors. Joanna Briggs Institute Reviewer’s Manual. The Joanna Briggs Institute; 2017. https://reviewersmanual.joannabriggs.org/.

  27. Hernán MA, Hernández-Díaz S, Robins JM. A structural approach to selection bias. Epidemiology. 2004;15:615–25.

    Article  PubMed  Google Scholar 

  28. Vandenbroucke JP, Broadbent A, Pearce N. Causality and causal inference in epidemiology: the need for a pluralistic approach. Int J Epidemiol. 2016;45:1776–86.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Honaryar MK, Lunn RM, Luce D, Ahrens W, ’t Mannetje A, Hansen J. et al. Welding fumes and lung cancer: a meta-analysis of case-control and cohort studies. Occup Environ Med. 2019;76:422–31.

    Article  PubMed  Google Scholar 

  30. IARC. Welding, molybdenum trioxide, and indium tin oxide. Lyon: International Agency for Research on Cancer; 2018. 320 p.

  31. Cherrie JW, Levy L. Managing occupational exposure to welding fume: new evidence suggests a more precautionary approach is needed. Ann Work Expo Health. 2020;64:1–4.

    PubMed  Google Scholar 

  32. HSE. Change in enforcement expectations for mild steel welding fume: health and safety executive, UK; 2019. https://www.hse.gov.uk/safetybulletins/mild-steel-welding-fume.htm.

  33. NTP. Report on carcinogens, 14th edition. Research Triangle Park, NC: U.S. Department of Health and Human Services, Public Health Service, National Toxicology Program; 2016.

  34. Huss A, Peters S, Vermeulen R. Occupational exposure to extremely low-frequency magnetic fields and the risk of ALS: a systematic review and meta-analysis. Bioelectromagnetics. 2018;39:156–63.

    Article  PubMed  Google Scholar 

  35. Steenland K, Barry V, Savitz D. Serum perfluorooctanoic acid and birthweight: an updated meta-analysis with bias analysis. Epidemiology. 2018;29:765–76.

    Article  PubMed  Google Scholar 

  36. Verner MA, Loccisano AE, Morken NH, Yoon M, Wu H, McDougall R, et al. Associations of perfluoroalkyl substances (PFAS) with lower birth weight: an evaluation of potential confounding by glomerular filtration rate using a physiologically based pharmacokinetic model (PBPK). Environ Health Perspect. 2015;123:1317–24.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. Johnson PI, Sutton P, Atchley DS, Koustas E, Lam J, Sen S, et al. The Navigation Guide – evidence-based medicine meets environmental health: systematic review of human evidence for PFOA effects on fetal growth. Environ Health Perspect. 2014;122:1028–39.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Lam J, Koustas E, Sutton P, Johnson PI, Atchley DS, Sen S, et al. The Navigation Guide - evidence-based medicine meets environmental health: integration of animal and human evidence for PFOA effects on fetal growth. Environ Health Perspect. 2014;122:1040–51.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Lawlor DA, Tilling K, Davey Smith G. Triangulation in aetiological epidemiology. Int J Epidemiol. 2016;45:1866–86.

    Article  PubMed  Google Scholar 

  40. Vlaanderen J, Lan Q, Kromhout H, Rothman N, Vermeulen R. Occupational benzene exposure and the risk of lymphoma subtypes: a meta-analysis of cohort studies incorporating three study quality dimensions. Environ Health Perspect. 2011;119:159–67.

    Article  CAS  PubMed  Google Scholar 

  41. Checkoway H, Pearce N, Kriebel D. Selecting appropriate study designs to address specific research questions in occupational epidemiology. Occup Environ Med. 2007;64:633–8.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Morgan RL, Thayer KA, Santesso N, Holloway AC, Blain R, Eftim SE, et al. A risk of bias instrument for non-randomized studies of exposures: a users' guide to its application in the context of GRADE. Environ Int. 2019;122:168–84.

    Article  PubMed  Google Scholar 

  43. WHO. Environmental noise guidelines for the European region. Copenhagen: World Health Organization; 2018.

    Google Scholar 

  44. National Research Council. Critical aspects of EPA's IRIS assessment of inorganic arsenic: interim report. Washington, DC: National Academies Press; 2013.

    Google Scholar 

  45. Marshall G, Ferreccio C, Yuan Y, Bates MN, Steinmaus C, Selvin S, et al. Fifty-year study of lung and bladder cancer mortality in Chile related to arsenic in drinking water. J Natl Cancer Inst. 2007;99:920–8.

    Article  CAS  PubMed  Google Scholar 

  46. Leogrande S, Alessandrini ER, Stafoggia M, Morabito A, Nocioni A, Ancona C, et al. Industrial air pollution and mortality in the Taranto area, Southern Italy: a difference-in-differences approach. Environ Int. 2019;132:105030.

    Article  CAS  PubMed  Google Scholar 

  47. IARC. Benzene. Lyon: International Agency for Research on Cancer; 2018. 309 p.

  48. Welling R, Beaumont JJ, Petersen SJ, Alexeeff GV, Steinmaus C. Chromium VI and stomach cancer: a meta-analysis of the current epidemiological evidence. Occup Environ Med. 2015;72:151–9.

    Article  PubMed  Google Scholar 

  49. Gatto NM, Kelsh MA, Mai DH, Suh M, Proctor DM. Occupational exposure to hexavalent chromium and cancers of the gastrointestinal tract: a meta-analysis. Cancer Epidemiol. 2010;34:388–99.

    Article  CAS  PubMed  Google Scholar 

  50. Yuan X, Zhu C, Wang M, Mo F, Du W, Ma X. Retraction: night shift work increases the risks of multiple primary cancers in women: a systematic review and meta-analysis of 61 articles. Cancer Epidemiol Biomark Prev. 2019;28:423.

    Article  Google Scholar 

  51. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.

    Article  PubMed  PubMed Central  Google Scholar 

  52. von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61:344–9.

    Article  Google Scholar 

  53. Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Syst Rev. 2014;3:74.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Marshall IJ, Kuiper J, Wallace BC. Automating risk of bias assessment for clinical trials. IEEE J Biomed Health Inf. 2015;19:1406–12.

    Article  Google Scholar 

  55. Rooney AA, Cooper GS, Jahnke GD, Lam J, Morgan RL, Boyles AL, et al. How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards. Environ Int. 2016;92-93:617–29.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Hempel S, Xenakis L, Danz M. Systematic reviews for occupational safety and health questions: resources for evidence analysis. Santa Monica, CA: RAND Corporation; 2016. 102 p.

  57. Rooney AA, Boyles AL, Wolfe MS, Bucher JR, Thayer KA. Systematic review and evidence integration for literature-based environmental health science assessments. Environ Health Perspect. 2014;122:711–8.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the following people for their thoughtful reviews and comments on this manuscript, which have improved it greatly: Tara Hartley, Kathleen MacMahon, Robert Daniels, Kris Thayer, Tom Luben, Craig Steinmaus, Andrew Rooney, Christina Parks, and Abee Boyles.

Author information

Authors and Affiliations

  1. Integrated Laboratory Systems, Morrisville, NC, USA

    Whitney D. Arroyave & Pam Schwingl

  2. Division of the National Toxicology Program, National Institute of Environmental Health Sciences, Research Triangle Park, NC, USA

    Suril S. Mehta, Kyla W. Taylor & Ruth M. Lunn

  3. International Agency for Research on Cancer, World Health Organization, Lyon, France

    Neela Guha & Nadia Vilahur

  4. Office of Environmental Health Hazard Assessment, California Environmental Protection Agency, Oakland, CA, USA

    Neela Guha

  5. Center for Public Health and Environmental Assessment, Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC, USA

    Barbara Glenn, Elizabeth G. Radke & Rebecca M. Nachman

  6. World Trade Center Health Program, National Institute for Occupational Safety and Health, Cincinnati, OH, USA

    Tania Carreón

Authors
  1. Whitney D. Arroyave
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Suril S. Mehta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Neela Guha
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pam Schwingl
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kyla W. Taylor
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Barbara Glenn
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Elizabeth G. Radke
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Nadia Vilahur
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Tania Carreón
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Rebecca M. Nachman
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Ruth M. Lunn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ruth M. Lunn.

Ethics declarations

Conflict of interest

The authors declare that they have no conflicts of interest.

Disclaimer

The views expressed are those of the authors and do not necessarily represent the official position of the Office of Environmental Health Hazard Assessment, the California Environmental Protection Agency or the State of California, nor any of the U.S. Government (National Toxicology Program, U.S. Environmental Protection Agency, and the Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health) or International agencies (International Agency for Research on Cancer) with which some of the authors are affiliated.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Arroyave, W.D., Mehta, S.S., Guha, N. et al. Challenges and recommendations on the conduct of systematic reviews of observational epidemiologic studies in environmental and occupational health. J Expo Sci Environ Epidemiol 31, 21–30 (2021). https://doi.org/10.1038/s41370-020-0228-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41370-020-0228-0

Keywords

This article is cited by

Search

Advanced search

Quick links