Systematic reviews are powerful tools for drawing causal inference for evidence-based decision-making. Published systematic reviews and meta-analyses of environmental and occupational epidemiology studies have increased dramatically in recent years; however, the quality and utility of published reviews are variable. Most methodologies were adapted from clinical epidemiology and have not been adequately modified to evaluate and integrate evidence from observational epidemiology studies assessing environmental and occupational hazards, especially in evaluating the quality of exposure assessments. Although many reviews conduct a systematic and transparent assessment for the potential for bias, they are often deficient in subsequently integrating across a body of evidence. A cohesive review considers the impact of the direction and magnitude of potential biases on the results, systematically evaluates important scientific issues such as study sensitivity and effect modifiers, identifies how different studies complement each other, and assesses other potential sources of heterogeneity. Given these challenges of conducting informative systematic reviews of observational studies, we provide a series of specific recommendations based on practical examples for cohesive evidence integration to reach an overall conclusion on a body of evidence to better support policy making in public health.
Subscribe to Journal
Get full journal access for 1 year
only $19.83 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
IARC. IARC monographs on the identification of carcinogenic hazards to humans: preamble. Lyon: International Agency for Research on Cancer; 2019.
NTP. Handbook for preparing the report on carcinogens monographs. Research Triangle Park, NC: National Toxicology Program; 2015.
Radke EG, Glenn B, Galizia A, Persad A, Nachman R, Bateson T, et al. Development of outcome-specific criteria for study evaluation in systematic reviews of epidemiology studies. Environ Int. 2019;130:104884.
Guyatt GH, Oxman AD, Schünemann HJ, Tugwell P, Knottnerus A. GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol. 2011;64:380–2.
Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.
Woodruff TJ, Sutton P. The Navigation Guide systematic review methodology: a rigorous and transparent method for translating environmental health science into better health outcomes. Environ Health Perspect. 2014;122:1007–14.
Cooper GS, Lunn RM, Ågerstrand M, Glenn BS, Kraft AD, Luke AM, et al. Study sensitivity: evaluating the ability to detect effects in systematic reviews of chemical exposures. Environ Int. 2016;92-93:605–10.
Howard J, Piacentino J, MacMahon K, Schulte P. Using systematic review in occupational safety and health. Am J Ind Med. 2017;60:921–9.
Lash TL, Fox MP, MacLehose RF, Maldonado G, McCandless LC, Greenland S. Good practices for quantitative bias analysis. Int J Epidemiol. 2014;43:1969–85.
Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.
NTP. NTP monographon the systematic review of traffic-related air pollution and hypertensive disorders of pregnancy. Research Triangle Park, NC: National Toxicology Program; 2019. Report No.: NTP Monograph 7.
Hamra GB, Laden F, Cohen AJ, Raaschou-Nielsen O, Brauer M, Loomis D. Lung cancer and exposure to nitrogen dioxide and traffic: a systematic review and meta-analysis. Environ Health Perspect. 2015;123:1107–12.
Armstrong BG. Effect of measurement error on epidemiological studies of environmental and occupational exposures. Occup Environ Med. 1998;55:651–6.
NTP. Draft report on carcinogens monograph on night shift work and light at night. Peer Review Draft. Research Triangle Park, NC: National Toxicology Program; 2018.
Radke E, Glenn B, Braun J, Cooper G. Phthalate exposure and female reproductive and developmental outcomes: a systematic review of the human epidemiological evidence. Environ Int. 2019;130:104580.
Johns LE, Cooper GS, Galizia A, Meeker JD. Exposure assessment issues in epidemiology studies of phthalates. Environ Int. 2015;85:27–39.
Herbison P, Hay-Smith J, Gillespie WJ. Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol. 2006;59:1249–56.
Savitz DA, Wellenius GA, Trikalinos TA. The problem with mechanistic risk of bias assessments in evidence synthesis of observational studies and a practical alternative: assess the impact of specific sources of potential bias. Am J Epidemiol. 2019;188:1581–5.
Ioannidis JPA. Meta-analyses in environmental and occupational health. Occup Environ Med. 2018;75:443–5.
National Research Council. Review of EPA's Integrated Risk Information System (IRIS) process. Washington, DC: National Academies Press; 2014.
Eze IC, Hemkens LG, Bucher HC, Hoffmann B, Schindler C, Künzli N, et al. Association between ambient air pollution and diabetes mellitus in Europe and North America: systematic review and meta-analysis. Environ Health Perspect. 2015;123:381–9.
Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa, Canada: The Ottawa Hospital; 2009. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp.
Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol. 2010;25:603–5.
Stang A, Jonas S, Poole C. Case study in major quotation errors: a critical commentary on the Newcastle-Ottawa scale. Eur J Epidemiol. 2018;33:1025–31.
Yu L, Wang B, Cheng M, Yang M, Gan S, Fan L, et al. Association between indoor formaldehyde exposure and asthma: a systematic review and meta-analysis of observational studies. Indoor Air. 2020. ePub, ahead of print.
Moola S, Munn Z, Tufanaru C, Aromataris E, Sears K, Sfetcu R, et al. Chapter 7: Systematic reviews of etiology and risk. In: Aromataris E, Munn Z, editors. Joanna Briggs Institute Reviewer’s Manual. The Joanna Briggs Institute; 2017. https://reviewersmanual.joannabriggs.org/.
Hernán MA, Hernández-Díaz S, Robins JM. A structural approach to selection bias. Epidemiology. 2004;15:615–25.
Vandenbroucke JP, Broadbent A, Pearce N. Causality and causal inference in epidemiology: the need for a pluralistic approach. Int J Epidemiol. 2016;45:1776–86.
Honaryar MK, Lunn RM, Luce D, Ahrens W, ’t Mannetje A, Hansen J. et al. Welding fumes and lung cancer: a meta-analysis of case-control and cohort studies. Occup Environ Med. 2019;76:422–31.
IARC. Welding, molybdenum trioxide, and indium tin oxide. Lyon: International Agency for Research on Cancer; 2018. 320 p.
Cherrie JW, Levy L. Managing occupational exposure to welding fume: new evidence suggests a more precautionary approach is needed. Ann Work Expo Health. 2020;64:1–4.
HSE. Change in enforcement expectations for mild steel welding fume: health and safety executive, UK; 2019. https://www.hse.gov.uk/safetybulletins/mild-steel-welding-fume.htm.
NTP. Report on carcinogens, 14th edition. Research Triangle Park, NC: U.S. Department of Health and Human Services, Public Health Service, National Toxicology Program; 2016.
Huss A, Peters S, Vermeulen R. Occupational exposure to extremely low-frequency magnetic fields and the risk of ALS: a systematic review and meta-analysis. Bioelectromagnetics. 2018;39:156–63.
Steenland K, Barry V, Savitz D. Serum perfluorooctanoic acid and birthweight: an updated meta-analysis with bias analysis. Epidemiology. 2018;29:765–76.
Verner MA, Loccisano AE, Morken NH, Yoon M, Wu H, McDougall R, et al. Associations of perfluoroalkyl substances (PFAS) with lower birth weight: an evaluation of potential confounding by glomerular filtration rate using a physiologically based pharmacokinetic model (PBPK). Environ Health Perspect. 2015;123:1317–24.
Johnson PI, Sutton P, Atchley DS, Koustas E, Lam J, Sen S, et al. The Navigation Guide – evidence-based medicine meets environmental health: systematic review of human evidence for PFOA effects on fetal growth. Environ Health Perspect. 2014;122:1028–39.
Lam J, Koustas E, Sutton P, Johnson PI, Atchley DS, Sen S, et al. The Navigation Guide - evidence-based medicine meets environmental health: integration of animal and human evidence for PFOA effects on fetal growth. Environ Health Perspect. 2014;122:1040–51.
Lawlor DA, Tilling K, Davey Smith G. Triangulation in aetiological epidemiology. Int J Epidemiol. 2016;45:1866–86.
Vlaanderen J, Lan Q, Kromhout H, Rothman N, Vermeulen R. Occupational benzene exposure and the risk of lymphoma subtypes: a meta-analysis of cohort studies incorporating three study quality dimensions. Environ Health Perspect. 2011;119:159–67.
Checkoway H, Pearce N, Kriebel D. Selecting appropriate study designs to address specific research questions in occupational epidemiology. Occup Environ Med. 2007;64:633–8.
Morgan RL, Thayer KA, Santesso N, Holloway AC, Blain R, Eftim SE, et al. A risk of bias instrument for non-randomized studies of exposures: a users' guide to its application in the context of GRADE. Environ Int. 2019;122:168–84.
WHO. Environmental noise guidelines for the European region. Copenhagen: World Health Organization; 2018.
National Research Council. Critical aspects of EPA's IRIS assessment of inorganic arsenic: interim report. Washington, DC: National Academies Press; 2013.
Marshall G, Ferreccio C, Yuan Y, Bates MN, Steinmaus C, Selvin S, et al. Fifty-year study of lung and bladder cancer mortality in Chile related to arsenic in drinking water. J Natl Cancer Inst. 2007;99:920–8.
Leogrande S, Alessandrini ER, Stafoggia M, Morabito A, Nocioni A, Ancona C, et al. Industrial air pollution and mortality in the Taranto area, Southern Italy: a difference-in-differences approach. Environ Int. 2019;132:105030.
IARC. Benzene. Lyon: International Agency for Research on Cancer; 2018. 309 p.
Welling R, Beaumont JJ, Petersen SJ, Alexeeff GV, Steinmaus C. Chromium VI and stomach cancer: a meta-analysis of the current epidemiological evidence. Occup Environ Med. 2015;72:151–9.
Gatto NM, Kelsh MA, Mai DH, Suh M, Proctor DM. Occupational exposure to hexavalent chromium and cancers of the gastrointestinal tract: a meta-analysis. Cancer Epidemiol. 2010;34:388–99.
Yuan X, Zhu C, Wang M, Mo F, Du W, Ma X. Retraction: night shift work increases the risks of multiple primary cancers in women: a systematic review and meta-analysis of 61 articles. Cancer Epidemiol Biomark Prev. 2019;28:423.
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.
von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61:344–9.
Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Syst Rev. 2014;3:74.
Marshall IJ, Kuiper J, Wallace BC. Automating risk of bias assessment for clinical trials. IEEE J Biomed Health Inf. 2015;19:1406–12.
Rooney AA, Cooper GS, Jahnke GD, Lam J, Morgan RL, Boyles AL, et al. How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards. Environ Int. 2016;92-93:617–29.
Hempel S, Xenakis L, Danz M. Systematic reviews for occupational safety and health questions: resources for evidence analysis. Santa Monica, CA: RAND Corporation; 2016. 102 p.
Rooney AA, Boyles AL, Wolfe MS, Bucher JR, Thayer KA. Systematic review and evidence integration for literature-based environmental health science assessments. Environ Health Perspect. 2014;122:711–8.
The authors would like to thank the following people for their thoughtful reviews and comments on this manuscript, which have improved it greatly: Tara Hartley, Kathleen MacMahon, Robert Daniels, Kris Thayer, Tom Luben, Craig Steinmaus, Andrew Rooney, Christina Parks, and Abee Boyles.
Conflict of interest
The authors declare that they have no conflicts of interest.
The views expressed are those of the authors and do not necessarily represent the official position of the Office of Environmental Health Hazard Assessment, the California Environmental Protection Agency or the State of California, nor any of the U.S. Government (National Toxicology Program, U.S. Environmental Protection Agency, and the Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health) or International agencies (International Agency for Research on Cancer) with which some of the authors are affiliated.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Arroyave, W.D., Mehta, S.S., Guha, N. et al. Challenges and recommendations on the conduct of systematic reviews of observational epidemiologic studies in environmental and occupational health. J Expo Sci Environ Epidemiol 31, 21–30 (2021). https://doi.org/10.1038/s41370-020-0228-0
- Environmental health policy
- Exposure assessment
- Alternatives assessment