The reliability and validity of six experts’ exposure ratings were evaluated for 64 nickel-exposed and 72 chromium-exposed workers from six Shanghai electroplating plants based on airborne and urinary nickel and chromium measurements. Three industrial hygienists and three occupational physicians independently ranked the exposure intensity of each metal on an ordinal scale (1–4) for each worker’s job in two rounds: the first round was based on responses to an occupational history questionnaire and the second round also included responses to an electroplating industry-specific questionnaire. The Spearman correlation (rs) was used to compare each rating’s validity to its corresponding subject-specific arithmetic mean of four airborne or four urinary measurements. Reliability was moderately high (weighted kappa range=0.60–0.64). Validity was poor to moderate (rs=−0.37–0.46) for both airborne and urinary concentrations of both metals. For airborne nickel concentrations, validity differed by plant. For dichotomized metrics, sensitivity and specificity were higher based on urinary measurements (47–78%) than airborne measurements (16–50%). Few patterns were observed by metal, assessment round, or expert type. These results suggest that, for electroplating exposures, experts can achieve moderately high agreement and (reasonably) distinguish between low and high exposures when reviewing responses to in-depth questionnaires used in population-based case-control studies.
Exposure assessment in population-based studies often requires experts to review study subjects’ responses to questionnaires designed to collect occupational information to provide exposure estimates for use in epidemiologic analyses.1, 2 The reliability and validity of these expert ratings are important to characterize because exposure misclassification can mask exposure-disease associations. Several studies have evaluated factors that can affect the reliability of experts’ ratings within population-based studies.3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 Of these, only five studies evaluated the experts’ validity.4, 8, 9, 11, 14 Validity studies for industry-based studies are somewhat more plentiful;10 for example, one of the earliest evaluated semi-quantitative estimates of methylene chloride and styrene for jobs in a small polyester factory.15 A previous review of these studies found that the experts’ validity, based on kappas or intraclass correlation coefficients (ICCs), varied widely from poor to excellent, with a median of ∼0.6.10 Most validity studies have compared the ratings with airborne measurements despite the fact that multiple routes of exposure are relevant for many agents. To date, we identified only three studies that have evaluated experts’ ratings compared with urinary measurements.11, 16, 17
No study has evaluated the reliability and validity of experts’ ratings in the electroplating industry. Previous studies of electroplating workers have often reported poor/no correlations between air and urine metal concentrations,18, 19, 20 although some studies have observed moderate to high correlations (range: r=0.48–0.96, median: r=0.68).21, 22, 23, 24 The relationship between air and urinary measurements likely varies due to the extent of dermal exposure, the use of personal protective equipment, and personal behaviors such as smoking that may transfer the contaminants from hand to mouth, as well as the time of day the urinary measurements are collected in relation to the air measurements.21, 22, 25, 26 Subject-specific variations in uptake and metabolic/excretion rates could also reduce the correlation with the post-work shift urinary concentrations. Therefore, our primary objective was to characterize the reliability and validity of experts’ ratings of nickel and chromium exposures within the context of a case-control study design for workers’ current jobs within an electroplating setting in relation to both airborne and urinary measurements of exposure. Our secondary objective was to evaluate the experts’ ratings in relation to the availability of participants’ responses to two types of questionnaires typically used in population-based studies (an occupational history questionnaire (OH) and an electroplating industry-specific questionnaire (EIQ)) and by type of expert (industrial hygienists versus occupational physicians).
MATERIALS AND METHODS
Study Subjects and Self-Reported Occupational Information
We recruited 64 nickel-exposed workers and 72 chromium-exposed workers from six electroplating plants (nickel-exposed workers from plants 1 to 3; chromium-exposed workers from plants 4 to 6) in Shanghai, China. Subjects were selected on the basis of their willingness to participate from those workers who had held their current job for at least 6 months and were expected to remain in their current job for at least 6 more months. Because this study was meant to mimic the type of information available to experts within a case-control study, each study subject completed an OH only for his/her current job. The OH included open-ended questions on job title, employment dates, products made or services provided by employer, primary work tasks and activities, tools and equipment used, and chemicals and materials used. Each subject also completed an EIQ for the same job, which asked more detailed questions (predominantly with categorical responses) about specific tasks, time spent in work locations within the plants, proximity to the source of metal aerosols, use of personal protective equipment, presence of ventilation systems and the subjects’ impression of operating efficiencies, and contact with the liquids from the plating tank. The OH and EIQ can be obtained from the corresponding author. Participation was voluntary and undertaken according to protocols approved by the Institutional Review Board of National Cancer Institute and the Shanghai Centers for Disease Control (hereafter, Shanghai CDC).
Air and Urinary Measurements
Personal airborne and urinary nickel and chromium measurements were collected on four occasions per subject over two seasons (summer and winter), from June 2002 to August 2003. The airborne samples collected total particulates in the workers’ breathing zones on mixed cellulose ester filters (pore size=0.8 μm) with a 37-μm cassette using a portable sampling pump and were analyzed according to National Institute for Occupational Safety and Health (NIOSH) Method 7300.27 Each worker provided one spot (∼50 ml) urine sample at the end of the work shift on each measurement day, using NIOSH Method 8310.28 These methods measure all insoluble and soluble forms, and all valences, of aerosols of the respective metals.
Expert Exposure Ratings
Three industrial hygienists and three occupational physicians affiliated with the Shanghai CDC or the US National Cancer Institute estimated subject-specific average (arithmetic mean (AM)) exposure intensity over a work shift to nickel or to chromium (depending on the plant in which the subject worked) for each subject’s current job in two separate assessment rounds. The experts were asked to estimate the AM because it is generally considered the most relevant metric for chronic health effects.29 In the first assessment round, the experts provided ratings based solely on the OH responses (OH round). In the second assessment round, the experts provided ratings based on both the OH and EIQ responses (OH/EIQ round). In each round, each expert independently assigned an intensity rating for nickel or chromium using an ordinal scale of 1 (very low) to 4 (high) without access to the exposure measurements. The categories were not anchored to specific exposure levels. The experts had no prior measurements available for these worksites with which to anchor their assignments, however, the experts were advised to consider the categories as approximately <10%, 10–50%, 51–100%, and >100% of the occupational exposure limit in place at the time of the evaluation.
To evaluate whether the experts were able to accurately estimate the AM exposure of each study subject, all comparisons were made to the AM of the subject-specific inhalation and urinary concentrations. The subject-specific inhalation concentration was calculated as the AM of the four airborne measurements. Similarly, the subject-specific urinary concentration was calculated as the AM of the four urinary measurements. All analyses were conducted using Stata 11.1 (StataCorp, College Station, TX, USA).
Descriptive statistics of the subject-specific inhalation and urinary concentrations were calculated, overall and by plant. The Spearman correlation (rs) between the airborne and urinary subject-specific means was also calculated for each metal, overall and by plant. ICCs that indicate the contrast in exposures between subjects (between-subject variance/(between-subject variance+within-subject variance)) were calculated from variance components obtained from random-effects models with subject ID included as the random effect.
Agreement between each pair of the six experts (15 pairs) was calculated using two metrics: proportion of agreement and weighted kappa (κw). We reported the mean and range of each kappa metric observed across the 15 pairs. To interpret the kappa values, we arbitrarily categorized kappa values <0.2 as poor, 0.2–0.4 as fair, >0.4–0.6 as moderate, >0.6–0.8 as moderately high, and >0.8 as high based on categories originally proposed by Landis and Koch..30
Validity of expert ratings
We evaluated the validity of expert ratings (ordinal scale, 1–4) compared with the subject-specific AMs of the airborne and urinary concentrations (used as an approximate ‘gold standard’ of each subject’s average exposure) using the Spearman correlation measure (rs), which is a non-parametric comparison that does not assume a linear relationship between the expert ratings and the measured subject-specific AMs. For each metal and sample media, we report the mean and range of the correlations observed across the six experts. We also calculated the Spearman correlation coefficient between the subject-specific means for each metal and sample media and the mean of the six experts’ ratings (‘group rating’; continuous scale, range 1–4), the mean of the three industrial hygienists’ ratings, and the mean of the three occupational physicians’ ratings. Confidence intervals (95% CI) were calculated based on Fisher’s transformation. To interpret the correlations, we used the cut points described above.
Sensitivity and specificity of a two-category scale
The sensitivity and specificity of the group rating compared with the subject-specific mean concentrations of airborne and urinary nickel and chromium exposure were calculated based on a two-category scale. For this calculation, group ratings ≤2.5 (the midpoint of the four-category scale) and subject-specific means less than the median were categorized as ‘low exposed’ and group ratings >2.5 and subject-specific means greater than median were categorized as ‘high exposed’.
The reliability and validity analyses described above were stratified by exposure agent, assessment round, and type of expert. Some analyses, identified in the results, were also stratified by plant. Measures of central tendency (mean, range) are reported because we focused on the performance of a ‘typical’ expert rather than a specific expert.
For nickel, the overall AM of the subject-specific means was 7.4 μg/m3 for air and 30.1 μg per g creatinine for urine (Table 1). Plant 1 had the highest airborne nickel AM; in contrast, plant 3 had the highest urinary nickel AM. For chromium, the overall AM of the subject-specific means was 3.0 μg/m3 for air and 76.3 μg per g creatinine for urine. Plant 5 had the highest airborne and urinary chromium AMs. Poor correlations were observed between the air and urine concentrations overall for nickel (rs=−0.28) and for chromium (rs=0.09) and by plant for both metals (rs range=−0.39–0.30).
Between- and within-subject variance components are reported in Table 2, overall and by plant. For airborne concentrations, ICCs ranged from <0.001 to 0.57, indicating poor-to-moderate contrast in airborne concentrations among subjects overall and within individual plants. For urinary concentrations, ICCs ranged from <0.001 to 0.26, indicating that nearly all variability in urinary concentrations was within-subject variability.
The overall means of the expert ratings were similar for both metals and for both assessment rounds (Table 3). On the basis of weighted kappa, moderately high agreement among experts was observed for both metals in the OH (κw: nickel=0.60, chromium=0.64) and OH/EIQ (κw: nickel=0.60, chromium=0.61) rounds. Agreement was only fair to moderate for both metals and assessment rounds when evaluated based on the proportion of agreement (means 0.47−0.57). For all measures, the agreement was somewhat higher for chromium than for nickel. No differences were observed by assessment round for either metal. The industrial hygienists had, on average, somewhat higher agreement among themselves than that observed among the occupational physicians for nickel but not for chromium (Supplementary Table S1 and S2).
Validity of Expert Ratings
For nickel, the mean Spearman correlation between each expert’s rating in the OH round and the subject-specific AM was −0.30 (rs range: −0.48 to 0.04) for air and 0.38 (rs range: 0.27 to 0.47) for urine. For chromium, the mean Spearman correlation between each expert’s rating in the OH round and the subject-specific AM was −0.02 (rs range: −0.11 to 0.06) for air and 0.04 (rs range: −0.06 to 0.15) for urine. Similarly, poor correlations in the EIQ round were observed (data not shown).
The Spearman correlations between the subject-specific means and the group ratings are shown in Table 4 for the OH round. Overall, poor-to-moderate correlations (rs=−0.37 to 0.46) were observed and did not vary by assessment round or type of expert. However, plant-specific differences were observed. For airborne nickel, we observed a good correlation for plant 1 (rs=0.70), but poor correlations for plant 2 (rs=0.15) and plant 3 (rs=0.19). For urinary nickel, poor correlations were observed in all three plants. For chromium, poor correlations (−0.67 to −0.04) were observed for both airborne and urinary concentrations in all three plants. For nickel, the industrial hygienists’ ratings had higher validity than the occupational physicians’ ratings in all plants based on the airborne measurements and in plants 2 and 3 based on urinary concentrations. For chromium, ratings from both types of experts had similarly low validity for both airborne and urinary concentrations.
We explored the relationship between the group ratings and the subject-specific means further visually. Figure 1 shows the scatter plot between the group rating and the subject-specific means of airborne nickel exposure in the OH and OH/EIQ rounds for plants 1, 2, and 3 (Figure 1a–c) and each plant’s corresponding distribution of the airborne nickel subject-specific AMs (Figure 1d–f). We found that the distribution of the subject-specific AMs for airborne nickel exposure was much wider and more evenly spread for plant 1 (Figure 1d), where we also observed moderately high validity, than for plants 2 and 3 (Figure 1e and f), where we observed poor validity. For urinary nickel and for airborne and urinary chromium, we observed narrow distributions for the subject-specific AMs in each plant (shown in Supplementary Figures S1, S2, and S3) that were similar to the distributions shown for plants 2 and 3 for airborne nickel (Figure 1e and f).
For nickel, the two-category scale derived for the group rating had higher sensitivity and specificity when compared with the urinary measurements (sensitivity=75% in OH round, 78% in OH/EIQ round; specificity=66% in both rounds) than when compared with the airborne measurements (sensitivity=25% in OH round, 28% in OH/EIQ round; specificity=16% in both rounds). For chromium, the two-category scale also had slightly higher sensitivity and specificity with the urinary measurements (sensitivity=50% in OH round, 47% in OH/EIQ round; specificity=58% in both rounds) than with the airborne measurements (sensitivity=47% in OH round, 39% in OH/EIQ round; specificity=50% in both rounds). No consistent pattern was observed across assessment rounds for these metrics. The two metals showed somewhat opposite patterns, with 9–12% higher sensitivity than specificity for nickel and 8–11% lower sensitivity than specificity for chromium.
In this study, experts’ ratings for nickel- and chromium-exposed workers based on OH and OH/EIQ responses were evaluated against repeated airborne and urinary measurements. The experts’ moderately high agreement among themselves (κw=0.60–0.64) was comparable to the median agreement reported in previous studies.9, 10, 12, 13, 14, 31, 32 Despite their moderately high reliability, the experts’ ratings had only poor-to-moderate validity overall in relation to both airborne and urinary measurements and was lower than most previous validity studies.9, 11, 17, 33 For example, Hertzman et al.17 found much higher correlations (>0.65) between experienced workers’ estimates of exposure and urinary chlorophenate levels in a large cohort study of the lumber industry. However, Tielemans et al.11 observed similarly poor-to-fair agreement (kappa<0.4) between estimates based on expert review of job-specific questionnaires (which has similarities to the EIQ used here) and urinary measurements (that is, methylhippuric, hippuric and chromium) in a population-based study. The validity observed here was also much lower than previously reported for two other Shanghai industries (textile industry, rs=0.30–0.65; foundry industry, rs=0.65–0.85) evaluated using nearly the same study design but based solely on airborne measurements.9 The group rating provided minimal improvement to the experts’ validity in this study, unlike previous studies that observed improvements in the validity when the estimates of multiple raters were averaged.9, 12, 34, 35 Moderate to moderately high validity, however, was observed when the experts’ four-category rating scale was dichotomized.
The poor overall validity observed here was likely, at least in part, a function of the limited exposure distribution between subjects within the same plant. Our plant-stratified analyses found much higher validity for plant 1, which was the only plant where the subject-specific means were relatively evenly distributed across a relatively wide range of exposure levels. All other plants had subject-specific means that were skewed and/or were narrowly clustered in the low exposure range (see Figure 1). Thus, the experts assessed workers’ exposures on a four-category scale when, for some plants in this study, there was little contrast (ICCs ranging from 0 to 0.5) among the exposures of most of the assessed workers. This finding is consistent with our previous study that observed poorer validity in the textile industry (with a highly skewed exposure distribution that clustered at low concentrations) than in the foundry industry (with more evenly distributed exposure concentrations).9 This finding may also point to the difficulty of asking experts to provide exposure ratings in the absence of any exposure measurements on which to anchor their estimates, which has previously been shown to improve experts’ validity.8, 10, 15 The availability of at least some measurements for these workplaces in advance of the rater evaluations may have revealed the lack of exposure contrast.
Although evaluating whether the experts’ ratings were better associated with air or urine concentrations was our primary objective, our findings are inconclusive because of the overall poor validity observed here. The experts’ ratings had better sensitivity and specificity for the urinary concentrations than the airborne concentrations, which provides some, but not conclusive, evidence that the experts may have considered all routes of exposure. This result was somewhat surprising because dermal exposure is more difficult to assess, has not been as well studied as airborne exposure, and little is known about the contribution of various dermal exposure determinants.36 These analyses also demonstrated that assessing exposure using a two-category scale (low and high) is easier than using a four-category scale when between-worker difference in exposure is small. However, the sensitivities (nickel: 75–78%; chromium: 47–50%) and specificities (nickel: 66%; chromium: 58%) based on the urinary concentrations remained only moderate to moderately high for both metals in both assessment rounds, suggesting that the experts were only partly successful in distinguishing exposures based on a two-category scale in this setting. The poor validity on both the four- and two-category scales suggests that even a limited, a priori, characterization of exposure may help determine appropriate semi-quantitative categories for the exposure range and the number of exposure categories to use for expert assessment in a workplace or study before requesting experts to provide their exposure estimates. For example, syntheses of the data in the published literature37, 38, 39 or from inspection measurements40, 41, 42, 43, 44 and other exposure databases45 can provide useful information on exposure variability and help anchor the experts’ ratings to a concentration scale.
The use of the EIQ responses had little impact on the overall measures of reliability or validity of the experts’ ratings in this study, likely (at least in part) due to the limited contrast in exposure concentrations between participants within the same plant. However, our finding is consistent with our similar evaluations in the textile and foundry industries9 and with previous studies that showed that additional information did not improve raters’ reliability and validity,12, 34, 46 although some improvements were observed by Tielemans et al.11 with the use of similar types of questionnaires. The advantage of the EIQ, however, is that within-job differences are systematically captured and thus more easily used in programmable decision rules.47
No consistent patterns were observed in the reliability and validity of the experts’ ratings by type of rater, similar to our previous findings in the textile and foundry industries.9 Industrial hygienists had somewhat higher reliability than the occupational physicians for both metals in most comparisons. However, both types of experts generally had similar poor-to-fair validity with airborne and urinary nickel and chromium exposures overall. Plant-specific differences were observed, with the industrial hygienists’ ratings having higher validity than the occupational physicians’ ratings in some comparisons. The similarity of the two types of raters likely relates to these occupational physicians’ regular work site visits and familiarity with exposure monitoring and may not be generalizable to occupational physicians without this expertize. For instance, several studies have reported that substantial field experience is critical to provide valid exposure estimates.10, 16, 34, 48
This study has several limitations. As in Friesen et al.,9 this study design straddled the population-based and industry-based study designs by using questionnaires designed for a typical case-control study but evaluating the performance in a single industry for a single year. Thus, this study cannot predict how well these experts would have performed if asked to estimate these same exposures across a wide time span or across multiple industries with a wide range of exposures. In addition, the small amount of contrast in exposures in these plants limited our ability to assess the validity of experts’ ratings in the electroplating industry. The contrast in case-control studies is likely to vary by the agent, time period and population being assessed. In addition, the airborne and urinary measurements may be ‘alloyed’ gold standards because the measurements did not account for the metal’s solubility, valence, form, or particle size and four measurements may not be sufficient to fully characterize the subjects’ average exposures.
In this study, despite moderately high reliability among experts, experts’ ratings had low validity in relation to airborne and urinary exposure measurements, which was likely attributable to low contrast in exposures. As a result, our evaluations of the influence of responses to industry-specific questionnaires on the validity of the experts’ ratings in relation to airborne and urinary measurements were inconclusive. However, this study provides some insight into the challenges for designing a study to evaluate the validity of exposure ratings. Validity was generally better when there was a wider and more even distribution of exposure and poor when there was low contrast. In addition, this study showed that the experts were reasonably able to distinguish between high and low exposures, but not finer categories. Both findings indicate that prior knowledge of the variability in exposure may be necessary for setting exposure categories for experts’ ratings. In population-based studies, however, information on historical exposure and its variability is generally limited or non-existent, and thus we will likely need to rely on syntheses of publicly available data to anchor experts’ estimates, which do not necessarily reflect the working conditions of the job being assessed. Our results also point to the continued need for methods to increase the ability of experts to evaluate exposures. For instance, training experts on the interpretation of exposure distributions and determinants49 and developing transparent, programmable decision rules to systematically capture exposure differences that can be reviewed and refined by multiple experts47 may improve the accuracy of expert judgment.
Siemiatycki J, Day N, Fabry J, Cooper JA . Discovering carcinogens in the occupational environment: a novel epidemiologic approach. J Natl Cancer Inst 1981; 66: 217–225.
Stewart PA, Stewart WF, Siemiatycki J, Heineman EF, Dosemeci M . Questionnaires for collecting detailed occupational information for community-based case control studies. Am Ind Hyg Assoc J 1998; 59: 39–44.
Rybicki BA, Peterson EL, Johnson CC, Kortsha GX, Cleary WM, Gorell JM . Intra- and inter-rater agreement in the assessment of occupational exposure to metals. Int J Epidemiol 1998; 27: 269–273.
Benke G, Sim M, Forbes A, Salzberg M . Retrospective assessment of occupational exposure to chemicals in community-based studies: validity and repeatability of industrial hygiene panel ratings. Int J Epidemiol 1997; 26: 635–642.
Siemiatycki J, Fritschi L, Nadon L, Gérin M . Reliability of an expert rating procedure for retrospective assessment of occupational exposures in community-based case-control studies. Am J Ind Med 1997; 31: 280–286.
McGuire V, Longstreth WT, Nelson LM, Koepsell TD, Checkoway H, Morgan MS et al. Occupational exposures and amyotrophic lateral sclerosis. A population-based case-control study. Am J Epidemiol 1997; 145: 1076–1088.
Goldberg MS, Siemiatycki J, Gerin M . Inter-rater agreement in assessing occupational exposure in a case-control study. Br J Ind Med 1986; 43: 667–676.
Fritschi L, Nadon L, Benke G, Lakhani R, Latreille B, Parent ME et al. Validation of expert assessment of occupational exposures. Am J Ind Med 2003; 43: 519–522.
Friesen MC, Coble JB, Katki HA, Ji BT, Xue S, Lu W et al. Validity and reliability of exposure assessors’ ratings of exposure intensity by type of occupational questionnaire and type of rater. Ann Occup Hyg 2011; 55: 601–611.
Teschke K, Olshan AF, Daniels JL, De Roos AJ, Parks CG, Schulz M et al. Occupational exposure assessment in case–control studies: opportunities for improvement. Occup Environ Med 2002; 59: 575–594.
Tielemans E, Heederik D, Burdorf A, Vermeulen R, Veulemans H, Kromhout H et al. Assessment of occupational exposures in a general population: comparison of different methods. Occup Environ Med 1999; 56: 145–151.
Steinsvåg K, Bråtveit M, Moen BE, Kromhout H . Inter-rater agreement in the assessment of exposure to carcinogens in the offshore petroleum industry. Occup Environ Med 2007; 64: 582–588.
Rocheleau CM, Lawson CC, Waters MA, Hein MJ, Stewart PA, Correa A et al. Inter-rater reliability of assessed prenatal maternal occupational exposures to solvents, polycyclic aromatic hydrocarbons, and heavy metals. J Occup Environ Hyg 2011; 8: 718–728.
Mannetje A, Fevotte J, Fletcher T, Brennan P, Legoza J, Szeremi M et al. Assessing exposure misclassification by expert assessment in multicenter occupational studies. Epidemiology 2003; 14: 585–592.
Post V, Kromhout H, Heederik D, Noy D, Duilzentkunst RS . Semiquantitative estimates of exposure to methylene chloride and styrene: the influence of quantitative exposure data. Appl Occup Environ Hyg 1991; 6: 197–204.
Teschke K, Hertzman C, Dimich-Ward H, Ostry A, Blair J, Hershler R . A comparison of exposure estimates by worker raters and industrial hygienists. Scand J Work Environ Health 1989; 15: 424–429.
Hertzman C, Teschke K, Dimich-Ward H, Ostry A . Validity and reliability of a method for retrospective evaluation of chlorophenate exposure in the lumber industry. Am J Ind Med 1988; 14: 703–713.
Guillemin MP, Berode M . A study of the difference in chromium exposure in workers in two types of electroplating process. Ann Occup Hyg 1978; 21: 105–112.
Kiilunen M, Utela J, Rantanen T, Norppa H, Tossavainen A, Koponen M et al. Exposure to soluble nickel in electrolytic nickel refining. Ann Occup Hyg 1997; 41: 167–173.
Pierre F, Diebold F, Baruthio F . Biomonitoring of two types of chromium exposure in an electroplating shop. Int Arch Occup Environ Health 2008; 81: 321–329.
Lumens ME, Ulenbelt P, Géron HM, Herber RF . Hygienic behaviour in chromium plating industries. Int Arch Occup Environ Health 1993; 64: 509–514.
Tola S, Kilpio J, Virtamo M . Urinary and plasma concentrations of nickel as indicators of exposure to nickel in an electroplating shop. J Occup Med 1979; 21: 184–188.
Liu CS, Kuo HW, Lai JS, Lin TI . Urinary N-acetyl-beta-glucosaminidase as an indicator of renal dysfunction in electroplating workers. Int Arch Occup Environ Health 1998; 71: 348–352.
Caglieri A, Goldoni M, Acampa O, Andreoli R, Vettori MV, Corradi M et al. The effect of inhaled chromium on different exhaled breath condensate biomarkers among chrome-plating workers. Environ Health Perspect 2006; 114: 542–546.
Bavazzano P, Bolognesi R, Cassinelli C, Gori R, Li Donni V, Martellini F et al. Skin contamination and low airborne nickel exposure of electroplaters. Sci Total Environ 1994; 155: 83–86.
Makinen M, Linnainmaa M . Dermal exposure to chromium in electroplating. Ann Occup Hyg 2004; 48: 277–283.
NIOSH. Manual of Analytical Methods (NMAM): Method 7300 4th edn US Department of Health and Human Services: Cincinnati, OH, USA. 1994.
NIOSH. Manual of Analytical Methods (NMAM): Method 8310 4th edn US Department of Health and Human Services: Cincinnati, OH, USA. 1994.
Smith TJ, Kriebel D . A Biologic Approach to Environmental Assessment and Epidemiology. Oxford University Press: New York, NY, USA. 2010 pp 77–79.
Landis JR, Koch GG . The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159–174.
Ciccone G, Vineis P . Inter-rater agreement in the assessment of occupational exposure to herbicides. Med Lav 1988; 79: 363–367.
Correa A, Min YI, Stewart PA, Lees PS, Breysse P, Dosemeci M et al. Inter-rater agreement of assessed prenatal maternal occupational exposures to lead. Birth Defects Res A Clin Mol Teratol 2006; 76: 811–824.
Cherrie JW, Schneider T . Validation of a new method for structured subjective assessment of past concentrations. Ann Occup Hyg 1999; 43: 235–245.
de Cock J, Kromhout H, Heederik D, Burema J . Experts’ subjective assessment of pesticide exposure in fruit growing. Scand J Work Environ Health 1996; 22: 425–432.
Semple SE, Proud LA, Tannahill SN, Tindall ME, Cherrie JW . A training exercise in subjectively estimating inhalation exposures. Scand J Work Environ Health 2001; 27: 395–401.
Vermeulen R, Stewart P, Kromhout H . Dermal exposure assessment in occupational epidemiologic research. Scand J Work Environ Health 2002; 28: 371–385.
Bakke B, Stewart PA, Waters MA . Uses of and exposure to trichloroethylene in U.S. industry: a systematic literature review. J Occup Environ Hyg 2007; 4: 375–390.
Park D, Stewart PA, Coble JB . A comprehensive review of the literature on exposure to metalworking fluids. J Occup Environ Hyg 2009; 6: 530–541.
Pronk A, Coble J, Stewart PA . Occupational exposure to diesel engine exhaust: a literature review. J Expo Sci Environ Epidemiol 2009; 19: 443–457.
Lavoue J, Friesen MC, Burstyn I . Workplace measurements by the US Occupational Safety and Health Administration since 1979: descriptive analysis and potential uses for exposure assessment. Ann Occup Hyg 2013; 57: 77–97.
Froines JR, Baron S, Wegman DH, O’Rourke S . Characterization of the airborne concentrations of lead in U.S. industry. Am J Ind Med 1990; 18: 1–17.
Friesen MC, Coble JB, Lu W, Shu XO, Ji BT, Portengen L et al. Combining a job-exposure matrix with exposure measurements to assess occupational exposure to benzene in a population cohort in Shanghai, China. Ann Occup Hyg 2012; 56: 80–91.
Burstyn I, Jonasi L, Wild TC . Obtaining compliance with occupational health and safety regulations: a multilevel study using self-determination theory. Int J Environ Health Res 2010; 20: 271–287.
Koh DH, Bhatti P, Coble JB, Stewart PA, Lu W, Shu XO et al. Calibrating a population-based job-exposure matrix using inspection measurements to estimate historical occupational exposure to lead for a population-based cohort in Shanghai, China. J Expo Sci Environ Epidemiol 2014; 24: 9–16.
Peters S, Vermeulen R, Olsson A, Van Gelder R, Kendzia B, Vincent R et al. Development of an exposure measurement database on five lung carcinogens (ExpoSYN) for quantitative retrospective occupational exposure assessment. Ann Occup Hyg 2012; 56: 70–79.
Stewart PA, Carel R, Schairer C, Blair A . Comparison of industrial hygienists’ exposure evaluations for an epidemiologic study. Scand J Work Environ Health 2000; 26: 44–51.
Pronk A, Stewart PA, Coble JB, Katki HA, Wheeler DC, Colt JS et al. Comparison of two expert-based assessments of diesel exhaust exposure in a case–control study: programmable decision rules versus expert review of individual jobs. Occup Environ Med 2012; 69: 752–758.
Kromhout H, Oostendorp Y, Heederik D, Boleij JS . Agreement between qualitative exposure estimates and quantitative exposure measurements. Am J Ind Med 1987; 12: 551–562.
Logan P, Ramachandran G, Mulhausen J, Hewett P . Occupational exposure decisions: can limited data interpretation training help improve accuracy? Ann Occup Hyg 2009; 53: 311–324.
This study was funded by the Intramural Research Program of the Division of Cancer Epidemiology and Genetics, National Cancer Institute, National Institutes of Health.
The authors declare no conflict of interest.
Supplementary Information accompanies the paper on the Journal of Exposure Science and Environmental Epidemiology website
About this article
Current Environmental Health Reports (2019)
Validity of Expert Assigned Retrospective Estimates of Occupational Polychlorinated Biphenyl Exposure
The Annals of Occupational Hygiene (2015)
International Journal of Molecular Sciences (2014)