Objective: The validation of dietary assessment methods is critical in the evaluation of the relation between dietary intake and health. The aim of this study was to assess the validity of a food frequency questionnaire by comparing energy intake with energy expenditure measured with the doubly labelled water method.
Design: Total energy expenditure was measured with the doubly labelled water (DLW) method during a 10 day period. Furthermore, the subjects filled in the food frequency questionnaire about 18–35 days after the DLW phase of the study was completed.
Subjects: Twenty-one healthy, non-pregnant females volunteered to participate in the study; only 17 subjects completed the study.
Results: The group energy intake was on average 10% lower than the energy expenditure, but the difference was not statistically significant. However, there was a wide range in reporting accuracy: seven subjects were identified as acceptable reporters, eight as under-reporters and two were identified as over-reporters. The width of the 95% confidence limits of agreement in a Bland and Altman plot for energy intake and energy expenditure varied from −5 to 3 MJ.
Conclusion: The data showed that there was substantial variability in the accuracy of the food frequency questionnaire at the individual level. Furthermore, the results showed that the questionnaire was more accurate for groups than individuals.
In epidemiological studies self-administered food frequency questionnaires are frequently used to assess habitual nutrient intake and to rank subjects according to intake. Poor validity of the questionnaire has implications for the interpretation of studies on diet and health. A key problem is how to identify those individuals who have provided data of poor quality, which can include under- or over-reporting.
In validation studies data from food frequency questionnaires have often been compared with data from a reference method, eg weighed food records or repeated 24 h recall. However, these reference methods may also be subject to error. To date, there are only a limited number of objective measures that are sufficiently precise and unbiased in themselves to validate dietary assessments (Bingham, 1991). Under conditions of weight stability, energy expenditure provides an estimate of energy intake that can be used to assess the validity of results from a dietary assessment method. The method of choice for estimating free living energy expenditure is the doubly labelled water (DLW) technique (Prentice, 1990).
Energy expenditure obtained using the DLW method has been used to validate energy intake, calculated from food records, diet recalls and diet history, in several studies (Black et al, 1991, 1996, 1997, 2000). These studies have identified considerable bias towards underestimation of energy intake. However, there have been relatively few studies comparing energy expenditure measured by the DLW method and energy intake obtained by food frequency questionnaires which aim to measure habitual energy intake (Kaskoun et al, 1994; Sawaya et al, 1996; Kroke et al, 1999).
In the present study the main aim is to evaluate a quantitative food frequency questionnaire developed for use in epidemiological studies. Energy intake estimated from the questionnaire is compared with energy expenditure measured with the DLW method in a group of healthy young women.
Subjects and methods
Subjects and design
Twenty-one healthy, non-pregnant females volunteered to participate in the study. They were selected from university students who had responded to an announcement of the project within the university area. After having been informed about the general purpose, procedure and risks of the experiments each subject gave written consent. The regional ethical committee approved the study protocol.
Total energy expenditure (TEE) was measured with the DLW method during a 10 day period, starting on days 6–7 of the menstrual cycle. The subjects filled in the quantitative food frequency questionnaire 18–35 days after the DLW phase of the study was completed. Bodyweight was measured at start of the DLW period. The basal metabolic rate (BMR) was measured and a second bodyweight recorded 18–25 days after the DLW period
Quantitative food frequency questionnaire
A 11-page optical mark readable questionnaire was designed to capture the habitual food intake among adults; this contained questions about 180 food items grouped together according to the Norwegian meal pattern. The traditional Norwegian meal pattern consists of three bread-based meals (breakfast, lunch and supper) and one hot meal (dinner). The options on the frequency of consumption of particular food types vary from several times a day to once a month, with portion-size choices based on typical household units: slices, glasses, cups, pieces, spoons and ladle. A detailed description of the questionnaire has been presented elsewhere (Nes et al, 1992; Johansson et al, 1997). Questions about use of dietary supplements, such as cod liver oil, fish oil capsules and some vitamins/minerals were included in the questionnaire. The questionnaire has previously been validated against weighed dietary intakes, fatty acids in serum and adipose tissue (Nes et al, 1992; Andersen et al, 1996, 1999).
The daily intake of energy and nutrients was computed using a food database and software systems developed at the Institute for Nutrition Research, University of Oslo. The food database is mainly based on the official Norwegian Food Table (1995). The table is continuously supplemented with new data and supplied with information on fatty acids (Norwegian Nutrition Council, 1988). Cod liver oil, vitamin and mineral supplements were included in the nutrient calculations.
Measurements of BMR, weight and height
Early in the morning (before 7.30 am) the women arrived at the institute after an overnight fast (10–12 h). All subjects rested for at least 30 min before oxygen consumption and carbon dioxide production were measured for 30 min in an open hood calorimetric system (Deltatrac™ II Metabolic Monitor, DATEX, Finland).
Body weight was measured with the females wearing their underwear. Body mass index (BMI) was calculated by body weight/height-squared (kg/m2).
Measurement of energy expenditure by the doubly labelled water method
After an overnight fast (day 0) the subjects received a preweighed dose of doubly labelled water followed by 50 g normal water. The dose given was 0.16 g 100% 18O per kg body weight and 0.18 g 100% 2H per kg body weight. Blood samples were taken before drinking the labelled water (baseline) and 3 h later. An aliquot of urine was obtained from the second voiding of the day throughout the DLW period. The plasma samples and urine samples from day 1 and day 10 were analysed for 18O and 2H on SIRA-10 and Series-II isotope ratio mass spectrometers (VG, Middlewich, UK) relative to a series of laboratory reference waters previously calibrated against international standards; Vienna Standard Mean Ocean Water and Standard Light Antarctic Precipitation (Haggarty et al, 1994a, b). Fat-free mass (FFM) was calculated from the H218O dilution space assuming that body water=H218O space/1.01, that FFM=0.732×body water and body fat=body weight−FFM. The mean proportion of water loss which was fractionated was estimated (Haggary et al, 1994b, 1997). The respiratory quotient (RQ) was taken to be equivalent to the food quotient (Black et al, 1986) determined from the food frequency questionnaire. Energy expenditure was calculated from CO2 (1/day) production and RQ using the following rearrangement of the Weir (1949) equation;
To identify the validity of the questionnaire the subjects were identified as under-reporters (UR), acceptable reporters (AR) or over-reporters (OR) from their ratio EI:EEDLW. The 95% of confidence limits of agreement between EI and EEDLW were calculated as proposed by Black et al (2000):
CVEI is the coefficient of variation for daily energy intake (23%), CVEE is the coefficient of variation for DLW energy expenditure (8.2%) and d is the number of days of diet assessment. Since the questionnaire refers to the habitual intake the number of days could be taken as infinity, the expression of CVEI disappears. AR were defined as having the ratio EI:EEDLW in the range 0.84–1.16, UR as EI:EEDLW<0.84 and OR as EI:EEDLW>1.16.
The following equation (Cole, 1997) was used to calculate sample size needed to detect a 16% difference between energy intake and energy expenditure (if we expect an energy intake of 9 MJ in this group of women 16% difference amounts to 1.4 MJ):
where Z1-α/2=1.96 (type I error α=0.05), Z1-β=0.84 (type II error β=0.2, power 80%), f=D/s.d., D=expected difference. The standard deviation of energy intake is about 2 MJ, so to be sure of detecting a difference of 1.4 MJ with 80% power at 5% significance we had to include 16 individuals.
Statistical analyses were performed using SPSS (9.0). The data are expressed as mean and s.d. Mean differences between methods were analysed using the Student t-test for paired samples. Results were considered to be statistically significant at P<0.05. The agreement between methods was analysed by the method proposed by Bland and Altman (1986), using a plot of the difference between the two methods against the average of the measurements. This type of plot shows the magnitude of disagreement, spots outliers and any trend. Furthermore, the agreement between the two methods was evaluated by Spearman rank correlation coefficient.
Three subjects had higher than normal values for isotope elimination rates and implausibly low values for the DLW-derived energy expenditure (EEDLW/BMR=1.10, 0.86 and 0.40, respectively). A blind re-analysis of the same samples from these subjects confirmed the initial values for energy expenditure, suggesting possible mislabelling or contamination of samples rather than analytical error. One person did not fill in the food frequency questionnaire. These four subjects were excluded from subsequent analysis. The characteristics of the remaining 17 subjects are presented in Table 1. Isotopic data for the calculation of EEDLW are shown in Table 2.
Information on the EEDLW and reported energy intake and the difference between the two methods is shown in Table 3. The accuracy of the reported energy intake was calculated by expressing the ratio EI:EEDLW. There was a wide range in reporting accuracy; seven subjects were identified as acceptable reporters (AR), eight as under-reporters (UR) and two were identified as over-reporters (OR). The group average energy intake deviated from the measured EEDLW by −10%. The Bland and Altman plot showing the difference between energy intake from the questionnaire and the energy expenditure from the DLW method plotted against the mean of the two methods are shown in Figure 1. The plot illustrates the problem with both under-reporting and over-reporting of energy intake among individuals. The width of the 95% confidence limits of agreement varied from −5 to 3 MJ (±2 s.d.) indicated wide discrepancies between the two methods for individual subjects. The plot did not indicate that differences tended to increase as absolute energy intake increased. The Spearman rank correlation coefficient between reported energy intake and expenditure was 0.36 (P=0.15).
Spearman correlation coefficients between some subject details and reporting accuracy are shown in Table 4. Both the body weight and BMI showed negative correlations with reporting accuracy, although neither was significant. Smilarly the percentage of energy from fat from the diet showed no relation to reporting accuracy. A small mean net weight loss of 0.5 kg (P=0.112) was observed over the study period. There was no correlation between the amount of weight change and reporting accuracy (Table 4).
The data from this study showed that the questionnaire was more accurate for groups than individuals. The energy intake was on average 10% lower than the EEDLW while there was substantial variability in the accuracy of the food frequency questionnaire at the individual level.
A limitation in the interpretation of the results is that participants in this study were a selected, highly motivated group, including only one gender. For this reason the validity of the FFQ observed in this study may have been overestimated compared to what would have been found in a group of the general population. Moreover, the exclusion of four subjects from the original small sample reduced the power of the study. However, with a study sample of 17 subjects we should still be able to detect the difference of at least 16% chosen as a limit.
The principle behind the validation of energy intake using EE measured by the DLW method is that of energy balance. The mean weight changes in our group of subjects were small (0.5 kg), therefore the subjects were likely in weight balance. Furthermore, the short-term (10 day) EE measurement should represent habitual energy expenditure since the energy intake estimated from the FFQ should capture the usual intake from the preceding year. The participants were instructed not to change their physical activity and dietary intake during the 10 day EE measurement period. However, the observed difference between EEDLW and reported energy intake might partly be explained by the incomplete coverage of the FFQ data by the EE measurement period. The time lag between measurements of EEDLW and EI (2½–5 weeks) included in this study should not interfere with the results since the FFQ aimed to cover the dietary intake during the last year.
Within individuals, the quite large discrepancies between energy intake and energy expenditure may of course also be due to errors in the DLW method. Although there is no reason to believe that there would be any significant bias in the DLW-derived group mean energy expenditure (Haggarty et al, 1994a, b), the technique is not perfect. Some of the within-individual disagreement when comparing the two methodologies can be ascribed to the expected variability in DLW derived energy expenditure (Davidson et al, 1997).
Comparison between energy intake and EEDLW
Several food record validation studies using the DLW method as reference method have observed a varied degree of under-reporting (Black et al, 1991). The same has been observed in the few validation studies of food frequency questionnaires, which have used the DLW method as a reference method. In the study by Sawaya et al (1996), comparing energy intake estimated by two food frequency questionnaires (Willett and a FHCRC/Block questionnaire) and EEDLW among 10 young women, the observed average under-reporting of 16–28% was larger than in our study. In a similar study among 28 men and women Kroke et al (1999) observed an under-reporting of 22% on average. In 1992 we conducted a similar study to the one presented here among a group of elderly women (n=12, age 67.0–76.0 y) using a very similar food frequency questionnaire (Andersen, 1992). In that study we observed a small over-reporting of energy intake by the questionnaire of 6% on average.
Correlation coefficient is widely accepted as an inappropriate way of assessing the level of agreement between two measurements. Therefore, the mean difference is calculated to obtain information on bias in the group estimate, and limits of agreement (±2 s.d. of mean difference) indicate the scatter of individual results (Altman and Bland plot). Applying the Bland–Altman plot to the energy data, we showed a mean difference with a small bias and a wide scatter of differences between self-reported energy intake and energy expenditure. The wide scattering of the differences showed clearly that some subjects under- or over-reported their energy intake more than others. Under-reporting was a larger problem than over-reporting—nearly 50% were identified as under-reporters while 12% were identified as over-reporters. The limits of agreement observed in the study by Kroke et al (1999) were similar to what we observed.
Differential under-reporting depending on subject characteristics could have serious implications for risk estimates of diet–disease relations. Unfortunately our sample size was too small to detect significant determinants (Table 4).
The ability of the FFQ to rank individuals according to energy intake was evaluated using the Spearman correlation coefficient. We observed a correlation coefficient of 0.36 between individual values for EEDLW and self-reported energy intake, which indicates no association. A higher correlation was observed among the elderly women (r=0.56, P=0.06; Andersen, 1992). Among 10 young females Sawaya et al (1996) observed a significant association between individual values for EEDLW and energy intake from the two questionnaires; however, no association was observed among 10 elderly women. Kroke et al (1999) found a significant correlation between energy intake and expenditure (r=0.48).
From the study we cannot conclude whether the difference we discover is due to mis-reporting of food items or difficulties in reporting correct portion sizes in the FFQ. Probably both problems introduce error in the FFQ.
In summary, data from this study show that there is substantial variability in the accuracy of the food frequency questionnaire at the individual level. However, the results indicate that the questionnaire can provide an accurate measure of the mean energy intake of a group.
Andersen, LF ((1992)). Energy intake measured with a quantitative food frequency questionnaire. Absolute and relative validation of the method among elderly women. Masters thesis. Oslo: University of Oslo
Andersen, LF, Solvoll, S & Drevon, CA ((1996)). Very-long-chain n-3 fatty acids as biomarkers for intake of fish and n-3 fatty acid concentrates. Am. J. Clin. Nutr., 64, 305–311.
Andersen, LF, Solvoll, K, Johansson, L, Salminen, I, Aro, A & Drevon, CA ((1999)). Evaluation of a food frequency questionnaire with weighed records, fatty acids and alpha-tocopherol in adipose tissue and serum. Am. J. Epidemiol., 150, 75–87.
Bingham, SA ((1991)). Validation of Dietary Assessment through Biomarkers, 41–52, Rotterdam: Smith-Gordon
Black, A, Prentice, AM & Coward, WA ((1986)). Use of food quotients to predict respiratory quotients for the doubly-labelled water method of measuring energy expenditure. Hum. Nutr. Clin. Nutr., 40C, 381–391.
Black, AE ((1986)). Physical activity levels from meta-analysis of doubly labeled water studies for validating energy intake as measured by dietary assessment. Nutrition Reviews, 54, 170–174.
Black, AE, Goldberg, GR, Jebb, SA, Livingstone, MBE, Cole, TJ & Prentice, AM ((1991)). Critical evaluation of energy intake data using fundamental principles of energy physiology: 2. Evaluating the results of published surveys. Eur. J. Clin. Nutr., 45, 583–599.
Black, AE, Bingham, SA, Johansson, G & Coward, WA ((1997)). Validation of dietary intakes of protein and energy against 24 hour urinary N and DLW energy expenditure in middle-aged women, retired men and post-obese subjects: comparison with validation against presumed energy requirements. Eur. J. Nutr., 51, 405–413.
Black, AE, Welch, AA & Bingham, SA ((2000)). Validation of dietary intakes measured by diet history against 24 h urinary nitrogen excretion and energy expenditure measured by the doubly-labelled water method in middle-aged women. Br. J. Nutr., 83, 341–354.
Bland, JM & Altman, DG ((1986)). Statistical methods for assessing agreement between methods of clinical measurements. Lancet, 1, 307–310.
Cole, TJ ((1997)). Sampling, study size, and power. In Design Concepts in Nutritional Epidemiology, ed. BM Margetts & M Nelsonpp 64–86, Oxford: Oxford University Press
Davidson, L, McNeill, G, Haggarty, P, Smith, JS & Franklin, MF ((1997)). Free-living energy expenditure of adult men assessed by continuous heart rate monitoring and doubly labelled water. Br. J. Nutr., 78, 695–708.
Haggarty, P, Franklin, MF, Fuller, MF, Mcgaw, BA, Milne, E, Duncan, G, Christie, SL & Smith, JS ((1994a)). Validation of the doubly labeled water method in growing pigs. Am. J. Physiol., 267, R1574–R1588.
Haggarty, P, McNeill, G, Manneh, MK, Davidson, L, Milne, E, Duncan, G & Ashton, J ((1994b)). The influence of exercise on the energy requirements of adult males in the UK. Br. J. Nutr., 72, 799–813.
Haggarty, P, Valencia, ME, McNeill, G, Gonzales, NL, Moya, SY, Pinelli, A, Quihui, L, Saucedo, MS, Esparza, J, Ashton, J, Milne, E & James, WPT ((1997)). Energy expenditure during heavy work and its interaction with body weight. Br. J. Nutr., 77, 359–373.
Johansson, L, Solvoll, K, Bjørneboe, G-EA & Drevon, CA ((1997)). Dietary habits among Norwegian men and women. Scand. J. Nutr., 41, 63–70.
Kaskoun, MC, Johnson, RK & Goran, MI ((1994)). Comparison of energy intake by semiquantitative food-frequency questionnaire with total energy expenditure by the doubly labeled water method in young children. Am. J. Clin. Nutr., 60, 43–47.
Kroke, A, Klipstein-Grobusch, K, Voss, S, Möseneder, J, Thielecke, F, Noack, R & Boeing, H ((1999)). Validation of a self-administrated food-frequency questionnaire administered in the European Prospective Investigation into Cancer and Nutrition (EPIC) Study: comparison of energy, protein, and macronutrient intakes estimated with the doubly labeled water, urinary nitrogen, and repeated 24-h dietary recall methods. Am. J. Clin. Nutr., 70, 439–447.
Nes, M, Frost Andersen, L, Solvoll, K, Sandstad, B, Hustvedt, B-E & Løvø, A ((1992)). Accuracy of a quantitative food frequency questionnaire applied in elderly Norwegian women. Eur. J. Clin. Nutr., 46, 809–821.
Norwegian Food Table ((1995)). National Nutrition Council and Norwegian Food Control Authorities.
Norwegian Nutrition Council ((1988)). Fatty Acid Content in Food, Oslo: The National Association for Nutrition and Health
Prentice, AM ((1990)). The doubly-labelled water method for measuring energy expenditure, Technical recommendations for use in humans. A concensus report by IDECG Working Group
Sawaya, AL, Tucker, K, Tsay, R, Willett, W, Saltzman, E, Dallal, GE & Roberts, SB ((1996)). Evaluation of four methods for determining energy intake in young and older women: comparison with doubly labeled water measurements of total energy expenditure. Am. J. Clin. Nutr., 63, 491–499.
Weir, JB & De, V ((1949)). New methods for calculating metabolic rate with special reference to protein metabolism. J. Physiol., 109, 1–9.
The DLW analysis was carried out with the support of SERAD
About this article
Cite this article
Andersen, L., Tomten, H., Haggarty, P. et al. Validation of energy intake estimated from a food frequency questionnaire: a doubly labelled water study. Eur J Clin Nutr 57, 279–284 (2003) doi:10.1038/sj.ejcn.1601519
- doubly labelled water
- food questionnaire
Adherence to the Healthy Nordic Food Index and the incidence of acute myocardial infarction and mortality among patients with stable angina pectoris
Journal of Human Nutrition and Dietetics (2019)
Creatinine, total cysteine and uric acid are associated with serum retinol in patients with cardiovascular disease
European Journal of Nutrition (2019)
Relative validity of a semi-quantitative food frequency questionnaire for Singaporean toddlers aged 15–36 months
BMC Nutrition (2018)
Relative validation of a pre-coded food diary in a group of Norwegian adults – Comparison of underreporters and acceptable reporters
PLOS ONE (2018)