In November 2018, we presented the results of a large macronutrient feeding study at The Obesity Society’s Blackburn Symposium  and simultaneously in BMJ, with peer review available online . In that study, total energy expenditure (TEE) measured by doubly-labeled water (DLW) at post-weight-loss and at 10 and 20 weeks of weight-loss maintenance was 200–280 kcal/day greater on a low- versus high-carbohydrate diet, consistent with the Carbohydrate-Insulin Model of obesity . To facilitate scientific discourse, we made the full database immediately available on Open Science Framework.
At the Blackburn Symposium, and in subsequent comments elsewhere, Kevin Hall raised a series of concerns about our study, including the possibility of baseline instability in weight, randomization failure for other reasons, and inaccuracy of DLW due to isotope sequestration by de novo lipogenesis [4, 5]. We aimed to refute those criticisms with evidence to show baseline weight stability, a lack of difference in weight change between diet groups before and after randomization, and insignificant rates of de novo lipogenesis on relevant diets [6,7,8].
In their current reanalysis, Hall et al.  reiterate several previous concerns and raise new ones. Here, we respond to three key questions related to our study. To facilitate this response, we obtained more accurate and precise data on energy intake during the weight-loss maintenance phase of our study .
What is the appropriate baseline for studying metabolism during weight-loss maintenance?
In earlier versions of this reanalysis (before Speakman joined as coauthor) , and elsewhere , Hall and Guo argue that the pre- rather than post-weight-loss measurement is the more appropriate baseline for examining diet effects on TEE. However, that approach disregards important biological variation among individuals regarding how body composition and metabolism change in response to weight loss, which averaged 9.6 kg in our study. Variation in these major confounders would introduce imprecision in statistical models involving inter-individual comparisons in this parallel design. As a general rule, baseline data should be collected as close to randomization as possible to decrease error arising from any time-varying covariate (the pre-weight-loss measurement would entail a 4-month lag until randomization). For these reasons, weight-loss-maintenance studies like Diogenes typically use the post-weight-loss timepoint.
We explored this issue by comparing the pre-weight-loss (Fig. 1a) or post-weight-loss (Fig. 1b) measurement of TEE with TEE measured at 10 and 20 weeks after randomization. As expected, the post-weight-loss timepoint yielded a stronger correlation. Furthermore, the results of the ANCOVA analysis by Hall et al. , without use of either baseline, yielded a diet effect similar to our change models with the post-weight-loss baseline.
A related concern of Hall and Guo  is the possibility of weight instability and ongoing metabolic adaptations during the post-weight-loss baseline measurements. However, as previously considered , weight change during the 15-day assessment period was small, averaging 23 g/day, with no significant difference between individuals who would be assigned to the three diet groups. Moreover, the experimental design protects against type I (false-positive) error due to ongoing metabolic adaptations (or any other prerandomization factor).
According to a basic principle of statistical analysis, the most powerful method available should be chosen to avoid type II (false-negative) error. Clearly, use of the post-weight-loss timepoint provides the most precise and least biased estimate.
Was the change in our analysis plan involving choice of baseline proper?
Hall et al.  criticize us for revising the final analysis plan to the post-weight-loss baseline, after our initial registry specified the pre-weight-loss timepoint. As previously discussed , the pre-weight-loss specification was an inadvertent holdover from a prior crossover study , in which no post-weight-loss timepoint was obtained (or needed). Unlike in parallel studies, use of a pre-weight-loss baseline would not introduce major confounding in crossover studies because of the within-individual nature of the comparisons. Biological changes during weight loss for each participant, with randomization, would affect all diet periods equally.
We identified this misspecification during preparation of our final analysis plan under supervision of our statistician, made the change before breaking the data blind, and disclosed the registry change in the manuscript. An analysis with the pre-weight-loss baseline is available in online peer review , and we released the full database to facilitate alternative analyses.
In the CALERIE study, the final analyses were similarly “prespecified before initiation of analyses” , even though data had accumulated during the study . Arguably the three largest macronutrient diet studies of the last decade—CALERIE, DIETFITS, and Diogenes—each made changes involving the primary outcome from initial registry posting to final publication. With the greater methodological heterogeneity of diet versus drug studies , registry changes and discrepancies are more the rule than the exception. Hall should appreciate this point. The registry for his only macronutrient RCT  did not specify a primary outcome, among other deficiencies that remain as of the 58th revision in February 2019.
Thus, our registry procedures are entirely consistent with standard practice in nutrition research and carry little risk of bias.
Could nonadherence to the test diets in our study explain the primary finding?
Before addressing this question, the notion of “unaccounted energy” introduced by Hall et al. warrants examination. In the reanalysis, they compared reported energy intake and expenditure in our study, identifying discrepancies as violating the “physical law of energy conservation.” But this treatment disregards substantial cumulative error arising from measurement of the various components of energy balance, each with recognized imprecision and temporal variation. In the recent study by Hall et al. of ultra-processed food , conducted in the optimal environment of a metabolic ward, mean energy discrepancy on one diet was large (382 kcal/day) and “unaccounted energy” exceeded 250 kcal/day for most participants (Fig. 1c).
Recognizing this inherent variability and imprecision in the measurement of both energy intake and TEE, we can see why exclusions involving their difference or ratio (as in Fig. 1b of Hall et al.) would produce highly misleading results. Whereas individuals with low energy intake relative to TEE might have been nonadherent (i.e., unobserved food intake), they would also tend to be at the upper end of the natural distribution for TEE (related to true biological differences or randomly distributed measurement variation). Therefore, eliminating them would deplete the cohort of those with the greatest TEE denominator, deflating the diet effect.
We can demonstrate this phenomenon in three ways. First, we conducted the converse analysis, sequentially eliminating individuals with “unaccounted energy” arising from low TEE to energy intake (here, TEE resides in the numerator). As illustrated in Fig. 1d, the diet effect now increases with the progressive threshold because individuals at the lower end of the TEE distribution are eliminated, leaving a residual cohort enriched for hyper-responders. However, these models, involving postrandomization variables inextricably linked to the outcome, violate a basic principle of statistical inference and should be discarded as fatally flawed.
Second, we divided the per protocol (weight stable) group into tertiles, based on the ratio of energy intake to TEE (Fig. 1e). In an unadjusted model, those in the lowest tertile (i.e., those eliminated in the analysis of Hall et al.) demonstrated a substantially larger diet effect. However, they were also more likely than those in the other two tertiles to have a baseline TEE above the median (OR 2.7 [95% CI 1.2–6.1], p = 0.02). With adjustment for baseline TEE and other relevant covariates, the differences between the tertiles for diet effect diminished.
Hall et al. modeled CO2 production (rCO2) in our cohort to circumvent the need for respiratory quotient (RQ), deviating from well-established DLW methodology and introducing severe bias against the low-carbohydrate diet. Because food quotient (FQ) equals RQ during weight (and body composition) stability, as applies to our per protocol group, a third approach is to conduct a sensitivity analysis examining how varying degrees of nonadherence would affect FQ and thereby TEE. As shown in Table 1, the low- versus high-carbohydrate diet comparison remained statistically significant through 50% nonadherence. Of particular interest, the diet effect relative to carbohydrate proportion remained remarkably stable throughout the range of assumed nonadherence, and consistently above the hypothesized 50 kcal/day for every 10% decrease in the proportion of energy as carbohydrate . Moreover, among participants in the lowest tertile of energy intake to TEE (for whom estimates of FQ may be least accurate), the unadjusted change in rCO2 was itself significantly greater on the low- versus high-carbohydrate diet (10.3 vs −47.0 L/day, p = 0.01). That is, the diet effect on TEE in this subgroup was so large as to require no assumptions about FQ, providing further evidence against nonadherence as an explanation for study findings.
As stated in the BMJ article, our preliminary estimates of energy intake, used by Hall et al., “would tend to selectively underestimate those with high energy expenditure” and were not intended to be definitive. With more precise and accurate data , we found that energy requirements for weight stability (i.e., by calorie titration) showed a similar magnitude of effect (≈200–300 kcal/day) and hierarchical order (low > moderate > high carbohydrate) among diets as TEE, as predicted by the Carbohydrate-Insulin Model. Due to imprecision involved in these (and all) methods for determining outpatient energy intake and expenditure, the magnitude of effect should be interpreted cautiously.
Hall et al. set a high bar for the Carbohydrate-Insulin Model by stating that “[p]roponents of low-carbohydrate diets have claimed that such diets result in a substantial increase in … [TEE] amounting to 400–600 kcal/day”. However, the original source for this assertion, Fein and Feinman , characterized this estimate as a “hypothesis that would need to be tested” based on extreme assumptions about gluconeogenesis, with the additional qualification that “we [do not] know the magnitude of the effect.” An estimate derived from experimental data—and one that would still hold major implications for obesity treatment if true—is in the range of 200 kcal/day . At the same time, they set a low bar for themselves, citing a 6-day trial  (confounded by transient adaptive responses to macronutrient change ) and a nonrandomized pilot study  (confounded by weight loss ) as a basis for questioning DLW methodology. Elsewhere, Hall interpreted these studies as sufficient to “falsify” the Carbohydrate-Insulin Model —but they do nothing of the kind. Indeed, a recent reanalysis of that pilot study suggests an effect similar to ours (≈250 kcal/day) .
Finally, we agree with Hall et al. that the component(s) of energy expenditure that might underlie our findings remain unknown. The diet effects on resting energy expenditure (REE) and moderate- to vigorous-intensity physical activity were of borderline significance in the hypothesized direction (p = 0.06–0.09) for pair-wise comparisons. Because individual components of energy expenditure might each contribute <100 kcal/day to the diet effect, our study lacked power to examine these secondary outcomes. In our crossover study , we found a significantly greater REE on a very-low-carbohydrate versus low-fat (high-carbohydrate) diet in the fasting state, when the thermic effect of food would have dissipated.
We aim to show that the latest series of criticisms by Hall et al., like those previously addressed, have little merit. Contrary to their claims, the data from our BMJ study, together with new data on energy intake, provide substantial support for the Carbohydrate-Insulin Model. Nevertheless, we recognize that these results require replication and that the relative advantages of dietary carbohydrate- versus fat-restriction on a population basis have not been established.
Of course, debate and criticism lie at the heart of science and should be encouraged, including with public posting of full databases. However, in the new era of open science, with widespread availability of raw data, the inevitable deficiencies in trials will be on full display. To minimize distraction and promote constructive discourse, it will be critical to distinguish inconsequential discrepancies and omissions from flaws that pose high risk of bias. Thus, we must all resist the admittedly natural tendency to hold a double standard for studies that support versus oppose our own views.
The protocol and data set for the original trial findings study are available at Open Science Framework (https://osf.io/rvbuy/). New primary data on energy intake and body composition will be posted upon publication of the related findings in a peer-review journal.
George L. Blackburn Symposium to feature weight loss studies. News Release, The Obesity Society; 14 November 2018. https://www.globenewswire.com/news-release/2018/11/14/1651612/0/en/George-L-Blackburn-Symposium-to-Feature-Weight-Loss-Studies.html. Accessed 9 Jul 2019.
Ebbeling CB, Feldman HA, Klein GL, Wong JMW, Bielak L, Steltz SK, et al. Effects of a low carbohydrate diet on energy expenditure during weight loss maintenance: randomized trial. BMJ. 2018;363:k4583. https://doi.org/10.1136/bmj.k4583.
Ludwig DS, Ebbeling CB. The carbohydrate-insulin model of obesity: beyond “calories in, calories out”. JAMA Int Med. 2018;178:1098–103.
Hall KD, Guo J. No significant effect of dietary carbohydrate versus fat on the reduction in total energy expenditure during maintenance of lost weight. BMJ Rapid Resp. 2018. https://www.bmj.com/content/363/bmj.k4583/rr-16.
Hall KD, Guo J, Chen KY, Leibel RL, Reitman ML, Rosenbaum M, et al. Methodologic considerations for measuring energy expenditure differences between diets varying in carbohydrate using the doubly labeled water method. Am J Clin Nutr. 2019;109:1328–34.
Ludwig DS, Ebbeling CB, Feldman HA. Choice of baseline for primary endpoint. BMJ Rapid Resp. 2018. https://www.bmj.com/content/363/bmj.k4583/rr-11.
Ludwig DS, Ebbeling CB. Author response to Hall and Guo regarding data reanalysis and other criticisms. BMJ Rapid Resp. 2018. https://www.bmj.com/content/363/bmj.k4583/rr-17.
Ludwig DS, Ebbeling CB, Wong JMW, Wolfe RR, Wong WW. Methodological error in measurement of energy expenditure by the doubly labeled water method: much ado about nothing? Am J Clin Nutr. 2019. (In press).
Hall KD, Guo J, Speakman JR. Do low-carbohydrate diets increase energy expenditure? Int J Obes. 2019. https://www.nature.com/articles/s41366-019-0456-3.
Ebbeling CV, Bielak L, Lakin PR, Klein GL, Wong JMW, Luoto PK, et al. Higher energy requirement during weight-loss maintenance on a low- versus high-carbohydrate diet: secondary analyses from a randomized controlled feeding study. medRxiv. 2019. https://doi.org/10.1101/19001248. [preprint].
Hall KD, Guo J. Carbs versus fat: does it really matter for maintaining lost weight? Version 5. bioRxiv. 2019. https://doi.org/10.1101/476655. [preprint].
Ebbeling CB, Swain JF, Feldman HA, Wong WW, Hachey DL, Garcia-Lago E, et al. Effects of dietary composition on energy expenditure during weight-loss maintenance. JAMA. 2012;307:2627–34.
Ravussin E, Redman LM, Rochon J, Das SK, Fontana L, Kraus WE, et al. A 2-year randomized controlled trial of human caloric restriction: feasibility and effects on predictors of health span and longevity. J Gerontol A Biol Sci Med Sci. 2015;70:1097–104.
Doubly-Labeled Water (DLW) Procedures for the CALERIE study. 1–13. https://calerie.duke.edu/sites/calerie.duke.edu/files/14.0_double-labeled_water_dlw_procedures.pdf. Accessed 9 Jul 2019.
Ludwig DS, Ebbeling CB, Heymsfield SB. Improving the quality of dietary research. JAMA. 2019. https://doi.org/10.1001/jama.2019.11169.
Hall KD, Bemis T, Brychta R, Chen KY, Courville A, Crayner EJ, et al. Calorie for calorie, dietary fat restriction results in more body fat loss than carbohydrate restriction in people with obesity. Cell Metab. 2015;22:427–36.
Hall KD, Ayuketah A, Brychta R, Cai H, Cassimatis T, Chen KY, et al. Ultra-processed diets cause excess calorie intake and weight gain: an inpatient randomized controlled trial of ad libitum food intake. Cell Metab. 2019;30:67–77.
Fine EJ, Feinman RD. Thermodynamics of weight loss diets. Nutr Metab. 2004;1:15.
Hall KD. A review of the carbohydrate-insulin model of obesity. Eur J Clin Nutr. 2017;71:323–6.
Friedman MI, Appel S. Energy expenditure and body composition changes after an isocaloric ketogenic diet in overweight and obese men: a secondary analysis of energy expenditure and physical activity. bioRxiv, Version 5. 2019. https://www.biorxiv.org/content/10.1101/383752v5.
We thank Arne Astrup, Henry Feldman, Steven Heymsfield, and Walter Willett for reviewing the manuscript.
The trial considered here was funded by Nutrition Science Initiative (made possible by gifts from the Laura and John Arnold Foundation and Robert Lloyd Corkin Charitable Foundation), New Balance Foundation, Many Voices Foundation, and Blue Cross Blue Shield. DSL was supported by a mid-career mentoring award from the National Institute of Diabetes and Digestive and Kidney Diseases (K24DK082730). The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of the study sponsors.
Conflict of interest
DSL and CBE have conducted research studies examining the Carbohydrate-Insulin Model of obesity funded by the National Institutes of Health and philanthropic organizations unaffiliated with the food industry; DSL received royalties for books on obesity and nutrition that recommend a low-glycemic load diet. The other authors declare that they have no conflict of interest.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Ludwig, D.S., Lakin, P.R., Wong, W.W. et al. Scientific discourse in the era of open science: a response to Hall et al. regarding the Carbohydrate-Insulin Model. Int J Obes 43, 2355–2360 (2019). https://doi.org/10.1038/s41366-019-0466-1
The Journal of Nutrition (2020)
The Journal of Nutrition (2020)
Effects of Dietary Carbohydrate Content on Circulating Metabolic Fuel Availability in the Postprandial State
Journal of the Endocrine Society (2020)
Testing the carbohydrate-insulin model of obesity in a 5-month feeding study: the perils of post-hoc participant exclusions
European Journal of Clinical Nutrition (2020)
Energy Requirement Is Higher During Weight-Loss Maintenance in Adults Consuming a Low- Compared with High-Carbohydrate Diet
The Journal of Nutrition (2020)