“It is better to know nothing,” wrote French physiologist Claude Bernard in An Introduction to the Study of Experimental Medicine (1865), “than to keep in mind fixed ideas based on theories whose confirmation we constantly seek.”

Embracing a fixed idea is one of the main dangers in the evolution of any scientific discipline. Ideally, errors will be uncovered in the trial-by-fire of rigorous testing and the science will right itself. In rare cases, however, an entire discipline can be based on a fundamental flaw.

As a science journalist turned science historian, I have written at length about how and why this may have happened in obesity research. I have suggested that the discipline may be a house of cards — as, by extension, may much research into the chronic diseases associated with obesity, such as diabetes.

Before the Second World War, European investigators believed that obesity was a hormonal or regulatory disorder. Gustav von Bergmann, a German authority on internal medicine, proposed this hypothesis in the early 1900s.

The theory evaporated with the war. After the lingua franca of science switched from German to English, the German-language literature on obesity was rarely cited. (Imagine the world today if physicists had chosen to ignore the thinking that emerged from Germany and Austria before the war.)

Instead, physicians embraced the ideas of the University of Michigan physician Louis Newburgh, who argued that obese individuals had a “perverted appetite” that failed to match the calories that they consumed with their bodies' metabolic needs. “All obese persons are alike in one fundamental respect,” Newburgh insisted, “they literally overeat.” This paradigm of energy balance/overeating/gluttony/sloth became the conventional, unquestioned explanation for why we get fat. It is, as Bernard would say, the fixed idea.

This history would be no more than an interesting footnote in obesity science if there were not compelling reason to believe that the overeating hypothesis has failed. In the United States, and elsewhere, obesity and diabetes rates have climbed to crisis levels in the time that Newburgh's energy-balance idea has held sway, despite the ubiquity of the advice based on it: if we want to lose fat, we have to eat less and/or move more. Yet rather than blame the advice, we have taken to blaming individuals for not following it 'properly'.

The alternative hypothesis — that obesity is a hormonal, regulatory defect — leads to a different prescription. In this paradigm, it is not excess calories that cause obesity, but the quantity and quality of carbohydrates consumed. The carbohydrate content of the diet must be rectified to restore health.

There is compelling reason to believe that the overeating hypothesis has failed.

This conclusion is based on endocrinology that has been understood for 50 years: insulin regulates fat accumulation, and blood levels of insulin are effectively determined by carbohydrate intake. The more easily digestible are the carbohydrates we eat (the higher their glycaemic index) and the sweeter they are (the higher their fructose content) the higher are our blood insulin levels, and the more fat accumulates.

If this is true, it suggests that the obesity epidemic was caused at least in part by the research community's failure to understand the nature of the disease, and by the food industry's exploitation of that failure.

But is it true? Or is it the case, as conventional wisdom has it, that these competing hypotheses of obesity have been rigorously tested, and the energy-balance hypothesis has simply won out?

Over the past year, with physician Peter Attia and support from the Laura and John Arnold Foundation of Houston, Texas, I have co-founded the non-profit Nutrition Science Initiative (NuSI) in San Diego, California, with the goal of resolving this controversy.

Among our first tasks was to comb the medical literature back to the 1930s, identifying all studies relevant to the question of whether carbohydrates or excess calories cause obesity. We found much ambiguous smoke, but none of the fire of rigorous experimental evidence necessary to establish definitively the truth or falsehood of either hypothesis. This is unacceptable, considering the critical public-health problem presented by obesity and diabetes. (The studies and our conclusions are available at http://nusi.org.)

The trials share many shortcomings. One common flaw is true of all free-living diet trials: the investigators simply fail to control what the participants actually eat. The evidence suggests that few participants comply with the dietary advice, yet the researchers interpret the results as somehow speaking to the fundamental cause of obesity. It is as if we drew conclusions about whether smoking causes lung cancer on the basis of trials (poorly controlled ones, at that) of the efficacy of different methods of smoking cessation — nicotine patches, say, versus nicotine gum. This problem must be solved to establish reliable knowledge.

NuSI aims to fund and facilitate the trials necessary to rigorously test the competing hypotheses, beginning with inpatient feeding studies that will rigidly control dietary interventions for participants so that we know unambiguously the effects of macronutrients — protein, fat and carbohydrates — on weight and body fat. These studies will be done by independent, sceptical researchers. This may be an idealistic dream, but we have committed ourselves to the effort.