Ordering diet soda significantly reduces the number of calories customers eat when they visit fast food restaurants, according to a recent study. The paper is an example of scientists reaching the right conclusion for the wrong reasons.
We've been told for many years that consuming diet soda encourages overeating. Reasonably designed clinical trials, in contrast, have shown that people actually tend to reduce their calorie intake when they switch to sugar-free drinks. As a result, I'm inclined to dismiss counter-intuitive research and the sensational headlines—“Diet soda may prompt food cravings”—it generates. But it's also possible for scientists to reach the correct conclusion for the wrong reasons. This recent study published in the Journal of Public Policy & Marketing offers a textbook example.
Based on the self-reported fast food ordering habits of a few thousand people, the researchers concluded that ordering diet soda reduced the total number of calories customers ate when they visited McDonald's. According to the university press release that accompanied the study,
"… [T]he researchers found no evidence that consumers use diet drinks to justify other indulgences. In fact, after controlling for drink size and consumer demographics, they found that people who bought diet drinks had significantly lower overall calorie counts: 298 fewer calories for those who ordered a large diet drink … and 156 fewer calories for those ordering a medium … What’s more, the total food calories they ordered did not go up when they ordered a diet drink."
Something approximating this result is probably accurate. For reasons we'll explore, however, I don't think this study was able to pin down the calorie reductions with such specificity. This is a good opportunity to remind ourselves to approach new research with skepticism, especially if its results bolster a conclusion we already accept.
Garbage in, garbage out
I apologize to the researchers for such a harsh sub-headline, but it's justified given a number of assumptions they made about the data they utilized. Here's the first problem:
“We use data from the NPD Group’s Consumer Reports on Eating Share Trends (CREST) panel for the years 2000–2007 from Canada to estimate our model. NPD collects the data using diary-based surveys from rolling panels of representative samples of consumers.”
As we discussed recently, participants in epidemiological studies do not accurately report their exposures. Whether its food consumption, smoking habits, or even church attendance, the combination of faulty memory and dishonesty all but assures that self-reported data don't accurately reflect human behavior in the real world. But this just the beginning of the problem in this case:
“... [W]e assume that food ordered is closely related to food consumption. Similarly, in a study of orders and consumption at a university campus fast-food restaurant, Schwartz et al. (2012) found that there was very limited wastage in a meal, even when the amount of food in the order varied.
The difference is that Schwartz et al. was based on “three field experiments” in which customers were invited to downsize their meals and monitored to see how their behavior shifted following the decision to order fewer starchy side dishes. The present study design allowed nothing close to that kind of verification. Here's arguably the biggest gap in the data, though:
“The CREST database does not provide calorie or nutritional information for meal items ordered. Thus, to supplement this data set we merged it with data on calorie content from the McDonald’s Nutrition Facts website and U.S. Department of Agriculture databases.”
So, the researchers compared the self-reported menu items to the calorie content of each of those items. Do you see the blind spots here? They had to assume participants accurately reported their ordering habits, assume the orders reflected how much people ate, then estimate the calorie content of the orders. The authors themselves acknowledged the flaws in their design, though couched in the implication that they had no other option:
"We recognize that data based on actual observations of food and drink consumption rather than on self-report of an ordered meal would provide more accurate information, but such observational studies would be difficult in a natural field setting."
A reasonable model
Using these numbers, the researchers built a model to estimate how “the choice of caloric intake can be explained by the choice of drink type (i.e., diet and regular) and drink size (small, medium, and large) while controlling for mean-centered covariates.” Those covariates included gender, income, age, meal occasion (breakfast, lunch, or dinner) and the number of people in a party who visited the restaurant.
This seems like a reasonable approach because these variables do indeed influence eating habits. For instance, women tend to eat healthier diets than men; higher income and age also exert a moderating influence on the amount of food and drink an individual consumes. It therefore seems likely that people who order diet drinks probably do so with the intention of cutting their calorie consumption. The researchers have probably uncovered a real relationship, but I don't think they can reasonably quantify it given the data they had access to.
The bigger problem beyond any methodological limitation is that the authors (and public health experts more generally) are trying to interpret these “results causally and to formulate policy recommendations.” As long as the USDA subsidizes the production of dairy products, buys the surplus back from farmers, and helps fast food companies market seven-cheese pizza to the public, I have no faith in any scheme to “nudge” consumers toward healthier dietary choices. The same point could be made in relation to corn subsidies that artificially depress the price of sugary drinks.
If we want people to consume less fast food and soda, let's stop offering a financial incentive to buy those items.
