By now, most of us are aware of the recent Annals of Internal Medicine’s publication of 4 interlocking meta-analyses of the effect of red and processed meat on our health. If nutritional guidelines are the Emperor, the new studies clearly indicate that the Emperor’s "clothing" of evidence is woefully lacking; and the guidelines authors, like the Emperor’s courtiers, are left sputtering in protest.
“This is sure to be controversial, but it is based on the most comprehensive review of the evidence to date. Because that review is inclusive, those who seek to dispute it will be hard pressed to find appropriate evidence with which to build an argument.” [1]
I think that this may be a game-changer for conducting nutritional research; it is worth a closer examination.
Each of the four meta-analyses looks at a comprehensive sampling of the accepted literature and taken as a whole form the basis for the “inflammatory” recommendations. The working group used standard techniques to survey the literature, and their reported findings are consistent with the evidence used in formulating our nutritional guidelines. The scientific facts are the same. They differ from the dietary authorities of eminence in reporting absolute differences in outcome and pairing those effects with a different, but widely accepted measure of the certainty of evidence. The change is not in the scientific dots, it is in how they are connected.
Absolute versus relative
Publishing relative versus absolute differences is a standard technique in the medical literature and represent two different ways of expressing a result. A quick example, if the baseline risk of an adverse outcome is 10%, and the treatment results in an improved, 8% outcome, you can report the absolute difference, 2%, or the relative difference 25% - which is a more powerful number? If you were writing a paper, which value would you be inclined to report? I have reformatted the study’s table of findings below.
The source of the table is the primary meta-analysis, Red and Processed Meat Consumption and Risk for All-Cause Mortality and Cardiometabolic Outcomes. The asterisk indicates that follow-up was median years.
In nearly every case, reduction in red and processed meat intake by 3-servings a week reduced those adverse health outcomes. The “new” studies found the same scientific dots as the old. But look at the difference in the absolute (yellow column) and relative (green column) effect of reducing our intake of red and processed meats. They both express the same finding; the current authors described the absolute response and called it a “small,” the guideline writers describe the relative response and call it “moderate.” For my money, the effect is small. You can reach your own decision
Certainty
How sure can we be that these outcomes, small or medium, are correct? Determining the certainty of the evidence is also problematic. Many studies rely on statistical inferences, especially confidence levels. For example, we might be confident that 95% of the time, our reported outcome of 0.9, really lies between 0.87 and 1.0. After several statistics classes, I understand the concept, but it remains challenging to communicate. The authors of the current work used recognized criteria, Gradings of Recommendations, Assessment, Development, and Evaluation or, GRADE. It is considered a best practice and is used globally. It is a subjective process that raises and lowers the certainty of the evidence, based on components of the study design, and expresses certainty in everyday language.
There are a number of reasons for lowering our “certainty” about scientific studies.
- Risk of bias – “inherent limitations in the design or conduct of a study.” This is nutritional research’s greatest weakness. The most frequent limitation is gathering information on a participant's dietary intake by self-reported surveys or only at the initiation of a 10 years study; after all, can you say with confidence that your diet hasn’t changed over that time?
- Imprecision – when within the statistical confidence levels, the outcome shifts from a benefit to no benefit.
- Inconsistency – when several studies used in the analysis show varying effects
- Indirectness – when the population studied is different than the people for whom recommendations are being made, as seen in genetic studies of Europeans, which have less bearing on the genetics of Asians. Or when surrogate end-points are used in place of specific outcomes, e.g., a lower LDL standing in for lower cardiovascular mortality
- Publication bias – when results influence the decision to publish, it is a frequent complaint concerning industry-funded research.
In nearly every instance, the articles reviewed were found to have low, “the true effect is probably markedly different from the estimated effect;” or very low, “the true effect might be markedly different from the estimated effect” certainty. [Italics added] Studies were most frequently downgraded when the dietary information was collected only initially during the study period. Other studies were downgraded because they were imprecise, with the outcome was both beneficial and not beneficial within reported confidence intervals. I think these are reasonable assessments. What do you think?
Using the GRADE system, there are far fewer reasons to upgrade the certainty of our belief in a study’s findings.; When there is a large effect that should make us feel that we are seeing signal, not noise, increasing our certainty; but none of the underlying studies showed large effects. Even the Emperor’s dithering advocates describe outcome effects as moderate.
You can increase your confidence when a study demonstrates a “dose-response gradient” – where one serving has a small effect, and several servings have a more substantial impact. Medications display dose-response gradients. But can be demonstrated. For example, if one serving has a small effect on the outcome, and several servings a much more significant impact. Medications display dose-response gradients. But as reported by the New York Times, Dr. Hu of Harvard states, “Nutrition studies, he added, should not be held to the same rigid standards as studies of experimental drugs.” But I would point out that nutritional studies are used to set national guidelines, regulations, taxations, and subsidies – I want a little more confidence in the basis of these decisions.
The next two studies, Reduction of Red and Processed Meat Intake and Cancer Mortality and Incidence: A Systematic Review and Meta-analysis of Cohort Studies and Patterns of Red and Processed Meat Consumption and Risk for Cardiometabolic and Cancer Outcomes: A Systematic Review and Meta-analysis of Cohort Studies demonstrate similar small adverse effects and low or very low certainty for the reducing our intake of red and processed meats for cancer and cardiovascular disease whether looked at in isolation or as a part of a dietary pattern.
The recommendations are directed to individuals, not policymakers
“The target audience for our guidance statement was individuals who consume unprocessed red meat or processed meat as part of their diet. The panel took the perspective of individual decision making rather than a public health perspective.”
Unlike other guidelines and dietary recommendations, the primary audience is not policymakers and other scientists, they are individuals, the patient sitting across my desk.
Dietary choices have a strong cultural component. Most guidelines ignore these preferences. Rather than paternalistically assuming that they know what is best for you because of their education and experience, the authors conducted a final meta-analysis, Health-Related Values and Preferences Regarding Meat Consumption: A Mixed-Methods Systematic Review to inform their opinion with our preferences. [2] What a practical idea, tailoring advice to individuals rather than populations.
Once again, they point out that the evidence gathered has a low certainty. But the omnivores amongst us,
“consider meat an essential component of a healthy diet, they enjoy eating meat, they feel that meat is a part of their traditions, and they believe they lack the knowledge and cooking skills to prepare an adequate meal without meat. Study participants’ willingness to change meat consumption in response to health concerns is generally low. … Our results highlight the inappropriateness of assuming that informed persons would choose to reduce meat consumption on the basis of small and distant health benefits, particularly if the benefits are uncertain.” [Emphasis added]
As I pointed out previously, 17 years of nutritional guidelines have barely altered American eating habits.
The Recommendation
“For adults 18 years of age or older, we suggest continuing current unprocessed red meat [and processed meat] consumption (weak recommendation, low-certainty evidence). Eleven of 14 panel members voted for a continuation of current processed meat consumption, and 3 voted for a weak recommendation to reduce processed [and red] meat consumption.”
Given the press coverage, is that what you expected to read? Pretty humble statement in my view, acknowledging up front that the recommendation is weak, the evidence uncertain, and there was disagreement, not consensus. And this leads us to another distinction from prior nutritional guidance, the real transparency of their findings. They evidenced that effort in two ways. First, the assembled panel had no acknowledged conflicts of financial or intellectual interest. That last one means flexibility of thought, no predefined agenda. Most guideline committees are made up of experts in the field, and when you dedicate your career to establishing one hypothesis, you are making a deep intellectual investment. Just consider how Ptolemy must have felt when Copernicus and that upstart Galileo challenged his body of work showing the Earth was the center of the solar system. The authors went so far as to report the dietary servings of red or processed meat by each panelist.
Second, and perhaps more importantly, they laid out in lay terms the rationale for their recommendation.
“Panelists considered 3 servings per week as a realistic reduction in meat consumption (for example, moving from 7 to 4 servings, or 4 to 1 serving) on the basis of the average intake of 2 to 4 servings per week in North America and Western Europe. …
The rationale for our recommendation to continue rather than reduce consumption of unprocessed red meat or processed meat is based on the following factors. First, the certainty of evidence for the potential adverse health outcomes associated with meat consumption was low to very low, … Second, there was a very small and often trivial absolute risk reduction based on a realistic decrease of 3 servings of red or processed meat per week. Third, if the very small exposure effect is true, given peoples’ attachment to their meat-based diet, the associated risk reduction is not likely to provide sufficient motivation to reduce consumption of red meat or processed meat in fully informed individuals, and the weak, rather than strong, recommendation is based on the large variability in peoples’ values and preferences related to meat.”
The Protest by the Emperor’s supporters
The research and development of nutritional guidelines remain an active area of study, providing $1.8 billion in NIH funding for 2018 (an amount comparable to NIH spending on cardiovascular disease), as well as establishing the career paths for many scientists. Gina Kolata, writing in the New York Times Science reported,
“The American Heart Association, the American Cancer Society, the Harvard T.H. Chan School of Public Health and other groups have savaged the findings and the journal that published them.
Some called for the journal’s editors to delay publication altogether. In a statement, scientists at Harvard warned that the conclusions’ harm the credibility of nutrition science and erode public trust in scientific research.’”
Delaying publication, harming credibility (of the protester’s work), eroding trust, irresponsible, unethical – all terms we can echoing from the past, when Galileo stood before Pope Urban VIII’s inquisitors, having to recant for his heretical stance about the Earth and Sun.
I concur with the editorial when they write,
“We have saturated the market with warnings about the dangers of red meat. It would be hard to find someone who doesn’t “know” that experts think we should all eat less. Continuing to broadcast that fact, with more and more shaky studies touting potential small relative risks, is not changing anyone’s mind.
The study provides an example of what we should be doing in communicating nutritional information. This study takes that decision back from the policymakers and puts it squarely in the hands of individuals. People choose to not eat meat for a variety of reasons, their health, ethical concerns over animal welfare or intelligence, and the environmental impact of industrial-scale farming. It is your call, but the health benefit appears to be overhyped.
[1] Editorial that accompanied the studies by Drs. Aaron Carroll and Tiffany Doherty
[2] Actually, the preferences describe those of meat-eating members of Western High-Income Countries
