It Can Be Difficult to Know What to Believe, Even in Medical Articles

Reading the medical literature – that is, articles reporting studies of various aspects of health and disease – can be enlightening.  It’s how we docs keep up with new knowledge and progress, after all, but it can also be frustrating – for example, when studies are poorly designed or performed, the statistical analysis is flawed, or for reasons that might be unclear, the conclusions are simply implausible. When I was in graduate school, during the Q&A after a seminar, an eminent professor was asked why a competing research lab kept getting different results from his group. His terse response: “There are a thousand ways to do an experiment wrong.”

In addition, there are some studies whose results are so intuitively obvious that they need not be done at all. Another problem is journalists’ reporting on trials that misrepresents the results.

Here are some examples.

Long COVID Symptoms

“Long COVID,” or what doctors refer to as “post-acute sequelae of COVID-19” (PASC), is marked by the persistence of COVID signs and symptoms, or the emergence of new ones, for at least four weeks after recovery from the acute infection.

The CDC cites 19 possible long COVID symptoms, and warns that even that is “not a comprehensive list,” and there might be more. Yale Medicine lists 22 symptoms, while the Mayo Clinic counts 10. And the U.K.’s National Health Service (NHS) describes 16.

In order to better define what constitutes the long COVID syndrome, investigators at the University of Missouri analyzed a large patient cohort to determine health effects of COVID-19 infection. Their database contained profiles of 52,461 COVID-19 patients from 122 health care facilities across the U,.S. and they compiled the 47 most frequently reported health long COVID symptoms.

The investigators compared these 47 long COVID symptoms to symptoms shared by other viral respiratory diseases in three subgroups:

• Patients diagnosed with COVID-19 who did not have any common viral respiratory infections (e.g., influenza)
• Patients with common viral respiratory infections but without COVID-19
• Individuals not diagnosed with COVID-19 or any other common viral respiratory infection

The investigators found only seven symptoms specific to COVID-19 infection: rapid heartbeat, hair loss, fatigue, chest pain, shortness of breath, joint pain, and obesity.

The trouble is, nobody I’ve spoken to who actually treats long COVID patients believes these findings. The most common omissions mentioned are dysgeusia and anosmia, the loss of sense of taste and smell, respectively.  Although they can occur rarely following other respiratory infections, the practitioners say they’re seeing them far more frequently post-COVID.  Another frequently mentioned symptom missing from the University of Missouri list is “brain fog,” which is cognitive distortion, confusion, and/or difficulty concentrating. That certainly can occur rarely with flu or even a bad cold, but it would be highly unusual for it to last for weeks, months or even years. 

Why the discrepancy between what was found and what appears true from clinical practice? These are likely issues of study design.  There was no standard method of determining who had (and more important) did not have COVID, and there was no standardization of how symptomatology was defined.  Rather, pieces of data were pulled from an electronic medical record.  This likely means that there are a lot of mistakes in how diseases and symptoms are defined, leading to erroneous conclusions.

Tylenol vs Morphine for Pain

This was a randomized, 54-patient clinical trial of pain control following rib fractures, which are notoriously painful. Conducted in an emergency department in Iran (strike one against it), it compared the ability of intravenous Tylenol (1000 mg) and morphine (0.1 mg per kilo of body weight) to control pain.  

In the article describing the trial, which was published in the journal Emergency (Tehran) (strike two), the investigators concluded that Tylenol worked as well as IV morphine for relief of pain from rib fractures.  Except that the data didn’t actually show that. The difference between the two groups was not statistically significant and, therefore, could have been due to chance. 

What differentiated this one from thousands of other poor studies was that Dr. Jerome Adams, the U.S. Surgeon General at the time, failed to appreciate those shortcomings and publicly endorsed the authors’ conclusion – one that any third-year medical student who had completed his rotation on General Surgery would know was absurd.

After receiving a firestorm of criticism, especially from my colleague, the astute and acerbic Josh Bloom, Adams backpedaled furiously. I just hope no patients were provided inadequate analgesia based on the Adams-endorsed Iranian article.

Moral: Totally implausible claims should be treated with skepticism.

Do parachutes prevent injuries in people who jump out of airplanes?

And then there are questions related to public health or medicine that don’t require randomized clinical trials. My favorite article illustrating this is a tongue-in-cheek one that purported to “to determine whether parachutes are effective in preventing major trauma related to gravitational challenge,” by performing a “systematic review of randomized controlled trials.”

The results: “We were unable to identify any randomized controlled trials of parachute intervention.”

The conclusions: “As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomized controlled trials. Advocates of evidence-based medicine have criticized the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence-based medicine organized and participated in a double blind, randomized, placebo controlled, crossover trial of the parachute.”

I couldn’t have said it better. Making it a crossover trial is a nice touch: That means that eventually everyone enrolled in the trial will get to be in the “no parachute” control group.

Do masks prevent respiratory diseases like COVID?

Aside from the wry humor in the parachute article, there is an important, often-ignored point in it that can be summed up thusly, “The absence of evidence is not evidence of absence.” In other words, the fact that there is no testing data on the effectiveness of parachutes for people jumping out of planes is not evidence that they don’t work.

And yet, it’s remarkable how many people fall into that trap. A recent example is an awful, angry article about a meta-analysis performed by the Cochrane Library that, according to one journalist, showed that wearing masks doesn’t prevent respiratory diseases. The author, John Tierney, concludes:

This verdict ought to be the death knell for mask mandates, but that would require the Centers for Disease Control (CDC) and the rest of the public-health establishment to forsake “the science” -- and unfortunately, these leaders and their acolytes in the media seem as determined as ever to ignore actual science.

First of all, on its face, I would place such a conclusion in the same category – highly implausible -- as the Tylenol vs morphine clinical trial, above.

Second, let's talk about who's ignoring the science. Beginning with the headline and sub-hed and extending to the quote above from his article, Tierney distorts the findings of the Cochrane analysis.

Here are Tierney’s headline and sub-hed:

                                                      Approximately Zero

Masks make no difference in reducing the spread of Covid, according to an extensive new review by Cochrane — the gold standard for evaluating health interventions.

Not so fast. The Cochrane authors identify these as the “key messages” of their analysis: “We are uncertain whether wearing masks or N95/P2 respirators helps to slow the spread of respiratory viruses based on the studies we assessed,” and “[h]and hygiene programs may help to slow the spread of respiratory viruses.” Period.

And expanding on those key messages, here are the first four sentences of the Cochrane authors’ conclusions:

The high risk of bias in the trials, variation in outcome measurement, and relatively low adherence with the interventions during the studies hampers drawing firm conclusions. There were additional RCTs [randomized, controlled trials] during the pandemic related to physical interventions but a relative paucity given the importance of the question of masking and its relative effectiveness and the concomitant measures of mask adherence which would be highly relevant to the measurement of effectiveness, especially in the elderly and in young children.

There is uncertainty about the effects of face masks. The low to moderate certainty of evidence means our confidence in the effect estimate is limited, and that the true effect may be different from the observed estimate of the effect.

(Emphasis added.)

Moreover, in the authors' discussion of their findings concerning the effectiveness of the highest quality, most effective masks -- N95/P2 respirators -- there is no mention at all of COVID specifically:

N95/P2 respirators

Four studies were in healthcare workers, and one small study was in the community. Compared with wearing medical or surgical masks, wearing N95/P2 respirators probably makes little to no difference in how many people have confirmed flu (5 studies; 8407 people); and may make little to no difference in how many people catch a flu-like illness (5 studies; 8407 people), or respiratory illness (3 studies; 7799 people). 

Michael Osterholm, the director of the Center for Infectious Disease Research and Policy (CIDRAP) and Regents Professor at the University of Minnesota, and his colleagues provide a scholarly, two-part discussion of the nuances of masks to prevent respiratory infections (here and here). 

Perhaps the most persuasive evidence is found in the table below that illustrates the degree of protection against an aerosol afforded by different types of masks when worn by both source and receiver.  It demonstrates "that an N95 FFR [filtering faceplate respirator, such as an N95], even if not fit-tested, offers more benefits as both source control and personal protection than any mask," as measured by time before an infectious dose is received:

But there's more.

As Drs. Katelyn Jetelina and Kristen Panthagani point out in an excellent article on the ability of masks to prevent the transmission of viral respiratory diseases:

Viral transmission in a population is exponential. Even if masks only reduce the risk of transmission for each individual by a small fraction, when a community masks, those small effects compound exponentially across a population, making a big dent in cases. Just like compounding interest—a small change in the percentage makes a big difference down the road.

Those observations provide what might be called the theoretical basis for predicting the effectiveness of masks, but the authors go on to provide actual real-world evidence of protection against COVID:

• "Mask wearing corresponded to a 19% decrease in the R(0) in one study. In other words, masks helped reduce transmission.

• "In Bangladesh villages were randomized to be provided free masks. Villages that got the intervention had more than double the mask usage than villages that didn’t (13% vs. 42%). This resulted in a 9% reduction in cases in the mask-wearing villages. 

• "In the U.S., a 10% increase in mask wearing was associated with greater control of transmission.

• "In Germany, mask mandates reduced spread by 45%.

Finally, they discuss the palpable weaknesses of the Cochrane meta-analysis so beloved by Tierney, including:

• the trials in the Cochrane analysis “had a number of problems and, given the limited number of RCTs [randomized controlled trials] on COVID-19, do not represent the totality of evidence.”
• the Cochrane analysis encompassed trials with different viruses: “When a virus is less contagious, an effect is harder to detect. Many of the RCTs evaluated influenza, which is far less contagious than COVID-19. This means that if we combine them, the impact of masks may be underestimated. (Another scientist, separate from this review, removed the flu studies and reran the meta-analysis. He found masks protected against SARS-CoV-2.)”
• The Cochrane analysis combined studies performed in widely disparate settings, making interpretation difficult. “Studies ranged from suburban schools to hospital wards in high‐income countries, crowded inner city settings in low‐income countries, and an immigrant neighborhood in a high‐income country.”

I will leave it to readers to judge (1) the efficacy of masks, and (2) Mr. Tierney’s veracity and integrity.

The bottom line

Don’t take everything you read – especially journalists’ accounts of published medical articles – at face value. Be circumspect about which interpretations you accept, whose advice you take, and whether there's a consensus among experts. Life can be complicated.