A new study from JAMA shows that when initial findings about experimental drugs or devices sound too good to be true, they probably are. Stanford University researchers statistically analyzed nearly 230,000 trials compiled from a variety of disciplines. Unfortunately, researchers found that study results claiming a "very large effect" rarely held up when other research teams tried to replicate them.
Although researchers found that initial study results are rarely disproven, they often are shown to be less significant than first thought. In follow-up studies, 90 percent of the first trials that showed a large benefit failed to show such a significant effect. Furthermore, 98 percent of subsequent studies that had shown a large benefit failed to replicate that response. Authors of the study attribute small study size as a contributing factor to the initially inflated benefits.
"Beware of small studies claiming extraordinary benefits or extraordinary harms of medical interventions; the truth about these may be more modest," said Dr. John Ioannidis, a professor of medicine, health research and policy and statistics at Stanford's Prevention Research Center in California.
In addition to small sample size, Dr. Ioannidis and colleagues also found that the studies that claimed a very large effect tended to measure intermediate effects rather than incidence of disease or death itself, outcomes that are more meaningful in assessing medical treatments.
Unfortunately, adds ACSH s Dr. Josh Bloom, You can t have it both ways. People want newer, safer drugs, and they want them now. But traditional clinical studies with more definitive outcomes take many years. It will be quite tricky to balance these two competing needs. For example, the Prescription Drug User Fee Act, enacted this summer, includes designed to allow patients access to cancer drugs at a much earlier time than before. While this is good news for many cancer patients, it cannot help but add to the uncertainty of clinical studies.