It s hard to figure out what to believe these days. As Drs. Henry Miller (a physician-scientist) and Stan Young (a statistician) point out in their piece on Forbes.com, one day coffee causes cancer and the next it prevents it. Or you ll read that cellphones cause brain tumors, or that everyone should be taking statins. The reason for this confusion, they say, may have a lot to do with current state of science. They go on, Scientists themselves are becoming increasingly concerned about the unreliability that is the lack of reproducibility of many experimental or observational results.
Researchers may be employing techniques in order to ensure they get the results they are looking for. For example, in an observational study, researchers ask subjects hundreds or thousands of questions, banking on the fact that statistical significance will occur five percent of the time. In experimental studies, researchers may tinker with their experimental design until they get the result they want and then rush to publish without replicating their own work, often choosing to drop specific animal subjects from their results. This is especially worrisome given that drug companies will often use results from these animal studies to begin to look into the effectiveness of drugs in humans. The failure to reproduce the studies creates huge roadblocks and Miller and Young point out that 80 to 90 percent of the claims coming from scientific studies are not reproducible.
According to an expose of these practices by Gina Kolata in the New York Times, the journals published by some of the worst offenders are nothing more than cash-generating machines that eagerly, uncritically accept essentially any submitted paper. And Miller and Young add, Given the incentives to publish research findings quickly whether they are reproducible or not and the ease of getting them into print, the disincentives to performing replication studies, and the general lack of interest in this subject from publishers and editors, it is difficult to be optimistic about reversing current trends and making science more reliable.
ACSH s Dr. Gilbert Ross had this comment: This analysis of the current, sad state of academic publication conforms pretty closely to what we ve been saying here at ACSH for quite a while: Whom can you trust? We have seen study after study obviously cobbled together to fit an agenda, yet being not only accepted and published as though by the Nobel Prize committee, but being trumpeted by activist/advocacy groups to prove their point (yes, I am thinking of that bizarre Sertalini paper on GMOs and rat cancer the fact that it was evicted by the journal seems to have had no effect on those publicizing it!). And there s no clear solution in sight. I guess it s like Democracy: the worst possible system except for all the others.
Sounds pretty grim. Read the full piece here.