First, a statistics lesson from Hell. Even when a statistical analysis is done correctly, a result can appear statistically significant just by chance. Using the conventional 5% cutoff, this will happen about once every 20 times when there is really no effect.
Here is a summary of a typical food frequency questionnaire study. Have a large number of people answer a survey about what they eat. A food frequency questionnaire might ask about 61 foods, or, more recently, as many as 200. Wait some time. Record the health conditions that develop. There could be 20 or more conditions. Count. We now have 20 × 61 = 1,220 questions at issue. But we could also look at male/female, young/old, etc. Each pair increases the potential analysis search space by a factor of 2. Suppose there are five such analytical choices. We now have 61 × 20 × 2⁵ = 39,040 possible analyses.
There is more. The level of food eaten could be categorized as low, medium, or high. The researcher can adjust the number of groups and the cut points used to separate them.
You get the picture. The food frequency questionnaire researcher has many options for their analysis. They can write a paper about the result that is “statistically significant.” With 39,040 possible analyses, even exploring a fraction of them creates plenty of opportunities to stumble across a statistically significant result by sheer chance.
Do they make stories? Well, yes. The Nurses’ Health Study is an ongoing Harvard study that began in 1976 with
121,700 female nurses. A Google Scholar search for “Nurses’ Health Study” returned about 4.35 million results.
[1] Another 1,100 papers have “Nurses’ Health Study” in the title. Yes, it is crazy. A publication machine. Your money turned into junk science.
In our hypothetical example, if all 39,040 possible analyses were performed and there were really no associations, as many as 1,952 could cross the 5% significance threshold by chance alone. Add the publish-or-perish motivation to come up with a plausible story.
You are now free. Always ask: How many analyses could have been computed? Even simpler, when a food frequency questionnaire study reports yet another statistically significant association between something you eat and some disease, be skeptical.
Very skeptical.
Note:
[1] The 4.35 million figure is Google Scholar’s approximate number of search results for the exact phrase “Nurses’ Health Study”; it is not the number of papers produced by the Nurses’ Health Study.