How to Spot Pseudoscience (Part 2): The Truth About Study Design and Data Bias
In Part 1 of ‘How to Spot Pseudoscience’, we looked at the journals and the system that decides what gets published. But even in a reliable journal, some studies are not reliable or telling the whole story. If you are like me and want to look at the research yourself, you have to know how to evaluate it. So in Part 2, we are going to discuss what questions to ask and what to look for in individual studies.
The Hierarchy of Evidence
When it comes to the health and wellness sphere, many people use “studies” to convince you of something or sell you something. They will often say, “A study says…. blah, blah, blah.” The very next thing you should ask is, “What kind of study was it?” This is because different studies carry different weight in the strength of their conclusions. This is referred to as the hierarchy of evidence.
Case Studies: Anecdotes and Storytelling
These are the science equivalent of “I know a guy who tried this and feels better,” but presented in a more structured scientific way. They generate hypotheses and test nothing. They are the equivalent of anecdotes and are often the type of research cited in many health and wellness spaces.
In Part 1, I shared a story about a patch someone was trying to sell me. Her evidence was user testimonials, which some people treat as case study level research. Whether it is a testimonial or case study, it is not evidence or proof of anything in scientific research.
Observational/Retrospective Studies: Causation vs. Correlation
The main purpose of these studies is to look for correlation. Could two different things be related in some way? They do nothing to establish causation. The example I always use is the fact that ice cream sales and drowning rates increase during the summer. Obviously, eating more ice cream does not cause drownings. There is no established causation, but they correlate because of temperature. When temperatures rise, people buy more ice cream and venture into the water more. Observational and retrospective studies are great for identifying possible patterns that can be tested more rigorously.
Randomized-Controlled Trials (RCTs): The Gold Standard for Cause and Effect
This is where we are starting to get into more reliable evidence. RCTs randomly assign participants into an intervention group and a control group. This is the best method we have for determining cause and effect. Ideally, they are double-blinded and placebo-controlled, meaning neither the researchers nor the participants know if they are receiving the intervention or a placebo. But this is not always possible.
They are powerful studies but not perfect. If a study has a small number of participants, a short follow-up period, or selective reporting, the results may be distorted. The other disadvantage of these studies is they are very expensive. The more participants and controls you have, the more researchers and money you need.
Systematic Review & Meta-Analyses (SRMAs): The Study of Studies
A well-done SRMA is the gold standard in scientific evidence. Researchers conduct a detailed literature search to find high-quality papers on a specific topic and then pool the statistics. As we discussed, an RCT’s conclusions can be questionable when the number of participants is small. An SRMA can pool RCTs done on the same topic together to strengthen the statistical analysis. When done well, SRMAs reduce the noise of individual studies and provide a more stable estimate of the truth.
The one caveat is that SRMAs are only as good as the studies they include. If you combine data from a bunch of poorly done, garbage studies, the output will be well-polished garbage.
Below is a figure that represents the hierarchy of scientific evidence.
Statistical Smoke and Mirrors
Relative vs. Absolute Risk
Most news headlines and big claims hinge on how statistics are reported. “Something reduces the risk of something terrible by 50%!” What is being reported is often the relative risk, which is the percentage of change between groups. The absolute risk tells the actual difference in outcomes.
For example, if the intervention group demonstrates a risk reduction from 2 in 1,000 to 1 in 1,000, that is a 50% relative reduction. It’s only a 0.1% absolute reduction. It’s the same data with very different interpretations.
Relative risk makes the effect look larger, but absolute risk tells you whether it’s actually meaningful. Marketers love using relative risk.
P-Hacking
In science, a “p-value” is an established statistical value that determines if something happened by chance versus as a result of the intervention being tested. Sometimes, a study doesn’t meet that threshold and cannot demonstrate statistical significance. Researchers may then start comparing a bunch of other variables to find a relationship that demonstrates statistical significance. This is called p-hacking and is a huge red flag.
Every well done study will provide their primary outcome (the main reason why they are doing the study) along with secondary and tertiary outcomes (the “oh-by-the-way” goals). If the primary outcome showed no statistical significance and the study’s conclusion makes a very specific claim that was not a part of the original primary goal, be skeptical.
This is often a sign of p-hacking. If you read the methodology and the results sections more closely, you will find language of additional analyses done to find some kind of significant finding to report. A well-done study will acknowledge it found no difference, explain why, and suggest how future studies could be designed.
Conflicts of Interest
Legitimate journals require every researcher to report their conflicts of interest and funding sources. This language is very important. First of all, if you don’t find it reported anywhere, it is a red flag. When you do find it, what it says matters as well. This gives you insight into whether there is a special interest or bias behind the published results. For example, if one of the authors has a vested interest in a certain industry, or a personal brand built on the idea that “all of Western medicine is garbage,” it is extremely unlikely they will publish a paper whose conclusion does not support their industry or brand.
The source of funding is also important. If a certain industry funded the research, it is unlikely they will support publication of a paper that conflicts with their financial priorities. This is often where people comment on “Big Pharma.” I hope by now I’ve established that I am not a Big Pharma favoring doctor. But I do need to point out something about Big Pharma-funded papers. These studies are extremely expensive to conduct. They are typically large, double-blinded, randomized controlled trials with a large number of participants. Outside of Big Pharma, no one else can afford to fund them. So I invite skepticism when the results are in favor of pharma, but if the study was done well and the analysis is transparent, then it’s good science.
Scientific Study BS Detector
Whether it’s a news headline, social media influencer, or someone trying to sell you a sugar and salt patch at a wellness event, anytime someone cites studies and evidence, ask these questions:
What type of study is it?
Are the results reporting relative risk or absolute risk?
Does the conclusion address the primary outcome?
Is there a conflict of interest?
Then use the tools you learned here to determine if the evidence is reliable or garbage science.
Bottom Line
Science is important and imperfect. When you lack the skills to appraise evidence and rely on others to spoon-feed you results, you are vulnerable to being misinformed or lied to. The only way to make sure you truly understand the data is to learn how to appraise it yourself.
I have zero expectation that anyone will read this week’s and last week’s posts and suddenly be an expert in statistics and data analysis. But I do hope that with the knowledge you’ve gained over the last two weeks, you can protect yourself from being bamboozled by someone whose primary intent is to make money off you. When we feel terrible, it is easy to be tricked into buying the next best thing. When you have the knowledge to read and understand the science behind the next best thing, you can at least protect your wallet.
Science is a process, not a destination. It’s okay for “the truth” to change as we get better data. But by understanding the system and the study types, you can stop being a victim of marketing and start making decisions based on actual evidence.