Research basics
Not every study counts the same.
Every grade on this site rests on one idea: some kinds of study can show that something works, and most cannot. This page explains the difference in plain language. No medical background needed.
The one question
Did it cause the result, or just travel with it?
A supplement study is trying to answer one question: if people take this, does something good happen to them that would not have happened otherwise? The hard part is that last word, otherwise. People who take a supplement are rarely the same as people who do not.
An everyday example shows the trap. People who carry a lighter get lung cancer more often than people who do not. Lighters do not cause cancer. Smoking does, and smokers carry lighters. A study that simply compares lighter carriers with everyone else will find a real pattern and draw the wrong conclusion.
The same thing happens with vitamins. People who take them tend to exercise more, smoke less, eat better and see a doctor more often. If they live longer, the vitamin gets the credit for the whole lifestyle. Researchers call this the healthy user effect, and it is one of the most common reasons a supplement looks better in a headline than it does in a trial.
Each kind of study deals with this problem differently, and some do not deal with it at all. That is what the ladder below is about.
The ladder of evidence
Five kinds of study, from the bottom up.
Researchers rank study designs by how well each one separates cause from coincidence. EBL uses five tiers, the same five described on How grades work. Here is each one, what it can show, and where it falls short.
-
Tier 5
Lab and animal studies
What it is. Experiments on cells in a dish, or on animals, usually mice.
What it can show. Whether something has a biological effect worth testing, and how it might work. Nearly every idea in medicine starts here.
Where it falls short. A mouse is not a person, and a cell in a dish has no liver, no gut and no rest of the body around it. Many things that work in mice never work in people. A claim such as "it activates longevity genes" or "it raises NAD+ Nicotinamide adenine dinucleotide, a molecule cells use in energy metabolism. Its level falls with age in some tissues. in cells" usually comes from this tier.
On EBL: never enters a grade. A supplement whose only evidence is a lab or animal study is Grade D.
-
Tier 4
Case reports and expert opinion
What it is. A doctor writes up what happened to one patient, or to a handful of patients in a case series. Expert opinion is an authority saying what they believe from experience. Testimonials and before-and-after stories belong here too.
What it can show. That something unusual happened and is worth a closer look. Case reports are often the first sign of a rare side effect.
Where it falls short. There is no comparison group. The person might have improved anyway, with time, from some other change, or simply from expecting to improve. A story cannot tell those apart, however sincere it is.
On EBL: never enters a grade.
-
Tier 3
Observational studies
What it is. Researchers watch large groups of people over time without assigning anyone a treatment, then look for patterns. A cohort study An observational study that follows a group of people over time and compares what happens to them. follows people forward from a starting point. A case-control study starts with people who already have a disease and looks back at what they did differently.
What it can show. Associations across very large numbers of people over many years, which no trial could afford to follow. Observational Research that watches what people already do rather than assigning a treatment. It can show that two things travel together, not that one causes the other. research is good at raising questions and at spotting harms.
Where it falls short. The lighter problem. The people being compared chose their own habits, so something else they share may explain the difference. Statistics can adjust for what was measured, but not for what was not.
On EBL: published as context beside a letter, never the basis for one.
-
Tier 2
Randomized controlled trials
What it is. Volunteers are assigned by chance either to the treatment or to a comparison, usually a placebo An inactive substitute made to look like the real treatment, so participants cannot tell which one they received., and then followed to see what happens to each group.
What it can show. Cause and effect. Because chance decides who gets what, the smokers, the exercisers and everyone else end up spread evenly across both groups. The only systematic difference left between them is the treatment itself.
Where it falls short. A trial can be too small, too short, run only once, or measure a number in the blood rather than something people feel or live with. The next two sections cover how a trial works and what to read in one.
On EBL: the minimum for any grade above D.
-
Tier 1
Systematic reviews and meta-analyses
What it is. A systematic review A review that searches for every study on a question using a method set in advance, then assesses them together. gathers every trial on a question using search rules written in advance, so the authors cannot pick only the trials they like. A meta-analysis A study that pools the numbers from several trials into one combined estimate. then pools the results of those trials into one combined estimate.
What it can show. The overall picture across many trials, with more precision than any single trial can give.
Where it falls short. A review is only as good as the trials inside it. Pooling ten small, weak trials produces a precise-looking answer that still rests on small, weak trials. The top of the ladder is strong only when the rungs underneath it are.
On EBL: the strongest tier, when it pools randomized human trials.
Inside a randomized trial
Six parts, each closing a door.
Each part of a well-run trial exists to rule out one way of fooling ourselves.
- 01Recruit the right people
Volunteers who fit the question, for example adults over 60 with low muscle strength. A result applies most clearly to people like the ones who were studied.
- 02Randomize
A computer, in effect a coin toss, decides who gets the treatment and who gets the comparison. Neither the researchers nor the volunteers choose, so neither can stack the groups.
- 03Use a placebo
The comparison group gets something that looks and tastes the same and contains nothing active. Feeling better because you expect to is real, and a placebo makes sure both groups get that boost equally.
- 04Blind it
In a double-blind Neither the participants nor the people assessing them know who received the real treatment until the trial ends. trial neither the volunteers nor the people measuring the results know who got what, so hope and expectation cannot tilt the scoring.
- 05Name the outcome in advance
The trial states before it starts what it will measure, ideally in a public registry. Otherwise a team could measure twenty things and report the one that happened to look good.
- 06Compare
At the end the two groups are compared. If the treated group did meaningfully better, by more than chance would plausibly produce, the treatment is the likely cause.
What can still go wrong
Randomization is the strongest design there is, and it still does not make every trial strong. A trial of twenty people can produce a striking result by luck. A trial of eight weeks says little about eight years. A single positive trial can be a fluke, which is why the same result from independent research teams matters so much. Funding from the company selling the product does not make a result wrong, but it makes independent repetition matter more. And a trial of one form of an ingredient, at one dose, says nothing direct about a different form or dose on a store shelf.
Why the standard exists
Two times the pattern pointed the wrong way.
Beta-carotene and lung cancer
Observational research found that people who ate more beta-carotene, the orange pigment in carrots, had less lung cancer, and a 1981 review in Nature asked whether it could materially reduce human cancer rates (PMID PubMed ID: the permanent number the US National Library of Medicine gives a published paper. Each one links to that paper's record on PubMed. 7010181).
Randomized trials followed. In the ATBC trial, 29,133 male smokers were assigned by chance to beta-carotene or placebo, and the beta-carotene group had 18 percent more lung cancer, not less (PMID 8127329). A second trial, CARET, was stopped early when lung cancer and deaths rose in the treated group (PMID 8602180). The pattern had been real. The explanation was not: people who eat a lot of vegetables differ in many ways, and a pill of one pigment did not reproduce what the diet appeared to do. EBL grades supplemental beta-carotene D, tested and harm found, in its record.
Hormone therapy and the heart
The Nurses' Health Study followed 48,470 postmenopausal women for up to ten years. Women taking estrogen had about 44 percent less major heart disease than women who were not (PMID 1870648). For years that pattern was read as a sign that hormones might protect the heart.
The Women's Health Initiative then assigned 16,608 healthy postmenopausal women by chance to estrogen plus progestin or placebo. The heart protection did not appear, and the trial was stopped early because the overall risks, including breast cancer, stroke and blood clots in the lungs, exceeded the benefits (PMID 12117397). One leading explanation is the healthy user effect again: the women who chose hormone therapy were, on average, healthier and better cared for to begin with. The trial removed that difference. The observational study could not. The EBL record for combined hormone therapy carries the full grade and the later analyses.
Neither story means observational research is useless. It found the questions. It took randomized trials to answer them.
What a trial measures
A better number is not always a better life.
Trials differ in what they measure. Some measure what people care about directly: living longer, heart attacks, strokes, broken bones, or a function recorded by an instrument, such as grip strength on a dynamometer. Others measure a stand-in, a number that tends to travel with the outcome, such as cholesterol, blood pressure or blood sugar. Researchers call these surrogate markers A lab value or measurement used in place of the outcome that matters, such as LDL cholesterol standing in for heart attacks. Moving a surrogate does not always move the outcome..
Some stand-ins are well established and genuinely useful. Still, a treatment can improve the number without improving the outcome, which is why a trial on a stand-in counts for less than a trial on the outcome itself.
One measurement deserves its own warning. A study showing that a supplement raises its own level in the blood shows that the supplement was absorbed. It does not show that it did anything.
How people say they feel matters, and trials record it. It is also the measure most easily moved by expectation, which is why blinding matters most when the result comes from a questionnaire.
From the ladder to the letter
Why a supplement with dozens of studies can still be a C.
EBL applies this ladder strictly. Lab and animal studies never enter a grade. Case reports and expert opinion never enter a grade. Observational studies can appear beside a grade as context and never set one. Only randomized trials in humans can lift a record above D, and how far depends on how many there are, whether independent teams found the same thing, and what the trials measured.
So a long list of studies is not the same as strong evidence. If most of those studies are in mice, or small, or never repeated, the letter stays low however long the list. A letter describes the human evidence, not the ingredient's worth. The exact rules for each letter are on How grades work.
Words you will see on this site
A short glossary.
- Randomized
- Chance, not choice, decided who got the treatment.
- Placebo
- A look-alike with nothing active in it, given to the comparison group.
- Double-blind
- Neither the volunteers nor the people measuring results knew who got what.
- Endpoint The outcome a trial measures to decide whether the treatment worked, such as heart attacks, blood pressure or a symptom score.
- The outcome a trial set out to measure, such as heart attacks, grip strength or blood pressure.
- Surrogate marker
- A measurable stand-in for an outcome, such as cholesterol standing in for heart disease.
- Replication The same finding reproduced by a separate research group. A result seen once can be chance or bias; a replicated one is harder to explain away.
- A separate research team running a similar trial and getting the same result.
- Meta-analysis
- A study that pools the results of several trials into one combined estimate.
- PMID
- The number PubMed, the U.S. National Library of Medicine's index of medical research, assigns to each published paper. The PMIDs on this site link to those listings, so a claim can be checked against its source.
Go further
From the basics to the letters.
A mouse study is a reason to run a human trial. It is not a result.
Grades are set from the published evidence. Educational use only, not medical advice.