Research Literacy
How to read a meta-analysis
"A meta-analysis found..." gets treated as the end of an argument. Whether it should be depends entirely on what went into it.
What a meta-analysis actually is
A meta-analysis statistically combines results from multiple separate studies into one pooled estimate, usually as part of a systematic review that first searches for and screens all the relevant studies on a question. The appeal is straightforward: a single small study might be noise, but if fifteen studies point the same direction, that's a much stronger signal — in principle.
In practice, a meta-analysis is only as good as the studies that go into it. Pooling fifteen weak, biased, or poorly designed studies doesn't produce one strong conclusion — it produces a more precisely wrong number.
Heterogeneity: were the studies even measuring the same thing?
If the included studies used different doses, different populations, different follow-up lengths, or different outcome definitions, averaging their results together can obscure more than it reveals. Meta-analyses report a heterogeneity statistic for exactly this reason — high heterogeneity is a signal to read the individual studies rather than trust the pooled number at face value.[1]
Publication bias distorts meta-analyses too
If studies with null or unflattering results are less likely to get published — a well-documented pattern — then a meta-analysis built only from published studies will systematically overstate the effect, no matter how careful the statistical pooling is. Researchers check for this with a funnel plot, which should look roughly symmetric if publication bias isn't distorting the picture; a lopsided funnel plot is a red flag.[1]
Questions worth asking before trusting the headline number
- How many studies, and how large? — a meta-analysis of five small trials is much weaker than one of thirty large ones
- Were the included studies mostly observational or mostly randomized? — pooling observational studies can pool their shared confounders too, not cancel them out
- Was heterogeneity high? — if so, the single pooled number may be hiding real disagreement between studies
- Who funded the included studies, and who funded the meta-analysis? — the same funding-bias questions from industry-funded research apply here, potentially multiplied across every study included
None of this means meta-analyses are untrustworthy — a well-conducted one is still generally stronger evidence than any single study. It means the phrase "a meta-analysis found" is a starting point for evaluation, not a substitute for it.
Frequently asked questions
Is a meta-analysis always stronger evidence than a single study?
Generally yes, but only if the underlying studies are sound. Pooling many weak or biased studies produces a more precise-looking but still biased result, so the quality of the included studies matters more than the number of them.
What is heterogeneity in a meta-analysis?
Heterogeneity measures how much the results of the included studies vary beyond what would be expected by chance. High heterogeneity suggests the studies may not be measuring comparable things closely enough to be meaningfully averaged together.
How do researchers check for publication bias in a meta-analysis?
One common method is a funnel plot, which plots study effect size against study size. A roughly symmetric funnel suggests publication bias is unlikely; a lopsided one suggests smaller unfavorable studies may be missing from the published record.