Suppose a study reports that an intervention produced a “statistically significant improvement.” It is natural to hear that as “the improvement was substantial.” That is not what statistical significance means.
What a p-value is trying to tell you
In a common hypothesis-testing setup, researchers specify a null model, often representing no difference or no association. The p-value describes how incompatible the observed data, or more extreme data, would be with that model under the assumptions of the analysis.
The American Statistical Association’s statement on p-values was published partly because the metric is so often misunderstood. The ASA emphasized that scientific conclusions should not be based only on whether a p-value crosses a particular threshold.
What statistical significance does not tell you
- It does not tell you the probability that the hypothesis is true.
- It does not tell you how large the effect is.
- It does not tell you whether the effect matters in practice.
- It does not guarantee that the study was well designed.
- It does not guarantee replication.
Large samples can make tiny effects statistically detectable
With enough observations, a very small difference can produce a low p-value. That may be scientifically interesting, but the practical importance depends on the magnitude and context.
For example, a program could produce a reliably measured change that is too small to matter to most users. Calling it “significant” without giving the size can create the wrong impression.
Small samples can hide meaningful uncertainty
The reverse also happens. A study can fail to reach a conventional significance threshold because the sample is small and the estimate is imprecise. The data may still be compatible with meaningful benefit, little effect, or even harm.
That is why “not statistically significant” should not automatically be translated to “no effect.”
Confidence intervals show the range of estimates reasonably compatible with the data
A confidence interval is not a perfect summary of uncertainty, but it provides far more information than a binary label. A narrow interval around a small effect tells a different story from a wide interval spanning several practically different possibilities.
| Result | What to ask next |
|---|---|
| p < .05 | How large is the effect? How precise? Is the design credible? |
| p > .05 | Is the interval narrow enough to rule out effects that would matter? |
| Very large sample | Could a trivial effect become statistically detectable? |
| Very small sample | Is the result simply too imprecise to distinguish important possibilities? |
Multiple testing can create false positives
If researchers test many outcomes, subgroups, time points, or analytic variations, some may cross a significance threshold by chance. Good research plans address this through prespecification, corrections, replication, or transparent reporting.
As a reader, be especially cautious when the headline focuses on one subgroup or secondary outcome that was not the study’s main question.
Statistical significance sits inside a bigger reasoning process
A result should be interpreted alongside study design, measurement quality, prior evidence, plausibility, bias, and practical importance. NIH’s guidance for communicating research specifically warns against overstating statistical significance and recommends explaining context and limitations. That communication principle reflects a deeper scientific one.
A better sentence than “it was significant”
Whenever possible, describe:
- the estimated effect;
- the uncertainty around it;
- the population and outcome;
- the study design;
- and whether the magnitude is practically meaningful.
That gives the reader something they can reason with.
Confidence intervals add information a threshold cannot
A confidence interval can show the range of effect sizes that are reasonably compatible with the data under the model being used. That makes it easier to see whether an estimate is precise, whether very small or very large effects remain plausible, and whether the practical conclusion is actually clear.
Two studies can have similar p-values but very different confidence intervals and therefore different levels of uncertainty. Looking at the estimate and its interval usually tells you more than sorting results into “significant” and “not significant.”
For a broader reading workflow, see how to read a health study. To understand how results combine across papers, see systematic reviews and meta-analyses.
