“Significant” has two meanings that easily get mixed up
In everyday language, significant often means important. In statistical testing, it usually means a result met a prespecified threshold, often based on a p-value. The American Statistical Association’s statement emphasizes that this label does not measure effect size or the importance of a result.
The difference becomes practical when a headline says a treatment produced a significant improvement. A reader still needs to know the outcome, how much it changed, how precisely that change was estimated, and what burdens accompanied it. The adjective alone cannot supply those details.
Two fictional results
Imagine a study measuring a symptom on a 100-point scale. It estimates an average improvement of 0.4 points, with a confidence interval from 0.2 to 0.6. The interval excludes zero, but the difference may be too small to be noticeable or worthwhile in the context. That judgment requires evidence about the scale and what changes matter to patients.
Now imagine a second study estimating a 7-point improvement, with an interval from a 2-point worsening to a 16-point improvement. That result is more uncertain. It includes no benefit and possible harm, but it also includes a potentially important benefit. Calling it simply “no effect” would discard much of what the interval shows.
These are invented examples, not clinical thresholds. Their purpose is to show why a small precise effect and a larger uncertain effect raise different questions. The first concerns practical importance; the second concerns unresolved uncertainty about both direction and size.
Where meaningful-difference thresholds come from
For some outcomes, researchers use a minimally important difference or related threshold. Such thresholds may be informed by patient judgments, clinical context, or statistical approaches. They are not universal constants that automatically transfer across populations, baseline severity, and purposes.
An average group difference also differs from an individual person’s change. A threshold designed to interpret within-person improvement should not be applied to between-group averages without checking whether that use is appropriate. People within each group may experience a wide range of outcomes even when the group average is stable.
The Cochrane Handbook discusses interpreting effects in context, including the size of benefits and harms and the certainty of evidence. That broader interpretation is necessary because one cutoff cannot capture every tradeoff relevant to a decision.
Absolute effects help connect the result to its scale
Suppose a fictional intervention reduces an event rate from 2% to 1%. That is a 50% relative reduction and a 1 percentage-point absolute reduction over the stated period. Describing only the relative change can make it difficult to understand how many events differ between groups.
Whether that reduction is important depends partly on the seriousness of the event, the intervention’s burdens, and the population’s starting risk. A small absolute change in a very serious outcome may matter greatly. A larger change in a minor outcome might be less compelling when the intervention is burdensome. The arithmetic informs the question without settling it for everyone.
Large samples do not make importance automatic
A larger sample can make a small difference easier to distinguish statistically from zero. That is useful when small differences matter, but it does not make all detectable differences consequential. Conversely, a small study may have an interval too wide to distinguish a meaningful effect from no effect.
Sample size also does not erase bias. A precisely estimated association can still reflect confounding or systematic measurement problems. Clinical interpretation depends on the study design as well as the numerical result.
Why a nonsignificant result is not an equivalence result
An ordinary superiority test asks whether the data provide evidence against a specified null hypothesis. Failure to reject that null does not establish that two options are sufficiently similar for a practical purpose. An equivalence study instead specifies a range of differences considered acceptably small and evaluates evidence against that framework.
For example, an interval spanning substantial benefit and substantial harm does not establish similarity merely because it crosses zero. An interval tightly concentrated around zero could be more informative about small differences, but the meaning still depends on the prespecified margin and design.
What a more useful result statement contains
A readable report can state the estimated difference, the uncertainty interval, the outcome’s scale, and the main limitations. It can then explain whether the estimate is large enough to matter under a stated interpretation, while acknowledging that people may weigh benefits and burdens differently.
This keeps the statistical result intact without letting a technical label stand in for the whole judgment. The question becomes how much changed, how certain that estimate is, and what the change could mean in the setting studied.
