Sample Size, Statistical Power, and Precision: How They Differ

“Large study” is not a complete description

A study with 10,000 records sounds substantial. Whether it answers a question well depends on what those records represent, how outcomes vary, which comparison is being made, and how the data were collected. Ten thousand independent participants are different from ten thousand repeated measurements from ten participants.

Sample size is therefore a count with a definition. It is not itself a measure of validity, and it does not establish that an effect is important. The count becomes useful when connected to the design and the uncertainty of the intended estimate.

Power describes a test under an assumption

A power calculation asks how often a specified statistical test would reject its null hypothesis if a particular alternative were true under the assumed design. For example, a study could be planned for 80% power to detect a specified difference at a chosen significance level.

That does not mean the completed study has an 80% chance of being correct. It also does not mean there is an 80% probability that the assumed effect exists. Power is conditional on the effect size, variability, sample structure, test, and other assumptions used in the calculation.

If the true effect is smaller than the planning value, power will generally be lower under otherwise comparable conditions. If variability is greater or more data are missing than expected, the plan may also overstate the information ultimately obtained.

Precision describes the estimate

Precision concerns how much uncertainty surrounds an estimate. A narrow confidence interval can indicate greater statistical precision under the model. It does not automatically indicate less bias or more practical importance.

NIST’s confidence-interval formulas show the familiar square-root relationship for simple independent samples. Under comparable conditions, increasing the sample fourfold approximately halves the standard error. Doubling the sample does not usually halve it. The exact calculation depends on the design and estimator.

This diminishing relationship helps explain why modest gains in precision can require substantial additional data. It also explains why the size of a dataset should be judged against the question rather than against an arbitrary threshold for “big.”

An invented comparison

Suppose two studies estimate the same five-unit difference. One reports an interval from minus eight to eighteen; the other reports an interval from three to seven. The point estimates match, but the second is much more precise. The first remains compatible with a wider range of effects, including opposite directions.

Now suppose a third study estimates a difference of 0.2 units with a very narrow interval excluding zero. Its statistical evidence may be precise while its practical importance remains uncertain. The outcome’s scale and context determine whether 0.2 units matters.

These examples separate three questions: what was estimated, how uncertain it is, and whether the magnitude is meaningful. A significance label cannot answer all three.

Clusters change the amount of information

People within the same school, clinic, household, or other group can have correlated outcomes. Repeated observations from the same person can also be correlated. Treating all observations as independent can make uncertainty appear smaller than warranted.

A fictional study with 1,000 measurements from a small number of clinics may have less information for comparing clinic-level interventions than the raw number suggests. The number of clusters, their sizes, and the correlation structure all matter. Appropriate design and analysis account for that structure.

The same issue explains why collecting more measurements from existing units may not substitute perfectly for observing more independent units. They contribute different kinds of information.

More data do not repair the wrong measurement

If a device systematically overestimates a quantity, collecting more readings can estimate that biased average very precisely. If a survey excludes an important part of the target population, a large respondent count does not automatically restore representation.

This is the distinction between random uncertainty and systematic problems. Increasing sample size can reduce some random uncertainty under suitable conditions. Measurement validity, selection, confounding, and missingness require separate attention.

What a useful sample-size explanation contains

A clear report identifies the target effect or precision, the assumptions used, and the actual information collected. It explains losses, clustering, and any departure from the plan. The final estimate and uncertainty then show what the study supports.

Power is useful for planning; precision is useful for interpreting the estimate; sample size contributes to both. Keeping the three separate makes it easier to understand why one study is informative and another remains uncertain, without treating a large number as a substitute for good design.