A convincing result can still leave a practical question
Suppose a fictional trial finds that appointment reminders improve attendance at a clinic. The trial is carefully randomized, attendance records are complete, and the comparison supports an effect in that setting. A different clinic wants to know whether the same reminders would help its patients.
That second question is not an accusation that the first study was flawed. The new clinic may have different appointment availability, language needs, transport options, or existing reminder practices. Even a trustworthy result needs interpretation before it becomes a prediction about another setting. Internal and external validity separate those two tasks.
Internal validity concerns the inference being made
For a causal question, internal validity is threatened when the observed difference could be explained by systematic differences other than the intervention being studied. Confounding, problems with allocation, biased outcome measurement, and certain patterns of missing data can interfere with the comparison.
The appropriate safeguards depend on the design. Randomization helps create comparable groups at assignment, but it does not guarantee perfect balance in every finite sample or prevent problems afterward. An observational study requires its own account of why the measured comparison can support the intended conclusion.
Precision is a related but different issue. A small study can produce an imprecise estimate even when its procedures avoid important sources of bias. A much larger study can estimate a biased quantity very precisely. More observations do not automatically repair a mismatch between the study design and the causal claim.
Random assignment is not random sampling
Random assignment allocates enrolled participants to study groups. Random sampling concerns how people are selected from a population. A trial can randomize treatment among volunteers without having recruited a representative sample of everyone who might eventually receive that treatment.
| Process | Main question it addresses |
|---|---|
| Selecting the study sample | Who enters the evidence base? |
| Assigning an intervention | How are study groups made comparable at the start? |
| Measuring and following participants | What information remains available for the comparison? |
| Applying the result elsewhere | Which differences between the study and target setting matter? |
These stages can work well or poorly in different combinations. A sample that resembles the target population does not by itself eliminate confounding. Conversely, a randomized comparison does not establish that the participants resemble every population of interest.
“Generalizable” needs a destination
A result may be relevant to one target population and uncertain for another. Asking whether a study is generalizable without naming that population leaves the claim unfinished. The intended outcome and intervention also matter: transferring an estimate about appointment attendance is not the same as transferring an estimate about long-term health.
Return to the reminder example. If the second clinic already sends effective reminders, adding another message may have a smaller effect than replacing no reminders at all. The comparator changed. If appointments are consistently unavailable, remembering to attend may not be the main limitation. The mechanism linking the intervention to the outcome changed.
Neither difference establishes that reminders cannot help. They identify reasons the original effect might change and what additional information would make the comparison more informative. This is more useful than treating all contextual differences as equally important or dismissing a study because its setting is not identical.
Population averages can change without a contradiction
Consider a hypothetical intervention with a larger effect among people facing one specific barrier. If that barrier is common in the study but uncommon in the target population, the overall average effect could differ even if effects within comparable groups were the same.
Researchers sometimes use weighting or standardization to address measured differences between study and target populations. These methods require relevant information and assumptions about which characteristics explain variation in effects. They cannot create evidence for a group entirely absent from the available data without additional assumptions.
This is one reason representation and measurement both matter. Merely knowing that two samples have different demographic distributions does not show exactly how an effect will change. The connection between those differences, the intervention, and the outcome needs a substantive argument.
What a careful conclusion sounds like
“The reminders improved attendance among participating patients in this clinic” is narrower than “reminders improve attendance everywhere.” It can still be a useful result. A follow-up comparison might examine whether the effect persists with different delivery methods, schedules, or populations.
Evidence becomes more informative as the boundaries of a claim become clearer. A study does not need to answer every question to contribute something valuable. The important distinction is between what the data support directly and what must be assumed to carry that result into another person’s circumstances.
