Two versions of a fictional survey
A survey about a community service is sent only through its app. People unable to use the app are less likely to appear in the data. If the aim is to describe everyone's access experience, the recruitment method may omit an important group.
Now suppose the survey reaches the intended population, but asks, “How much did our excellent service help you?” The question's wording may influence responses. That is a measurement problem even if recruitment was appropriate.
The first example changes whose experience is observed. The second changes how an observed experience is recorded.
The proposed fix should match the problem
Recruiting more people through the same app might increase precision while leaving the access-related selection problem. Rewording the question would not bring missing app nonusers into the sample.
CDC's Field Epidemiology Manual distinguishes selection and information bias when interpreting associations. Naming the mechanism matters because it helps identify what design or analysis changes could address it.
A large sample can still be biased
Increasing sample size can reduce some random uncertainty. It does not automatically correct systematic differences in who is included or how a variable is measured. A very narrow interval around a biased estimate can therefore be confidently misleading.
What to look for in a paper
For selection, follow the flow from eligible people to those analyzed. For information, examine the measurement method, timing, definitions, and whether procedures differed between groups.
Identifying a possible bias is not enough to declare the result invalid. The next questions are whether the mechanism is plausible, how large its effect could be, and whether the study investigated it. A named concern should lead to a reasoned assessment, not a reflexive dismissal.
