Does a Low Survey Response Rate Mean the Results Are Biased?

The missing responses do not all have the same meaning

A fictional community survey invites 1,000 eligible people to report their usual travel time to a local service. Two hundred respond. The simple response fraction is 20%. That number describes participation, but it does not reveal whether the respondents live closer to the service, farther away, or at roughly the same distances as everyone else.

If people facing very long journeys are particularly motivated to respond, the estimated average could be too high. If they have less time or access to the survey, it could be too low. If response is unrelated to travel time, the missing responses might have less effect on that estimate. The participation percentage alone cannot distinguish these possibilities.

A numerical example of what remains unknown

Suppose the 200 respondents report an average journey of 20 minutes. If the other 800 eligible people also average 20 minutes, the full population average is 20. If they average 40 minutes, the full average is 36 minutes: 200 times 20 plus 800 times 40, divided by 1,000.

Both scenarios have the same response rate and the same observed mean. The difference lies in information the survey did not collect. These invented numbers illustrate the missing-data problem; they are not a method for guessing the nonrespondents’ actual journeys.

The example also explains why the direction of bias is not a property of “low response” alone. A survey might overestimate one outcome and underestimate another. People who respond can resemble nonrespondents on some characteristics and differ on others relevant to the question being asked.

What counts as a response rate?

Real survey calculations can be more complicated than completed questionnaires divided by invitations. Some sampled people may be ineligible, some may only partially complete the survey, and eligibility may remain unknown for others. AAPOR provides standardized definitions that account for different case outcomes and assumptions.

A completion rate among people who opened a questionnaire is not necessarily the response rate among everyone selected. Nor does a high participation rate in an opt-in online poll show that the poll represents the broader public. The denominator and recruitment method are essential parts of the claim.

For example, “90% completed the survey” could describe 90 of 100 people who began, even if thousands saw an invitation and most never started. That figure may be useful for evaluating questionnaire completion, but it answers a narrower question than how well the final sample represents the population of interest.

Weighting can help when relevant information exists

Survey weights can account for unequal selection probabilities and adjust for some differences between respondents and the target population. CDC’s NHANES weighting guidance describes adjustments for its complex sampling design, nonresponse, and population totals. Those procedures are tied to that survey’s design rather than being a universal repair formula.

Imagine that response differs by neighborhood and that population counts for each neighborhood are known. Weighting can restore the neighborhoods’ intended representation. But if long-distance travelers respond differently even within each neighborhood, and the adjustment does not capture that difference, bias in travel time may remain.

Weights can also increase the variability of estimates when a small number of respondents represent many people. The relevant uncertainty calculation needs to account for the design. A large raw count of responses does not alone describe the effective information in a weighted estimate.

A high response rate does not fix the question

Even if nearly everyone replies, ambiguous wording can produce answers to different interpretations of the same question. “How long does it take to get there?” might refer to a typical trip, the most recent trip, driving time, or the entire journey including waiting. High participation cannot settle that ambiguity.

Coverage is another issue. If the invitation list excludes people without a particular account or address, responding individuals may accurately represent the list while the list misses part of the intended population. Response rate applies within a recruitment process; it does not certify that process as complete.

Evidence beyond the participation percentage

Comparisons with known population characteristics, information about nonrespondents, follow-up studies, and sensitivity analyses can help assess the possible consequences of missing participation. None is a universal guarantee. Agreement on a few demographic totals can coexist with differences in an outcome that was never measured among nonrespondents.

The most informative report makes its target population, recruitment, response definition, weighting, and limitations visible together. That allows a reader to distinguish a well-described uncertain estimate from an unsupported claim. A response rate is useful evidence about the survey, but it is the beginning of that interpretation rather than the final verdict.