The people who stay may already differ
In a fictional app study, frequent users improve more than infrequent users. It is tempting to conclude that extra use produced the difference. But frequent users might have more available time, fewer access barriers, or early improvement that encouraged them to return.
The direction could also run the other way: people with more persistent difficulties may use an app more often. A simple correlation between use and outcomes does not settle these possibilities.
Two study questions
“Do people offered the app do better than a suitable comparison group?” evaluates the offer under the study's conditions. “Among app users, who spends more time in it?” describes engagement patterns. Both can be worthwhile, but they do not estimate the same effect.
CDC's evaluation framework distinguishes questions about implementation from questions about outcomes. For an app, successful onboarding is an implementation result. An improvement in function measured later is a different result, requiring its own evidence.
When less use might be good
Suppose a tool helps someone prepare a question list before a medical appointment. Finishing in five minutes could be better than needing twenty. A dashboard that treats every extra minute as success would reward friction as well as usefulness.
A practical evaluation can record: - Whether the person completed the intended task. - Whether the task was useful and accessible. - Whether a relevant outcome changed. - How people who stopped using the app were included in the analysis.
Streaks and session counts can help describe the experience. They become misleading when substituted for the result the app promises. The best interpretation starts with what the tool is supposed to help someone do, then asks whether that happened.
