Start with the intended decision
“The model works” is too vague. A more useful question is whether a particular system can support a defined task for a specified population within an acceptable workflow.
The criteria should reflect that task's consequences, not a generic accuracy target copied from another product.
Include more than a model score
| Area | Example criterion type |
|---|---|
| Performance | Appropriate metric with uncertainty and comparator |
| Coverage | Eligible cases receiving a usable result |
| Timing | Output available before the relevant decision |
| Workload | Review burden the service can actually support |
| Failure handling | Tested behavior for missing or stale inputs |
The exact thresholds require clinical and operational judgment. This table is a planning structure, not a universal pass standard.
Decide how inconclusive evidence is handled
A small pilot may show feasibility without estimating rare harms or subgroup performance precisely. “No observed harm” is not equivalent to demonstrating that harm is absent.
Specify what additional evidence would be needed and whether use should remain limited while it is gathered. A pilot is more informative when it can produce an honest “not yet enough information,” rather than being designed so that any completed demonstration becomes approval for broad deployment.
