Better accuracy can come from a smaller task
If a model answers only the easiest cases, its accuracy among answered cases may improve. That does not necessarily mean the full service improved. The remaining cases still need assessment.
Report coverage alongside performance.
A hypothetical comparison
System A answers 95% of eligible cases. System B answers 60% with higher accuracy on that subset. Choosing between them requires knowing the consequences of errors, the nature of the unassessed cases, and the capacity of the fallback service.
The higher percentage alone cannot decide.
The fallback belongs in the design
- Who receives an abstained case?
- Is the handoff timely and visible?
- Are certain groups disproportionately sent to the fallback?
- Can the user distinguish abstention from a negative result?
An uncertainty score used to trigger abstention also needs evaluation. A confident model can still be wrong, especially on unfamiliar inputs.
Avoid rewarding hidden exclusions
Predefine the abstention policy and include all eligible cases in reporting. Otherwise, removing difficult examples can make a product appear stronger without explaining the reduced scope. A transparent “unable to assess” can be valuable when it leads to an appropriate next step rather than leaving the person unserved.
