Agreement is not always success
If a clinician changes a correct initial judgment to match an incorrect model, agreement rises while decision quality falls. A deployment metric based only on acceptance can therefore reward the wrong behavior.
A confident explanation or polished interface can also make an output easier to accept without making it more accurate.
Test the difficult interaction
An evaluation should include cases where the system is wrong or incomplete, with appropriate ethical safeguards. Useful observations include whether users notice conflicting evidence, seek clarification, and retain the ability to act independently.
A randomized study of physicians exposed to erroneous model recommendations illustrates why training alone should not be assumed to eliminate overreliance. Its findings still need to be interpreted within its study setting and sample.
Design questions worth asking
- Can the user inspect the relevant original information?
- Are uncertainty and intended-use limits visible at the decision point?
- Is disagreement possible without an unreasonable burden?
- Are errors reviewed as system problems rather than automatically blamed on the user?
The goal is calibrated reliance: using helpful information while retaining appropriate judgment. Neither unquestioning acceptance nor automatic rejection is a meaningful definition of successful clinical AI use.
