A stable average can hide a changed process
A hospital might switch a laboratory method while the average result remains similar. A model may receive fewer measurements for one group, or a clinical practice change may alter the relationship between an input and the outcome.
Not every important change appears as a dramatic shift in the overall mean.
Watch several layers
| Layer | Example question |
|---|---|
| Data feed | Are expected fields arriving on time? |
| Population | Has the eligible case mix changed? |
| Output | Has the distribution of predictions changed? |
| Performance | Do predictions still align with later outcomes? |
| Workflow | Are users responding as intended? |
Outcome labels may arrive slowly, so some monitoring signals are early warnings rather than definitive performance measures.
Assign a response, not just a chart
A monitoring plan needs an owner, review frequency, investigation process, and conditions for limiting or suspending use. A dashboard without a response pathway can document a problem while allowing it to continue.
Retraining is not the automatic answer to every change. A broken feed, altered definition, or routing problem may need a different fix. First identify what changed and whether the original clinical purpose still fits the current setting.
