Calibration vs. Discrimination in Health AI: Two Different Kinds of Accuracy

A ranking can be right while the numbers are wrong

Consider a fictional model predicting a particular event within one year. It assigns 10% risk to a lower-risk group and 40% to a higher-risk group. In a large evaluation sample, the observed frequencies are 5% and 20%. The model correctly ranks the groups, but its probabilities are approximately twice the observed frequencies.

That is the central distinction. Discrimination concerns ordering or separation. Calibration concerns the meaning of the numerical probability. A model can be useful for one purpose and misleading for another if these are treated as the same property.

The methodological review by Van Calster and colleagues describes calibration as a frequently overlooked part of predictive model evaluation. Its relevance is particularly clear when a decision depends on crossing a probability threshold rather than simply being ranked above someone else.

What discrimination measures

For a binary outcome, a commonly reported discrimination measure is the area under the receiver operating characteristic curve, or AUROC. In a usual interpretation, it describes the probability that a randomly selected person with the outcome receives a higher score than a randomly selected person without it, with appropriate treatment of ties.

An AUROC does not say that the predicted probabilities are correct. Nor does it identify one operating threshold. A system can have the same ranking after every predicted probability is transformed in a way that preserves order. Its discrimination may remain unchanged while its probability estimates become much too large or too small.

This is why “90% accurate” is an incomplete description of a risk model. It may refer to classification accuracy at one threshold, a discrimination statistic, or something else. Each measure answers a different question, and none should be silently substituted for the others.

A calibration example with 1,000 people

Suppose 1,000 fictional people each receive a predicted 10% probability of an event. If about 100 experience it over the stated period, that group-level observation is consistent with calibration at that risk level. If 250 experience it, the model substantially underestimated frequency in that group.

This does not mean the model can identify which specific 100 people will experience the event. A probability is not a promise about an individual outcome. Someone assigned 10% risk can experience the event without disproving calibration, just as someone assigned 90% risk can avoid it.

One group-level average is also insufficient. A model could overestimate in a lower-risk group and underestimate in a higher-risk group while matching the overall event count. Calibration across the range of predictions, with uncertainty around the estimates, gives a fuller picture than one average comparison.

Population changes can alter calibration

A model developed in one setting may encounter a different baseline event frequency elsewhere. Measurement practices, referral patterns, treatment availability, and outcome definitions may differ too. The relation between predictors and outcomes can therefore change, even when the software itself has not.

Imagine a model trained among patients referred to a specialist clinic being applied to everyone visiting a community service. The populations were selected differently. Good results in the specialist setting would not establish reliable probabilities in the broader setting. External evaluation asks whether performance transfers rather than assuming it does.

Updating a model’s probability estimates may improve calibration in a new setting, but that is an empirical question. It does not automatically resolve missing predictors, poor measurement, or weak discrimination. A correction that works overall can also leave important subgroup differences.

Why the decision threshold changes the stakes

In a hypothetical workflow, a service offers an additional assessment above a predicted 15% risk. The earlier model assigning 40% where 20% is observed would still place that group above the threshold. But assigning 10% where 5% is observed could matter if the threshold were 7%. The practical consequence depends on both the size of the error and the action attached to it.

An additional assessment can have benefits, burdens, costs, and false alarms. Evaluating a model therefore extends beyond whether the probabilities look good on a graph. It includes whether using the output improves decisions compared with available alternatives in the intended workflow.

What a complete performance claim would describe

A useful model description names the outcome and prediction period, identifies the evaluation population, and distinguishes discrimination from calibration. It also reports uncertainty and makes clear whether the evaluation data were separate from development data.

For a reader, those details turn an abstract accuracy claim into something interpretable. The question becomes whether this model produces useful probabilities for this population and this decision. That is narrower than calling a system intelligent or accurate, but it is much closer to the evidence needed to understand what it can actually do.