A split can look more independent than it is
Suppose a fictional dataset contains several visits from each patient. Randomly dividing visits can put one person's records in both training and testing. The test then asks partly how well the model handles people it has already encountered.
A patient-level split asks a stronger question about new patients. A later-time split asks about a later period. A different-hospital evaluation asks about a different setting. None is a universal substitute for the others.
The scikit-learn documentation explains data leakage and why information from evaluation data must not influence model fitting. That principle includes preprocessing and feature selection, not only the final model.
The hospital can be part of the signal
Imagine a test is ordered frequently at Hospital A but rarely at Hospital B. A model that uses the presence of that test may be learning part of Hospital A's workflow. Applying it elsewhere changes the meaning of an input, even if the laboratory units match.
Likewise, a model evaluated on carefully completed records may encounter incomplete records in routine use. “New data” is not enough description to assess this difference.
What a useful validation report tells you
- Which people, sites, and dates were excluded from development.
- Whether the model and threshold were fixed before evaluation.
- How outcomes and missing values were defined.
- Performance for relevant groups, with uncertainty.
An external evaluation that finds lower performance is useful evidence. It identifies a boundary that a development score could not show. If the model is then adjusted using that external dataset, the adjusted version needs another independent evaluation before the same dataset can be called an untouched test.
