A billing code and an adjudicated diagnosis differ
A billing code can be useful evidence, but it is produced for a particular administrative process. A specialist review, laboratory result, or follow-up outcome provides a different kind of reference. None should be selected merely because it is easiest to extract.
The reference should fit the intended clinical question.
Timing can create hidden leakage
If the label or an input includes information recorded after the intended prediction time, the evaluation may use knowledge that would not have been available in practice. A seemingly excellent model may partly be recognizing a later decision.
Questions for the label process
- Who or what assigned the label?
- What information was available?
- How were uncertain cases handled?
- Were reviewers blinded where appropriate?
- Did all participants receive comparable verification?
Disagreement can reveal an ambiguous task rather than simply poor annotators.
Report uncertainty honestly
If the model predicts agreement with a particular coding process, say that. Do not automatically turn it into a claim of disease detection. A clear reference definition lets readers assess whether the result addresses the clinical need and where label error might influence both training and evaluation.
