One model, two fictional setups
In one setup, a model answers from the conversation alone. In another, it searches a document collection before answering. The second system may have access to newer or more specific information, but it also depends on retrieval finding the right documents.
If retrieval misses a relevant page, the model may never see it. If a tool returns an incorrect value, the final answer may inherit the error. These failures cannot all be described as properties of the model's training alone.
Trace the path to the answer
Input passes through product choices before and after model generation. A file may be converted to text, shortened, split into passages, or filtered. An output may be checked, formatted, or used to call another tool.
The original model-card framework encourages reporting intended uses and evaluation conditions. NIST's generative-AI profile also considers risks in deployed systems. Together they support asking what was actually tested, rather than relying on a model name as a complete specification.
Questions a comparison should answer
- Did both systems receive the same information?
- Could one search or calculate while the other could not?
- Were different instructions or output limits used?
- Was success measured on the final task or only the generated text?
These details can explain a performance difference without assuming the comparison was unfair. Different products may intentionally serve different needs.
Evaluate the task people will perform
A document assistant should be tested on the actual document workflow, including file handling and source links. A scheduling assistant needs evaluation of permissions and completed actions. A strong language benchmark can be relevant background, but the complete product is what a person ultimately uses. Its surrounding components belong in the assessment.
