A better AI score does not prove the model improved
By Joe Zhou ยท
- AI Agents
- AI Governance
- AI Testing
- Model Evaluation
Your product team upgrades the model behind a customer-facing assistant. The new version scores higher, the demonstration looks stronger and the release appears ready to approve.
Then someone notices that the evaluator changed too.
The new judge uses a different rubric. Perhaps the parsing or score aggregation changed too. The result may be useful, but it no longer answers a question: did the product improve, or did the measurement change?
For an AI or SaaS team, the practical response is not to reject automated evaluation. It is to separate the system change from the measurement change. Run the current and candidate versions on comparable cases, score both under the same evaluation configuration, inspect important disagreements and retain enough information for another reviewer to understand the comparison.
If the evaluator needs fixing, fix it. Then re-score the baseline as well as the candidate.
Treat the evaluator as part of the experiment
An AI evaluation result depends on more than the model being tested. It can also depend on the test cases, expected outcomes, prompt, tools, environment, scoring rubric, judge model, parser and aggregation method.
AWS's 11 November engineering article on its Universal Agent Evaluation Framework makes the operational requirement clear: comparing changes to a model, prompt or tools requires consistent metrics and a shared record of previous results. It also separates quality into several dimensions rather than relying on one accuracy number.
This is not only a tooling concern. A July 2026 paper, When the Judge Changes, So Does the Measurement, held candidate responses fixed and found that replacing the language-model judge could change the measured result. Its experiments do not establish how a particular evaluator will behave in your product, but they demonstrate why a judge upgrade cannot be assumed to preserve the same measuring standard.
The same problem can arise without an LLM judge. Changing a pass threshold, expected answer, parser or weighting can alter the result even when the product output has not changed.
Make one comparison answer one question
Before running the evaluation, write down the release question.
For example:
Does the candidate model improve customer-reply quality without increasing unsupported commitments or incorrect account actions?
That question identifies more than a target average. It also names failures that should remain visible even if other results improve.
Then decide which components belong to the product and which belong to the measurement:
| Layer | Examples | What to do for the comparison |
|---|---|---|
| System under test | Model, prompt, retrieval, tools and product configuration | Record the baseline and candidate versions and the intended change |
| Evaluation cases | Inputs, conversation history, expected outcomes and expected tool calls | Use the same controlled set unless a documented scope change requires a new version |
| Evaluation method | Rubric, judge model, prompt, parser, weighting and threshold | Apply the same version to both output sets |
| Run conditions | Tool mocks, data snapshot, temperature, retries and relevant environment settings | Keep comparable and record meaningful differences |
| Review evidence | Per-case results, critical failures and human decisions | Retain the detail behind the summary, not only the average |
This makes the basis of the comparison visible without claiming to eliminate every source of variation.
A practical five-step comparison
1. Freeze the decision and critical failures
Define what the release is meant to improve and what must not regress. A customer-support assistant may need to give a complete answer while avoiding fabricated refunds, delivery promises or account changes.
Do this before seeing the candidate score. Otherwise, it is easy to choose the interpretation that makes the new version look successful.
2. Generate baseline and candidate outputs on common cases
Run the current and proposed versions against the same cases and expected outcomes under comparable conditions. Include ordinary work, difficult cases and examples of the failures that matter to customers.
Store the outputs separately from their scores. This allows you to re-evaluate the same evidence when the measurement method changes.
3. Score both sets with one evaluation configuration
Use the same rubric, judge version, judge prompt, parser, weighting and threshold for the comparison. If the evaluator is stochastic, repeat or sample sufficiently for the decision you are making and retain the run details.
NIST's January 2026 initial public draft on automated benchmark evaluations recommends versioning benchmarks and identifying breaking changes where results before and after the change are no longer properly comparable. It also recommends retaining full evaluation logs and exact system-version information alongside summary results.
That draft is not a law, certification requirement or final standard. Its versioning principle is still useful for a small product team: name the measurement method so you know which results can legitimately share a trend line.
4. Inspect paired failures, not just the average
Put each baseline output next to the corresponding candidate output. Review cases where the verdict changed, where the evaluator disagrees with a human-reviewed example, and where a critical failure appears.
Consider a hypothetical assistant that drafts customer emails. The candidate produces warmer, more complete replies. It also promises delivery by Friday when the source record contains no confirmed delivery date. If the new evaluator rewards helpful tone but no longer penalises unsupported commitments, the overall score can rise while a material customer risk becomes harder to see.
The release discussion should identify that regression explicitly. A higher average cannot make an unsupported commitment acceptable.
5. Record the decision and the comparison boundary
Keep a short release record containing:
- the product change and versions compared;
- the test-set version and expected outcomes;
- the evaluator, rubric, parser and threshold versions;
- results by important failure category, not only the overall score;
- human-reviewed disagreements and unresolved limitations; and
- the release, remediation, staged rollout or rollback decision and its owner.
This is evidence for a specific decision under stated conditions. It is not proof that the system will behave correctly in every production situation.
What to do when the evaluator is wrong
Consistency does not mean preserving a bad test forever.
If reviewers find that the evaluator rewards unsupported claims, misreads a valid answer or misses an important failure, update the evaluation method. Treat that as its own controlled change:
- document the defect and intended correction;
- assign a new evaluator or benchmark version;
- re-score both the stored baseline and candidate outputs;
- check the revised method against a small set of human-reviewed anchor cases; and
- mark earlier results as non-comparable where the change is material.
Do not silently splice the new scores onto the old trend. The revised evaluator may be better, but the historical measurement basis has changed.
The practical decision rule
Before approving an AI upgrade, ask:
Can we explain the candidate's improvement using the same cases, expected outcomes and evaluation configuration used for the baseline, and can we see whether any critical customer failure became worse?
If the answer is no because both the product and evaluator changed, re-establish the comparison. If the evaluator was flawed, correct it and re-score both versions. If an important regression remains, make that risk visible in the release decision rather than averaging it away.
This is a narrow piece of governance, but it is useful governance. It gives product, engineering and leadership a common record of what improved, what regressed and why the release was approved.
If your enterprise customers are asking how AI changes are tested and approved, Complyd's AI governance and Virtual Compliance Officer support can help establish the agreed review, evidence and ownership around your product programme. Book a Compliance Strategy Call to discuss the release requirement or recurring governance burden driving the work.