Change the model.Recheck the work.

Name the change before you measure it

“The new model seems better” is a useful observation, but a poor release criterion. Better at what: finding an approved procedure, extracting a reference, asking for missing information or preparing a draft that needs less correction? Choose the outcome before looking at the results.

Record the model and agent-runtime versions, instructions, tool definitions, retrieval configuration, memory policy and output schema. Include reasoning settings where the selected model exposes them. Keep the source collection and test environment identifiable too. If the prompt, index and model all move together, a different result will tell you very little about which change caused it.

Anthropic’s January 2026 evaluation guidance treats the agent as the model and its surrounding software working together. It distinguishes what an agent says from the final state it produces, and recommends repeated trials because outputs vary.¹ That is the right starting point for a release decision.

Make the test concrete

Consider an illustrative assistant that finds an internal procedure and prepares a work-request draft. Its test collection could contain ordinary requests, an obsolete procedure with a familiar title, two conflicting revisions, a missing record and a document the requester cannot access. These are proposed test cases, not reported client results.

Write the expected behaviour beside each case. For a missing reference, the useful outcome may be a precise follow-up question. For a conflict, it may be a pause with both sources shown. A fluent draft is a failure if it quietly chooses the wrong revision.

1. Freeze a comparison. Give the current and candidate versions the same permitted inputs and a clean starting state.
2. Run more than once. Record each attempt rather than keeping the most attractive answer.
3. Check the result. Inspect the draft fields, supporting passages and any resulting system state.
4. Read the disagreements. Investigate cases where the scores and a reviewer’s judgement diverge.

Separate quality from controls

Some checks have a definite answer: a required field is present, an identifier exists, an unauthorised operation is rejected. Others need judgement: whether a summary preserves a material exception or a clarification question is useful. Use explicit criteria and human review for those decisions; a second model can assist, but its score also needs checking.

OWASP’s 2026 misinformation guidance connects unsupported output to downstream decisions and calls for grounded claims, validated preconditions and checks before action.² Apply that distinction here. A model-quality score cannot establish that approval is enforced, and a passing permission check cannot establish that the draft is accurate.

Keep critical boundaries outside the average score. An improvement in twenty easy summaries must not outweigh an unauthorised write in one difficult case. Decide which failures block release, which require investigation and which are tolerable within the agreed use.

Read the operational cost

Measure end-to-end time, tool calls, retries and reviewer effort as well as answer quality. A shorter model response may still create extra work if people must repair the output. Compare the same completed task, and keep any cost estimate tied to the actual provider terms and usage measured at the time.

Retain a small set of cases outside routine prompt tuning. If every awkward example becomes a prompt instruction, the comparison can gradually reward familiarity with the test rather than handling a new request. Add fresh cases when the workflow or source collection changes.

Make a release decision someone can revisit

The decision record should name the change, the evidence reviewed, the remaining weaknesses, the person accepting them and the rollback trigger. Keep the previous configuration available where the provider supports it. If that version is being withdrawn, the fallback may instead be a reduced workflow or manual handling.

A bounded rollout can reveal conditions absent from the test set. Monitor corrections, failures and unexpected tool use, with someone responsible for acting on them. Reverting the model does not undo actions already committed; those need their own recovery plan.

Sources and further reading

1. Demystifying evals for AI agentsAnthropic↩
2. LLM07:2026 MisinformationOWASP GenAI Security Project↩