LLM model migration: catch regressions before rollout

Before switching an AI model, compare the current and proposed configurations on the same, business-reviewed test cases. Look specifically for tasks that worked before the change and fail afterwards. These regressions can disappear inside an improved overall success rate.

A release decision should answer three questions: which cases improve, which deteriorate, and whether the new failures are acceptable within the intended scope.

Record exactly what changes

A model migration can require other changes: a different system prompt, revised tool definitions, new parameters or different handling of structured outputs. Record them. Otherwise, an improved or degraded result cannot be attributed to a specific cause.

Capture at least:

• provider and exact model identifier,
• prompt and workflow versions,
• available tools and their schemas,
• generation settings and output limits,
• data sources and retrieval configuration,
• permitted actions, roles and approvals.

Start with the changes required for migration. Assess additional optimisations separately. If a provider migration requires several changes together, document them as one new configuration.

A retirement deadline does not establish task quality

Providers publish retirement dates and replacement recommendations. OpenAI maintains a deprecation overview; Anthropic documents model status, retirement dates and recommended replacements. Sources checked on 22 October 2026.

A replacement recommendation addresses availability. Whether the replacement performs your workflow well enough needs to be tested against your tasks.

Allow time for comparison, adaptation and a controlled rollout before the deadline. If the old model becomes unavailable, it cannot remain your fallback.

Compare individual cases as well as aggregate scores

The following example is entirely fictional. It is neither a provider benchmark nor a customer case.

Two configurations process the same 100 tasks:

The current configuration passes 92 tasks; the new one passes 94. The new configuration still introduces four regressions. If they include wrong recipients, unauthorised writes or business-critical amounts, migration may be unsuitable despite the higher overall score.

Review the six improvements and four regressions individually. An average quality gain does not offset a critical boundary violation.

Define scoring rules before the comparison

Each test case needs an identifiable acceptance condition. Depending on the task, this may be a correct value, an evidence-backed answer, a necessary clarification or an action the system must explicitly avoid.

Assess three dimensions:

• Business correctness: the task was completed correctly.
• Process boundaries: permissions, approvals and permitted actions were respected.
• Operating limits: runtime, cost and stopping conditions stayed within agreed limits.

Scoring rules must fit the task. Exact matching is useful for a structured identifier, but usually too restrictive for an open-ended summary. Qualitative assessment needs a clear rubric and business-reviewed reference cases.

OpenAI's evaluation best practices recommend calibrating automated scoring against human judgements and including typical, edge and adversarial cases. This does not establish a universal pass rate for your system. Thresholds must reflect the intended use.

Keep the comparison baseline controlled

A useful test set covers supported tasks and known failure classes. Where relevant, include different input lengths, languages, data sources and permission roles.

Keep the test set and scoring rules constant during the comparison. Correct outdated or invalid references, then evaluate both configurations against the same corrected baseline.

For the final assessment, also use cases that were not used during adaptation. Repeat selected cases according to a predefined plan to expose noticeable variability. A few repetitions do not establish a rare-event failure rate.

For RAG applications, distinguish changes in retrieved evidence from changes in how the model uses that evidence. The RAG evaluation guide explains this separation.

For agents, inspect the actions taken

A correct final message does not show whether an agent used permitted tools with correct parameters.

Also evaluate:

• the selected action and its parameters,
• required clarifications and approvals,
• retries and stopping conditions,
• the actual state reached in the target system.

Run comparisons in a controlled environment. Block writes or direct them to suitable test systems. An additional comparison run must not create a second production booking or message.

For a broader assessment of the execution path, see AI agent evaluation.

The fallback must still work after retirement

“We will switch back if there is a problem” is useful only while the old version remains available. Verify that availability against provider documentation.

After retirement, you need another route: an already evaluated alternative, limited operation without the affected function, or manual processing.

Define what happens to in-flight work. Switching the model back does not undo actions already performed.

Document the release decision on one page

This template is a suggested working aid:

A migration can be suitable for some tasks but not yet for others. Document that boundary explicitly. For the economic comparison, the guide to AI cost per successful task complements the quality assessment.

Start with the next affected workflow

Choose an application with a clear task and traceable configuration. Record its model version, verify the availability deadline and compare the candidate against existing cases.

If useful test cases are missing, first establish a small, business-reviewed baseline.

If your existing system lacks a comparison baseline and release criteria, I can help through a focused AI Production Readiness assessment.