Upgrading the Model Means Retesting the Workflow

Changing a model identifier can be a one-line edit. It is not necessarily a one-line change to the system's behavior.

The same task may now produce different searches, tool arguments, clarification questions, stopping decisions or recovery attempts. The final code can improve while one of those surrounding behaviors becomes less suitable for the workflow.

I would treat a model upgrade as a change to an engineering dependency, not as permission to carry every previous assumption forward.

That isn't an argument for staying on an old model. It is a reason to make the benefit and the compatibility work observable.

The application contract is larger than the final answer

A coding agent operates inside a harness: prompts, tool definitions, permissions, context management, execution loops and output handling. The model is one consequential part of that arrangement.

The contract may require the agent to inspect the current base before editing, use a particular tool for a sensitive action, stop when authorization is missing, and distinguish an unavailable check from a successful one.

Those requirements don't disappear because the candidate model produces a better-looking patch.

Some differences are API compatibility issues. Others are valid behavior changes inside an otherwise compatible API. A request succeeding doesn't establish that the application still gets the kind of progress events or tool-use pattern it expects.

There is current, concrete evidence for taking that distinction seriously. Anthropic's Fable 5.1 migration guide, checked on October 23, 2026, documents model-specific changes involving forced tool choice, conversation-history handling and agent-loop behavior. Those details apply to the documented migration, not to every model or managed product.

The general lesson isn't to memorize today's parameter list. It is to read the migration contract for the exact integration being changed. A managed agent product may handle details that a custom API harness must manage itself.

First name what is actually changing

An editor update can change the model, tool descriptions and execution behavior together. A provider can update a model alias. A team can revise its system prompt while testing a new model.

If all of those happen at once, an improved result does not identify which change helped. A regression doesn't identify which component broke either.

For a controlled comparison, I would record the model identity available to the integration, harness revision, prompt revision, tool schemas, relevant settings and fixture state. If the exact underlying model is hidden, I would record that limitation instead of inventing a stable identity.

Keep task inputs and starting repositories comparable. A new base commit, warmed cache or already-fixed fixture can turn a model comparison into a comparison of different problems.

One variable at a time is a useful diagnostic ideal, not always a deployment option. When a product bundles changes, evaluate the bundle and describe the attribution limit honestly.

Build cases around decisions that matter

I would start with a small regression set drawn from the workflow's consequential boundaries.

A clear local fix should remain clear and local. An ambiguous requirement should produce a useful question rather than an invented product decision. A failed integration check should not be summarized as a successful task because the unit suite passed.

Another fixture might change the PR head during review. The expected outcome is to notice that the reviewed artifact has changed, not merely attach the old conclusion somewhere the API accepts it.

These are proposed evaluation cases, not reported experiments or results from my work. Their value comes from defining the acceptable outcome before reading the candidate's explanation.

I would avoid requiring one exact tool-call sequence unless the sequence itself enforces a real constraint. Two valid investigations may read different files. The evaluation should distinguish a different route from a violation of the task's boundaries.

Conversely, a plausible final patch shouldn't erase an unauthorized intermediate action. Evaluate consequential actions and the final artifact, not just the last paragraph.

Grade observable behavior, with room for uncertainty

Some outcomes can be checked mechanically: whether a forbidden mutation occurred, whether the final diff contains unrelated files, or whether a claimed test command actually completed.

Other outcomes need a domain reviewer: whether a clarification was necessary, whether a suggested integration belongs in scope, or whether the proposed test establishes the intended behavior.

A model-based grader can help organize evidence, but it isn't automatically an independent authority. I would calibrate it against reviewed examples and inspect disagreements. A score without an understandable basis is another opaque output.

OpenAI's evaluation documentation describes datasets, annotations and graders. At the October 23 check, it also carries a deprecation notice for the hosted Evals platform. I am not recommending a new dependency on that retiring service. Representative cases and explicit grading criteria remain useful regardless of which supported runner stores them.

The tooling should serve the comparison. It should not become a reason to keep a fragile evaluation tied to an obsolete interface.

Recovery deserves its own test

A clean success path is a friendly demonstration. The more revealing cases often involve unavailable tools, authentication failures, partial edits and interrupted work.

I would check whether the candidate preserves the task boundary after an error. Does it ask for a secret in an inappropriate channel? Does it retry an irreversible operation without checking the previous outcome? Does it weaken a test merely to complete the loop?

These concerns aren't claims that a particular model does those things. They are reasons to observe recovery rather than assume it inherits the quality of the model's successful answers.

Permissions should remain enforced outside the model. An evaluation can detect undesirable behavior; it shouldn't be the only thing preventing a consequential action.

The same reasoning applies to compaction and handoffs. A workflow that works for five turns may fail when a summary loses an important unresolved condition. A longer-running fixture can expose that boundary without requiring an enormous benchmark.

Compare the cost of an accepted result

Tokens and response latency are useful measurements. They don't describe the whole task if the candidate requires more correction, publishes noisier reviews or leaves harder-to-understand changes.

I would record human interventions and rework alongside execution cost, using actual observations rather than estimated productivity claims. Passive waiting and a person's active effort are different quantities.

Repeated runs matter where outputs vary. One candidate success and one baseline failure are anecdotes, not a stable rate. The amount of repetition should match the consequence of the decision and the available resources.

The result can be mixed: better implementation on one class of task, more clarification on another, a compatibility repair needed in the harness. That is a useful migration finding, not a disappointing failure to produce a winner.

Keep a way back

Before changing a default, I would keep the previous configuration identifiable and limit initial exposure. That might mean a small authorized task class or shadow review without automatic publication.

Rollback has limits too. A pinned model may later be retired, and a managed product may not expose an exact old configuration. Record what can actually be restored rather than promising a permanent switch.

The existing evaluation article asks whether the workflow produces useful outcomes. A model upgrade is a concrete occasion to ask that question again, with the old assumptions made visible.

I want to use better models. I also want to know which part of my working agreement with the agent changed when I selected one.

What would make a model upgrade fail acceptance in your workflow even if its final code looked better? I'd like to hear which behavioral regressions your evaluations can actually detect.