Green Checks Only Cover What You Checked

The build passes. The tests pass. The dependency update is small.

The integration check disagrees.

I do not find that especially surprising. A local check and an integration check can ask different questions about the same change. The awkward part begins when the agent treats the first green result as evidence for a larger conclusion than the check supports.

In The Work Around the Agent Is Engineering Work, I described the work around an agent as part of the engineering itself. One useful responsibility in that work is keeping the scope of evidence visible.

A fictional contract makes the distinction easier to see

The following is an illustrative composite. Names, fields and implementation details are invented; it is not a reconstruction of an identifiable system or an internal contract.

A booking service consumes a delivery window from a planning service. The contract previously used an ISO timestamp. An updated client expects an integer timestamp in milliseconds.

The booking service's local fixture is updated too:

type DeliveryWindow = {
  startsAtMs: number;
};

const window: DeliveryWindow = {
  startsAtMs: 1_800_000_000_000,
};

The compiler accepts the fixture. The local tests confirm that the booking code handles that fixture correctly.

Neither observation tells us which representation the deployed planning service actually returns. If it still returns the previous shape, the local result is not wrong. It is simply narrower than "these services remain compatible."

This is one reason I would not call local validation a miniature version of the whole production system. Its exclusions are often exactly what lets it run quickly.

Evidence has a scope and a revision

For each check, ask what it exercised and which code it observed.

A test against a mock exercises the mock contract. A test against a generated client exercises the declaration and the path using it. An integration gate may inspect a combined contract or contact a real environment. A package's exported type can differ from its runtime behavior or from what another participant still expects.

The checks have different value. They do not automatically inherit each other's coverage.

I would keep a small evidence record:

Check: booking service unit tests
Observed revision: current task commit
Established: local handling of the declared delivery window
Not established: compatibility with the deployed planner

Check: shared integration contract gate
Observed inputs: declared participating revisions
Established: compatibility within that selected set
Not established: behavior of every deployed client

This is not a demand for a document after every test command. It is a way to make the distinction explicit when the check is being used to justify a consequential decision.

A green integration result also needs an honest boundary. It can be based on a particular environment, selected revisions, or a contract model. It is stronger evidence for some questions, not proof of every runtime property.

Read the package, not only its version label

When a dependency change is involved, inspect what was installed: package metadata, declarations, runtime exports, and the relevant contract representation.

A requested version in a manifest is not the full observation. The lockfile, registry artifact, workspace resolution and build output determine what code is actually used. A stale installation can make an agent diagnose a version that is not the one under test.

This does not mean dumping the entire dependency tree into context. Start with the disputed boundary and follow the few inputs that decide it.

I want the agent to compare evidence it can point to, rather than expand an explanation around a guessed package behavior. More confident prose does not resolve disagreement between an artifact and a declaration.

Several repositories do not form one snapshot

Giving an agent access to related repositories is helpful. It does not make their branches, local changes, and deployed versions coherent automatically.

One repository may be on a current base. Another may have an old checkout. A third may contain a pending fix that already changed the intended migration path.

Before reasoning across them, identify the revisions and the purpose of each source: current main, active task branch, proposed change, or deployed release. A locally visible implementation can be real without being the implementation production uses.

Repository memory should help locate those questions, not answer them from an earlier session. The source of truth for current compatibility is the current relevant state.

Recheck the problem before repairing the diagnosis

The composite continues. Further investigation finds a compatible change already delivered through another path. The original dependency update is now obsolete.

There are two different next actions available: make the old proposal more elaborate, or stop pursuing it.

An agent can be quite good at the first. It has a task, a branch and an explanation to complete. That momentum is useful when the task remains valid. It is not useful when the current system no longer needs the proposed work.

The correct result can be closing the obsolete PR and recording why it was superseded. Do not describe that as delivering the other change. Do not rewrite the history so the incomplete initial diagnosis becomes a clean success.

The useful outcome is a rejected unsuitable proposal and a better freshness check for the next task.

Before asking whether the patch works, ask whether it is still the patch the system needs.

This is particularly valuable with agents because implementation is relatively easy to continue. Deciding not to continue still requires evidence and an accountable owner.

Correct the workflow at the missing boundary

If the initial diagnosis relied on stale or incomplete sources, merely fixing the current diff leaves the same failure available next time.

Move the relevant current-base or related-change check earlier. If a shared contract matters, make the integration gate part of the verification criteria. If the package contents were assumed, inspect the actual artifact before proposing a fix.

Keep the correction proportional. One incident does not justify requiring every task to scan every repository. A small single-repository CSS change and a shared dependency migration do not need identical discovery work.

The rule should describe the condition that makes the check useful: cross-service contracts, dependency behavior, version skew, or an active related change. Otherwise a lesson becomes another universal instruction the agent follows even when it has no bearing on the task.

Separate a failed assumption from a failed person

An incomplete AI diagnosis deserves correction. It does not need a theatrical explanation about agents being incapable, nor a reassuring story in which the model secretly intended the right thing.

Record the assumption, the contradicting evidence, the correction, and the remaining uncertainty. That is enough to learn from the task.

It also gives credit accurately. A person may notice the stale base. CI may reject the shared contract. The agent may then help compare sources and prepare the explanation. Those contributions can all be real without being compressed into "AI solved it."

Keep green useful by keeping it specific

I want local checks to stay fast and valuable. Their usefulness does not depend on pretending they cover the whole system.

A build confirms something about the selected sources and toolchain. A test confirms behavior under its chosen conditions. A related contract check adds another layer. Current-state investigation tells us whether the task itself still makes sense.

The result is not certainty. It is a decision with evidence that matches its scope.

I would rather close one well-explained obsolete proposal than merge an increasingly polished implementation of yesterday's requirement. There is less to show in the diff, but more of the system has been understood.

The next article, The Command Succeeded. The Scope Changed., looks at another boundary green checks do not automatically cover: what automation changed after the initial edit.

If a check in your workflow is routinely used to justify more than it tested, which claim gets attached to it? That gap is usually worth examining before adding another test suite.