Guardrails for a Runaway Agent

Everything in this cluster has circled the same failure: an agent that runs longer, spends more, and drifts further than you intended, with nothing to stop it. The multi-agent orchestra fractures context, the token bill balloons, the scheduled loop burns all night. Underneath all of them is one uncomfortable property, and Anthropic name it directly: agents are stateful, and their errors compound. One bad step doesn't just produce one bad result — it changes the state the next step reasons from, so the agent quietly walks off in a completely different direction and has no idea it's lost.

That's the thing traditional software doesn't really do. A bug usually breaks a feature and stops. An agent's bug can send it on a confident, expensive journey to nowhere. So the design question isn't "how do I make the agent never fail?" — you can't. It's "when it fails, how do I make sure it fails small?"

You don't grant an agent autonomy by trusting it. You grant it autonomy by bounding what it can do when you're wrong to trust it.

Cap the loop

The first guardrail is the dumbest one and the most important: a hard ceiling on how many steps or iterations a run can take. Anthropic build exactly this into their agents — a maximum number of iterations as a stopping condition, so a task that can't converge terminates instead of spinning. This is the single line that turns the overnight-burn story into a non-event: the agent tries twenty times, hits the cap, stops, and tells you it's stuck. "Burned all night" becomes "gave up before breakfast."

Budget the spend

The step cap bounds count; a spend cap bounds cost directly. Give a run a token or dollar budget and a circuit breaker that trips when it's exceeded. Even if you got the iteration ceiling wrong, the budget is a second, independent wall between a stuck agent and a catastrophic invoice. Two different limits guarding the same failure is not redundancy — it's the point. The whole philosophy here is that any single guardrail might be the one you misconfigured.

Notice when it stops making progress

A cap stops an agent eventually; progress-detection stops it sooner. If the agent takes the same action, gets the same result, and tries again — that's not work, that's a loop, and you can detect it. Repeated identical tool calls, the same error over and over, no change in state across steps: these are all signals that the run has stopped advancing and should be cut short. It's the difference between letting it exhaust twenty iterations and killing it at the third pointless repeat.

Put a human at the sharp edges

Not every action deserves the same freedom. Reading a ticket, running a test, opening a draft PR — low consequence, let it run. Deleting data, pushing to a shared branch, spending real money, touching production — these are where autonomy should pause and check with a person. This isn't timidity; it's the same negotiated boundary that makes "do it as usual" safe to run unattended. The agent owns the reversible actions and stops at the irreversible ones. You decide where that line sits per action, deliberately.

Shrink the blast radius

The guardrails above assume the agent tries something bad; sandboxing assumes it succeeds and makes that survivable. Least privilege is the whole idea: an agent that physically cannot reach production can't take it down, no matter how confused it gets. Run in an environment where the worst thing it can do is still recoverable — a scratch branch, a staging database, scoped credentials. Then a runaway is an annoyance, not an incident.

See why it ran away

You can't tighten a guardrail you can't see through. Anthropic lean hard on tracing agent decisions for exactly this reason — when an agent "can't find obvious information" or spirals, the only way to fix it systematically is to look at what it actually did, step by step. Observability isn't a nice-to-have bolted on at the end; it's how you learn which guardrail was missing. Every runaway you can inspect becomes a rule you can add.

The reframe

There's a temptation to read all of this as distrust — as clipping the agent's wings. I see it the opposite way. Guardrails are what let me hand an agent more autonomy, not less, because I know the failure modes are bounded. The reason I can say "do it as usual" and walk away is precisely that a stuck run caps out, a runaway can't reach anything that matters, and I can see afterward what went wrong. The bounds are what make the trust affordable.

An agent without guardrails isn't a bold, high-trust setup. It's an unbounded process with a credit card and access to your systems, and the only thing standing between it and a bad night is luck. Design the blast radius first, and autonomy stops being a gamble. That's the through-line of this whole cluster: the goal was never a more impressive agent. It was a sane one — one you can actually let run, because you decided in advance how it's allowed to fail.

Take one agent you run unattended and write down what happens if it gets stuck right now: does it stop on its own, and what can it reach while it's confused? If either answer is uncomfortable, you've found your next guardrail. Tell me which one — I'd bet it's the iteration cap, because it almost always is.