The blast radius of an action

There is a question people ask about autonomous agents that sounds rigorous and is almost useless. The question is: how good is it? How often does it get the answer right, choose the right tool, take the sensible action instead of the reckless one? This is worth knowing, but it is the wrong governing question, because it describes the agent on an average day. Governance is not built for average days. It is built for the worst hour of the worst run, when the model has misread its instructions and is executing confidently in the wrong direction. The question that governs is not how good the agent is. It is how far a single wrong action can reach before something stops it.

Borrow the term from the discipline that has thought about this longest. In security engineering, the blast radius of a component is the extent of the damage if that component is compromised — how much it can touch, corrupt, or destroy before containment kicks in. You do not secure a system by assuming its parts will behave. You secure it by assuming one of them will not, and bounding what that failure can reach. Least privilege, isolation, the whole architecture of containment exists to make the answer to the question what if this goes wrong as small as possible. Agentic systems are importing that discipline, and they should — but not only as a security measure. Blast radius is an accountability primitive. It is the true measure of the authority you have handed over.

Worst case, not average case

Consider two agents. The first can draft and send a single email to one recipient. The second can move money between accounts, delete customer records, and act on thousands of accounts in a loop without pausing. Suppose, for the sake of argument, that both are equally competent — same model, same average error rate, same tendency to do the sensible thing nine times out of ten. They are not remotely the same object to govern, and the difference has nothing to do with their competence. It is entirely a difference of reach. The first agent's worst possible action is one wrong email. The second agent's worst possible action is a financial and record-keeping catastrophe executed faster than a human can read the first alert. Competence is identical; blast radius is not; and blast radius is what you are actually deciding about when you grant authority.

This is why competence is not containment, and why measuring an agent by its average behavior quietly hides the thing that matters. An agent that is usually right and can, when it is wrong, touch everything is not a safe agent with a small flaw. It is a large-blast-radius system that has been lucky. The authority you granted it was never the authority to do the useful thing well. It was the authority to do the worst thing at all — and that authority was in force the entire time, on every run, whether or not it was ever exercised. What you granted is defined by the ceiling, not the mean.

You do not control an agent by trusting its judgment; you control it by bounding what a single mistake is allowed to touch.

Scope, rate, and stage

Once you see authority as reach rather than reputation, the controls that bound it stop looking like security hygiene and start looking like the substance of governance. Least privilege — the principle that a component should hold only the authority its immediate task requires, and no more — is usually framed as a defense against attackers. It is also, and more fundamentally here, the thing that makes an agent's action answerable. An action taken inside a narrow, recorded grant has an author and a boundary; you can say what it was permitted to touch and confirm it touched nothing else. An action taken under a broad, standing grant of everything has no such boundary, and so no clean account of what was and was not authorized. Scope is not merely how you keep an agent from reaching too far. It is how you make the reach of any given action a stated, checkable fact.

Rate is the second bound, and it is the one autonomy makes urgent. A human acting badly is limited by the pace of hands; an agent acting badly is limited only by the loop. Caps on how many times an action can fire, how much value can move, how many records a single run can alter convert an unbounded failure into a bounded one. The point is not that the agent will exhaust the cap; most runs never come near it. The point is that the cap sizes the worst case in advance, so the maximum reach of a malfunction is a number you chose rather than a number you discover. Staged execution does the same work along a different axis: an action of consequence is proposed, checked, and released in steps, so there is a place for a limit or a person to intervene between deciding and having-done. The stage is where reach gets interrupted.

And all of this has to land in the record. It is not enough that a scope, a cap, a staging boundary was in force; the account of the action must capture which scope was in force at the moment it acted. Authority drifts — grants are widened for a migration and never narrowed, caps are raised under deadline and left raised. A record that says only what the agent did, and not the authority it did it under, cannot answer the question that governance actually asks after something goes wrong: was this action inside the reach it was permitted, or had the reach quietly grown? The scope that bounded the act is part of the act's account, or the account is incomplete.

Sizing the remedy to the radius

The last move is the one that ties reach to accountability directly. A large-blast-radius action does not merely need tighter scope going in; it needs a stronger record and a faster abort coming out. The account and the brakes should be sized to what a single action can reach, not to how often the agent is expected to be right. An agent that can send one email can be governed lightly, because the most it can do is small and reversible. An agent that can move money or act across thousands of accounts needs a record built to be contested, a standing that names an answerable party, and an abort that can arrest a runaway loop before it completes — because the worst it can do is large, fast, and sometimes irreversible, and the remedy has to be able to reach as far as the harm.

This is the point of contact with the two ideas the rest of this family has argued. Authority is the account: a narrowly scoped, just-in-time grant is what makes an autonomous action answerable at all, and blast radius is the measure of what that grant costs if it fails. And the executed action does not wait: an agent closes the gap between deciding and doing where an error used to be caught, so the record has to do the work the missing pause once did. Containment is how those two truths stay compatible. It is how you keep an unaccountable action from becoming an unbounded one — how you ensure that when the pause is gone and the account is the only witness, the most that can happen between the mistake and the stop is something you already decided you could live with.

So the governing question is not whether your agent will act well. Assume, for the purpose of building, that it will sometimes act badly, and that no competence retires that assumption. Ask instead how far a single wrong action can reach before a limit or a person arrests it — and then scope, rate, and stage the authority until that reach is small enough to survive. The blast radius is the real specification. Everything else is a hope about the average day.

— Dispatches · Summit Cognitive