The burden of the builder

Almost every conversation about holding automated systems to account begins one stage too late. It begins with the operator — the organization that switches the system on, points it at real people, and lives with what it does. We ask the operator to be transparent, to monitor outcomes, to answer for the decisions the system makes on its behalf. These are reasonable demands, and they are also, in a quiet way, the wrong place to start. By the time a system is in the operator's hands, the choices that determine whether it can be held to account at all have already been made by someone else, somewhere upstream, and they cannot be unmade from the operator's chair.

The builder makes those choices. Not the builder as a heroic individual, but the builder as a role — whoever decides what the system records, what it discards, what it can reconstruct about its own behavior, and what is gone the instant it acts. Every one of those is a decision about future accountability, made long before any decision the system itself will ever produce. Whether a determination can later be examined depends on whether the builder arranged for the relevant facts to be kept. If they were not kept, no amount of good faith from the operator can summon them back. The operator can only be transparent about what the builder left them to be transparent with.

This is why the operator-first framing quietly lets the most consequential party off the hook. We hold the deployer responsible for the system's conduct while treating the system's capacity to be examined as a fixed property of the world, like gravity, rather than as something a person chose. But it was chosen. Someone decided, in a design review or a sprint or the simple absence of either, whether this system would be able to account for itself. That decision is the one that matters most, and it is made by the party we are least likely to name.

A system that cannot be made accountable after the fact was made unaccountable in advance — quietly, by default, and usually without anyone deciding to.

The choice is made before the first decision

There is a tempting fiction that accountability can be added later, that a system can be built first for function and fitted afterward with the machinery of explanation when someone asks for it. For some properties this is true. For this one it is not. The evidence a decision actually consulted, the rules that were actually active when it ran, the state needed to replay it — these exist only at the moment of the decision. If they are not captured then, they are not anywhere. You cannot retrofit a record of what was true at an instant that has passed. The window in which accountability becomes possible is the same window in which the decision happens, and it does not reopen.

So the builder is not choosing whether to add accountability later. The builder is choosing, at design time, whether later is even an option. A system that captures provenance and freezes its rules and preserves enough to replay can be held to account whenever someone needs it to be, on a timeline set by them. A system that does not is sealed against examination the moment it ships, and the seal cannot be broken afterward by anyone, however well-intentioned. The operator inherits whichever of these two systems the builder handed them, and inherits it as a settled fact.

The uncomfortable consequence is that unaccountability is almost never an act. It is a default. No one writes a requirement that says this system must be impossible to question. It simply turns out that way, because contestability was not specified, and what is not specified does not get built. The result is identical to sabotage and arrives without any of the intent — a system that cannot answer for itself, produced by people who would have been genuinely surprised to hear that this was the design. The absence of a malicious choice does not undo the effect of the structural one.

A duty that precedes deployment

This gives the builder a duty that is prior to, and independent of, anything the operator owes. The operator's duty is to use the system well. The builder's duty is to make a system that can be used accountably at all — to ensure that the capacity to be held to account is present in the artifact before it is ever pointed at a person. This is not a duty to predict every future challenge or to guess which decisions will be disputed. It is the narrower, more tractable duty to preserve, at the time of each decision, the things that any future challenge would need: the inputs, the governing rules, the replayable state. Get those right, and you do not have to know in advance what will be contested, because whatever it is, the means to examine it will be there.

This reframes a Decision Receipt as something the builder owes rather than something the operator generates on demand. It is the form the builder's duty takes — a commitment, made at design time and discharged automatically at every decision, that each consequential action leaves behind an artifact capable of being interrogated later by people the builder will never meet. The operator runs the system; the receipt is the builder's promise that running it does not extinguish the possibility of an account. One party acts in the present. The other has already decided whether the present can be revisited.

None of this dissolves the operator's responsibility, which remains real and substantial. It relocates the heaviest part of the burden to where the decisive choice is actually made. When a system cannot answer for a decision that harmed someone, the first question should not be why the operator failed to explain. It should be why the builder built something that could not. Accountability that is designed in can be honored by anyone who runs the system; accountability that was designed out cannot be honored by anyone at all. The builder is the only party positioned to make the difference, which is exactly why the burden belongs there first — and why a system that cannot be held to account should be read not as an oversight but as an answer, given in advance, to a question its makers chose not to ask out loud.

— Dispatches · Summit Cognitive