The standard nobody chose

We tend to imagine that the rules governing a new kind of power get written deliberately. Someone convenes a body, hears arguments, weighs interests, and emerges with a standard that bears the marks of having been chosen. That is the flattering story we tell about how norms come to be, and it is true often enough to keep us believing it. But it is not how most standards arrive. Most arrive by default — accreted out of countless small, unexamined decisions made by people who did not think they were deciding anything at all, and who would be surprised to learn that the sum of their choices had hardened into a rule everyone now treats as settled.

This is the situation we are in with machine-made authority. As automated systems take over decisions that used to be made by people who could later be asked to account for them, the live question is not whether those systems will be powerful. They already are. The question is whether the exercise of that power will leave a record anyone can examine — and that question is being answered, right now, not by deliberation but by accumulation. Every time an engineer wires up a system that decides something and chooses, often without a second thought, whether to retain the evidence and the rules that produced the decision or to let them fall away, that engineer is casting a vote in an election no one announced.

The votes are not being counted anywhere. There is no tally, no published threshold, no moment at which someone declares the result. But the result is forming all the same, and it has the same force a deliberated standard would have. When enough systems are built the same way, the way they are built stops looking like a choice and starts looking like the nature of the thing. People will say that is just how automated decisions work — not because anyone decided they should work that way, but because that is how the ones in front of them happened to be made.

Default is not the absence of a decision

The seductive thing about a default is that it feels like nobody chose it. A default presents itself as the natural resting state, the thing that happens when no one intervenes — and so it seems to carry no responsibility, because responsibility attaches to choices and a default is precisely what you get when you do not choose. This is an illusion, and a costly one. A default is a choice that has been made invisible by repetition. Someone, at some point, decided what the system does when no one is paying attention, and everyone who builds on top of that decision without revisiting it ratifies it again.

Consider the most consequential default in this whole domain: whether a system that makes a decision keeps the materials that would let anyone reconstruct it. Whether it retains the evidence it actually consulted, the rules that were active when it decided, enough state to replay the decision and see whether it holds. None of that retention is free; all of it takes deliberate engineering. So the path of least resistance — the default — is to discard it. To let the decision happen and let the basis for the decision evaporate the instant it is no longer needed to produce the output. A system built that way is not neutral about accountability. It has quietly chosen against it, and called the choice an oversight.

A standard arrived at by default is still a standard. We will inherit it as if someone had chosen it — and no one will be able to say who.

This is what makes the present moment dangerous in a way that is easy to miss, because nothing about it looks like a crisis. There is no villain deciding that machine decisions should be unaccountable. There is only a long sequence of reasonable-seeming build-time choices, each defensible in isolation, that together are setting the terms on which a generation of automated authority will be allowed to operate. By the time the standard is visible enough to argue about, it will already be load-bearing, and the cost of changing it will have grown to match.

Inheriting a rule no one wrote

The trouble with a standard set by default is not only that it might be the wrong standard. It is that there is no author to hold to account for it, and no record of the reasoning that produced it, because there was no reasoning — only drift. We will inherit it the way you inherit a path worn across a field: it goes where it goes because that is where feet went, and the fact that a slightly different path would have been better is invisible to everyone now using it. The path becomes the route. The route becomes the map. The map becomes the territory.

I do not think the answer is to slow down the deployment of automated decisions, which is neither possible nor obviously desirable. The answer is to recognize that the default is itself a decision, and to make it on purpose. The systems that decide things about people should be built so that each decision carries its own account — the evidence, the active rules, enough to replay it — not because regulation will eventually demand it, though it may, but because the alternative is to let the most important governance question of the decade be settled by whoever happened not to think about it. A Decision Receipt is, at bottom, a refusal to let the default win by inertia. It is the act of choosing, explicitly, that the record will be kept.

The norms governing machine-made authority are being set this year and next, in code that will outlast the people who wrote it and the assumptions they did not examine. Someone will inherit those norms and treat them as given, the way we treat as given a hundred standards that were themselves once nobody's deliberate choice. The only thing to decide now is whether, when that inheritance lands, it will be a record of what we chose, or merely a fossil of what we failed to.

— Dispatches · Summit Cognitive