The accountability machine

It is tempting to believe that the obligation to explain oneself is as old as power itself — that the powerful have always owed the rest of us a reckoning, and that the only thing changing now is the technology of the powerful. This is a comforting story, and it is wrong. For most of human history, power answered to almost no one. A king did not keep minutes. A lord did not show his working. The idea that a consequential decision should come attached to an account of how it was reached, an account that a wronged party could inspect and dispute, is not a natural feature of authority. It is an invention. More precisely, it is a long sequence of inventions, each of them hard-won, each of them resisted, each of them built deliberately against the grain of power's strong preference to act without explaining. The remarkable thing is not that accountability sometimes fails. It is that it ever exists at all.

I want to trace that sequence, because I think it tells us something urgent about the present. We are living through the arrival of a new kind of consequential power — automated decision-making at scale — and we keep arguing about it as though it were unprecedented, as though no one had ever faced the problem of an authority that acts faster than anyone can watch and leaves no account of itself by default. But we have faced exactly this problem before, repeatedly, in different costumes. Each time, the answer was the same in form even when it differed wildly in detail: someone built a machine for holding the new power to account. The machinery looked nothing like the power it constrained. A courtroom looks nothing like a crime. A ledger looks nothing like a fortune. And the apparatus we now need looks nothing like a model. But the move is the same move, and recognizing it is the difference between feeling helpless and knowing what to build.

The court, and the discipline of admitting evidence

Consider first the oldest of these machines: the court, and specifically the slow accretion of rules about what may be allowed to count. It is easy to forget that the law of evidence is an invention, not a discovery — that for long stretches of history, disputes were settled by ordeal, by oath, by the relative standing of the men swearing, by methods that asked nothing about where a claim came from or whether it could be tested. The rules of evidence were built, over centuries and against considerable resistance, to replace that. They are a discipline for deciding which claims are even permitted to enter the record: rules about provenance, about who is offering a thing and whether they have firsthand knowledge of it, about whether the other side gets to test it, about the difference between something seen and something merely repeated.

The deep move there is worth naming precisely, because it recurs in every machine that follows. A court does not promise to establish the truth directly. It cannot rerun the events in question; the crime is over, the contract was signed long ago, the witnesses are fallible. So instead of guaranteeing truth, the court governs the conditions under which a claim earns the right to be relied upon. Admissibility is not a weaker substitute for truth — it is the standard a system adopts once it honestly admits it cannot inspect the truth and must instead discipline the process by which claims are allowed to bear weight. This is the founding insight of accountability machinery, and every later instrument is a variation on it: when you cannot verify the answer directly, build a procedure that makes the answer contestable, and trust the procedure rather than the assertion.

The ledger, the experiment, the audit

The same move appears, in a wholly different domain, with the spread of double-entry bookkeeping through the commercial cities of Renaissance Italy. Before it, a merchant's wealth was a matter of assertion and memory; you took his word, or you took your chances. Double-entry did something subtle and revolutionary. By requiring that every transaction be recorded twice, as both a debit and a credit, it built a structure that could be checked against itself — a ledger that, if it did not balance, announced its own error. It made the books contestable from the inside. A fortune was no longer simply claimed; it was accounted for, in a form whose internal consistency could be tested by anyone who understood the method. This was not a tool for honest merchants to feel good. It was a machine for making dishonesty visible, and it spread because the institutions that adopted it could be trusted in a way that the ones who did not could not.

Then the scientific revolution built its own version of the same instrument, aimed at a different kind of power: the authority of the person who claims to know how nature works. The empirical method that took institutional form in places like the early Royal Society added a requirement that now seems obvious and was then radical — that a claim about the world be backed by an experiment that others had witnessed and, crucially, could reproduce. The point of insisting on a witnessed, repeatable experiment was not ceremony. It was to move the warrant for a claim out of the speaker's reputation and into a procedure anyone could run. You no longer had to trust the natural philosopher's word that he had seen what he said he saw. You could, in principle, do it again and watch the result land where he said it would. Reproducibility is the scientific name for replay, and it does precisely what replay does in a court: it lets the conclusion be checked without trusting the one who reached it.

Each of these machines made the same move. When you cannot inspect the answer, build a procedure that makes the answer contestable — and trust the procedure, not the assertion.

The modern audit, the minutes of a meeting, the paper trail of a working bureaucracy — these are the same instrument again, refined for the institutions of the industrial age. The audit answered the problem of the firm too large for any one person to oversee: an independent party, with standing to demand the records, who reconstructs what happened from evidence rather than accepting the management's account of it. The minutes answered the problem of the decision made by a group, where afterward everyone remembers a flattering version; they freeze the deliberation as it stood, so that what was actually known and said at the time cannot be quietly rewritten. The bureaucratic file, for all the contempt heaped on it, answered the problem of the official who acts in the name of the state — by leaving a trail that a later inquiry, or a wronged citizen, could follow back to the moment the decision was made. None of these were inevitable. Each was built against power's preference to be opaque, and each survives only because we keep choosing to maintain it.

The power that escaped the machine

Now hold all of that in mind and look at what automated decision-making does. A model denies the loan, flags the claim, ranks the résumé to the bottom of the pile, sets the price, routes the case, scores the risk — and it does so in a fraction of a second, thousands of times an hour, with no minute taken, no ledger entry, no witness, no file. This is the first form of consequential power in modern history that has, so far, escaped the accountability machinery entirely. Not because anyone decided it should be exempt. Because it arrived faster than the old instruments could fasten onto it, and because, by default, it keeps no account of itself at all.

Notice exactly which capacities fail. The court's machinery assumes a decision slow enough to be witnessed and a decider who can be summoned and questioned; an automated decision is neither. The ledger assumes a record kept as the events occur; the model keeps none unless someone deliberately builds one. The witnessed experiment assumes the result can be reproduced; but a system that is not engineered to preserve the exact inputs and the exact state it ran against cannot be rerun, and so cannot be checked the way a published experiment can. The audit assumes a trail to reconstruct from; the minutes assume a deliberation to freeze. Automated decision-making, left to its defaults, supplies none of these. It is power that acts and forgets — and forgetting, here, is not a bug to be patched but the natural state of the thing unless accountability is built into it on purpose.

This is the uncomfortable recognition. We did not lose accountability because anyone attacked it. We lost it because we deployed a new kind of power before we re-tooled the machinery that holds power to account — and the machinery does not transfer on its own. A courtroom cannot cross-examine a weight matrix. An auditor cannot demand minutes that were never kept. The old instruments were built for human and institutional decision-makers who left human and institutional traces, and they are simply the wrong shape for a decision that happens in a half-second and vanishes. The demand they served has not changed at all. The instruments that served it have stopped fitting.

Re-tooling the oldest demand

So the work in front of us is not the invention of a new principle. It is the same work the bookkeepers did, and the experimentalists, and the people who first wrote down that hearsay should not count and that a fortune should balance against itself. It is the deliberate construction of an accountability machine for a power that arrived without one. And because the move has been made so many times before, we already know the shape of what has to be built. It has four parts, and each is the old demand re-tooled for a decision that decides too fast to be watched.

The first is provenance — the re-tooling of the rule about where a claim came from. An accountable automated decision has to carry an account of what it actually drew on: the specific inputs in front of the process at the instant it ran, recorded as they were rather than summarized after the fact. This is the ledger's discipline and the evidence rule's discipline at once. A decision whose provenance is unknown has not been made humbly; it has structurally opted out of being checked.

The second is replay — the re-tooling of the reproducible experiment. The machinery has to let the decision be run again against what was known at the time, so that the conclusion can be reproduced rather than merely narrated. An explanation is the system's opinion about its own reasoning; a replay is a fact you can establish without trusting the system at all. When the rerun reproduces the outcome, the record has earned its standing. When it diverges, the divergence is the finding.

The third is standing — the re-tooling of the rule about who may demand the record. The audit was only powerful because someone had the right to compel the books. An automated decision owns nothing and can be summoned by no one, so the machinery has to specify, in advance, which parties are entitled to pull the record, challenge it, and make it answer. A record no one may demand is a sealed file, not accountability.

The fourth is contestability — the property that binds the other three and gives them their point, exactly as it did in the courtroom. Provenance, replay, and standing exist so that a decision can be argued against by someone with reason to. A record assembled to reassure is a different object from a record assembled to be refuted, even when the two look identical at a glance. The whole tradition I have been tracing is a tradition of building records of the second kind. This is the heart of what we mean by a Decision Receipt, and the deeper argument for why admissibility, not truth, is the successor question for machine-made authority: not a summary that asks to be believed, but a record built to survive contact with someone who disagrees.

We are not being asked to imagine a new kind of accountability. We are being asked to do, for automation, what every accountable age before us did for its own new power.

I find the historical frame steadying rather than alarming, and I want to end on why. It would be easy to read the escape of automated power from the accountability machine as a defeat — as evidence that the old protections are obsolete and that we are entering a period in which consequential decisions simply will not answer to anyone. But that reading mistakes a gap for a verdict. Every machine I have described was, at the moment a new power arrived, exactly this kind of gap: a stretch of time in which the powerful could act without an account, before someone built the instrument that closed it. The merchants got their ledger. The natural philosophers got their reproducible experiment. The firms got their auditors and the states got their files, and none of it was easy or inevitable, and all of it was built by people who refused to accept that the new power was simply beyond reach.

We are at the same point with automation, and the same choice is open. We can ratify, by default, a standard in which a decision that arrives fast enough and sounds sure enough need answer to no one — or we can build, deliberately and against the grain, the accountability machine that automation skipped. The demand is the oldest one we have: that power keep an account, in a form a wronged party can inspect and dispute. The instrument is new, because the power is new. But the work is not unprecedented. It is the most precedented work there is. We have done it in every age that learned to hold its own new power to account, and the only real question is whether we will be the age that decided, uniquely, not to bother.

— Dispatches · Summit Cognitive