The corrigible institution

Every institution is wrong sometimes. This is not cynicism; it is arithmetic. An organization that makes many decisions across many years will, with certainty, make some bad ones — from faulty inputs, from a rule that turned out to be crude, from a judgment that looked sound and was not. So the interesting question about an institution was never whether it errs, because that answer is fixed. The interesting question is what happens next. When the error has been made and the harm has been done, can the institution find it, name it, and fix the thing that produced it? Or can it only deny that anything went wrong, until the pressure of denial finally breaks it? That difference — between an institution that can be corrected and one that cannot — is, I think, the deepest thing there is to know about any organization that holds power over people.

Borrow a word from a narrower context and let it earn a broader one: corrigibility. A corrigible institution is one that can absorb the discovery that it was wrong and metabolize it into a repair. It can locate the specific decision that failed, trace back to the rule or the input or the assumption that caused the failure, change that thing, and carry on more sound than before. An incorrigible institution cannot do this. Confronted with its own error it has only two moves available — deny that the error occurred, or, when denial finally fails, come apart. It cannot repair, because repair requires knowing precisely what to fix, and it does not know.

What separates the two is not virtue or intent. It is a record. Corrigibility is not a moral quality an institution has more or less of; it is a mechanical capacity that depends entirely on whether the institution kept enough of a trace of its own decisions that an error, once suspected, can actually be located and isolated. You cannot correct what you cannot find. And you cannot find, in a system of any size, without a record that preserves each decision well enough to be examined after the fact.

An institution that cannot say why it decided cannot discover that it decided wrongly — and cannot repair what it cannot discover.

Automation tends toward the incorrigible

Here is the uncomfortable part. The direction automation pushes an institution is, by default, toward incorrigibility. Automated systems are fast, they are confident, and they are — in their raw form — nearly silent about why. They produce a verdict without producing the reasoning that would let anyone check it, and they produce it at a volume that makes case-by-case human reconstruction impossible. Each of these traits is a virtue for throughput and a poison for correction. Speed means errors propagate before anyone notices. Confidence means the output carries no signal of its own uncertainty. And silence about reasoning means that when an error is finally suspected, there is nothing to reason back through. The institution has decided a million times and can account for none of them.

This is why an automated institution can look, for a while, like the most reliable kind — until the day it is wrong, at which point it reveals itself as the least fixable kind. Its errors cannot be located, because it kept no map of how any particular decision was reached. And an error that cannot be located cannot be repaired, only denied. So the organization that most needed the discipline of correction ends up least able to practice it, having traded away, in the name of efficiency, the very traces that correction depends on. The failure mode is not that automated systems make more mistakes. It is that they make their mistakes uncorrectable.

None of this is inevitable. Incorrigibility is a design outcome, and a different design produces a different outcome. A system built so that every consequential decision leaves a real record — the evidence it consulted, the rule as it stood at the time, enough state to replay the path from inputs to verdict — is a system whose errors have a location. When such an institution is told it got a case wrong, it can go to that case, reconstruct exactly what happened, and see whether the failure was in the input, the rule, or the reasoning. That is the moment correction becomes possible: not when the institution wishes to be better, but when it can point to the precise thing that was wrong.

Corrigibility is a property you build

The record does more than locate a single error. It converts a one-off failure into a repair of the rule that produced it. Because if you can trace a bad decision back to the specific rule it followed, then fixing that rule fixes not only the case in front of you but every future case the rule would have governed. This is the difference between apologizing for an outcome and correcting a system. The apology addresses the instance; the corrected rule addresses the class. And only the record makes the second possible, because only the record tells you which rule, of the thousands in force, was the one that led here.

There is a further consequence worth stating plainly. An institution that can be corrected is an institution that can be trusted with more, precisely because being wrong is not catastrophic for it. Its errors are survivable, because they are findable, because they are fixable. The incorrigible institution, by contrast, has to treat every error as an existential threat — because for it, an error it cannot repair is an error it can only hide, and hidden errors accumulate until they surface all at once. Corrigibility is what lets an institution be wrong safely. It is the shock absorber that turns a fatal error into a survivable one, and the record is the shock absorber's working part.

So when we ask what to demand of the automated institutions now being built, the answer is not that they be right — no institution can promise that, and one that does is lying. The answer is that they be corrigible: that they keep, for each decision, enough of a record that when they are wrong, and they will be, the error can be found, named, and repaired at the level of the rule. An institution that can do this can be wrong and still be trusted. An institution that cannot can only be confident until it collapses. Between those two, the record is the whole of the difference.

— Dispatches · Summit Cognitive