Correction is an institutional capability

We tend to praise institutions for being right, and to judge them by how often they are. But rightness is a poor test, because no institution of any size stays right for long. The world moves, the inputs shift, a rule that fit last year's conditions begins to land wrong on this year's cases, and the question is never whether an institution will err but what it does when it has. The property that actually separates a trustworthy institution from an untrustworthy one is not accuracy. It is corrigibility — the capacity to discover its own mistake, name it precisely, and change the rule that produced it. And corrigibility, unlike rightness, is something you can design for.

Consider what correction actually requires. To fix an error you must first be able to find it, which means being able to point at a specific decision and say: this one, here, came out wrong. Then you must be able to say why — not vaguely, that something went awry, but exactly which input, which rule, which threshold produced the outcome that should not have stood. Only then can you change the thing that needs changing rather than flailing at the whole apparatus. Every step of that sequence depends on a record. Without one, you cannot locate the error, cannot diagnose it, and cannot verify that your fix actually addressed it rather than something adjacent. An institution with no record of how it decided is not merely slow to correct itself. It is structurally unable to.

This is why the incorrigible institution has only two moves available when confronted with its own failure, and both are bad. It can deny — insist that no error occurred, because it has no way to demonstrate that one did and no appetite for admitting what it cannot examine. Or it can collapse — concede that something is wrong somewhere but, unable to isolate it, throw out the whole system in a spasm of overcorrection that discards the parts that were working. Denial and collapse are the two failure modes of an institution that cannot read its own past. Correction is an institutional capability has a third option, which is simply to fix the specific thing.

An institution that cannot examine its own decisions can only deny them or abandon them. The middle path — correction — is the one that requires a record.

Automation bends toward the incorrigible

There is a particular reason to worry about this now. Automated systems are, by default, tilted toward incorrigibility, and the tilt is easy to miss because it comes disguised as competence. Such systems are fast, they are consistent, they are confident — and none of those virtues has anything to do with being correctable. A system can produce a million fluent, self-assured decisions and retain nothing about how it reached any of them. When one of those decisions turns out to be wrong, there is nowhere to look. The system cannot tell you what it did, because it did not keep what it did. Its confidence and its incorrigibility are not in tension; they grow from the same root, which is that the system was optimized to produce outputs, not to account for them.

This matters more as the decisions compound. When yesterday's automated outputs become the inputs to today's, an uncorrected error does not stay put. It propagates, quietly, through every downstream decision that trusted it, and by the time anyone notices the pattern, the original mistake is buried under a thousand others built on top of it. In a corrigible system this is recoverable, because the record lets you trace the pattern back to its source and cauterize it there. In an incorrigible one it is not, because there is no thread leading back. The error simply becomes part of the system's inherited behavior, indistinguishable from the parts that were always meant to be there.

Corrigibility is a design property, and the record is its foundation

The encouraging thing about all this is that corrigibility is not a matter of institutional character, or courage, or humility, though those help. It is a property you can build into the machinery, and the load-bearing element is the record. A Decision Receipt that preserves the inputs a decision rested on, the rule in force at the time, and enough state to replay it is precisely the instrument correction needs: it lets you find the wrong decision, see exactly what produced it, change the rule, and then replay the old cases through the new rule to confirm that you fixed what you meant to and did not break something else. That last step is what turns correction from a hopeful gesture into a verifiable one. You are not asserting that you fixed the problem; you are demonstrating it against the actual record of the decisions that went wrong.

Notice how this reframes what a record is for. We usually justify decision records by appeal to the affected party — the person who wants to contest, to appeal, to be heard. That justification is real and enough on its own. But there is a second beneficiary who is easy to overlook: the institution itself. The same record that lets an outsider challenge a decision is what lets the institution examine, diagnose, and repair its own behavior. Accountability to the outside and corrigibility on the inside are not two requirements competing for the same budget. They are the same capacity, described from two directions. A record built to answer the person who was harmed is, by construction, a record the institution can learn from.

So the honest measure of an institution operating at machine scale is not how confident its systems are, or how rarely they visibly fail, or how clean the story sounds when something goes wrong. It is whether, when a decision lands badly on someone, the institution can find that decision, say precisely why it came out as it did, change the rule, and prove that the change worked. An institution that can do that will make mistakes and remain trustworthy, because its mistakes are correctable. An institution that cannot will eventually face an error it can neither deny nor survive — and it will discover, too late, that the record it never kept was the only thing that could have let it recover.

— Dispatches · Summit Cognitive