The first draft of an injustice

Every serious institution that decides things about people has, historically, built in a pause between the decision and its finality. A first instance and an appeal. A ruling and a review. A determination and a window in which it can be contested before it sets. This pause is not inefficiency; it is the recognition that a decision made once, by a fallible process, is a draft — a proposed answer that has not yet earned the right to be the final one. The draft becomes final only by surviving the chance to be challenged. That is what makes finality legitimate: not that the decision was made, but that it was made and then left standing after someone had the opportunity to knock it down.

Automation quietly removes the pause. Not by deciding anything different than a person would, but by deciding it so fast, and presenting it with such finished confidence, that the draft and the verdict collapse into the same instant. The system produces an output, the output is applied, the consequence lands, and at no point is there a gap in which the decision is understood to be provisional. It arrives already wearing the clothes of a settled conclusion. And because there is no person standing behind it whom you could ask to reconsider, there is often no obvious place to even lodge the objection that, with a human decision, the structure of the institution would have invited.

The result is a strange inversion. We adopted automated decisions partly for their speed, and speed is genuinely valuable. But speed applied to the wrong thing is dangerous. It is good to reach a draft quickly. It is bad to reach finality quickly, because finality is precisely the thing that should be slow — it should wait for the contest. When automation compresses the two, it does not make justice faster. It makes error permanent faster, which is the opposite of what we wanted.

An error and an injustice are not the same thing

It is worth separating two ideas that the collapse of the pause tends to fuse. An error is a decision that got it wrong. Errors are ordinary; every decision process produces them, human or machine, and a system that never erred would be a system that never decided anything hard. An injustice is something else. An injustice is an error that has been allowed to stand — that the affected party could not get reopened, could not contest on its merits, could not get anyone with authority to look at again. The error is the mistake. The injustice is the closing of the door behind the mistake. And the door does not close because the error was especially bad. It closes because nothing in the process treated the decision as still open.

This distinction matters enormously, because it tells you where to intervene. You cannot eliminate error; no amount of care will. What you can do is keep error from curdling into injustice, and the way you do that is by keeping the decision contestable for long enough, and with enough of a record, that a genuine mistake can be surfaced and corrected before it sets. The harm of an automated system is rarely that it errs more than a person would. The harm is that, built carelessly, it forecloses the correction — it treats its first draft as a final verdict, discards the basis that would let anyone reopen it, and so converts each ordinary error into a small permanent wrong.

An error becomes an injustice at the moment it stops being contestable. Automation does not make more errors; it just closes the door behind them faster than anyone can get a foot in.

To treat a decision as a first draft is to make a few specific commitments about how it is allowed to operate. It means the decision arrives marked as provisional rather than presented as settled. It means the affected party has standing to contest it — a real, structured way to say this is wrong, and here is why, that lands somewhere obligated to receive it. And it means the decision carries enough of a record — the evidence consulted, the rules active, enough to replay it — that the contest can be more than a complaint. A draft you cannot examine is not really a draft; it is a verdict you have merely been invited to dislike. The provisionality has to be backed by the materials that make revision possible.

Provisional by design, not by grace

The temptation is to handle all of this with goodwill — to say that of course the decisions can be appealed, that of course someone will look again if you ask nicely, that the system's operators are reasonable people who will fix obvious mistakes. But goodwill is not a structure, and the whole problem with a collapsed pause is structural. If contestability depends on the operator choosing, in each case, to treat the decision as still open, then it will hold only as long as attention and patience hold, which is to say not very long and not at scale. The provisionality has to be built into how the decision is made and recorded, so that it does not rely on anyone remembering to be generous about it.

That is what it means to design a decision as a first draft rather than to merely permit it to be appealed. The record exists because the system makes it, not because someone asked. The standing to contest is part of the process, not a favor extended to the persistent. The decision is revisable because it was built revisable — because the materials needed to reopen it were preserved at the moment it was made, on the assumption that reopening might be necessary, rather than discarded on the assumption that it never would. A Decision Receipt, in this light, is just the artifact that keeps a decision in draft long enough for the contest the institution owes. It is the foot in the door.

The choice in front of every institution deploying these systems is therefore not whether their automated decisions will sometimes be wrong. They will. The choice is whether a wrong decision will remain a draft — visible, contestable, revisable, an error that can still be caught — or whether the speed and finality of the machine will be allowed to do what speed and finality always do, and convert it into a verdict before anyone could object. The first draft of an injustice looks exactly like an ordinary mistake. What makes it the first draft of an injustice is that nothing was built to let anyone write the second.

— Dispatches · Summit Cognitive