The precedent of a machine

There is an old and hard-won idea at the center of how legitimate institutions decide things: like cases should be decided alike, and the reason a case came out the way it did should be written down where the next decider can find it. We call the result precedent. It is not merely a habit of consistency. It is consistency with a memory — a chain of recorded reasons that lets a later decision-maker see how an earlier one thought, agree or disagree on the record, and either follow the reasoning or give a reason for departing from it. The writing-down is not bureaucratic residue. It is the thing that turns a string of similar outcomes into a body of law.

Automated systems are consistent in a way no human institution ever managed. Feed the same inputs to the same model under the same configuration and you get the same answer, every time, without fatigue or mood or the slow drift of a tired judge at the end of a long docket. This looks, at first, like precedent's dream finally realized — perfect evenhandedness, the same case decided the same way without exception. But it is worth looking harder at what kind of consistency this actually is, because it is missing the one ingredient that made precedent worth wanting in the first place.

It is consistency without a recorded rationale. The machine produces the same output for the same input not because it is following a written reason that the next hard case can be measured against, but because it is the same function applied twice. There is no because in it that a person could read. The inputs go in, the answer comes out, and the regularity is real — but no one can say why this input maps to this answer, and no one can point to the reasoning that should govern the next case that sits awkwardly between two others. The system is repeating itself, faithfully, with nothing written down about what the repetition means.

Precedent is consistency you can argue with. A machine without a recorded rationale gives you consistency you can only obey. The first is law; the second is just a pattern that happens to hold.

Repetition is not justice

The difference matters most exactly where it is hardest to see — at the margin, in the case that does not cleanly resemble the ones before it. When a human institution meets a novel case, precedent gives it somewhere to stand. The decider reads the recorded reasons behind the nearest prior cases, judges how far they reach, and decides whether the new case falls inside or outside them — and then writes down that judgment, extending the chain. The reasoning is the bridge from the settled to the unsettled. A machine has no such bridge. It maps the novel case to an answer with the same blind regularity it brings to the routine one, and produces no account of whether the prior logic should have reached this far. It does not extend a line of reasoning, because there is no line of reasoning to extend — only a function that returns a value.

This is why mechanical consistency, however perfect, is not the same as justice. Justice was never only about treating like cases alike; it was about being able to say why two cases were alike, and being willing to defend that judgment to the person it bound. A system that returns identical answers to identical inputs while saying nothing about why those inputs deserve that answer has achieved the form of evenhandedness and abandoned its substance. It is fair the way a turnstile is fair — it does the same thing to everyone — and that is not the kind of fairness a person can appeal to, because there is no reasoning in it to appeal against.

A bad precedent can be overturned

Here is the part that should worry anyone who has watched these systems spread. A bad precedent, however entrenched, carries within it the seeds of its own correction. Because the reasoning was written down, a later court can read it, see exactly where it went wrong, name the flaw, and overrule it — and the overruling is itself reasoned and recorded, so the correction enters the same body of law and governs from then on. The whole apparatus is built to be wrong sometimes and to fix itself when it is, and it can only do that because the rationale is on the page. You cannot overrule a reason you cannot read.

A machine that is consistently wrong offers no such handle. There is no recorded rationale to examine, so there is nothing to identify as the flaw, nothing to name in an opinion that says here is where this went astray and here is the rule we adopt instead. You can change the model, retrain it, retune the threshold — but that is not overruling a precedent; it is replacing one silent function with another, and the new one will be just as mute about its reasons as the old one was. The error gets edited out without ever being articulated, which means it was never really understood, which means the same kind of error can return through a different door and no one will recognize it, because no one ever wrote down what the first one was.

The repair is not to make machines reason like judges. It is to insist that when a machine decides something consequential, the decision arrives with its rationale attached — the inputs that were actually consulted, the rule that was in force, the threshold that was applied, recorded at the moment of decision in a form a later reader can examine. A Decision Receipt is what turns a mute function into something that can hold precedent: not because the machine has learned to explain itself, but because the institution around it has finally written down what the decision rested on. Consistency we already have for free. What we have to build, deliberately, is the recorded reason that lets consistency be questioned, defended, and when it is wrong, overruled.

— Dispatches · Summit Cognitive