The decision that makes itself

There is a kind of error that does not announce itself, because by the time you go looking for it, it is no longer an error at all — it has become part of the ground beneath your feet. A model classifies a borderline case one way. That classification is recorded, stored, and folded back into the data the next model trains on. The next model, seeing the case already labeled, treats the label as fact. It was never fact; it was a judgment, and a close one. But the judgment has been promoted to evidence, and the evidence will now be cited in the judgments that come after it. The decision has begun to make itself.

This is the quiet structural feature of automated decision systems that almost nobody designs against, because it does not look like a failure mode. It looks like learning. The system improves; its outputs get folded into its inputs; it gets better at producing the outputs it already produces. From the inside, this is indistinguishable from competence. From the outside — from the position of the person whose case was the borderline one — it is something closer to a sentence with no appeal, because the original judgment has been laundered through enough downstream steps that no one can any longer point to where it was made.

The word for this is feedback, and feedback is not inherently bad. A thermostat uses feedback and keeps a room comfortable. The danger is specific: it is feedback that carries a decision forward as if it were an observation. When an output is treated as an input on the next pass, the system loses the ability to distinguish what it concluded from what it found. Those are different epistemic objects. One is a claim the system made and is responsible for; the other is a fact about the world that the system is merely reporting. Collapse them, and the system can no longer be held to account for anything, because everything it asserts now arrives pre-laundered as something it merely noticed.

How an error becomes ground truth

Consider the path a single early mistake takes. At the first step, the system produces a confident output on thin grounds — a label, a score, a recommendation. Nothing checks it, because the output is plausible and the volume is high. At the second step, that output is consumed by a downstream process that has no way of knowing it was a judgment rather than a measurement, so it is treated as settled. At the third step, the downstream result is itself recorded, and that record becomes part of the corpus the system learns from. By the fourth step, the original thin judgment is statistically reinforced: the model has now seen its own past output reflected back as confirmation, and it updates toward greater confidence in exactly the thing it should be least sure of.

The most troubling part is the direction of the confidence. A well-functioning belief should grow less certain when its only support is its own prior assertion. This loop does the opposite. Each pass through it increases the system's confidence precisely because the support is its own prior assertion — it mistakes the echo for corroboration. The system is not discovering that it was right. It is arranging the world so that it cannot discover it was wrong.

An output folded back in as an input stops being a claim the system can be wrong about. It becomes the floor it stands on.

What makes this hard to catch is that no single step in the loop looks irresponsible. The first output was reasonable on the evidence available. The downstream consumer was right to trust its inputs; it had no signal that they were untrustworthy. The training process did what training processes do. Each actor behaved correctly given what it could see, and the failure lives entirely in what none of them could see: the path the judgment took from one place to the next, and the moment it changed from a conclusion into a premise.

The record is the thing that can break the loop

You cannot break a loop you cannot find. This is the whole practical problem. If you suspect that a class of decisions has drifted — that the system is now confidently wrong about a category of cases it once handled tentatively — the first thing you need is to trace the lineage of a particular judgment back to the moment it was first made and ask what it actually rested on then. Without a record at each step, that trace is impossible. The judgment exists in the present as a fact, with no surviving memory of having ever been a choice. You can observe the output. You cannot observe its provenance, and provenance is the only thing that distinguishes a finding from an artifact of the loop.

A Decision Receipt at each step changes the shape of the problem. When every consequential output carries a record of the evidence actually consulted, the rules active at the time, and enough state to replay the call, the loop becomes legible. You can take a present-day judgment, follow it back to the output that seeded it, and inspect whether that seed was a measurement or a guess. You can find the borderline case that got promoted to fact and ask, at the source, whether it deserved to be. The record does not stop the system from learning from itself. It makes the learning auditable — it preserves the distinction between what the system concluded and what it found, so that the distinction can be checked rather than assumed.

This is the difference between a feedback loop that improves a system and one that calcifies its earliest mistakes. The first keeps a memory of its own reasoning, so a bad seed can be located and pulled. The second forgets, so a bad seed propagates undisturbed, gathering false confirmation at every pass until it is indistinguishable from truth. Provenance is what lets you tell which kind you have built, and replay is what lets you prove a particular conclusion does or does not survive being rerun against its real inputs.

The systems we are building now will, more and more, be trained on the records of their own past decisions. That is not avoidable and not, in itself, wrong. What is avoidable is doing it blind — letting outputs flow back into inputs with no record of the journey, so that the system quietly decides, again and again, that it was right, because it is now learning from nothing but its own remembered confidence. The cost of that is paid by whoever was on the wrong side of the first close call, long after anyone could say where the call was made. A record at each step is the difference between a system that can be corrected and one that has, without anyone deciding it should, become unappealable.

— Dispatches · Summit Cognitive