The machine that forgot why

There is a specific kind of forgetting that automated systems are unusually good at, and it is not the forgetting of data. Data they hoard. What they forget is why. A decision is made — an application scored, a claim routed, a price set — and the outcome is written down carefully, propagated everywhere it needs to go, retained for years. But the reasoning that produced it, the particular path from inputs to conclusion, is not written down at all. It happened in the running of the computation and then it was gone, the way a thought is gone the instant after you have it. The system keeps the verdict and forgets the deliberation. This is the amnesia of automation, and it is by far the most consequential thing most systems throw away.

What makes this forgetting so easy to overlook is that nothing appears to be lost. The output is right there, sitting in the record, complete and confident. Nobody notices the absence of the reasoning until the day someone asks why — and the honest answer turns out to be that no one knows, and no one can find out, because the only artifact that survived is the conclusion. The reasoning was never persisted. It was a transient state in a process optimized for producing outputs, not for remembering how it produced them.

A human decision-maker who forgot every reason for every decision the instant it was made would be considered unfit for any responsibility whatsoever. We would not let such a person near a consequential choice, because the inability to say why is the inability to be held to account. And yet we deploy systems with exactly this deficit at enormous scale and call them efficient.

A system that remembers what it decided but not why has kept the one part that cannot be questioned and lost the only part that could be answered for.

Why the reasoning is the part that matters

The output of a decision is, in a strict sense, the least informative thing about it. Two decisions can produce the identical output for opposite reasons — one because the evidence genuinely supported it, the other because of a bug, a stale input, a proxy variable standing in for something it should never have touched. From the outside, from the output alone, you cannot tell these apart. They look the same. The difference lives entirely in the reasoning, which is exactly the part the amnesiac system did not keep. So when the output turns out to be wrong, or contested, or merely questioned, there is no way to distinguish a defensible decision that happened to err from an indefensible one that happened to be caught. The information that would separate them was discarded at the moment it was generated.

This is why the forgetting is not a storage problem you can solve by keeping more outputs. You already have the outputs. Keeping ten years of them does not help, because they were never the thing you needed. What you needed was the account — the evidence that was actually consulted, the rule that was actually applied, the state that would let you rerun the decision and watch it land where it landed. None of that is in the output. All of it was in the reasoning, and the reasoning is what the system was built to forget.

There is a subtler cost, too, which is that a system incapable of remembering why cannot learn from its own errors in any principled way. To correct an error you have to locate it — to find the point in the reasoning where the wrong thing entered. An institution holding only outputs can see that an outcome was bad but cannot see where it went bad, and so its only options are to distrust the whole system or to keep using it and hope. Corrigibility is a property of remembering why. The amnesiac machine forecloses its own improvement along with its own accountability.

Building a machine that remembers

The remedy is not to make systems slower or to force them to narrate themselves in prose, which would only produce a plausible-sounding story after the fact — the wrong kind of memory, a reconstruction rather than a record. The remedy is to persist, at the moment of decision, the specific things that constitute the reasoning: the evidence in front of the decision, the rules as they stood, and enough state to replay. A Decision Receipt is precisely a refusal to forget why — an artifact that captures the deliberation while it is still happening, so that the account survives the computation that produced it. The point is not that the system explains itself. The point is that it kept what an explanation would have to be checked against.

This costs something, and the cost is exactly why systems forget by default: remembering the reasoning is more work than emitting the output, and the pressure of throughput always favors the cheaper path. But the economy is false. The reasoning is cheap to keep at the moment it exists and impossible to recover once it is gone. An institution that saves the cost of remembering is borrowing against the day it will be asked why, and on that day the debt comes due in full — as an inquiry it cannot answer, an error it cannot locate, a decision it cannot defend.

So the question to ask of any automated system is not does it work but can it say why it did what it did, from a record, without making something up. If it can only show you the answer, it has forgotten the reasoning, which means it has forgotten the part you would need to trust it, correct it, or hold it to account. A machine that remembers only its conclusions is not a mind that has learned. It is a mind that cannot be questioned — and those are not the same thing, however much the output tries to suggest otherwise.

— Dispatches · Summit Cognitive