The witness who forgets

There is an old and useful distinction in the law between a witness who lies and a witness who simply cannot remember. The first is dangerous but at least adversarial; you can impeach a lie, contradict it, catch it changing shape under pressure. The second is something stranger and, in its way, more final. A witness who has genuinely forgotten cannot be cross-examined, because there is nothing left to question. The event happened, the witness was present, and then the memory closed over like water. What remains is a gap that no amount of skilled examination can reopen. We are now building systems that make consequential decisions and then forget them in exactly this manner, and we have not yet reckoned with what that forgetting costs.

The forgetting is rarely deliberate in the sense of malice. It is deliberate in the sense of design. A system processes an input, produces an output, and moves on, retaining only what it needs to do the next thing. The intermediate state — the particular configuration of evidence and rules and intermediate judgments that produced this specific result — is transient by default. It exists for the duration of the computation and then it is gone, overwritten by the next computation, because keeping it would cost storage and complexity and someone, somewhere, decided those costs were not worth paying. The result is a system that can act but cannot account. It is a witness who forgets the moment it has spoken.

Testimony requires a record to testify from

We sometimes talk about machine accountability as though it were a matter of asking the system to explain itself after the fact, as if explanation were a faculty the system could exercise on demand. But a system can only testify to what it has retained. If the state that produced a decision has been discarded, then anything the system says afterward is not testimony — it is reconstruction at best and confabulation at worst. The honest answer to "why did you do this" is, for most systems as they are built today, "I no longer have the information that would let me tell you." And a system that cannot answer that question is not being evasive. It is being accurate about its own emptiness.

This is why retention is not a feature you add to an accountable system; it is the substrate that makes accountability possible at all. The capacity to be held to account is downstream of the capacity to remember what you did in enough detail that someone else could check it. A Decision Receipt is, at its core, a refusal to forget — a commitment that the state which mattered at the moment of decision will survive the moment, carried forward intact, available to anyone with standing to ask. Without that commitment, every other guarantee is hollow. You can promise contestability, provenance, replay; but a system that has discarded its state can deliver none of them, because all of them require the past to still be there to examine.

A system that cannot remember what it did cannot be questioned about it. The gap left by forgetting is not neutral; it is exactly the shape of the account that will never be given.

Forgetting is a posture, not an accident

It is tempting to treat the loss of state as a technical inevitability, the natural entropy of computation, something that happens to a system rather than something a system was built to do. But every discard is a choice. Somewhere in the architecture there is a decision about what to keep and what to let go, and that decision encodes a prior judgment about whether this system expects ever to be questioned. A system designed to retain its decision-state is a system that anticipates being asked to justify itself and has prepared to do so. A system designed to forget is a system that has decided, in advance, that no such justification will be owed — or that if it is owed, it will not be given.

Read this way, the architecture of forgetting is a moral posture wearing the costume of a storage policy. When a system overwrites the evidence it consulted, freezes nothing, and keeps no replayable trace, it is not merely economizing. It is arranging things so that the question of accountability can never be made to land, because the material that would answer it will not exist by the time anyone thinks to ask. This is the same logic that, in a human institution, we would recognize immediately and name without hesitation: the shredded file, the unminuted meeting, the conversation held deliberately off the record. We do not consider those neutral. We consider them tells.

The difference with software is that the forgetting is silent and continuous and built in, so it does not announce itself the way a shredder does. It looks like nothing happening, because in a sense nothing is happening — the state simply fails to persist. And this invisibility is precisely what makes it dangerous. An institution that wanted to be unaccountable used to have to take active steps to destroy its records, steps that themselves left traces and could be discovered. A system that forgets by default needs to take no steps at all. The unaccountability is the resting state. You have to build against it, on purpose, to get anything else.

So the question to ask of any system that affects people is not whether it can explain itself in the abstract, but whether it has kept the thing that explanation would have to draw on. Does it retain the evidence as it stood, the rules as they were, the path the decision actually took? Or has it moved on, clean and empty, ready to act again and unable to say what it did the last time? The first kind of system can be a witness. The second cannot — and a system that was built so that it could never be a witness was, whether anyone admitted it or not, built to be unaccountable. The forgetting is the answer. We just have to be willing to hear it as one.

— Dispatches · Summit Cognitive