The model is not the witness

A witness is not someone who knows the answer. A witness is someone who can be asked how they came to know it, and then asked again, differently, by someone who would prefer they were wrong. That second part is the whole of it. We do not trust testimony because it is confident or fluent or internally consistent. We trust it because it survived contact with a hostile question. Strip out the cross-examination and what remains is not a weaker form of testimony. It is a different thing entirely — an assertion — and we have four centuries of rules built precisely to keep the two apart.

This is the problem a model presents the moment its output starts deciding something that matters. The model will produce an answer. Ask it, afterward, how it got there, and it will produce that too — a clean, plausible account of its own reasoning, delivered in the same confident register as the answer itself. The trouble is that the account is generated by the same process that generated the decision, after the fact, with no privileged access to what actually happened inside. It is not a recollection. It is a second output, shaped to be satisfying. You cannot cross-examine it, because there is no it that was present at the decision and is now under pressure to be honest about it.

So the model is not the witness. It cannot be sworn, because an oath binds a party who persists across time and can be held to what they said. It cannot be impeached, because there is no prior testimony to contradict — only a fresh story each time you ask. Whatever it tells you about its own reasoning has exactly the evidentiary weight of a defendant narrating their own innocence: it may be true, but its being told by the interested party is the reason it cannot, by itself, count.

You cannot put a model under oath. You can only put under oath the record of what it actually had in front of it.

This is where the instinct goes wrong. Faced with an inscrutable decision, the reflex is to demand that the model explain itself — to make the system more articulate, more forthcoming, better at narrating its own interior. But a more eloquent witness who still cannot be cross-examined is not more trustworthy. They are more dangerous, because fluency reads as credibility. The fix is not a better account from the model. It is to stop asking the model to be its own witness at all.

What stands in for testimony

When the decider cannot testify, the law already knows what to do, because it has handled this case for as long as there have been decisions made by things that cannot speak. A ledger cannot take the stand. A thermostat cannot be cross-examined. A door log does not remember opening. And yet records like these decide cases every day, because the question shifts from what does the thing say it did to what was preserved, contemporaneously, of what it did — and that record can be tested without the thing's cooperation. We do not ask the lock to explain itself. We ask whether the log was kept honestly, kept at the time, and kept beyond the reach of anyone with a reason to revise it.

That is the substitution. What stands in for testimony, when the thing that decided cannot be put under oath, is the preserved record of the inputs it actually had, the rule it actually applied, and the state it actually ran against — captured as the decision was made, not reconstructed afterward in a tone of helpful explanation. The record is cross-examinable in the way the model is not. You can challenge how it was kept. You can show it was incomplete, or tampered with, or that the rule on file is not the rule that ran. Every one of those is a real question with a real answer, which is exactly what a model's account of itself can never offer.

The difference is not cosmetic

It is tempting to treat this as a distinction without much practical weight — the model explains, the record explains, both produce a paragraph about why the decision came out the way it did. But the two paragraphs answer to entirely different authorities. The model's explanation answers to nothing; it can be regenerated, reworded, made more agreeable, and there is no fact of the matter it can be measured against. The record answers to what was true at the moment of the decision, and it can be wrong in ways that are checkable. One is rhetoric. The other is evidence. The whole of decision admissibility lives in that gap.

The institutions now wiring models into consequential decisions are, mostly without noticing, choosing which of those two they intend to rely on. If the plan is to ask the system, later, why it did what it did, they have chosen the witness who cannot be cross-examined and dressed it as accountability. If instead they have built so that the decision leaves behind a contemporaneous, tamper-evident record of what it ran on, they have chosen the thing that can actually be put under oath. The model was never going to be the witness. The only question that was ever open is whether anyone bothered to keep one.

— Dispatches · Summit Cognitive