Replays, not explanations

Ask an automated system why it decided something, and it will tell you. That is the trouble. The answer arrives fluent, plausible, and shaped to satisfy the question — and you have no way to know whether it is true. An explanation is generated after the decision, by the same system whose conduct is in question, and there is nothing about being fluent that makes a statement accurate. We have spent years treating explainability as the goal of accountable automation. It is the wrong goal. The right one is duller and far harder to fake: reproducibility.

The distinction is easy to miss because both words sound like transparency. They are not the same act. An explanation is a narrative the system produces about its own reasoning — a description, after the event, of why it believes it did what it did. A replay is something else entirely. It takes the exact inputs that were in front of the decision, the exact rules that were in force, and runs the decision again, then checks whether it lands in the same place. The explanation asks you to believe the system. The replay asks you to believe nothing — it hands you the evidence and lets you check.

This is the difference, old and well understood outside software, between testimony and physical evidence. Testimony can be sincere and still wrong. It can be revised on the second telling, made more flattering, quietly corrected to fit what we later learned. A fingerprint cannot. It either matches or it does not, and when it does not, the mismatch is itself a finding. An explanation is testimony: the system's opinion about its own conduct. A replay is physical evidence: a fact you can verify without extending the system any trust at all.

An explanation is the system's account of itself. A replay is the decision happening again, in front of you, where the account can be checked against the world.

What a replay can and cannot prove

It is worth being precise about what reproducibility buys, because it is easy to oversell. A replay proves how a decision was made. It does not prove the decision was wise. You can have a perfectly reproducible record of a bad call — the same inputs, the same rules, the same wrong answer, every time. Reproducibility is not correctness, and anyone who tells you it is has confused the instrument for the judgment.

But hold the two failure modes side by side. A faithfully reproduced bad decision is diagnosable: you can see exactly what evidence was in front of it, exactly which rule turned that evidence into that outcome, and therefore exactly where to intervene — change the rule, fix the input, correct the policy that misfired. The decision is wrong, but the wrongness has an address. An unreproducible good decision has no such address. You cannot tell why it came out right, cannot tell whether it will come out right next time, cannot distinguish skill from coincidence. It was, as far as anyone can prove, luck. And luck is not an accountability mechanism. A wrong answer you can replay is worth more than a right one you cannot, because the first can be fixed and the second can only be hoped for.

This reframes the whole project. The point of accountable automation is not to extract a comforting story from the machine. It is to make consequential decisions checkable — by an auditor, a regulator, an affected party, a future version of your own engineering team — without requiring any of them to take the system's word. Explanation serves the system's reputation. Replay serves everyone else.

What replay actually demands

Replay is not free, and that is precisely why it is honest. To run a decision again, three things have to have been true at the moment it was made, and you cannot bolt them on afterward.

First, the inputs have to have been captured — not summarized, not described, but recorded as they actually were when the decision consumed them. A decision replayed against today's data is not a replay; it is a new decision wearing the old one's clothes. The inputs that matter are the ones that were in force then, frozen and addressable, so that the replay confronts what the original confronted.

Second, the rules have to have been frozen. Policies drift. A rule that governed on Tuesday may be amended by Thursday, and a decision evaluated against the amended rule is being judged by a standard it never operated under. A replay has to carry the governing rules with it, in the form they held at the time, or it tests the wrong thing. The question is never whether the decision satisfies today's policy. It is whether it satisfied the policy that was actually live.

Third, the decision has to be deterministic enough to land in the same place twice. Identical inputs and identical rules have to produce the identical outcome, or there is nothing to check against. This is the demand most automated systems quietly fail, because the convenient, fluent, generative parts of them are not built to repeat themselves. Determinism is a constraint you accept up front — it costs you some of the system's improvisational flair — in exchange for the only thing that makes the record mean anything. You give up a little freedom at the moment of decision to keep the decision honest forever after.

None of this is exotic. Captured inputs, frozen rules, deterministic execution — these are the unglamorous preconditions of being able to say run it again and see. They are also, not coincidentally, what separate a record that proves something from a log that merely asserts it. A log line says a decision happened. A replay lets you stand where the decision stood and watch it happen once more.

So when a system offers you an explanation, accept it for what it is: a hypothesis about its own conduct, possibly true, entirely unverified. Then ask the better question — not can you explain it, but can you run it again. The systems that can will be the ones we are still able to trust when the stakes are real. The systems that can only explain are asking, as they always have, to be trusted on their word.

— Dispatches · Summit Cognitive