The genealogy of a mistake

When an automated decision turns out to be wrong, the first question asked is almost always the least useful one. Someone wants to know who is responsible. It is an understandable reflex, and in some settings a necessary one, but as the opening move it is a mistake about the nature of the mistake. Blame is a verdict you reach at the end of an inquiry, if at all. Treated as the start, it does the opposite of what an inquiry needs: it points everyone toward defending a position rather than finding out what happened. The question of fault is a question about people. The question worth asking first is a question about the decision.

That question is: where did this come from? Not who let it happen, but what produced it — by what chain of inputs, transformations, models, and rules did a wrong answer get assembled and emitted as if it were a right one? A bad outcome is the last link in a chain, and the chain is the thing that needs reading. Every automated decision has a lineage. Some datum was ingested, then scored, then combined with others, then weighed against a threshold, then returned. The wrong answer did not appear from nowhere. It descended from somewhere, through identifiable steps, and the genealogy of how it got there is the only thing that tells you what to repair.

This is why I think of provenance as a forensic discipline rather than a compliance one. Forensics does not begin by asking who to punish. It begins by reconstructing the sequence of events that produced the body of evidence in front of you, link by link, until the path from cause to outcome is laid bare. Only then, and only sometimes, does the question of responsibility even become answerable. Provenance applied to a decision is exactly this: the reconstruction of the path from the inputs that entered the system to the determination that came out, traced backward from the failure until you reach the link where the failure was born.

Blame asks who should suffer for the error. Provenance asks where the error entered. Only one of those questions has an answer that prevents the next error.

Tracing backward through the chain

To see why the genealogy matters more than the verdict, follow what each approach actually yields. Suppose a decision was wrong and you have settled on someone to blame. You now know who. You do not know what. You do not know whether the fault was in a stale input, a miscalibrated model, a rule that was correct in general but wrong for this case, or an interaction between components each of which was behaving as designed. You have assigned responsibility without locating the defect, which means the defect is still there, and the next case that travels the same path will produce the same wrong answer with a different name attached to it.

Now suppose instead that you traced the outcome backward through its lineage. You followed the determination to the threshold that produced it, the threshold to the score it was applied against, the score to the model that generated it, the model to the inputs it consumed, and the inputs to their sources. Somewhere along that chain you found the link where a defensible step became an indefensible one — the stale record, the misread field, the rule that fired when it should not have. You have located the defect. And a located defect is a fixable defect, because you can see precisely which link to change and you can confirm, by re-running the chain, that changing it changes the outcome. That is the whole difference. A mistake with a genealogy can be corrected at its source. A mistake known only by its outcome can only be regretted.

This is where provenance and replay become two halves of one capability. Provenance gives you the chain — the recorded lineage of which input fed which transformation fed which decision. Replay lets you walk that chain under controlled conditions, holding everything fixed except the link you suspect, and watching whether the outcome moves. Without the recorded chain, replay has nothing to walk. Without replay, the chain is a hypothesis you cannot test. Together they turn a wrong answer from an accusation into a diagnosis, and a diagnosis is the only thing a repair can be built on.

A mistake without a genealogy will recur

The deepest reason to prefer provenance to blame is that the two have opposite relationships to the future. Blame closes a case. Provenance prevents its recurrence. When an organization responds to a bad automated decision by finding someone at fault and moving on, the system that produced the decision is unchanged, which means its disposition to produce that decision is unchanged. The error has been narrated, not removed. It is still latent in the same chain, waiting for the next case to arrive at the same junction, and it will. An error you have explained by reference to a person is an error you have agreed not to find.

An error you have traced through its lineage is an error you have a chance of ending. You know which link produced it, you can change that link, and you can verify the change by replaying the cases that the link affected. The correction is targeted, testable, and durable, because it operates on the mechanism rather than on the personnel. This is the practical payoff of treating provenance as forensics: it converts a one-time apology into a permanent fix, and it does so precisely because it refused, at the start, to ask the question everyone wanted to ask.

So when an automated decision goes wrong, resist the pull toward the verdict. The verdict can wait; it will be better-informed for waiting, and it may turn out that no person was the cause at all, only a chain that no one had been asked to trace. Begin instead with the genealogy. Read the lineage backward from the failure until you find where it entered, and you will have done the one thing that both serves the person who was harmed and protects the next person from being harmed the same way. A mistake you can trace is a mistake you can fix. A mistake you can only assign is a mistake you have promised to repeat.

— Dispatches · Summit Cognitive