The decision in the dark

When we imagine a decision going wrong, we usually imagine it acting on something false. It believed a bad number, trusted a corrupted source, was fed a lie and acted on it. This is a real failure and a familiar one, and much of the machinery of good judgment is built to guard against it — to check the inputs, to verify the sources, to make sure the facts a decision rests on are true. But there is a second kind of failure that this machinery does not catch, and it is quieter and in many ways more troubling. A decision can be built entirely on true facts and still be wrong, because the fact that would have changed the answer existed, and was available, and was simply never retrieved.

Call it the decision made in the dark — not because the light was bad but because the decision never turned toward the part of the room where the relevant thing was sitting. The record that would have mattered was in the file, but the file was not opened. The contradicting fact was one query away, but the query was never run. The context that would have reframed everything was accessible, and the decision proceeded without it, not because it was hidden but because nothing reached for it. The decision was not misled. It was under-informed, in a specific and correctable way: it failed to consult what it could have consulted.

The failure that leaves no fingerprints

What makes this kind of failure so hard to catch is that it is invisible in the outcome. A decision built on a false fact often betrays itself eventually; the false fact collides with reality somewhere down the line, and the collision is a clue. But a decision that simply failed to retrieve a true fact leaves no such clue. The outcome looks complete. It is internally consistent, defensible on its own terms, indistinguishable at a glance from a decision that considered everything. Nothing about the answer announces that it was reached without the one input that would have overturned it. The missing fact does not appear as an error. It appears as nothing at all, which is exactly why it is so dangerous.

This is the asymmetry at the heart of the problem. What a decision used is, at least in principle, present in the decision — it is there in the reasoning, visible in the result, available for inspection. What a decision failed to use is present nowhere. It is an absence, and absences do not show up in the thing that was produced; they show up only against the background of what could have been produced instead. You cannot find the unexamined fact by studying the decision, because the decision was made precisely by not touching it. You can find it only by comparing what the decision looked at against what was there to be looked at — and that comparison requires a record of the looking.

A false fact eventually argues with reality and gives itself away. A fact never consulted never argues with anything. It just quietly fails to have been there.

A record of what was looked at, and what was not

The remedy is a record not merely of what a decision concluded but of what it consulted — the specific evidence it actually retrieved and considered, in enough detail that the set of things it looked at is legible. This sounds like a modest thing, a log of sources. It is not modest at all, because a record of what a decision looked at is, by implication, a record of what it did not. Once you can see the bounded set of facts a decision actually reached for, you can hold it up against the facts that were available, and the gap between the two is the darkness the decision was made in. The unexamined fact, invisible in the outcome, becomes visible in the record — as the thing that was there, and available, and absent from the set consulted.

This is the only way to distinguish a decision made on the full picture from one made in the dark, and the distinction matters enormously even when the two produce, in a particular case, the same answer. A decision that considered everything relevant and reached its conclusion has earned that conclusion. A decision that reached the same conclusion while never consulting the record that might have changed it got lucky, and luck is not a property you can rely on across the next thousand decisions. The record of consultation is what lets you tell the earned answer from the fortunate one, and a system that keeps no such record cannot tell them apart even about itself.

The discipline of admissible reality treats this as central rather than incidental: a decision record has to capture the evidence actually consulted, so that the completeness of a decision — not merely its correctness on the facts it happened to use — can be examined afterward. The question the record makes answerable is not only was this true but was this all, and the second question is the one that the outcome alone can never answer. What was retrieved is written down. What was available but not retrieved can then be seen against it. And the fact that would have changed everything, which left no mark on the decision because the decision never reached for it, finally leaves a mark somewhere — in the space between what was consulted and what was there.

So when a decision affects someone and turns out to have been wrong, the first question is often the wrong one. We ask what false thing it believed. The better question, frequently, is what true thing it never looked at — and whether the record is detailed enough to tell us that it was there all along, waiting in the dark the decision never turned toward.

— Dispatches · Summit Cognitive