The cost of a clean story

There is a particular kind of explanation that arrives too clean. You ask why a decision went the way it did, and what comes back is seamless: a clear motive, a tidy chain of reasons, an outcome that follows from its premises like a proof. It is satisfying to hear, and that satisfaction is exactly the problem. Real decisions, the consequential ones, are made under pressure with partial information against a moving target. They have loose ends. An account with no loose ends has not captured a decision; it has replaced one with a story about it.

The reason is structural, and it has nothing to do with anyone lying. An explanation produced after the fact is optimized for a different thing than the decision was. The decision was optimized to be made — quickly, with whatever was on hand, good enough to act on. The explanation is optimized to be accepted. It is built, consciously or not, to satisfy a listener, to close a question, to make the asker stop asking. Those are plausibility objectives, not accuracy objectives, and a process tuned for plausibility will quietly discard whatever made the real decision look uncertain. The hesitation, the discarded option, the fact that two reasonable people argued for an hour — none of it survives, because none of it helps the story land.

So the smoothness is not a sign that the decision was sound. It is a sign that the account has been polished, and polishing is a subtractive process. Every burr it removes is a place where the decision might actually have been contestable. By the time the story is frictionless, the very features you would need in order to evaluate it — the alternatives weighed, the evidence that cut the other way, the moment the call could have gone differently — have been buffed off as noise. You are handed something with no handholds and told to trust it because it has none.

A decision you can argue with leaves you something to grip. A clean story is built to give your hand nothing to hold.

This inverts the instinct most institutions run on. The polished memo, the executive summary that resolves cleanly, the post-incident narrative with a single root cause and a single owner — these read as competence and get rewarded as such. But the cleaner the artifact, the more processing stands between you and what happened, and processing is where the contestable parts go to die. The tidiness you are admiring is a measure of how much has been removed, not how much has been understood.

Why machines make this worse

When the decision-maker is a system rather than a person, the clean story stops being a tendency and becomes a product. A model can be asked to produce a rationale, and it will, fluently, on demand — a paragraph of confident causation generated after the output it purports to explain. The fluency is not evidence that the paragraph describes the actual computation. It is evidence that the system is good at producing paragraphs that sound like explanations. The form is preserved perfectly; the connection to what was actually decided is exactly what is missing, and nothing about the smoothness will tell you that it is missing.

This is the case where the warning sign matters most, because the suspiciously tidy account is now cheap to generate at scale and arrives wearing the authority of the machine. For a decision to be admissible — to count as real once you press on it — the standard cannot be whether it produced a satisfying explanation. Satisfying is the thing the explanation was optimized for, and so it carries no information. The standard has to be whether the decision can be reconstructed from what was actually in front of it: the evidence it held, the rule it applied, the state it ran against. That is the opposite of a story. It is the raw material a story would have smoothed away.

How to use the tell

The practical move is to treat tidiness as a question rather than an answer. When an account is suspiciously clean, the right response is not relief but a specific demand: show me the part that did not fit. What was the closest alternative, and why was it dropped. What evidence pointed the other way. Where, exactly, could this have gone differently. A decision that was genuinely made can answer these, because the friction was real and the record of it exists. A story will stall, because the friction was the first thing sacrificed to make the story clean.

None of this is an argument for messiness as a virtue. It is an argument that the smoothness of an account should lower your trust by roughly the amount it raises your comfort, and that the two move together for a reason. The decisions worth scrutinizing are precisely the ones whose honest reconstruction would be a little rough — provisional, hedged, visibly made by something that did not know everything. Keep the roughness. It is the part that was true. The clean version is the part someone, or something, decided you should hear instead.

— Dispatches · Summit Cognitive