The chain of thought is not the reason

When an agent works through a problem, it can show you its thinking. It lays out the steps — considered this, weighed that, ruled out the other, arrived here — and the effect is remarkable. You feel, reading it, that you have been let into the room where the decision was made. The prose is orderly. The reasons connect. Each step appears to follow from the one before, and the action at the end appears to follow from all of them. It is the most convincing account of a machine's decision anyone has yet produced, and that is precisely the problem. It is convincing in the way a well-told story is convincing, which has never been the same thing as being the record of what happened.

Call this trace the chain of thought — the visible, step-by-step narration an agent emits alongside its work. It is genuinely useful. It helps a developer debug, helps a reviewer follow along, helps a person decide whether to trust the thing at all. I am not arguing that it is worthless. I am arguing against one specific, tempting mistake: taking the chain of thought to be the reason the agent acted — the account of record, the thing you would point to if someone asked why the action was taken and on what basis. It is not that. It looks exactly like that, and it is not that, and the gap between the two is where a great deal of misplaced trust is about to accumulate.

The story that sounds like a reason

Narrated reasoning seduces because it satisfies, fluently, the exact hunger an account is supposed to satisfy. When a consequential thing happens, we want to know why, and we recognize a good answer by its shape: premises, a line of inference, a conclusion that sits at the end like a destination. The chain of thought supplies that shape on demand, in complete sentences, without hesitation. It reads like the transcript of a deliberation. And because it is expressed in the first person of the system's own working — I should check this before doing that — it carries the intimate authority of a confession. You are not being told what the system did; you feel you are being shown what it was thinking as it did it.

But notice what the trace actually is. It is text the system generated in the course of producing the action — generated with the action, as part of the same forward motion, not extracted afterward from some separate ledger of causes. It is an output, sitting alongside the other outputs. The fact that it is phrased as reasoning does not make it the reasoning, any more than a novel written in the first person makes its narrator a real person who did the things described. A fluent account of a decision and the decision's actual determinants are two different objects that happen, here, to be produced by the same pen. We are being handed the memoir and invited to mistake it for the evidence.

Narration is not causation

The load-bearing point is simple and easy to lose under all that fluency: the narration can diverge from what actually drove the action, and you cannot tell from the narration alone whether it has. What moved the agent were the concrete things — the tool outputs it received, the context it was working from, the objective it was pointed at, the learned weights that turned all of that into a next step. The chain of thought is a story generated in the neighborhood of those forces. Sometimes it faithfully reflects them. Sometimes it reflects what a plausible account would look like given the situation, which is not the same thing and can come apart from it in exactly the cases that matter most — where the real driver was something the tidy story omits, or where the story rationalizes a step the actual computation reached by another route entirely.

This is why a record that is a self-portrait is worth so little when the stakes are real. A self-portrait is composed by its subject, for an audience, and it flatters by construction — not through dishonesty, but because a fluent account is optimized to read as reasonable, and reading as reasonable is a different objective than being the causal truth. This is the agentic version of a mistake the rest of this series has named repeatedly: trusting a system's account of itself. A system describing its own reasoning is the most interested party in the room, testifying about its own conduct, in language it is extraordinarily good at making sound like a reason. We would not accept that from a person under scrutiny. We would ask what the account can be checked against.

A system's account of its own reasoning is an autobiography, and no one has ever been acquitted on the strength of the defendant's memoir.

The danger is not that agents lie. It is subtler and worse: that they produce, effortlessly and in volume, accounts that are sincere-sounding, internally coherent, and untethered from what determined the act — and that we, starved for any window into machine decisions, will take the window for the room. The better the narration gets, the stronger the temptation, and the narration is getting better every quarter. A false record that announced itself would be harmless. This one arrives wearing the face of exactly the thing we were looking for.

The account that can be checked

The account that matters is not the one the system tells; it is the one you can reconstruct. The inputs the agent actually received. The tools it actually called and what they actually returned. The context that was actually in front of it. The rule or objective that was actually in force at the moment it acted. These are not the agent's opinion about its decision — they are the decision's circumstances, recorded independently of the agent's narration of them, and against them the action can be replayed. Reconstruction asks a question the memoir cannot answer: feed the same inputs back through, and does the same action follow? That is checkable by someone who trusts the system not at all, which is the only kind of check worth having when the thing being examined is the party under suspicion.

A replay beats an explanation for the same reason evidence beats testimony — one can be run again by a skeptic, the other can only be believed. This is what a Decision Receipt is for: not to capture the system's story about why, but to preserve enough of the real inputs, tools, context, and rule in force that the action can be reconstructed and contested without recourse to the system's self-description at all. Provenance over autobiography. The reconstructable circumstances, not the narrated ones.

None of this means the chain of thought should be thrown away. Keep it — as one artifact among many, a useful record of what the system said it was doing, sometimes a genuine clue to what it was in fact doing, always a thing worth having when you are debugging or deciding whether to trust the run. What it must never be is the reason. The moment the narration is promoted from artifact to account — the moment "here is what the agent said it was thinking" is allowed to stand in for "here is why the action was taken and on what basis" — you have quietly let the interested party write the record of its own conduct, in the most persuasive prose available, and called the result an explanation. The story is not the reason. The reason is the thing you can run again.

— Dispatches · Summit Cognitive