Show your confidence

Your system emits a verdict. Approved or denied, this class, this action, escalate or clear. It comes out clean and single-valued, a decision with no visible seam, and by the time it reaches the person it affects it looks exactly like every other decision the system has ever made. But behind that verdict there was a number, and the number was not always the same distance from the line. Sometimes the system was all but certain. Sometimes it was a hair past the threshold, resolving a call it could as easily have resolved the other way. The verdict erased that difference. It handed everyone downstream the answer and threw away the one fact that tells them how much to trust it. Stop doing that. Capture how close the call was, and carry it forward.

This is not a demand for more output. It is a demand for the piece of output you already computed and then discarded. The model produced a score before your code turned it into a verdict; the distance between that score and the cutoff is decision-relevant information, and it is information about this decision, not a generic property of the system. Discarding it is not neutral housekeeping. It is deleting the signal that would let a reviewer know which decisions deserve a second look, let a downstream system weight a shaky input appropriately, and let the affected person understand that their case was not the open-and-shut matter the bare verdict implies.

The verdict that erased the margin

Consider what a bare decision does to the three parties who inherit it. The human reviewer, handed a queue of verdicts that all look alike, has no way to tell the borderline calls from the certain ones — so either they re-examine everything, which does not scale, or they trust everything, which means the coin-flips sail through unexamined alongside the slam-dunks. The downstream system, consuming the decision as an input, treats a barely-past-the-line classification with the same weight as an overwhelming one, and compounds a shaky call into a confident chain of consequences. And the affected person, told only the outcome, cannot know that the system nearly went the other way — that on a slightly different day, or with the threshold set one notch looser, they would have been approved rather than denied. All three are blinded to the same thing, and it is the same discarded fact that would have unblinded all three.

The tell is that a borderline call and a near-certain one are rendered identically. That identity is a design choice, made silently, almost always by omission. Nobody decided that a 51 percent decision and a 99 percent decision should look the same to the people they land on; it just happened, because the schema had a field for the outcome and no field for the margin. The verdict is the part that acts, so the verdict is the part that gets persisted. The confidence behind it is the boring intermediate value, so it slips away — and with it goes the ability of anyone downstream to calibrate their trust to the actual strength of the call.

Confidence is decision-relevant

The directive follows directly. Record a measure of confidence alongside each consequential decision — not as decoration but as a first-class part of the record, captured at the moment the score becomes a verdict, bound to the same account as the outcome it qualifies. Then use it. Route the low-confidence decisions somewhere different from the high-confidence ones: to a second reviewer, to a human with authority to override, to a slower path that can absorb the extra scrutiny a shaky call warrants. The whole point of knowing a decision was close is to be able to treat it differently, and a confidence value you record but never route on is a confidence value you have not really used.

And where it helps the affected party understand and contest the decision, surface the margin to them too. A person told they were denied is owed less than a person told they were denied on a call the system rated as narrow — because the second person knows where to push. The margin tells them the decision was contestable in fact and not merely in principle, and it points them at the seam. This is continuous with what a Decision Receipt already owes the party who wants to fight it: the confidence is part of the evidence, and a record that hides how sure the system was has withheld a material fact about the decision it purports to account for.

A decision that hides how close it was is not being decisive; it is concealing the coin-flip from the person it landed on.

The honest hard parts

Now the parts that keep this from being a slogan. First: a raw model score is not a calibrated probability. A 0.9 output is not a ninety percent chance of being right — it is a number the model emits with its own internal geometry, and treating it as a probability is a category error that manufactures false precision. So the directive is calibrated confidence, honestly represented, not whatever the model happens to print. If you have not measured how often your 0.9-scored decisions are actually correct, you do not yet have a confidence to show; you have a raw score wearing a probability's clothing. Represent what you can honestly stand behind, label it for what it is, and do not dress up an uncalibrated number as a guarantee. Showing your confidence and inventing your confidence are opposites.

Second: confidence is not a justification. “The system was confident” is not a defense of a decision that was wrong — a confidently wrong call is still wrong, and high confidence attached to a bad outcome is an aggravating fact, not a mitigating one. The margin exists to inform trust, not to launder mistakes. The instant confidence becomes the reason a decision gets waved through unexamined, it has been inverted from a tool for scrutiny into a shield against it. Guard against that. The high-confidence path can move faster, but it cannot become the path where nobody ever looks.

Finally, remember what the margin is measured against. Confidence is distance from a threshold, and that threshold is a policy choice — write it down before you tune it — so “how close was this call” is always relative to a line someone chose to draw. Move the line and yesterday's comfortable margin becomes today's coin-flip. That is exactly why the confidence has to be recorded at the moment of decision, against the threshold in force at that instant: so that the closeness of a past call can be understood on its own terms, and not silently re-scored against a cutoff that has drifted since. Record the margin, route on it, surface it where it helps — and never let it stand in for the judgment it was only ever meant to inform.

— Dispatches · The Field Manual · Summit Cognitive