DISPATCHES · Summit Cognitive

← All dispatches

AccountabilityThe CasebookJuly 27, 20265 min read

The claim the model denied

When an automated system denies a claim, the policyholder meets a decision that was never made by anyone — and discovers that the promise they paid for was quietly conditioned on a reason no one will show them.

Consider a person who files a claim. It could be a health plan, a disability policy, a homeowner's line — the shape of the story is the same across all of them. Something went wrong that they had bought a promise against. They gathered the paperwork, they submitted it, and some days later a letter arrived. The letter was courteous and it was final. It said the claim did not meet the terms of the policy, and it cited a clause — a section number, a subsection, a phrase like not medically necessary or outside the scope of covered loss. What the letter did not say, because letters of this kind never do, was the actual thing that happened inside the system: that the claim had been scored, flagged, sorted against a model of what claims like this tend to be, and routed to denial before any person formed a judgment about it. The clause was printed afterward, to give the outcome a place to stand.

This is worth sitting with, because it describes a genuinely new situation and not just a familiar frustration dressed up in software. The policyholder has not been turned down by an adjuster who weighed the file and reached a hard conclusion they could be pressed to defend. The policyholder has been turned down by a determination that no one made and that no one, in the ordinary sense, holds. And the reason on the page is not the reason for the decision. It is the reason the decision is permitted to cite. Those are different objects, and the whole difficulty lives in the gap between them.

The denial no one made

Adjudication by model works by comparison. A claim is not so much read as located — placed against a distribution of prior claims and assigned a position: likely eligible, likely not, likely worth a second look, likely fraud. The position does the work. Whatever pushed the claim across the line — a diagnosis code that pattern-matches to denied claims, a repair estimate above some learned threshold, a timing coincidence that the model has learned to distrust — that specific trigger is the real basis of the outcome. And that is exactly the thing the denial letter will not contain.

What the letter contains instead is a clause. The clause is not a lie; the policy does say what it says, and the denial is very likely defensible under it. But the clause is a legal formality, not an account. It answers the question under what authority may we decline. It does not answer the question the policyholder is actually asking, which is why mine. Between those two questions sits everything the system knows and will not disclose: the score, the feature that drove it, the comparison that condemned the claim. The policyholder is handed the citation and denied the reason, and is then invited to appeal a determination whose grounds have never been stated.

A claim is a promise being called

To see why this matters, it helps to be precise about what insurance is. An insurance policy is not a service or a product in the ordinary sense. It is a promise — a paid-for commitment to pay on a defined contingency. The policyholder performs first, and performs continuously: the premiums are paid, the coverage is relied upon, the other precautions one might have taken are foregone because the promise is supposed to be there. The whole arrangement is a structure of reliance built on a commitment to be honored later. And the claim is the moment the promise is called. It is the single instant the entire relationship was arranged to produce. Everything before it was preparation for this.

Which means the denial is not one decision among many. It is the promise being tested — and, if it fails, the promise being broken. A promise that can be adjudicated by a model the promisor will not show is a promise held on terms the promisor has reserved the right to restate after the fact. The clause was fixed in advance; the operative reason was assigned at the moment of the claim, by a system whose criteria the policyholder never saw and cannot inspect. What was sold as a fixed commitment turns out to have a floating condition attached to it, and the floating condition is the one that decides.

The asymmetry here is the crux. This is not a decision to grant a favor, where a thin account might be forgivable because nothing was owed. It is a decision to withhold something owed, from the party that already performed. And a decision to deny what is owed demands a stronger record than a decision to grant what is not — the stakes are highest precisely where, in these systems, the account is thinnest. That inversion is the whole problem in a sentence. The moment that most needs an answerable reason is the moment engineered to produce none.

An insurer that denies by a model it will not show has not adjudicated your claim; it has amended your policy in the dark, and dated the amendment before you bought it.

What the account owes the policyholder

The remedy is not that a human must decide every claim, and it is not that models must be banned from adjudication. Volume makes automation inevitable and often makes it fairer than the tired adjuster at four in the afternoon. The remedy is that an adverse determination must carry an account addressed to the person it falls on. Three things, concretely.

First, the actual basis of the denial, rendered in a form the policyholder can rebut. Not the clause — the operative reason: the fact, the finding, the threshold the claim ran into. A reason you cannot contest is not a reason; it is a notice. If the estimate was judged excessive, the policyholder should be able to answer with a second estimate. If the treatment was judged unnecessary, they should be able to answer with the record that shows it was. You cannot argue against a comparison you were never shown.

Second, the rule that was actually in force, frozen into the record as it stood at the moment of decision — the coverage terms, the criteria, the version of the model's policy that applied — so that the question did this denial follow the commitment that was actually made can be answered later without anyone choosing, after the fact, which version of the rules to quote. A promise whose terms can be re-narrated at the moment it is called is not a promise anyone can hold.

Third, an appeal that reaches someone who can actually pay — a route to a party with the standing and the authority to reverse the outcome, not a queue that loops back to the same model with the same inputs and returns the same result wearing the word reviewed. Contestability that cannot change anything is theater, and everyone in the loop eventually learns to treat it as such.

There is an honest complication, and the argument is weaker if it hides from it: fraud models genuinely must stay partly confidential. Publish the exact triggers and you publish the manual for beating them. But that is a narrower constraint than it is usually made to carry. What must stay secret is the detailed operation of the detector. What need not — what must not — stay secret is that an adverse determination was made, on what basis in terms the claimant can meet, under which rules, and with an auditable trail behind it. Separate the two. The confidentiality of the method is not a license to deny the existence of a reason. A Decision Receipt can preserve the operative basis, freeze the rule, and stand up to an appeal without ever exposing the model's internals. The insurer keeps its detector. The policyholder keeps the promise. Those are not in tension; the pretense that they are is how the account gets withheld.

The scenario above is illustrative — a composite drawn to show a pattern, not an account of any real person, company, or event.

— Dispatches · Summit Cognitive

Continue from here

Turn the argument into a practice.

Get new dispatches, assess how your organization handles consequential decisions, or explore Summit Cognitive.