The uninsurable decision

There is a test of accountability more exacting than any regulator's, and it is not conducted in a hearing room. It happens at an underwriting desk, where someone whose own capital is on the line looks at a proposed risk and decides whether it can be priced. Regulators can be lobbied, standards can be softened, penalties can be treated as a cost of doing business. But an insurer asked to stand behind a risk it cannot measure does not argue about principle. It declines. That refusal, repeated across a market, is the most honest verdict any system of decisions will ever receive — not that it is wrong, but that no one can tell, and that not being able to tell is itself the disqualifying fact.

Insurers price what they can measure

Insurance is, at bottom, the business of converting uncertainty into a number. An underwriter takes a hazard nobody can predict at the level of the individual case — a fire, a crash, a lawsuit — and prices it at the level of the pool, where the law of large numbers makes the aggregate knowable even when the instance is not. This works because the underlying risk can be assessed: there is a history, a rate, a set of features that move the odds up or down. Underwriting is the discipline of measurement applied to the future. Where measurement is possible, almost anything can be covered, at some price.

The corollary is the part that matters here. Where measurement is not possible, the underwriter has only two moves. It can refuse the risk outright, or it can price it punitively — loading the premium so heavily to cover its own ignorance that coverage becomes, in effect, a refusal wearing a number. Neither move is a moral judgment. An unmeasurable risk is not necessarily a bad one; it may well be safer than risks the market insures cheerfully every day. But safety the underwriter cannot see is, from the underwriter's chair, indistinguishable from danger. What cannot be assessed cannot be priced, and what cannot be priced tends, by a logic no argument reverses, toward the uninsurable.

Now bring an automated decision system into that chair. As machine decisions begin to generate liability — for the loan denied, the claim rejected, the diagnosis missed, the applicant screened out — someone will be asked to insure against the losses those decisions produce. And the underwriter will ask what it always asks: let me see the risk. For a decision, seeing the risk means examining whether the decision was made soundly — on what evidence, under what rules, by a process that can be inspected after the fact. A system that can produce that account presents a measurable risk. A system that cannot produce it presents no risk the underwriter can name — only an unbounded one it must assume the worst about. The examinable decision is priceable. The unexaminable one is a hole in the ledger where a number should be.

The discipline of the counterparty's money

This is why insurability is a sharper instrument than regulation, and it is worth being precise about the reason. A regulator spends the public's authority and answers to politics; its rules can be delayed, diluted, grandfathered, or simply outlasted. An underwriter spends its own balance sheet and answers to its reserves. When it is wrong about a risk, it does not pay a fine — it pays the claim, out of capital it cannot get back. That is a form of seriousness no statute can legislate, because it is not borrowed conviction but the counterparty's own money talking. The pressure to make a risk legible comes not from a belief that legibility is virtuous but from the brute fact that illegible risk is where insurers go broke.

The precedent is old and physical. The reason factories came to have sprinklers and alarms was rarely conscience and rarely, at first, the law. It was the insurer, who would not write the fire policy — or would write it only at a ruinous rate — until the measurable safeguards were in place. The safeguard became a condition of coverage, and coverage was a condition of operating, so the safeguard became universal without anyone banning its absence. The lever reaching machine decisions is the same one. Coverage will come to require an examinable record of how decisions were made, for precisely the reason the fire policy came to require the sprinkler: not because the underwriter cares about the record for its own sake, but because without it the risk cannot be assessed, and an unassessable risk is one it will not carry.

The market will not ban the decision it cannot understand; it will simply decline to insure it, which for anything consequential amounts to the same thing.

Notice the shape of that withdrawal. The market does not prohibit; it abstains. It issues no ruling, holds no hearing, names no wrongdoing. It just declines to stand behind you — and for any activity consequential enough to need insurance, to be left standing alone is to be quietly removed from the field. This is discipline without adjudication, and it is far harder to appeal than a regulator's order, because there is no order to appeal. There is only a market that has looked at what you can show it and found nothing it can price.

Examinability as a condition of doing business

Follow the logic to its destination and the strategic picture reorders itself. The path to accountable AI may run less through legislatures than through the underwriters who will not otherwise carry the risk. If that is right, then the examinable account of a decision — the kind of record that lets an outsider check whether a decision was sound, the substance behind terms like admissibility and assurance — stops being a compliance nicety and becomes a condition of insurability. And insurability is, for most consequential activity, a condition of doing business at all. The firm that cannot produce an examinable account of its decisions will not be fined into producing one; it will find the market declining to stand behind it and will produce one to get coverage back, which is a more durable motive than any penalty.

Two honest qualifications keep this from being a fable. The first is that markets misprice. A soft market flush with capital and hungry for premium will cover bad risks for a while, waving through exposures it does not understand because someone else will if it does not. Insurability is not a smoke detector that goes off the moment danger appears; it lags, it errs, it is swayed by competition and optimism. The second is that insurability is a floor, not a ceiling. That a decision can be insured means only that its risk could be measured, not that the decision was good — plenty of terrible-but-legible risks get written every day. Examinability makes a decision assessable; it does not make it right. It is the precondition for judgment, not a substitute for it.

But grant the mispricing and the direction of travel still holds, because it is set by losses, not by opinion. A soft market can carry unmeasured risk only until the claims arrive. When they do — when the losses from unexaminable decisions accumulate into the actuarial record — the underwriting response is invariant: tighten the terms, and require whatever makes the risk measurable next time. That is how every hazard the industry now prices routinely made its way from uninsurable novelty to standard exclusion to condition-of-coverage. The unexaminable decision is early in that arc. The letter declining to renew has not been written yet for most who will one day receive it. But the desk it will come from is already staffed, and it is doing the one thing exhortation and regulation both struggle to do: making it cheaper, in the end, to be accountable than to be opaque.

— Dispatches · Summit Cognitive