The moral hazard of the machine that cannot be blamed

Moral hazard is a term the insurance trade invented for a problem it kept running into: a person who is fully insured against a loss behaves, on average, less carefully than a person who is not. Give someone comprehensive fire coverage and, at the margin, they are a little slower to replace the frayed wiring; give a driver a policy that pays for every dent and they park a little less cautiously. Nobody has to become reckless on purpose. The mechanism is quieter than that. When the cost of a bad outcome falls on someone else, the incentive to prevent it weakens, and behavior drifts — not because people are bad, but because care is expensive and the reason to pay for it has been removed. Economists generalized the insurance case into a broad law: whenever the party who takes a risk is not the party who bears its cost, the risk gets taken more freely than it should.

It is worth seeing automation through this lens, because it is one of the most powerful pieces of insurance an institution has ever bought, and almost nobody accounts for it that way. When a decision is made by a person, that person is exposed — nameable, answerable, potentially blamed. When the same decision is routed through a system, the exposure thins out. The loan officer who declines you is a human who chose; the model that declines you is a process that ran. If the decision proves wrong, there is no one who decided it — only a system that produced it, an operator who says the system produced it, and a vendor who says the operator deployed it. The blame that would have concentrated on a person diffuses across a supply chain until it lands nowhere. Automation, whatever else it does, sells the institution a policy against being blamed. And a policy against blame produces exactly what every policy against consequences produces: a quiet, structural decline in the care taken to avoid the thing insured against.

The insured decision

This is the moral hazard of the machine that cannot be blamed, and it is more corrosive than the ordinary kind because the coverage is invisible and nobody signed for it. A fire policy is a document; everyone knows the coverage exists and can reason about how it changes behavior. The blame-coverage that automation provides is nowhere written down. No one decided that routing a decision through a model would dilute responsibility for it; that is just a side effect of the routing. So the institution enjoys the coverage without ever acknowledging it holds a policy, and enjoys, therefore, the relaxation of care that the coverage induces, without ever noticing the relaxation. The wiring frays and nobody is even aware they stopped checking it, because no one framed the decision to automate as a decision to insure against blame — but that is part of what it was.

You can watch the care drain out in concrete ways. A threshold gets set and never revisited, because no one owns the consequence of it being wrong. A model degrades and the degradation is tolerated, because the loss it causes falls on applicants who cannot see it and cannot bill anyone for it. An edge case that a careful human would have escalated gets processed silently, because the system has no incentive to flag its own uncertainty and no one downstream is exposed enough to demand that it does. None of these is a scandal. Each is the perfectly ordinary behavior of an insured party economizing on care it is no longer paying for the absence of. The decisions are, in the precise economic sense, over-taken — made too freely, checked too little — because the party making them has been insured against their cost.

Nobody decided that routing a decision through a machine would dilute the blame for it. But that is part of what automation buys — and the care that blame used to enforce drains out through a policy no one admits they hold.

The record is the deductible

The insurance trade did not respond to moral hazard by lecturing the insured about carelessness; it knew that would fail, because the insured are behaving rationally given their coverage. It responded by redesigning the coverage so that the insured kept some exposure to the loss. The deductible is the central instrument: make the policyholder bear the first slice of every claim, and their incentive to prevent the loss returns, because now it is partly their loss again. Co-insurance does the same. The whole art is to re-attach the actor to a portion of the consequence they had been fully insured against, so that care, which is only ever bought by exposure, comes back.

The record is the deductible for the insured decision. What automation insures against is not the loss itself but the blame — the traceable connection between a bad outcome and the party who produced it. An account restores that connection. When a decision leaves behind a reconstructable record — what it rested on, what rule it followed, who set the threshold, who deployed the model — the diffusion of blame across the supply chain stops, because now the outcome can be traced to specific choices by specific parties. The operator can no longer hide behind the system produced it, because the record shows the system produced it under conditions the operator chose. The vendor can no longer hide behind the operator deployed it, because the record shows what the vendor supplied. Responsibility re-concentrates. And the moment it re-concentrates, the parties are re-exposed to the cost of getting the decision wrong — which is to say, their incentive to take care returns, for the same reason a deductible returns it: they are on the hook again for a slice of the loss.

This is the argument for the account that has nothing to do with the affected party's right to contest, real as that right is. It is an argument about the decider's own behavior. A decision system that leaves no reconstructable account is a fully insured system — its operators shielded from blame, and therefore, predictably, less careful than they would be if exposed. A decision system that records how each decision was reached is a system with a deductible: its operators know that any decision can be traced back to the choices that produced it, and that knowledge, all by itself, disciplines the choices. The record does not only let others hold the decider to account after the fact. It makes the decider more careful before the fact, because a party that expects to be traceable behaves like a party that bears its own risk — which, once the record exists, it does. Take the coverage away and care comes back. That is what the account is for, and it is why a system that cannot be blamed is not a safe system. It is an uninsured public wearing a fully insured operator's carelessness.

— Dispatches · Summit Cognitive

---