The premium the model set

Picture two people on the same block, in the same kind of house, driving the same year of the same car. Neither has filed a claim in a decade. Neither has a mark against them that either could name. When the renewal notices arrive, one premium is meaningfully higher than the other — not by a rounding error, by a sum that matters at the end of the month. Each of them sees a single figure and a line that says the rate reflects their risk profile. Neither is told what, in that profile, moved the number. There is nothing to read, because nothing was written down in a form a person could read.

Insurance is the pricing of risk; that is the honest thing about it, and I do not want to pretend otherwise. A premium is supposed to differ from person to person, because risk differs from person to person, and a company that charged everyone the same would be transferring money from the careful to the reckless. The difficulty is not that a model set two different prices. The difficulty is that no one can say why it set the two it did — and when a price is opaque, the door opens to a specific and well-understood failure that opacity is very good at hiding.

The price with no reason

Start with what the customer actually holds. Not a rate table, not the features the model weighed, not the comparison between their file and the neighbor's. A number. Modern pricing does not run on a short list of underwriting facts a person could recite; it runs on a wide field of signals — where you live down to the parcel, the pattern of your purchases, the device you file from, credit-like indicators assembled from sources you never dealt with directly. Individually, most of these are innocuous. No single one is the reason for anything. But a model does not price on any single one. It prices on the sum, and the sum has a shape that none of the parts announce.

This is what makes the opacity load-bearing rather than incidental. If the price came from three named facts, a customer could look at the three and argue with one of them. A hundred quiet proxies, each carrying a sliver of weight, produce a total that is real in its effect and unnamed in its basis. The customer cannot point to the input that hurt them because no input, taken alone, did. The reason exists — it is distributed across the model — but it exists in a place the person paying for it is never shown. A price that cannot be pointed at cannot be argued with, and a price no one can argue with is a verdict wearing the costume of a quote.

The forbidden basis, reconstructed

Here is where the innocuous inputs stop being innocuous. There are characteristics an insurer may not price on — some barred by law, some barred by any decent account of fairness: race, and the near-proxies for it that redlining once ran on; health status smuggled into a policy that is not supposed to weigh it; the raw fact of a neighborhood's poverty. The rule against pricing on these is not a technicality. It reflects a judgment that these are not things a person should be made to pay for. But a model does not need the forbidden field in its data to price as if it had it. Given enough correlated substitutes — the store you shop at, the hour you browse, the shape of your address, the texture of your credit — it can rebuild the forbidden characteristic well enough to charge on it, having never been handed it and never been told to want it.

A model does not need to be told to discriminate; it only needs enough innocent-looking clues to rebuild what it was forbidden to ask, and price you on it anyway.

This is the accountability problem in its exact form, and it is worth being precise about the boundary. Legitimate risk pricing distinguishes people on grounds connected to the risk they actually carry: a history of at-fault collisions, a house in a genuine flood plain. Proxy discrimination distinguishes people on a forbidden or unfair basis while dressing it in the clothing of correlated, permissible-looking data. The two can produce identical-looking rate sheets. The difference is not visible in the price. It is visible only in the basis — and the basis is the one thing the opaque model does not surrender. So the customer cannot see whether they were priced on their driving or on a laundered stand-in for who and where they are, and neither, often, can the insurer that bought or trained the model. Nobody in the transaction can rule out the thing the rule exists to rule out.

An examinable basis

What is owed here is not a lower price and not a promise that the model is fair. Promises are cheap precisely where they are hardest to check. What is owed is the actual basis of the price in a form the person who pays it — and a regulator standing behind that person — can examine. Which features moved this premium, in what direction, by how much; the evidence the model actually consulted about this customer; the rule set it was operating under, frozen as it stood when the price was struck. Not a marketing paraphrase of the reasons, but the reasons, rendered so that a proxy effect could be caught. This is the whole point of a record built to be examined rather than to reassure: proxy discrimination is invisible in the output and detectable in the derivation, so an account that exposes the derivation is the only kind that lets anyone find it.

An examinable basis changes what everyone in the system can do. The customer gains something to contest — a specific driver they can dispute, correct, or challenge as an illegitimate stand-in for a characteristic they have a right not to be priced on. The regulator gains something to audit across a whole book of business, where proxy effects show up as patterns no single file reveals. And the insurer gains the only defense of a price worth having: not "trust the model," but here is what the model weighed, and you can check that none of it is the thing we are not allowed to weigh. That is the difference between risk pricing done accountably and pricing on unexaminable proxies. Both set a number. Only one can show its work, and showing the work is what keeps a legitimate practice from quietly becoming an illegitimate one.

None of this asks insurance to stop being insurance. Risk should be priced; careful people should pay less than reckless ones; the model can be part of how that happens. What accountability requires is that a decision with this much weight on a person's month carry its reasons in a form that person can hold — so that a human being remains answerable for the pricing model, and so that the line between charging you for your risk and charging you for who the model guessed you are does not get to sit forever on the far side of a number nobody will explain.

The scenario above is illustrative — a composite drawn to show a pattern, not an account of any real person, company, or event.

— Dispatches · Summit Cognitive