The cost of a false alarm

Consider the arithmetic of catching something rare. Suppose that among every hundred thousand people who pass through a system, a handful are doing something the system is meant to stop. To catch that handful, the system casts a net, and any net wide enough to catch the rare few will also catch a great many who resemble them and are innocent. This is not a flaw in a particular detector; it is a property of the mathematics. When the thing you are hunting is rare and your test is anything short of perfect, the alarms that fire will be overwhelmingly false. The rare guilty are found, if they are found, buried in a much larger population of the wrongly flagged.

The institution rarely experiences this the way its subjects do. From inside, the false alarm is a line in an operating budget — a review cost, a support ticket, a tolerable rate of noise accepted in exchange for the occasional real catch. It is scored as vigilance working as intended. But the false alarm does not stay inside the institution. It exits and lands on a person, and to that person it is not a statistic. It is a transaction declined at a checkout, a benefit delayed while eligibility is re-confirmed, an application returned with a reason that is not a reason, a name that now carries a mark it did not earn. The cost the institution files under overhead is, at the far end, an experience of being treated as a suspect for having done nothing at all.

Who pays

The defining feature of the false alarm is that the party who benefits from the vigilance and the party who bears its cost are not the same party. The institution gets the security. The innocent flagged person gets the suspicion. Because the two are separated, the institution faces a standing temptation to tune its net toward catching more, since the marginal cost of another false alarm is borne almost entirely by someone else. A false-positive rate that would be intolerable if the institution felt it directly becomes comfortable when it is exported to strangers who cannot see it happening and have no standing to object.

An acceptable cost is only acceptable to whoever is not the one paying it. Ask who signs for the false alarm, and you will usually find it is not the party who called it acceptable.

What makes this worse than an ordinary unfairness is its invisibility to both sides. The flagged person often cannot see that they were flagged, only that something went wrong — a delay they attribute to bureaucracy, a denial they attribute to their own paperwork. And the institution, seeing only its aggregate catch rate, cannot see the individuals its net swept up by mistake. Each side is missing exactly the information that would let them correct the other. The person cannot clear themselves because they do not know what they are accused of. The institution cannot fix its net because it never learns which of its alarms were wrong.

The record closes the loop

What breaks this stalemate is a record of why each alarm fired. Not a summary rate, not a monthly dashboard of total flags, but a per-decision account: this person was flagged, on this basis, by this rule, at this threshold. A record of that kind does two things at once, and they are the two things the false alarm otherwise makes impossible. It gives the innocent person something to answer — a stated basis they can contradict with facts, rather than a fog of suspicion they can only protest. And it gives the institution, in the aggregate, the one thing its catch rate cannot tell it: what kinds of innocent behavior its net keeps mistaking for the thing it hunts.

The second of these is easy to overlook and matters enormously. An institution that keeps only its hit rate learns nothing about its misses. It cannot discover that a particular legitimate pattern trips its alarm every time, that a threshold set for one population lands unfairly on another, that the net has drifted until it catches mostly the wrong things. Those findings live entirely in the record of the false alarms, read together — and a system that discards its false positives as noise has thrown away the only evidence that could tell it the net needs mending. Vigilance without a record of its own errors is not vigilance improving; it is vigilance repeating, confident because it never counts what it gets wrong.

None of this argues against catching the rare bad actor. The handful the system exists to stop are real, and stopping them is legitimate. The argument is only that the cost of doing so should be visible to the party imposing it and answerable by the party bearing it. Reference the standard from Admissible Reality: a flag that affects a person is admissible only if that person can contest it, which means it must carry a basis they can read. A false alarm with a record is a mistake that can be cleared and, in the aggregate, corrected. A false alarm without one is a mistake that becomes permanent by default, paid for quietly by someone who never learns why the system decided, however briefly, to treat them as guilty.

— Dispatches · Summit Cognitive