The two kinds of wrong

There are two ways to be wrong about almost anything that matters. You can flag something that was fine — call a healthy account fraudulent, a safe applicant a risk, an innocuous message a violation. Or you can wave through something that was not — miss the fraud, approve the risk, let the violation stand. The first is a false positive, the second a false negative, and the deep fact about them is that they are not two sizes of the same mistake. They are different mistakes, with different victims, and no system can drive both to zero at once. Push hard against one and the other rises. This is not a flaw to be engineered away. It is the shape of the terrain.

Because the two errors trade off against each other, every consequential system sits somewhere on a line between them, and where it sits is a choice. A fraud filter set to catch nearly everything will also freeze a great many honest accounts; loosen it to spare the honest and more fraud slips through. A screening tool tuned to miss no real threat will stop a crowd of harmless people; relax it and some real threat passes. There is no setting that is simply correct. There is only a setting that decides, in advance, which of the two wrongs the system will commit more often — and therefore which group of people will bear the cost of its being wrong.

That last point is the one that tends to vanish from view. The two errors do not fall on the same person. A false positive lands on the one wrongly flagged — the honest applicant denied, the legitimate account frozen, the safe person stopped. A false negative lands somewhere else entirely — on the victim of the fraud that was missed, on whoever is harmed by the threat that passed. So tuning a system is not just a technical adjustment of a dial. It is a redistribution of harm from one set of people to another, made quietly, often by someone who will never meet either group, and recorded — if it is recorded at all — as a number in a configuration file.

Choosing which error to risk is choosing whose harm you are willing to tolerate. That is a value judgment wearing the costume of a parameter.

The dial is a moral instrument

We are trained to read the position of that dial as an engineering decision, because it is expressed in engineering terms — a threshold, a tolerance, a tradeoff between two rates. But the units are misleading. What looks like a tuning parameter is, in its effect, a judgment about whose interests the system will protect when it cannot protect everyone. Set to spare the falsely accused, it accepts more missed harms; set to catch every real case, it accepts more wrongful flags. Either way the system is taking a side in a conflict between two groups who will never be in the room, and it is taking that side at the moment the threshold is chosen, not at the moment any individual decision is made.

This is what makes the choice so easy to miss and so consequential to ignore. It is made once, upstream, by people thinking about aggregate rates, and then it silently governs thousands of individual outcomes downstream, each of which feels to the person on the receiving end like a judgment about them rather than the working-out of a policy about everyone. The applicant denied does not experience a tradeoff between error types. They experience a no. But the no was shaped, before they ever applied, by a decision about how much wrongful denial the system was willing to inflict in order to catch how much real risk. The value judgment came first. The individual decision is its echo.

Put the setting on the record

If the choice of which error to risk is a value judgment — and it is — then it belongs to the class of things a decision ought to disclose, not bury. Yet it is among the least visible decisions in the entire machine. The inputs to a decision sometimes get recorded. The rule sometimes gets recorded. But the stance the system took toward its own errors — which way it was leaning, which wrong it preferred to commit — almost never appears in the account given to the person affected, even though it may have mattered more to their outcome than any fact about their case. They are told they were declined. They are not told that the system was set, deliberately, to decline people like them rather than risk the opposite mistake.

A record worth the name should carry that setting forward. A Decision Receipt that names which error the system was tuned to risk — and, by implication, whose harm it was tuned to accept — turns an invisible value judgment into a reviewable one. It lets the affected party see not only that they were flagged, but that they were flagged by a system leaning hard in one direction, and it lets a later reviewer ask the question that the bare outcome forecloses: was that lean the right one for this kind of decision, given who pays for each mistake? Without the setting on the record, that question cannot even be posed. The tradeoff was made, the harm was distributed, and the only trace left is the outcome itself, which says nothing about the choice that shaped it.

None of this is an argument that systems should refuse to choose. They must choose; the terrain leaves no neutral ground, and a system that pretended otherwise would simply be making the choice without admitting it. The argument is narrower and harder to evade: that the choice, having been made, should be on the record where the people who live with its consequences can see it. The most consequential decision in a consequential system is often the one nobody is shown — the quiet setting of the dial that decides, before any case arrives, which kind of wrong the system has agreed to be. A record that hides that setting is hiding the decision that mattered most.

— Dispatches · Summit Cognitive