Discrimination by accident

When we imagine an unfair decision, we tend to imagine a decider with a motive — someone who wanted a particular group to lose and arranged for them to. That picture is satisfying because it locates the wrong in a will, which is the thing we know how to condemn. But it is, for automated systems, almost always the wrong picture. The unfairness that matters most is the unfairness nobody wanted, that no one would defend if you showed it to them, that arrived without anyone deciding it should. It is discrimination by accident, and it is both more common and harder to catch than the deliberate kind, precisely because there is no author to point to.

Accidents of this sort have recognizable shapes. A proxy variable, innocent on its face, that turns out to stand in almost perfectly for something the system was forbidden to consider — a postal code that tracks a protected characteristic, a gap in a history that correlates with circumstance rather than conduct. A training set that faithfully encodes an old injustice, so that the system learns to reproduce the past with mechanical fidelity and calls it prediction. A threshold set for a perfectly defensible reason that nonetheless lands harder on one group than another, because the underlying distributions differ in a way no one checked. None of these involves intent. Each of them produces, at scale, exactly the outcome that intent would have produced. The effect is indistinguishable from malice; the cause is indifference.

Why the accidental kind is the hard kind

Deliberate discrimination, for all its evil, has one merciful property: someone knows about it. There is a decision, a policy, a motive, a person who could in principle confess. Accidental discrimination has no one who knows, because by construction no one chose it. It lives in the relationship between an input and an outcome that no human ever inspected. The engineer who selected the proxy variable did not know it was a proxy. The team that assembled the training data did not see the injustice folded into it. The analyst who set the threshold optimized for accuracy and never asked who the errors would fall on. Each acted in good faith, and the wrong assembled itself out of their separate, reasonable choices.

This is what makes the accidental form so resistant to the usual remedies. You cannot interview your way to it, because the people involved sincerely do not know. You cannot find it by reading the policy, because the policy says nothing discriminatory — the discrimination is in the data and the math, not the words. You cannot even find it by examining outcomes alone, because an outcome that looks adverse might be entirely justified by factors you cannot see. To find accidental discrimination, you have to be able to ask a very specific question: what did this decision actually use? Which inputs, which features, which weights, which rule, applied to which person. And that question can only be answered if the answer was written down at the moment the decision was made.

You cannot find a bias no one chose by asking the people who made the decision. You can only find it in the record of what the decision actually consulted.

The record is the only place it shows

Here is the structural trap. Accidental discrimination is detectable only in the evidence of how decisions were made — and the evidence of how decisions were made is the one thing that systems built for speed tend not to keep. A system optimized to decide quickly and cheaply has every incentive to discard the working: the intermediate features, the inputs as they stood, the rule as it was active, the path from data to verdict. All of that is overhead from the perspective of throughput. So the very systems most likely to harbor an accidental bias are the ones least likely to retain the only evidence that could reveal it. The failure hides in exactly the place the design throws away.

A decision record changes what is findable. If each decision carries the inputs it actually used, the rule that was actually active, and enough state to reconstruct the path it took, then accidental discrimination stops being invisible and becomes measurable. You can ask whether a proxy is doing forbidden work by checking what the feature actually correlated with across the real population of decisions. You can ask whether a threshold falls unevenly by reading where it landed, person by person, group by group. You can ask whether the training data carried an old injustice forward by tracing which historical patterns the decisions reproduce. None of this is possible against a system that kept only its outputs. All of it is possible against a system that kept its reasoning.

There is a quiet moral inversion here worth naming. We tend to think a clean record is a sign of nothing to hide, and an absent record merely an inconvenience. But for accidental discrimination the absence is not neutral — it is the condition under which the bias survives. A system that keeps no record of what its decisions used has not avoided discrimination; it has only arranged that any discrimination it commits will be undiscoverable. The good faith of everyone involved is real and beside the point. Good faith did not prevent the accident, and without the record, good faith cannot even detect it. The remedy is not better intentions. It is the evidence of how the decision was actually made, kept deliberately, against the day someone thinks to look.

— Dispatches · Summit Cognitive