The decision behind the decision

When a system denies someone a loan, flags a claim, or lowers a score, we tend to treat the denial as the decision — the moment where judgment happened and accountability, if it exists anywhere, must attach. But the denial is the last link in a chain of much earlier decisions, made long before this particular person came anywhere near the system. Someone chose which model to use. Someone chose where to set the threshold that separates approve from deny. Someone chose what the system would optimize for and, by omission, what it would ignore. Those choices were made once, and they now govern every individual outcome the system produces. The denial is not where the decision was made. It is only where the decision arrives.

These are the decisions behind the decision, and they have a peculiar status. They are enormously consequential — a threshold moved by a fraction can change the fate of thousands of people — and they are almost entirely invisible to the people they affect. The person denied never meets the analyst who set the cutoff, never sees the meeting where one objective was chosen over another, never learns what was left out of the model because it was inconvenient or expensive to include. The meta-decision is made in one room, at one time, by people who will never encounter a single one of the individuals it will go on to govern. And then it recedes into the infrastructure, where it stops looking like a decision at all and starts looking like the way things simply are.

Why the meta-decision hides

Part of what makes these choices so hard to hold to account is that they do not feel like decisions when they are made. Choosing a threshold feels like a technical calibration. Choosing an objective feels like stating the obvious. Choosing what data to include feels like a matter of what was available. Each is framed as a matter of engineering or convenience rather than a matter of judgment with winners and losers, and so each passes without the deliberation, the record, or the sign-off that a decision of its consequence would otherwise attract. The individual denial, by contrast, feels like a decision precisely because a person is standing at the other end of it — which is exactly backward, because the individual denial is the one place where no fresh judgment occurred at all. The system merely applied choices already made.

The most consequential decision in any system is usually the one that has stopped looking like a decision and started looking like the weather.

The result is an inversion of accountability. The decision that is easiest to see — the individual outcome — is the one with the least discretion behind it, because it was determined by rules set earlier. The decisions with the most discretion — the model, the threshold, the objective — are the ones hardest to see, because they were made once, quietly, and then dissolved into the machinery. Anyone trying to contest an outcome is pointed at the individual decision, where there is nothing to contest, and steered away from the meta-decisions, where the real judgment lives and where no record was ever kept of it having been made.

Recording the choice as a choice

The remedy is not exotic. It is to record the meta-decision as a decision: to treat the choice of a threshold, a model, an objective, or an exclusion as an event with an author, a basis, a moment, and a rationale — the same things we would demand of any consequential decision made about a person. Who set this cutoff, when, on what evidence, toward what end, and against what alternatives that were considered and rejected. A threshold with that record behind it can be examined. Its author can be asked why they set it where they did. Its rationale can be tested against the outcomes it went on to produce. A threshold with no such record is a rule that governs thousands of lives and answers to no one, because there is no decision on file for anyone to point to.

This matters most when an outcome is contested. A person denied a loan is usually told, in effect, that the system applied its criteria and the criteria were not met. That answer treats the criteria as given, as though they fell from the sky. But the criteria are exactly what should be open to challenge, because the criteria are where the judgment was. If there is a record of how the threshold was chosen, the contest can reach the place where the discretion actually lives. If there is not, the contest terminates at a rule no one will claim authorship of — and the person is left arguing with a machine about a choice a human made and then declined to record having made.

Reference the standard from Admissible Reality: a decision that affects a person is admissible only if it can be traced to a basis that can be examined and contested. That standard cannot stop at the individual outcome, because the individual outcome is downstream of the choices that determined it. To make a system genuinely accountable, the trail has to run back through the denial, past it, to the meta-decisions that set the terms — the model chosen, the line drawn, the aim selected, the factor left out. Those are the decisions behind the decision, and until they are recorded as decisions, the most important choices in any automated system will remain the ones no one can find, and therefore the ones no one can be held to.

— Dispatches · Summit Cognitive