The right to be counted correctly

We argue about whether automated decisions are fair, accurate, contestable, and explainable, and these are the right things to argue about. But there is a decision that comes before all of them, so early in the process that it is rarely thought of as a decision at all. Before a system can decide whether you qualify, whether you are a risk, whether you are owed something or owe something, it has to decide who you are. It has to resolve the person in front of it to a record, match a name and a few attributes to a history, attach this living individual to a stored identity. Everything downstream depends on that match being right. And the match can be wrong.

This is the resolution step, and its quietness is exactly the problem. It does not feel like a judgment. It feels like a lookup — mechanical, clerical, beneath the dignity of the word decision. But it is a decision in the fullest sense, because it can go either way and it has consequences. The system is deciding that this person is that record, that the someone standing here is the someone described there. When the decision is right, it disappears, and all the attention flows to the substantive question that follows. When it is wrong, it does not announce itself. It simply hands the substantive question the wrong person, and the substantive question answers correctly about the wrong life.

The errors of the first step

The failures of identification have a grim familiarity once you look for them. The mismatched name, where two people share enough attributes that the system attaches one person's question to the other's history. The merged file, where two distinct individuals have been folded into one record, so that one person now carries the other's history as if it were their own, and neither can fully be pried loose. The someone-else's-history, where a record that was always about a different person gets matched to you because the matching criteria were loose and the stakes were treated as low. In each case the substantive engine downstream may be flawless. It is reasoning impeccably about a person who is not the one in front of it.

And here is what makes these errors distinct from ordinary mistakes: they are not corrigible by improving the decision that follows. If a lending model, a risk score, an eligibility check, or a flag is fed the wrong history because the wrong record was matched, then making that model more accurate makes the outcome worse, not better — it reasons more precisely about a life that is not yours. You cannot fix a misidentification by sharpening anything downstream of it. The error is at the root, and everything that grows from the root inherits it. A decision built on a misidentification is wrong from the first step, and no amount of accuracy in the later steps can repair what the first step got wrong.

If the system has decided you are the wrong person, every later thing it decides correctly is correct about someone who is not you. The accuracy is real, and it is aimed at a stranger.

A prior right, and what secures it

This is why I want to name a right that sits underneath the more familiar ones — the right to be counted correctly. It is prior to the right to a fair decision, prior to the right to contest, prior to the right to an explanation, because all of those are rights about a decision made about you, and they are empty if the decision was never about you in the first place. The right to be counted correctly is the right to have the system's claim that this is your record be true before anything is built on it. It is the foundation the other rights stand on. When it fails, the other rights do not protect you; they protect the stranger you were mistaken for.

What does it take to honor such a right? At minimum, the resolution step has to be treated as the decision it is — recorded, not assumed. A system that wants to be accountable cannot let the identification dissolve into an invisible lookup. It has to keep a record of the match: who the system decided you were, on what attributes, against what record, with what confidence, under what matching rule. That record is what makes the first step contestable like any other. Without it, a person harmed by a misidentification faces a special cruelty — they are being decided about as someone else, and they cannot even point to where the substitution happened, because no one wrote down that a substitution was made. The error is upstream of everything, and the evidence of it was never kept.

There is a temptation to treat identity resolution as too low-level, too plumbing, to deserve this kind of ceremony. The temptation should be resisted precisely in proportion to the stakes of what follows. The higher the consequence of the downstream decision, the more the integrity of the first step matters, because the first step is where the consequence gets aimed. A Decision Receipt that records only the substantive reasoning and treats the identification as given has documented the wrong thing carefully and the right thing not at all. To honor the right to be counted correctly, the record has to begin one step earlier than we are used to beginning it — at the moment the system decided who you are, which is the moment everything else became possible to get wrong.

— Dispatches · Summit Cognitive