The data that trained the thing that judged you

When a model decides something about you, there is a natural way to picture the encounter: the system is over there, finished and impartial, and you arrive in front of it with your particulars, and it renders a judgment. Two parties, cleanly separated — the instrument and the case. It is a comforting picture, and for a great many people it is wrong. The same behavior the system is now judging is, more often than anyone admits, part of what the system was built from in the first place. Your old records, your past choices, the traces you left across years of ordinary life — some fraction of that was gathered up, somewhere upstream, and folded into the training data that taught the model what people do. You are not standing in front of the instrument. You are, in part, inside it.

This is a stranger position than being judged by a machine, and it deserves its own name. The person on the receiving end here occupies two roles at once, and holds neither by choice. You are the subject the decision is about, and you are also raw material the decider was made from. Your history helped set the baseline against which your history is now measured. And at no point in this loop were you asked — not to be training data, not to be a case, not to have the one turned into a rule and applied to the other.

Subject and raw material at once

Consider what actually happens when a model is built. It learns from records of how people behaved — what they did, what followed, who ended up where. To learn a pattern is to generalize across many particular lives, compressing them into a sense of what someone in a given situation is likely to do or deserve. The people whose records went in do not survive in the model as themselves; they survive as weight, as a nudge to the model's expectations, averaged together with strangers into a shape. That shape is the instrument. And when the instrument later turns to judge one more person, it may well be judging someone whose earlier life was one of the many that gave it its shape to begin with.

So the affected party is not simply the object of a decision. They are, unknowingly, one of the authors of the standard — a co-builder conscripted into constructing the very thing that now stands over them. This is not a metaphor stretched for effect. It is the plain structure of how these systems come to be: they are made of us, and then they are pointed back at us. The double role is held silently, without announcement, and it is held without consent at both ends. You did not agree to donate your past to the making of a judgment engine, and you did not agree to be one of the cases it would later decide. The same person is both the ingredient and the meal.

You were fed to the machine before you were judged by it, and both times without being asked — first as an ingredient, then as a case.

No standing over either role

The injury is not that this happened; data has to come from somewhere, and no one seriously proposes that models be built from nothing. The injury is that the person at the center of it has no standing over either of the two roles they were placed in. Take them in turn. As raw material, you did not agree to be training data, and — this is the sharper part — you cannot see what of you went in. There is no receipt for your conscription. You cannot inspect which of your records were gathered, how they were labeled, what they were taken to represent, or whether they were even accurate when they were folded into the average. Your contribution to the instrument is real and invisible to you at once — the worst combination: consequential and unaccountable.

And as the subject of the decision, you cannot contest how your own past was generalized into a rule now bearing down on your future. The model does not tell you that the presumption it is applying to you was learned, in part, from people like you — perhaps from an earlier version of you. It presents its verdict as if it came from nowhere, or from everywhere, which amounts to the same unarguable thing. So your own traces have been used to manufacture a verdict about you that you had no part in shaping and cannot inspect. You are being held to a standard you helped build, without the standing to see your own contribution, and without the standing to challenge how it was turned against you.

It is worth being precise about how this differs from a related harm — being judged by a pattern you were never part of, the reason for your treatment resting on strangers' records rather than your own. That is a real grievance, and a distinct one: the injustice there is that the group whose history is being applied to you is not your group; you are being read through a template that was never fitted to you. Here the injury runs the other way and cuts deeper. You were part of the group — unknowingly, as raw material. The template was fitted to you, among others, and you were never told. Being generalized by strangers is one wound. Being generalized by a system that quietly used you to learn the generalization is another, and it stings in a way the first does not, because you cannot even locate yourself in it to object.

Standing over the lineage

If standing means anything on the receiving end, it has to reach the place the decision actually came from — and part of where it came from is what it was built out of. The usual account of a consequential decision covers the inputs in front of it and the rules it applied. But a model carries a third thing behind those: its lineage, the body of records it learned from, the material that gave it its sense of what people like you do. That lineage is part of the provenance a serious decision owes. A record that shows you the evidence and the rules but is silent about what the deciding instrument was made from has told you about the last mile and hidden the road.

Extending standing to the training-material role does not require handing anyone a copy of a model's internals. It is a claim about a person's relationship to their own traces: some visibility into whether and how your records were used to build the systems that decide about you, and some say over that use — the same standing you would expect over any other consequential handling of your history. The public vocabulary the series already uses is enough to name what is missing. Provenance should run all the way back, not stop at the threshold of the decision. Contestability should attach not only to the verdict but to the lineage that made the verdict possible. A Decision Receipt worth the name treats the model's ancestry as part of what it must account for, because a judgment you helped construct, and cannot see yourself inside, is not one you can be said to have received fairly.

The comforting picture — instrument over there, case over here — will not survive an honest look at how these systems are made. You are in the machine that judged you. The least a decision can do is admit that you are there, tell you how you got there, and give you a way to answer for the version of yourself you never agreed to become.

— Dispatches · Summit Cognitive