The self the system rewards

There is a moment, familiar to anyone who has lived inside a scored system, when you stop describing yourself and start assembling yourself. You have a form to fill, a profile to complete, a history that will be read by something you cannot see and cannot question. And so you begin to choose — not what is true, exactly, but what will read well. You leave out the year that would need explaining. You phrase the gap in the terms the reader is known to like. You do not lie, quite; you curate. And in that small, reasonable act of curation, something larger has already happened: you have started to become the person the system is looking for.

This is the quiet feature of being judged by a machine that the language of measurement never admits. A model that scores people is described, always, as an observer — neutral, downstream, reading behavior that exists independently of it. But the person on the receiving end knows better, because the person on the receiving end can feel the pull. The instant you learn you are being scored, the score stops being a measurement of you and becomes a target you are trying to hit. You are no longer just living; you are producing evidence.

Performing for the scorer

The mechanism is not mysterious, and it is not even irrational. If a system rewards a legible signal — a steady address history, a certain cadence of activity, a category of answer it knows how to score — then the sensible thing to do is to produce that signal. You emphasize the parts of your life the model can read and you suppress the parts it cannot. Not because those unreadable parts are shameful, but because they are useless to you here: a system that cannot parse them will, at best, ignore them, and at worst, count their illegibility against you. So they go. The irregular career that made sense in context, the obligation that does not fit a field, the particularity that would take a paragraph the form does not offer — all of it gets rounded off, because rounding off is what the situation rewards.

Economists have a name for the general phenomenon: once a measure becomes a target, it stops being a good measure. But Goodhart's law, stated that way, is a fact about metrics, observed from outside. What it feels like from inside is stranger and more intimate. It is not that you cynically game a number. It is that the number reaches back into how you present, how you choose, eventually how you think of yourself — and it does so under the honest cover of simply trying to do well. You are not cheating the test. You are letting the test tell you who to be, one reasonable accommodation at a time.

The system that manufactures its subject

Follow that pull far enough and the system's founding claim collapses. It said it would measure people. But a measurement, to deserve the name, must leave its object undisturbed — the thermometer does not warm the room to make the reading come out round. A scoring system disturbs its object constantly, because its object is a person who knows they are being read and adjusts accordingly. What the system records, then, is not who people are. It is who people have learned to perform in front of it. The score is not a photograph of behavior; it is behavior's response to being photographed.

A system that scores you does not find out who you are; it teaches you who to be, and then congratulates itself on the accuracy of the measurement.

And the direction of that teaching is never neutral. A model rewards what it can read, and what it can read is the standardized, the legible, the already-common. So the pressure it exerts is always toward predictability — toward the median case it was trained to recognize, away from the particular case it was not. Over enough time and enough decisions, this does not merely sort people. It shapes a population: it makes them more like one another, more like the model's idea of a good subject, more the kind of person the metric was built to reward. The system claimed to describe a distribution and instead began to narrow one. This is a coercion of identity, but it is the gentlest coercion imaginable, because at every step it felt like your own free choice to round yourself off and fit in.

It is worth being precise about what kind of harm this is, and how it differs from the more visible kind. Elsewhere in this series I have written about the person turned away at the point of entry — the applicant whose true situation had no box on the form, coerced in a single moment by a category that would not hold them. That is coercion you can point to; it happens once, at a door, and you feel the door close. This is slower and harder to name. No single door closes. Instead, over months and years of being scored, you quietly rebuild yourself to fit the scorer — and by the time the reshaping is complete, it does not feel like something that was done to you. It feels like your personality. That is what makes it the deeper injury to autonomy: not that you were refused, but that you were trained, and rewarded for the training, until the performance and the person could no longer be told apart.

Reckoning with what you reshape

An account that took this seriously would have to begin by giving up the pose of the neutral observer. A system that decides about people changes the people it decides about; this is not a defect to be denied but a fact to be governed. The first honest thing such a system can do is admit that its own presence is part of what it measures — that the behavior in its ledger is, in part, behavior it induced. A record built for the individual should be able to hold not just the decision and its inputs but this reflexive shadow: the recognition that the subject was performing for the very instrument doing the reading.

From there the obligation is to resist the thing the system does most naturally, which is to optimize people into legibility. It is easy to reward the readable and penalize the illegible, and it is easy to call the resulting uniformity a signal. But a metric that manufactures the behavior it measures has stopped measuring anything real. It has become a mechanism for producing the very predictability it then claims to detect — a closed loop that governs who people are allowed to be while reporting, in good faith, that it is only observing them. The discipline is to keep the illegible admissible: to build accounts with room for the particular fact that does not fit, standing for the person to assert the part of themselves the model cannot read, and contestability against a score that has confused the shape it rewards with the truth.

Because the alternative is a kind of accuracy no one should want. A system can drive its own error rate to nothing by teaching everyone to answer the way it expects — and then point, with real pride, at how well its predictions come true. That is not a system that has learned to see people clearly. It is a system that has trained people to disappear into its categories, and mistaken the silence for a good reading. The test of an accountable decision system is not whether the person matches the model. It is whether the person is still allowed to exceed it.

— Dispatches · Summit Cognitive