Deciding for someone who cannot object

Almost every safeguard we have built for automated decisions is addressed to a particular reader: an adult who notices that a decision has been made, understands roughly what it was, and has the standing, the time, and the composure to push back. Notice, explanation, appeal, review — each of these is a door, and each assumes someone on the other side capable of finding the handle. It is a reasonable assumption a great deal of the time. It is precisely wrong exactly where the stakes are highest, because the decisions that do the most to a person are disproportionately made about people who can do the least in response.

Consider who is actually on the receiving end of the gravest automated calls. A child moved through a school's discipline system or flagged by a welfare screen, who does not know the decision exists and could not read it if handed to them. A patient who is sedated, delirious, or cognitively impaired while a triage or eligibility model sorts them. An elderly person whose benefits are recalculated by a rule they were never in a position to follow. Someone in acute crisis, for whom the very condition being assessed is the thing that strips away the capacity to assess back. A person who does not speak the language the system speaks, receiving a reason in words that are, to them, no reason at all. For each of them, the sentence you may appeal this decision is not a right. It is a formality performed over their head.

The right that presupposes a rights-holder

Contestability — the principle that an automated decision must be open to challenge — is one of the better ideas in this field, and I have argued for it elsewhere. But contestability smuggles in a premise it rarely states aloud: that there is a contestant. The whole apparatus assumes a capable adversary who can perceive the decision, form a grievance, marshal the relevant facts, and lodge an objection within whatever window the system allows. Strip away any one of those capacities and the right does not weaken gradually; it collapses, because it was never designed to function without a person able to invoke it.

This is the difference between a right and its exercise, and it matters more here than almost anywhere. A right to counsel means little to someone who cannot ask for a lawyer. A right to object means nothing to someone who cannot object. We know this in law, which is why we appoint guardians for those who cannot speak for themselves and do not treat their silence as consent. Yet automated decision systems routinely treat the absence of an objection as the absence of a problem — as though a case that generated no appeal must have been decided correctly. For the people this essay is about, an unappealed decision is not a vindicated one. It is simply one whose subject was never in a position to fight.

To offer a right to object to someone who cannot is not a courtesy but a quiet cruelty; for them, the account has to arrive on its own, before it is asked for and without their having to ask.

Accountability that does not wait to be invoked

The deep flaw is one of placement. Most accountability mechanisms locate the burden of triggering them on the affected party — you must notice, you must request, you must appeal, you must supply the counter-evidence. That design is not neutral. It quietly loads the entire cost of oversight onto the person, and then it loads it most heavily onto the person least able to bear it. The competent, resourced, well-advised adult can absorb that cost; the child, the patient, the person in crisis cannot. A system built this way does not fail its most vulnerable subjects by accident. It fails them by construction, because it made the safeguard conditional on a capacity they were selected, in part, for lacking.

The correction is to stop treating review as something that has to be summoned. For the highest-stakes decisions about the least-able people, review should be automatic rather than opt-in — a second look that the system performs on its own initiative because of what the decision is and whom it lands on, not because someone with the strength to complain happened to complain. And where a decision genuinely turns on judgment, the standing to contest it has to be vested in someone with real authority to act: a guardian, an advocate, a clinician, a caseworker whose objection carries the same weight the affected party's would, precisely because the affected party cannot raise it. A proxy who can be ignored is not standing; it is decoration.

None of that is possible without a record built for the purpose. If a guardian or an advocate is going to challenge what was done to someone who cannot narrate it themselves, they need to reconstruct the decision from the outside — the inputs it rested on, the rules that were live at the moment, enough state to see whether the outcome actually follows. This is what a Decision Receipt is for, and it is worth noticing that the case for such a record is strongest here, where the subject cannot testify to their own experience of the decision. The record has to stand in for a voice that is absent. It has to let someone acting on behalf of the person do what the person cannot: see the decision plainly and press against it.

The measure of a system is its treatment of the voiceless

There is a comfortable illusion in accountability work, which is that a system besieged by challenges is a responsive one. Contestation feels like health. But a system that answers only to those who can fight it has not achieved fairness; it has achieved responsiveness to pressure, and those are different virtues that happen to look alike from the front. Such a system will be scrupulous toward the articulate complainant and negligent toward the silent case, and it will experience that negligence as calm — no appeals, no friction, no signal that anything is wrong. The absence of a fight is read as the presence of consent. That is the failure mode dressed as success.

The honest test runs the other way. Not how well a system defends itself against those equipped to attack it, but what it does, unbidden, for the person who will never attack it at all — who does not know the decision was made, could not read it, cannot reach anyone, has no one who can. What the system does for that person without being made to is the true measure of whether it is accountable or merely well-defended. A record that only produces itself when demanded is a record that abandons everyone unable to demand. Genuine standing does not depend on the affected party's capacity to assert it; it is held for them, exercised for them, and honored whether or not they ever knew to ask.

The dignity of the vulnerable, in the end, is not measured by the rights they are formally granted. It is measured by what a system gives them when they are in no position to insist — the review it performs anyway, the proxy it takes seriously anyway, the account it builds and keeps anyway, on the chance that someday someone will need to reconstruct what was done to a person who could not object at the time and cannot object now. A system that only rises to that occasion when compelled has already told you what it is. The one worth building answers to the person who cannot make it.

— Dispatches · Summit Cognitive