DISPATCHES · Summit Cognitive

← All dispatches

LegitimacyThe CasebookJuly 27, 20265 min read

The post the model removed

A platform that removes speech at the scale of millions cannot convene a hearing for each — but the person whose account vanished is owed more than a template citing a rule they cannot see applied to a decision they cannot inspect.

A seller runs a small shop through a large platform. She has posted the same kind of listing for years — a photograph, a short description, a price. One morning the listing is gone, and so, a few hours later, is the account it lived on. The notice arrives by email. It says the content was removed for violating the platform's community standards, that repeated violations may result in suspension, and that she can appeal by clicking a button. She reads it three times. She cannot tell which listing was the problem, which standard it broke, or what in the text or image tripped the wire. She clicks the button. Within a minute — faster than any person could have opened the file — the appeal comes back denied.

Consider what she actually received. Not a decision she can examine, but the announcement that a decision occurred. The rule is named but not shown applied; the standard is broad enough to cover a hundred behaviors, and she is left to guess which one the system believes it saw in hers. The appeal, which is supposed to be the remedy, turns out to be the same machinery running a second time — an automated check confirming an automated result, wearing the costume of review. She has been told that she did something wrong, denied the chance to see what, and denied the chance to argue she didn't. The account that carried a year of her work is simply not there anymore.

None of this means the platform lacked the right to act. It set the standards; she agreed to them; moderation is a condition of the space, not an intrusion into it. The authority to remove the listing is not the thing in question. What is in question is whether an authority that removes speech this way, at this scale, can still claim to be acting legitimately toward the person on the receiving end — or whether legitimacy is exactly what gets lost when the account vanishes and no account is given.

The account that vanished

Two things disappeared at once, and it is worth separating them. The first is the account in the platform's sense — the profile, the listings, the standing to keep posting. The second is the account in the older sense: the explanation, the reckoning, the answer to why. A person can lose the first and still be treated fairly if she is given the second. What makes the morning intolerable is that both were taken together, and that the platform behaves as though the first loss discharged its obligation to provide the second.

The generic notice is the tell. A message that could have been sent, unchanged, to any of a million users on any of a million days is not a communication about her decision; it is a template with her address on it. It cites a rule the way a parking ticket cites a statute number — as authorization, not as reasoning. The specific trigger, the thing the model flagged in her particular listing, is precisely the fact she needs and precisely the fact withheld. And the appeal completes the enclosure: a contest path that routes back to the system being contested is not a check on the decision but a repetition of it. She is invited to disagree with a wall.

Scale is why the platform automated the decision; it is not why the person deserves no account of it.

Scale is a reason to build it in, not skip it

The platform's defense is real and should be stated at full strength, because a weak version of it is too easy to knock down. Moderation at this volume genuinely cannot be done by hand. There is no room of reviewers large enough to read every post, weigh every report, and hold a hearing for every removal; the arithmetic forbids it. Automated judgment is not a shortcut the platform took to save money. It is the only way the space can be moderated at all. Anyone who pretends otherwise is arguing against the existence of large platforms, which is a different essay.

But notice what that constraint actually licenses. Scale makes human review of every action impossible. It does not make an inspectable action impossible. The reason a hearing cannot be held for each post is a fact about human attention — reviewers are finite, reports are not. The reason a decision arrives with no visible basis is a fact about how the decision was built. These are not the same limit, and the second does not follow from the first. A system can decide a million cases and still record, for each, what it saw and which rule it applied. The record is generated by the same automated process that made the call; it costs attention only when someone asks to read it. Impossibility of hearings is being used to excuse un-inspectable decisions, and the two have nothing to do with each other.

Legitimacy has never required that authority be exercised slowly or by hand. It requires that the exercise be answerable — that the person subject to it can find out what was decided about her and mount a real objection to it. A takedown can be instantaneous and legitimate at the same time, if it leaves behind something she can read and contest. What corrodes legitimacy is not the automation. It is the automation offered as a reason the answer need never come.

An account proportional to the stakes

The honest answer is not a hearing for every post; that concedes the platform's strongest point and loses. The answer is proportion. A removed comment on a public thread and a destroyed livelihood are not the same event, and the account owed should scale with what is at risk. For the low-stakes case, the minimum is modest and entirely automatable: a record of what specifically triggered the removal — the passage, the image, the pattern the model matched — and the exact rule in force at that moment, carried with the decision rather than pointed at from a distance. Not a citation to a standard. The standard, and the thing in her post it was applied to, set side by side.

As the stakes climb, so does what legitimacy demands. When the consequence is the loss of an account someone's income depends on, the contest path has to be more than the takedown run twice. It has to reach human judgment — a reviewer who can see the same preserved record she can, and who can be wrong in a way that is itself reviewable. This is where a decision built to be examined earns its name: a Decision Receipt that travels with the action, showing the evidence the system actually acted on and the rule it actually applied, so that the person, or a court, or the platform's own auditors can check whether the decision followed from its stated basis. Provenance is not a courtesy here; it is the difference between an authority that can answer for itself and one that hides behind the fact that it moves too fast to ask.

That is the line the whole case turns on. Moderation at scale can be built so that it accounts for itself at scale — every action leaving a trace the affected person has standing to open — or it can be built so that scale is the standing excuse for accounting for nothing. Both are automated. Both are fast. Only one treats the person whose account vanished as someone owed an answer rather than merely notified of a result. The platform's right to moderate was never the question. Whether it moderates in a way a person can understand and contest is the whole of it.

The scenario above is illustrative — a composite drawn to show a pattern, not an account of any real person, company, or event.

— Dispatches · Summit Cognitive

Continue from here

Turn the argument into a practice.

Get new dispatches, assess how your organization handles consequential decisions, or explore Summit Cognitive.