The tragedy of the commons of trust

When you accept that a machine made a decision about you — that a claim was denied, a limit was set, an application was declined — you are not, in that moment, evaluating the machine. You have no way to. What you are drawing on is a diffuse, inherited sense that decisions of this kind are usually made in good faith, that the systems producing them are usually competent, that “the system decided” usually means something was actually decided. That background willingness to extend belief is not yours alone and it was not built by the actor now invoking it. It is a common resource, accumulated over decades by everyone who ever made an automated decision that turned out to be sound — and it is spent, a little, by everyone who makes one that turns out to be worthless.

Economists have a precise name for a resource shaped like this, and a precise prediction about what happens to it. The resource is a commons: a pasture no one owns, that everyone may graze, whose grass is depleted by use. The prediction, made vivid by Garrett Hardin in 1968 and formalized long before by the logic of collective action, is that a commons open to individually rational actors will be overgrazed to exhaustion — not from malice, but because each herder captures the full benefit of one more animal while the cost of the thinning pasture is shared across all of them. The arithmetic favors defection at every step, right up to the moment there is nothing left to graze. Trust in automated decisions is a pasture of exactly this shape, grazed by exactly this logic.

Trust as a shared pasture

Consider what an actor actually gains by deploying a fast, cheap, unaccountable decision system. The savings are immediate and privately captured: no evidence retained, no rules frozen, no record built to be replayed or contested, no cost of standing up an account that could survive a challenge. The decisions still get made, and — this is the crucial part — they are still, at first, believed. They are believed because they arrive dressed in the general credibility that automated decisions enjoy, a credibility the deploying actor did nothing to earn. It is grazing. The system feeds on a stock of public trust that other actors, more careful ones, spent real money to build and maintain, and it pays nothing back into the pasture.

The cost of that grazing is real, but it does not land on the actor who did it. When one of these cheap decisions goes wrong — is exposed as arbitrary, unexplainable, impossible to contest — the damage is not confined to the firm that produced it. It seeps outward into the general category. The affected person learns something they cannot unlearn: that “the system decided” can mean nothing was meaningfully decided at all. That lesson generalizes. It attaches not to one vendor but to automated decisions as a class. The next actor’s sound, accountable, genuinely careful decision now meets a fractionally more skeptical public — a public that has been taught, at someone else’s convenience, to suspect the whole category. The saver banks the whole saving; the cost is smeared across everyone who relies on the trust.

This is the externality that the rest of this family keeps returning to, seen from a particular angle. The account that was skipped had a price, and the price was paid — just not by the party who skipped it, and not all at once, but by the commons, in the slow currency of eroded belief. Each opaque decision that fails is a withdrawal from a shared account no single actor is charged for drawing down.

How the commons collapses

The danger of a commons is not the first defector; it is the second, and the tenth, and the hundredth. Overgrazing is a cascade. Once some actors are visibly getting away with cheap, unaccountable decisions — capturing the savings, suffering none of the diffuse cost — the private calculus shifts for everyone watching. The careful actor, still paying to keep real accounts, now competes against rivals who don’t, and watches them draw on the same public trust for free. Carefulness starts to look like a tax only the naive pay, and the rational response, decision by decision, is to graze a little harder oneself. The race is not to the top; it is to the cheap.

Run that dynamic forward and you arrive at the collapse the arithmetic promised. Past some threshold of accumulated bad experience, the public stops extending the benefit of the doubt to automated decisions at all. “The system decided” stops functioning as a reason and starts functioning as an evasion — the phrase you reach for when you don’t want to answer. Every automated decision becomes suspect on arrival. And here is the bitterly instructive part: the actors who never abused the trust lose it anyway. Their sound decisions, their real records, their genuine willingness to be contested — none of it earns them the belief it should, because the category they belong to has been discredited by others. The pasture is bare for the careful herder and the reckless one alike. This is the specific cruelty of a commons collapse: it does not discriminate between the parties who caused it and the parties who tried to prevent it. Depletion is collective even when the grazing was not.

Every worthless receipt spends a little of the trust that made receipts worth anything — and the actor who prints them keeps the savings while everyone shares the debt.

Governing the commons of trust

The reason a commons cannot be left to individual incentive is not that the actors are wicked. It is that the incentives are misaligned with the collective interest by construction — each actor’s private ledger underprices a cost that is real but shared. This is the classic case for governance of a commons, and the classic solutions are not exotic: shared standards, verifiable records, and enforcement that together make the private cost of grazing approximate the collective cost. Elinor Ostrom spent a career documenting how real communities govern real commons without either collapse or central seizure — through agreed rules, monitoring that lets the community see who is drawing down the resource, and graduated consequences for those who take more than their share. Accountability infrastructure is that governance, applied to trust: a standard for what an admissible decision record is, a way to verify whether a given decision actually meets it, and consequences for the actors whose decisions cannot survive the check. Together these re-attach the cost of a worthless decision to the party who made it, which is the only thing that ever stops a commons from being grazed to death.

Honesty requires marking the limit of the metaphor, because a commons argument stretched too far becomes an argument for control, and that is not the argument here. Trust is not perfectly rivalrous the way grass is: my believing an institution does not use up your capacity to believe it, and a single spectacular fraud can deplete trust faster than a thousand quiet lapses. Reputation, too, is partly individual — an actor with a long, visibly accountable record retains some private credit even as the category around it erodes, which is precisely why building such a record is worth doing. So the claim is not that trust is a pure commons. It is that trust has a strong commons component — a shared stock that individual incentives systematically underprotect — large enough, and depleting fast enough, to warrant the standard remedy for undersupplied shared goods. That remedy is standards and governance, not central command. No one is proposing to nationalize belief; the proposal is the older, humbler one, that a resource everyone depends on needs rules that make the private cost of spoiling it land where the spoiling happens.

The cheapest actor is not, in the end, outside the pasture. The firm that skips the account is betting that automated decisions will keep being believed while it, specifically, declines to earn the belief. That bet only pays as long as enough others are still feeding the commons it grazes — and the whole trajectory of a tragedy of the commons is that this condition fails. When it does, the reckless actor’s decisions are disbelieved along with everyone’s. The case for accountability, made in the currency of incentives rather than virtue, is finally this: it is the fence and the herding agreement that keep a shared pasture alive, so that “a machine decided” can go on meaning something to the very actors who would otherwise have every private reason to make it mean nothing.

— Dispatches · Summit Cognitive