The moral hazard of opacity

The usual complaint about opaque decision systems is that we cannot see inside them. We do not know why the model denied the claim, flagged the account, ranked one applicant above another; the reasoning is unavailable, and so we are left to trust an outcome we cannot inspect. This is true, and it is the complaint the field has organized itself around. But it treats opacity purely as a problem of knowledge — a fog between us and a decision that was, presumably, made with the same care it would have been made under a clear sky. That assumption is the mistake. Opacity does not merely hide the quality of a decision. It changes it.

Economists have a name for what happens when an actor's conduct cannot be observed and the actor does not bear the full cost of that conduct. They call it moral hazard, and it is one of the most robust findings in the discipline. The insured party drives a little less carefully. The agent spending someone else's money negotiates a little less hard. Not because anyone is wicked — the mechanism does not require wickedness — but because care is expensive, and the discipline that normally forces its price back onto the careless party has been removed. Where conduct is unobservable and consequences are escapable, effort quietly falls to the level that serves the actor rather than the level that serves the outcome. Opacity in a decision system is exactly this condition, industrialized.

Opacity as a subsidy to carelessness

Consider who stands behind an automated decision: a firm that deployed the model, a vendor that built it, an operator who tuned and runs it. Each of these is an agent in the economic sense — a party acting with consequences that fall, in the first instance, on someone else. When their decisions can be reconstructed and questioned, the ordinary discipline of scrutiny applies: an error that can be traced back to a cut corner is an error someone will have to answer for, and the anticipation of that reckoning is priced into how the work gets done. Remove the traceability and you remove the pricing. The corner that no one can see becomes a corner it is rational to cut.

This is not a claim about bad actors; it is a claim about ordinary ones responding to ordinary incentives. Edge cases are costly to handle well, and if failures in the edge cases will never be attributed, the return on handling them well collapses. Data quality is expensive to maintain, and if no one can examine whether a decision rested on stale or unrepresentative inputs, the maintenance becomes a cost with no offsetting exposure. A threshold set for the vendor's convenience rather than the subject's protection is invisible if the basis of each decision is invisible. In every case the same thing has happened: the cost of carelessness has been severed from the careless party, and effort has drifted to the level that opacity rewards. The system is not just unexamined. It has been subsidized to be worse than it would otherwise be.

A decision no one can examine is a decision made by someone who knows no one is watching — and we already have a word for how carefully that work gets done.

Observability manufactures care

Reframe the value of an examinable record and something changes. We tend to justify records after the fact: when a decision is disputed, a Decision Receipt that carries its real evidence and its frozen rules lets the affected party contest it and lets a reviewer replay it. That is the ex post case, and it is a good one. But it undersells the instrument. The deeper value of examinability is ex ante — it operates before the decision is disputed, before it is even made, by changing the incentive of the party making it. A decider that knows its reasoning will be reconstructable is a decider that has been put back on the hook for the care it takes, and it behaves accordingly.

This is not a novel effect; it is the oldest trick institutions know. Audited accounts do not merely catch fraud after it occurs — the prospect of the audit disciplines the reporting all year, in every entry no auditor will ever read, because the reporter cannot know in advance which entry will be pulled. Inspected work is built to a higher standard than uninspected work by the same craftsman, because the inspection reprices sloppiness. The record does its most important work in the decisions that are never contested, by raising the baseline of care across all of them. Contestability, in other words, is not only a right the subject holds; it is a discipline the maker internalizes. The strongest argument for admissible records may be the preventive one: observability manufactures care, and it manufactures it precisely where after-the-fact justice can never reach — in the vast run of decisions that go unchallenged because they were made carefully enough not to invite a challenge.

Examine the decision, not the theater

There is an honest objection here, and it deserves its own move rather than a footnote. Observability can be gamed. Anything that is measured and rewarded invites the behavior Goodhart warned of — the measure ceases to be a good measure the moment it becomes a target. A regime that demands visible accountability without demanding the right kind produces its own pathology: defensive box-ticking, decisions padded with justification written for the reviewer rather than the subject, metrics optimized because they are watched rather than because they matter. Surveillance-theater induces care about the theater, not about the decision. It can even make things worse, by teaching the agent to spend real effort manufacturing the appearance of diligence instead of the substance.

So the aim has to be specified precisely. What must be examinable is the actual decision and its actual basis — the evidence that was in front of it, the rules that were live when it was made, enough state to replay it and watch where it lands — not a dashboard of proxy metrics that reward looking accountable. Care is manufactured by genuine accountability, the kind that exposes the real object and lets it be tested; it is not manufactured, and is often corroded, by accountability-shaped ritual. The distinction is the whole game. Aim at the theater and you get better theater. Aim at the decision itself — make the thing that was actually done reconstructable and contestable — and you reprice carelessness at its source, which is the only place the price ever mattered.

The epistemic complaint about opacity is real, but it is the smaller half of the problem. The larger half is that opacity is not a neutral fog we happen to be standing in; it is an incentive we are paying, a standing subsidy to the party that would rather not be careful. Examinability withdraws the subsidy. It does this not mainly by punishing the bad decision after it surfaces, but by changing the arithmetic of every decision before it is made — the same way audited books and inspected work have always done. The record is not only how we settle the account. It is how we make the account worth keeping in the first place.

— Dispatches · Summit Cognitive