The shape of an inevitable market

There is a difference between a market that could exist and a market that has to. The first depends on adoption, taste, and timing — it is a real possibility that may or may not arrive. The second is structural: given certain conditions, the market forms with the reliability of water finding its level, because the alternative is a state of affairs that the parties involved will not tolerate for long. A market for decision records belongs to the second kind, and the way to see this is not to forecast the future but to notice that the same sequence has already played out, more than once, in domains that had nothing in common except the underlying condition. The condition is this: a society begins making consequential decisions faster, or more opaquely, than it can watch them being made — and it responds, every time, by building an institution that keeps a record it can trust.

The sequence, stated once

The sequence has four movements, and they always occur in the same order. First, a capability spreads that lets consequential decisions be made at a scale or speed that outruns direct human oversight. Second, that capability produces failures — not because anyone is villainous, but because opacity plus scale reliably generates error and abuse that no one can see until it is large. Third, the failures cross a threshold of visibility and cost that makes them intolerable to the parties standing outside the decision: the harmed, the public, the state. Fourth — and this is the movement that matters — the response is not to slow the capability down, which never works, but to require that the decisions leave behind a record that an independent party can examine. The capability is allowed to keep running, on the condition that it becomes accountable after the fact. The record is the price of the speed.

What is striking is how little the domain matters. The sequence is indifferent to subject. It has run through the keeping of company accounts, through the certification of financial statements, through the investigation of catastrophic accidents, and through the safety of the systems we ride and fly in. In each case the specifics differ wildly and the shape is identical: a capability outran oversight, produced intolerable failures, and was answered with a mandated, standardized, independently held record. Once you have seen the shape a few times, you stop treating each new instance as a novelty and start treating it as a recurrence.

Three prior instances, briefly

Consider the modern audit. Corporations grew large enough, and their affairs complex enough, that the people who owned them could no longer see what the people who ran them were doing. The information asymmetry was the whole problem: capital was being allocated on the basis of accounts that the allocators had no way to verify. The response, hardened after financial catastrophe, was not to shrink the corporations. It was to require that their statements be examined and attested by an independent party operating to a common standard. The audit exists because ownership and control came apart, and a record verified by a disinterested third party was the only thing that could stitch trust back across the gap. Notice the structure: independence, standardization, a record made for outsiders to read.

Consider the investigation of a major accident. When mechanized transport began failing in ways that killed people at scale, societies did not respond only by punishing operators. They built standing institutions whose job was to reconstruct exactly what happened — to establish, from durable evidence captured at the time, the sequence of events and decisions that produced the outcome. The dedicated recorder carried aboard high-consequence vehicles is the physical embodiment of the same logic: the decisions and conditions leading to a catastrophe must survive the catastrophe, in a form an independent investigator can read, precisely because the operators who could otherwise explain them may be gone or interested. Again: a durable record, captured at the moment of decision, preserved for a party who was not present and does not simply trust the operator's account.

Consider financial market surveillance more broadly. As trading moved to machine speed, the human capacity to watch a market in real time evaporated, and the response was to require that the events leave behind reconstructable records — timestamped, ordered, retained — so that what could no longer be watched as it happened could at least be reconstructed after it did. The market was allowed to keep accelerating on the condition that it became auditable in retrospect. Speed was purchased with a record.

In every prior instance, the society did not choose between the capability and its oversight. It kept the capability and invented the record. The record is how modernity says yes to speed without saying yes to blindness.

Why the sequence is running again now

The condition that triggers the sequence — consequential decisions made faster than they can be watched — is exactly the condition that automated decision-making now produces, and produces more sharply than any prior technology. When software moves from advising a human to acting on its own authority, it does not merely speed up existing decisions; it makes decisions at a volume and velocity that forecloses real-time human oversight as a physical matter. You cannot watch a million automated determinations as they are made any more than you could watch a modern market by eye. The first movement of the sequence — capability outrunning oversight — is not a forecast here. It is the present tense.

The later movements are already visible in outline. Failures of opaque automated decisions — the wrong denial, the biased screen, the unexplainable outcome that no one inside the institution can reconstruct — are accumulating and, increasingly, surfacing. The parties outside the decision are beginning to find them intolerable, which is what the growing body of rules around automated decision-making is: the third movement, the crossing of the visibility threshold, expressed in law. What has not yet fully arrived is only the fourth movement — the settled, standardized, independently kept record that the prior instances all converged on. That is the part still being built. But its arrival is not in question, any more than the arrival of the audit was in question once ownership and control had come apart. The sequence does not have an exit before the fourth movement. It has never stopped at three.

What "inevitable" does and does not mean

Inevitability at the level of the category is not a promise about any particular company, and it should not be read as one. The sequence guarantees that a durable, standardized, independent decision record will become a required institution. It does not guarantee who builds it, what it is called, or which incumbents capture it. The audit profession consolidated around a handful of firms; the safety recorder standardized around specific formats and mandates; market surveillance settled into particular regimes. Each was inevitable as a category and contingent in its details. The same will be true here. The bet a builder or investor makes is not on whether the record becomes mandatory — the sequence settles that — but on the details the sequence leaves open: the shape of the object, the standard it converges on, and who is positioned when convergence happens.

This is what changes when you stop treating decision accountability as a speculative market and start treating it as the current instance of an old sequence. The question is no longer "will there be demand for this?" The four movements answer that. The question becomes the narrower, harder, more useful one: given that the record is coming, what will it have to be — durable enough to survive the decision, standardized enough to compose across systems, independent enough to be believed by a party who does not trust the institution that made the decision? Those are the properties every prior instance converged on, under pressure, after failure. A market that will converge on them is not a market you are hoping to create. It is a market you are early to.

— Dispatches · Summit Cognitive