The assurance industry

There is a pattern that recurs everywhere buyers cannot see what they are buying. A market produces some good whose quality is hidden — a company's true financial condition, the real risk in a bond, the safety of a drug, the seaworthiness of a hull — and rather than remain opaque forever, the market grows a second industry whose entire job is to look at the thing on the buyer's behalf and vouch for it. The auditor examines the accounts. The ratings agency grades the debt. The inspector approves the drug. The classification society surveys the ship and certifies that it will hold. These institutions are not decoration. They are the market's structural answer to the fact that trust does not scale and checking is expensive, and they arise on their own, without anyone planning them, the moment the cost of unverifiable quality gets high enough to be worth paying to reduce.

AI decisions are now the unverifiable good. A model denies a claim, prices a loan, flags a transaction, ranks a candidate — and the party on the receiving end has no way to see whether the decision was sound, and often the party deploying it does not fully know either. That is precisely the condition under which assurance intermediaries appear, and they are appearing. Firms are already forming to audit AI systems, to certify them against emerging standards, to underwrite the liability of the organizations that run them. An assurance industry for machine decisions is not a forecast; it is a thing in progress. The only open question is what kind of industry it will be.

And the answer to that question does not turn on the diligence of the auditors, or the rigor of the standards, or the good faith of anyone involved. It turns on something more basic and more decidable: the substrate the assurance industry is allowed to examine. What an auditor can certify is bounded, absolutely, by what an auditor can look at. Hand them a verifiable artifact and they can produce assurance that is worth something. Hand them a reputation and a folder of attestations and they can produce only the appearance of assurance — a stamp — no matter how conscientious they are. The substrate is upstream of everything.

An auditor who can only read what the audited chose to write is not a check on power; they are its letterhead.

The intermediary that trust builds

It is worth dwelling on how reliably this pattern shows up, because it tells you the AI case is not special. Wherever quality is hidden from the buyer, an assurance institution grows to reveal it, and the form is always the same: a third party, nominally independent, who inspects the good and issues a signal the buyer can rely on instead of inspecting for themselves. The financial audit exists because a company's own account of its finances is not, on its own, credible to someone deciding whether to invest. Securities ratings exist because the risk in a bond is not visible to the person buying it. Drug approval exists because a pill's safety cannot be seen by the patient swallowing it. Ship classification exists because a merchant chartering a vessel cannot personally survey its hull. In each case the market did not stay opaque and it did not collapse; it grew an intermediary. The intermediary is what trust builds when trust runs out.

So the emergence of AI assurance is not a surprise and not, by itself, reassuring. That an industry forms tells you the demand is real. It tells you nothing about whether the industry will supply what it promises. For that, you have to ask the harder question the history also answers: assurance intermediaries do not always work. Sometimes they decay into exactly the thing they were built to prevent.

The captured certifier

The failure mode is recurring and well understood, and it has two ingredients that tend to travel together. The first is that the certifier is paid by the party being certified. The second is that the certifier can only inspect what that party chooses to show them. Put those together and the incentives bend in a single direction. The certified party is also the customer; the customer would prefer a clean certificate; and the certifier, unable to independently reach the underlying facts, is left grading the paperwork the customer assembled to be graded. Assurance under those conditions does not measure quality. It measures the quality of the presentation of quality — the diligence of the document, not the soundness of the thing.

This is not hypothetical. It is the shape of the failure that discredited securities ratings before the financial crisis: instruments rated safe by paid graders who were assessing models and disclosures rather than independently verifiable exposure, until the gap between the rating and the reality became the crisis. It is a version of the same weakness that has periodically discredited financial auditing, where an audit can certify that a process was followed and a control existed without independently confirming the fact the control was supposed to guarantee. The pattern generalizes: when the certifier is captured by the certified and confined to the certified's own materials, the certificate stops being a check and becomes a laundering service — it takes the certified party's claim about itself and returns it with a third party's letterhead on top.

AI assurance is walking straight toward this failure mode, and the path is easy to see. If auditors of AI systems can only inspect process, policy documents, and vendor attestations — governance frameworks, model cards, the AI equivalent of a controls questionnaire — then they will certify the appearance of diligence, because the underlying facts of any particular decision are not in front of them to check. A vendor can hold every certification and still produce decisions no one can verify, for the same reason a company can pass every process audit and still be misstating what the process was meant to secure. Certification of process is not certification of outcome. It relocates the trust problem; it does not solve it.

Certify the record, not the reputation

There is only one durable escape, and it is not more rigorous auditors or stricter standards laid over the same soft substrate. It is to change what the auditor is handed. Give assurance a verifiable artifact — a decision record built to be independently replayed and contested, carrying the evidence that was actually consulted, the rules that were actually active, and enough state to reproduce the decision — and certification finally rests on checkable fact rather than on trust in the checker. The auditor is no longer grading a story the vendor wrote about itself. They are re-running the decision and observing whether it holds. That is the difference between an assurance you can check and an assurance you must believe, and it is the difference on which the entire value of the industry turns.

This reframes what assurance is even for. An audit is only as good as the artifact it can examine; assurance inherits the falsifiability of its substrate and can never exceed it. A record designed to be replayed gives the certifier something to actually verify. A reputation and a governance binder give them something to launder. Both can carry a stamp; only one earns it. And notice the ordering this implies. Building the record is upstream of building the audit — you cannot bolt credible assurance onto decisions that were never made checkable, because there is nothing underneath the stamp for the stamp to be about. The assurance industry that is forming will be exactly as trustworthy as the records it is handed, and no more. That is not a limit on the auditors. It is a fact about what they can see.

Which means the consequential choice is being made now, and it is not being made by the assurance firms. It is being made by everyone deciding what their AI systems will record. If decisions are logged as reputations and paperwork, the industry that grows to certify them will certify reputations and paperwork, and will fail the way paid graders of unverifiable goods have failed before. If decisions are captured as records that can be replayed and contested, the industry can become what audit is supposed to be — a check on power rather than its letterhead. The substrate decides. It always has. We are simply choosing it, this time, for a market large enough that the choice will be hard to unmake.

— Dispatches · Summit Cognitive