The audit you cannot fail

A market for auditing artificial intelligence is forming right now, and it is being built on a fault that has swallowed better-funded, better-regulated markets before it. The fault is not technical. It is not that the models are hard to inspect, though they are. It is structural, and it is embarrassingly simple: the party being examined is the party paying the examiner, and it is free to shop for the examiner most likely to pass it. Wherever that arrangement has taken hold, the audit has drifted from a test into a transaction — and the certificate at the end has drifted from a finding into a receipt for services rendered.

The promise of an audit is that someone independent looked hard and might have found something. Its entire value is the possibility of failure. Remove that possibility — quietly, through incentives rather than fraud — and you are left with a document that has the shape of assurance and the substance of a sale. This is the thing worth naming before the AI-audit market hardens around its conveniences: an audit that no one can fail is not a weak audit. It is a different object wearing the same clothes.

Paying the examiner you hope will pass you

The conflict is old and it recurs with almost mechanical reliability. Consider the corporate financial audit, where the firm being audited chooses its auditor, negotiates the fee, and can decline to rehire one it found disagreeable. The auditor is supposed to serve the investing public, but the public does not sign the check. The client does — and the client can leave. That single fact bends the incentive: the auditor who is easy to satisfy keeps the engagement, and the auditor who is hard to satisfy loses it. The great accounting failures of the last generation were not, for the most part, stories of stupidity. They were stories of a professional who understood exactly whom he was serving and had every reason to serve them well.

The same shape produced the credit ratings that blessed the instruments at the center of the last financial crisis. The issuer of a security paid the agency to rate it, and could take its business to whichever agency offered the friendlier grade. Predictably, the grades ran generous. When you pay for a rating and can walk to a competitor for a better one, you are not really buying an assessment. You are buying a stamp, and the agencies knew which stamp their clients had come to buy. The signal that a top rating was supposed to carry decayed because the economics were quietly instructing it to.

Now transplant that structure onto AI assurance, where the pressure to look responsible is intense and the means of actually verifying anything are immature. A company deploying a consequential model wants a clean bill. It will pay for one. It can choose among providers, and it can decline to return to a provider whose report was inconvenient. Everything that corrupted the audited financial statement and the pre-crisis rating is present at the creation of this market, before it has developed any of the countervailing scars the older markets acquired the hard way. An audit designed within that arrangement so that failing is rare — and costly to the auditor's own book of business — is the audit you cannot fail. It is sales dressed as assurance, and the dress is convincing.

An audit the client cannot fail is not being sold assurance; the client is buying an opinion, and everyone in the room knows the price.

The market that selects for leniency

What makes this durable — what makes it resistant to good intentions — is that no one has to be corrupt for it to happen. The failure is a property of the market, not of the people in it. Suppose most auditors are honest and want to do rigorous work. If a rigorous audit is more likely to fail the client, and a failed client is more likely to take its next engagement elsewhere, then the rigorous auditor systematically loses revenue to the lenient one. Over enough cycles, the lenient firms grow and the strict firms shrink, not because anyone chose leniency as a policy but because the market rewarded it and starved the alternative. Competition, which we normally trust to raise quality, here drives it down: the auditors compete to be the easiest to pass, because that is what the paying party is shopping for.

And as leniency spreads, the certificate stops meaning anything. A passing grade is only informative if some plausible version of the examined party would have failed it. When failure becomes vanishingly rare, the grade separates nothing — the good model and the negligent one carry the same clean report — and the signal decays into noise. The buyer of assurance who most needs to distinguish the careful vendor from the reckless one finds that the instrument built to make that distinction has been sanded smooth. Everyone passes, so passing tells you nothing, so the honest vendor gets no credit for being honest. That is the lemons dynamic wearing an auditor's letterhead.

This is why appeals to auditor ethics, though not worthless, cannot carry the weight placed on them. Ethics is a demand that individuals absorb a private cost to produce a public good, standing against the current of their own incentives, indefinitely, at scale. Some will. Enough will not, and the market will reward the ones who do not, and the average will find its level. You cannot moralize your way out of a mispriced market. The problem is not a shortage of virtue among auditors; it is an arrangement that taxes virtue and subsidizes its absence. Arrangements are the thing to fix.

Making the audit checkable

The fix is not a better auditor. It is a different incentive structure, and it has three parts, none of which asks anyone to be a saint. Separate who pays from who is judged, so that the auditor's continued business does not depend on the goodwill of the party under examination — the examiner should not be able to be fired for finding something. Standardize what a passing record must contain, publicly, so that the auditor is measuring against a fixed public bar rather than one negotiated privately with the client; a standard the client cannot renegotiate is a standard the client cannot quietly lower. These two moves reduce the conflict. They do not eliminate it, because a determined client and a captured auditor can still cooperate against a distant public.

Which is why the third part is the one that actually changes the game: make the underlying record independently verifiable, so that the audit can be checked rather than merely trusted. If the decision under examination leaves behind a replayable record — the evidence that was in front of it, the rules in force at the time, enough state to run the inputs back through and watch where they land — then a third party can rerun the check the auditor claims to have run. The auditor's opinion stops being the last word and becomes a testable claim. A captured audit that blessed a decision the record does not support can itself be audited, by anyone, after the fact. The examiner who signs off on something the replay contradicts is no longer merely trusted to have been honest; he is exposed to being caught.

This is the hinge on which the whole thing turns, and it connects this argument to the ones about the assurance industry and about insurance that surround it. Assurance is only ever as good as two things: the checkable artifact underneath it and the incentive behind the person vouching for it. Get the incentive wrong and the vouching decays to noise, as it did for the ratings and the accounts. But even a corrupted incentive is survivable if the artifact is checkable — because then the vouching is not the evidence, the record is, and a bad opinion can be overturned by anyone willing to look. An audit built on an unverifiable record is a chain of trust with a paid link in the middle. An audit built on a replayable record is a claim you can test, and a test is the one thing a purchased opinion can never survive. That is the difference between an audit you cannot fail and an audit worth passing.

— Dispatches · Summit Cognitive