The audit that audits nothing

There is a particular kind of audit that has become the default proof of seriousness, and it works like this. An assessor arrives with a list of controls. There should be a policy for how decisions are made; there is one, signed and dated. There should be an approval step; the log shows approvals were collected. There should be a periodic review; minutes confirm the review met. Each control is mapped, each box checked, and at the end a clean report attests that the process is in place. What the report never says — what it is not built to say — is whether any decision the process produced was actually right. It audits the existence of the machine, not the soundness of anything the machine decided.

This is not a failure of diligence. It is the design. Controls-based compliance was conceived to answer a narrow question: does an organization have the apparatus a reasonable institution is expected to have? That question is worth answering. The trouble begins when the answer gets quietly upgraded in the reading. A clean audit means the controls exist. It is received as if it meant the outcomes are trustworthy. The two are not the same claim, and the distance between them is exactly where consequential decisions go unexamined.

Consider what a controls audit actually inspects. It looks at whether a review happened, not at what the review concluded or whether the conclusion could survive a second look. It confirms an approval was recorded, not that the approver saw the evidence, understood it, or would reach the same call shown the same facts today. It verifies the policy is written down, not that the system followed it on the ten thousand occasions no human was watching. The control is a proxy for soundness, and like all proxies it can be fully satisfied while the thing it stands in for quietly fails.

A process can pass every control it has and still have no answer to the only question that matters: run this decision again on what was actually known, and does it hold?

There is a second audit that almost no one asks for, because until recently it was not possible to ask. Not did you have a process, but can you re-run this particular decision against the evidence it claims to have used, and does it come out the same way? That is an audit of the decision, not the apparatus. It does not care whether a review meeting occurred. It cares whether the inputs that decision rested on still exist, still mean what they meant, and still produce that output when put back through. The first audit asks whether you built the machine. The second asks whether the machine was right on the third Tuesday in March, in the case that is now in dispute.

Why the substitution holds

The reason the cheaper audit stands in for the dearer one is that, for most of the history of institutional decision-making, the dearer one could not be performed. You could not re-run a loan officer's judgment, a triage nurse's call, an analyst's assessment, against the precise state of the world they faced. The inputs were in their heads and in scattered paper; the moment passed and took its context with it. So we audited the next best thing — the conditions under which good judgment was likely — and called it accountability. Controls were a reasonable surrogate for a soundness check that was genuinely out of reach.

That surrogate is now being asked to certify systems for which the real check is reachable. When a decision is produced by software running against recorded inputs under an explicit rule, re-running it is not a metaphor. The evidence it used can be captured, the rule it applied can be stated, the state it ran against can be preserved, and the whole thing can be put back through and checked. The constraint that justified the substitution has lifted. What remains is the habit of accepting a controls audit as if the soundness audit were still impossible — paying for proof of process and booking it as proof of outcome.

The gap is about to be priced

For now the substitution survives because nobody adjacent is doing better, and a clean report is a clean report. That is a temporary condition. The moment a decision can be genuinely re-run, a controls audit and a decision audit stop looking like two grades of the same thing and start looking like two different products — one that certifies you owned a process, and one that certifies a specific call would survive being examined. The first has no answer when a single decision is challenged on its merits. The second is built for exactly that moment.

The institutions reading their compliance reports most comfortably are often the ones with the largest gap between the two. They have passed everything that was asked and confused the passing for assurance. When the question finally narrows from do you have a process to defend this one, the controls audit will have nothing to say, because it was never an audit of decisions. It audited the room the decision was made in, and pronounced the room compliant. That was always a different claim. It is about to be read as one.

— Dispatches · Summit Cognitive