The contract that cannot govern

Read a typical contract for an automated decision system and you will find it is a careful, serious document about the vendor. It specifies what the vendor must deliver, how available the service must be, what happens when it goes down, who owns the data, how disputes are escalated, and what the penalties are for falling short. Every clause is the product of someone imagining a way the relationship could go wrong and writing language to prevent it. It is, on its own terms, a competent piece of work. And it has, almost always, a hole in the middle of it large enough that no one notices it is there.

The hole is this. The contract governs the vendor's conduct exhaustively, and says nothing — or nothing that bites — about whether the buyer will be able to account for the decisions the system actually makes. It buys capability: the system will assess, score, approve, deny, flag, route, prioritize. It warrants uptime: the capability will be available at such-and-such a level. But between those two things lies the entire substance of what the system does to people, and the contract treats that substance as though it were a private matter between the buyer and the machine — a thing that happens, produces an output, and leaves no obligation behind it.

So you end up with an arrangement in which the vendor is thoroughly governed and the system is not governed at all. The vendor can be held to account for failing to deliver the capability. No one can be held to account for what the capability decided, because nothing in the contract requires that the decisions be accountable — that they arrive with a record of the evidence that produced them, the rules that were active when they were made, enough to reconstruct and contest them. The contract has confused governing the supplier with governing the thing supplied. They are different problems, and solving the first does nothing for the second.

Buying the capability is not buying the account

It is worth being precise about why this gap is so easy to miss, because the people writing these contracts are not careless. They are applying a model of procurement that works well for almost everything else. When you buy a capability — a service, a tool, a stream of outputs — the natural thing to specify is the quality and availability of the capability. Does it work? Is it up? Is it accurate enough? Those are the questions you would ask of any supplier, and they are the questions the contract is built to answer. The trouble is that an automated decision system is not only a capability. It is an exercise of authority over the people its decisions land on, and authority carries obligations that capability does not.

A capability owes you performance. An exercise of authority owes the affected party an account — a defensible answer to the question why was this decided about me, and on what basis. A contract that secures the first and ignores the second has bought a powerful instrument and declined to acquire the one thing that would let its owner stand behind what the instrument does. When a decision is challenged — and consequential decisions are challenged — the buyer will reach for the record that would let them defend or revisit it, and discover that nothing in the agreement ever required such a record to exist. The capability was delivered exactly as promised. The account was never part of the deal.

A contract can compel a vendor to keep the lights on and still be powerless to compel a single defensible answer about what was decided in the light. The first governs the supplier. Only the second governs the system.

Notice that this failure survives even a perfectly performing vendor. There is no breach to point to. The service was available; the outputs were produced; every warranted thing was warranted true. The buyer's inability to account for the decisions is not a consequence of the vendor falling short. It is a consequence of the contract never having asked for accountability in the first place. You cannot enforce a term you did not write, and a term that is not in the contract is not governed by the contract, however much everyone later wishes it were. The system is ungoverned not because anyone failed but because no one specified.

What a contract would have to compel

A contract that actually governed the system would treat the account as a deliverable, on the same footing as the capability and the uptime. It would require that each decision the system makes be accompanied by a record sufficient to defend or contest it — the evidence actually consulted, the rules active at the time, enough state to replay the decision and check whether it holds. It would specify that this record be producible on demand, in a form the buyer can actually use, by a party who is actually obligated to read it. And it would make the absence of such a record a breach, because a decision that cannot be accounted for is a decision the buyer cannot stand behind, and a buyer who cannot stand behind the decisions has not really bought a system they can use in the open.

This is not a more demanding contract for the sake of severity. It is a contract that matches its terms to what it is actually procuring. If you are buying a capability that makes decisions about people, then the ability to account for those decisions is not an add-on to the capability; it is part of what makes the capability fit to deploy at all. A Decision Receipt — a record built to be examined, frozen at the moment of decision, sufficient to reconstruct what happened — is exactly the deliverable the standard contract leaves out, and exactly the one without which the buyer is governing only their supplier while the system itself runs unattended.

So the question to ask of any agreement to deploy an automated decision system is not whether it holds the vendor to their promises. It will. The question is whether, when a decision the system made is challenged, the agreement gives the buyer the power to compel a defensible record of it. If it does, the contract governs the system. If it does not — if all it can compel is performance and availability — then it governs the vendor and leaves the system, and everyone the system decides about, in a space the contract was never written to reach.

— Dispatches · Summit Cognitive