The only witness is the system

You say you cancelled the subscription in July. The system says you never did, and the charges kept coming. You say the payment cleared — you remember the confirmation screen. The system says it failed, and the account lapsed. You say you never agreed to the new terms. The system says you clicked, on a Tuesday, at 9:14 in the morning. In each of these small, ordinary disputes you are certain of something, and the system is certain of the opposite, and there is a fact of the matter about which of you is right. The trouble is who gets to establish it. Whatever record exists of what actually happened, the system made. The system holds it. And the system decides what it says.

This is a strange position to be in, and we have grown so used to it that we no longer find it strange. In any dispute worth the name, we expect the evidence to sit somewhere neutral — with a clerk, an archive, a third party who has no stake in the outcome. What makes a dispute with an automated system different is that the evidence never leaves the hands of one of the parties. The system is at once the entity you are arguing against and the sole keeper of the record about the argument. It is the defendant and the court reporter. It is the party with the most to gain from a particular version of events, and it is the only party who can tell you what the record of those events contains.

Party and record-keeper at once

Strip away the software and the asymmetry is ancient. The powerful have always kept the books, and keeping the books has always been a form of power. The landlord's ledger, the company's accounts, the official's file — whoever holds the record holds an advantage in any quarrel about what the record shows, because they can produce it, withhold it, or read it aloud in the light most favorable to themselves. What is old here is the asymmetry. What is new is that automation makes it total and routine. A dispute that once required a person to pull a physical file, and left the file itself as a stubborn object anyone could later inspect, now resolves inside a system where the only account of the transaction is a set of entries the system wrote about itself, kept in a place only the system can reach.

Set the two of you side by side and the imbalance is plain. On one side stands the system, with timestamps, logs, event streams, a complete and confident narrative of what occurred, rendered in the flat authority of a machine that does not appear to have any reason to lie. On the other side stands you, with your memory — which you know to be fallible, which the system knows to be fallible, and which everyone in the process will discount accordingly — and perhaps a screenshot you had the presence of mind to take, which proves only what your screen showed, not what the system did. You are being asked to contest the official account of events with the one form of evidence the process trusts least, against an opponent who authored the account and controls the only copy.

You cannot get a fair hearing when your opponent is also the stenographer, the archive, and the only one permitted to read the file aloud.

Evidence you cannot see, held by your opponent

The deeper problem is not that the system's record might be wrong. Any record might be wrong. The problem is that sole custody by an interested party corrupts the value of a record even when it happens to be right, because there is no way for the affected person to tell the two cases apart. A record you cannot inspect, held by the one party who benefits from a particular reading of it, tells you nothing you could rely on. You cannot confirm it. You cannot challenge it. You cannot even establish what it says — you have only the system's report of what its own record contains, which is a claim, not evidence for the claim.

And the record is not fixed the way a paper file is fixed. It is data in a database the system owns, and data can be revised without leaving a mark. This does not require anyone to be a villain. It requires only that the possibility exist and be undetectable, because once the possibility of quiet revision exists and cannot be ruled out, the record loses its standing as evidence whether or not it was ever touched. A number that could have been changed after the dispute began, by the party the dispute is with, without anyone else being able to know, is not a fact about the world. It is that party's current position, wearing the costume of a fact. You are not being shown what happened. You are being shown what the system now says happened, by a system that had both the means and the motive to make those two things differ.

From the receiving end this is the whole of the experience. You are not disputing the evidence; you never reach the evidence. You are disputing your opponent's summary of evidence they will not let you see, cannot prove they have not altered, and are under no obligation to explain. There is no file you can point to, no clerk you can subpoena, no neutral copy that would settle the question. The record exists, and it is decisive, and it belongs entirely to the other side.

A record that does not belong to one side

What fairness requires here is not that the system be trusted less. It is that the record stop belonging solely to one party to the dispute. Three things follow, and none of them is exotic. The first is access: the person a decision lands on should be able to see the evidence about their own case, in the form the system actually relied on, not a paraphrase issued after the fact. A record no affected party can read is not a record in any sense a fair process would recognize; it is a private note the system keeps about you. The second is tamper-evidence: the record must be built so that the keeper cannot quietly revise it after the dispute begins without the revision becoming visible. This is the entire function of a chain of custody, imported from the one domain — the handling of evidence — that has thought hardest about what it takes to trust a record held by an interested party. The third, and the strongest, is independent verifiability: the record's integrity should rest on something a skeptic can check, not on the keeper's good faith. Good faith is exactly what is in dispute. A record whose only guarantee is the honesty of the party you are fighting has offered you no guarantee at all.

The difference this draws is between a log and evidence. A log is something a system keeps for itself, useful to the system, owned by the system, meaningful only on the system's word. Evidence is something a process can use — because the affected person can see it, because it cannot have been rewritten after the fact without detection, because its integrity can be established by someone with no stake in the answer. A Decision Receipt is worth the name only insofar as it is the second kind of thing and not the first. The point of provenance and tamper-evidence is not ceremony. It is to take the record out of the sole custody of the party to the dispute and turn it into something the person on the receiving end can actually hold up.

Until that happens, the disagreement is not a hearing. It is the system marking its own homework in a fight it is also in — writing the record, keeping the record, reading the record, and grading your objection against a copy you will never see. You can insist you cancelled, that the payment cleared, that you never clicked. But you are arguing against the only witness, and the witness works for the other side.

— Dispatches · Summit Cognitive