The machine cannot be embarrassed

For most of history, the person who decided your fate could be made to feel something about getting it wrong. The magistrate who misjudged a case might have to pass the man he ruined in the street. The official who denied a claim that should have been granted could be summoned to explain, in a room, to a face. We do not usually count this among the great instruments of justice, because it is informal and uneven and easy to corrupt. But it was always there, working quietly in the background — the prospect of being seen to have erred, of having to look at the consequence, of carrying the small private weight of having been wrong about someone who did not deserve it.

This discipline never appears in the official account of how decisions are kept honest. The official account speaks of rules and reviews and rights of appeal. But underneath the formal machinery ran an informal current of consequence: decision-makers were people, people can be embarrassed, and the anticipation of embarrassment shaped what they were willing to do. A great deal of restraint that we attribute to procedure was in fact attributable to this — to the simple, durable fact that a human being does not want to be the one who was obviously, demonstrably, face-to-face wrong.

An automated decision system removes this current entirely, and it removes it so cleanly that we have barely noticed it is gone. The machine cannot be embarrassed. It will not pass anyone in the street. It does not anticipate a review with dread, does not rehearse its justifications the night before, does not feel the particular discomfort of having to defend a judgment it secretly suspects was thin. It produces its output with exactly the same affect whether the output is its finest work or its worst, because it has no affect, and no stake, and no face to lose.

What shame was actually doing

It is worth being precise about the function that conscience and reputation served, because the point is not nostalgia for a more humane bureaucracy. Shame did not make decisions correct. Plenty of shameless people decided plenty of things wrongly, and plenty of conscientious ones erred in good faith. What shame did was narrower and more structural: it gave the decision-maker an internal incentive to prepare to be questioned. Knowing you might have to account for a judgment, to a person, in a way that could expose you, makes you assemble — if only privately, if only in your own mind — the grounds you would give. The dread of the review is what produces the readiness for it.

That readiness was a kind of record. Not a written one, usually, but a held one: the decision-maker carried in memory the reasons, the evidence weighed, the rule applied, the close calls and why they fell as they did. They carried it because they might be asked, and being asked and having nothing to say is the precise shape of the embarrassment they were avoiding. The internal discipline produced an internal account. We relied on that account constantly, often without realizing we were relying on something so fragile and so personal.

What conscience supplied was never correctness. It was the readiness to be questioned — and a system that feels nothing comes ready for nothing.

The machine carries no such account, and not because it is negligent. It carries none because there is nothing in it that would want one. The readiness-to-be-questioned that a human assembles out of self-regard simply has no analog in a system with no self to regard. The output appears; the grounds for it, if not deliberately preserved, dissolve the instant the computation ends. There is no equivalent of the official who, summoned a year later, can still recall why. There is only the output, and a silence behind it.

Rebuilding the discipline from outside

This is the real reckoning that automation forces, and it is not the one usually named. The familiar worry is that machines will decide unfairly. The deeper worry is that they will decide without the quiet apparatus of accountability that human decision-making smuggled in for free. We are removing the decider's conscience from the loop, and conscience was load-bearing in ways we never had to make explicit, because it came attached to the person and the person came attached to the job.

If the discipline will no longer arise from within the decider — and it will not, because the decider now has no within — then it has to be supplied from outside, deliberately, as something built. The internal account that a conscientious human held in dread of being questioned has to become an external record that exists whether or not anything fears the question. This is what a Decision Receipt is for, read in this light: it is the externalized form of the readiness that shame used to produce internally and for nothing. It carries the evidence the decision actually rested on, the rules that were active when it was made, and enough state to replay it — the very things a human would have assembled in anticipation of having to answer for the call.

The crucial property is that the record cannot wait to be motivated. A human assembles their justification because they might be asked; the machine will never be moved by the prospect of being asked, so the record has to be produced at the moment of decision, unconditionally, by the system itself. It must not depend on anyone caring, because the one entity in a position to produce it at the source is precisely the entity that cares about nothing. The discipline that used to live inside the decider, as a disposition, now has to live outside it, as an artifact — generated by default, retained by default, examinable by anyone with standing to look.

There is something almost consoling in stating it this way. We are not asking the machine to develop a conscience; that would be a category error and probably a worse outcome. We are recognizing that the thing conscience produced — a checkable account of why — was always the part that mattered for accountability, and that this part can be built directly, without the fragile and corruptible human feeling that used to be its only source. The machine cannot be embarrassed. So we stop relying on embarrassment, and we build, in its place, a record that does the work embarrassment did, for everyone, all the time, whether the system would have bothered or not.

— Dispatches · Summit Cognitive