The decision and the double bind

The phrase has a reassuring sound to it. A human in the loop. Whatever the system does, a person sees it, weighs it, and signs off — so there is always someone who decided, someone who can answer for it, someone whose judgment stands behind the outcome. It is offered as the answer to anxieties about automated decisions, and on its face it ought to be. A human reviewer is exactly the accountable agent that a machine cannot be. But the phrase conceals a question it never asks, which is whether the human in the loop is actually positioned to decide anything at all, or has merely been positioned to be blamed.

Consider the situation the reviewer is often placed in. The system produces an output — an approval, a denial, a flag, a score — and presents it for human confirmation. But the reviewer is given the output without the evidence that produced it, or with so much evidence and so little time that examining it is impossible. They are expected to clear a queue at a rate that allows seconds per item, in a process designed around the assumption that the system is usually right and the human is a formality. And then, when something goes wrong, the same arrangement that made real review impossible turns and asks: but a human approved this. The human is held to account for a decision they were never given the means to make.

The shape of the bind

This is a double bind in the precise sense — a situation engineered so that the person cannot win, because the demands placed on them contradict each other. They are told to exercise judgment and given no basis on which to exercise it. They are told to review and given no time to review. They are told the decision is theirs and shown nothing that would let them make it differently from the system's recommendation. And underneath all of it runs the implicit threat that whatever happens, they will own it. To approve is to ratify a decision they could not examine. To decline is to obstruct a system they are presumed to trust, at a speed that punishes hesitation. There is no available action that constitutes genuine review, because genuine review was not among the things the design left room for.

What this produces is not accountability but its appearance — accountability theater. The structure manufactures a responsible human at the point of decision, so that the institution can say a person decided, while ensuring that the person had no actual capacity to decide. It is worse than having no human in the loop at all, and this is the part that deserves to be stated plainly. An automated decision with no human attached is at least honest about what it is; everyone knows the machine decided, and the question of who is accountable is left open and pressing. An automated decision with a powerless human attached has been laundered. It now has a name on it, and the name absorbs the accountability that should have stayed with the system and its designers. The theater does not add a safeguard. It removes one, by satisfying the demand for a responsible party with a person who could not have been responsible.

To hold someone accountable for a decision they could not examine is not to give them responsibility. It is to give them the blame and keep the power.

What real review would require

None of this is an argument against human review. It is an argument that human review is a real thing with real preconditions, and that satisfying the form of it without the substance is the specific failure to avoid. If you want a human in the loop to mean what it sounds like it means, you have to give the human the two things the bind withholds. The first is a record they can actually use — not the bare output, but the evidence the decision rested on, the rule that was active, the basis on which the recommendation was made, presented in a form a person can read and weigh in the time available. A reviewer who cannot see what the system saw is not reviewing; they are rubber-stamping with extra steps.

The second is the time to use it. A review surface that takes a minute to read honestly, dropped into a queue that budgets ten seconds per item, is a review surface in name only. The pace of the loop has to be set by what genuine examination requires, not by what throughput would prefer. This is uncomfortable, because the entire appeal of automation is speed, and real human review is the place where speed has to yield. But the alternative is not faster accountable review; it is the theater, which is fast precisely because it is not review. You can have a human who actually decides, or you can have the throughput that makes deciding impossible. What you cannot have is both, and the double bind is what you get when an institution pretends otherwise.

So the test for any human-in-the-loop arrangement is simple to state and uncomfortable to apply. Ask whether the human could, on the materials and in the time they are given, have reached a different conclusion than the system's and known why. If the answer is yes, there is a reviewer, and the accountability that attaches to them is earned. If the answer is no, there is no reviewer — there is only a name placed where a decision should have been, waiting to be blamed for a decision it was never allowed to make. A record the reviewer can actually use, and the time to use it, are not refinements of human review. They are the difference between human review and its forgery.

— Dispatches · Summit Cognitive