The referee and the record

Consider the strangest role in the whole apparatus of modern knowledge: a person, usually anonymous, usually unpaid, who reads your work with the specific intention of finding out where it fails. This person is not your enemy and not your friend. They have no stake in your success and, ideally, none in your failure either. They were selected because they know the subject well enough to see the crack you cannot, and their function — the entire reason the institution went to the trouble of finding them — is to stand between your claim and the world and ask, before anyone else does, whether it should be let through. We call this refereeing, and we have grown so used to it that we forget it was an invention, and a strange one: an adversary built into the process on purpose.

The habit is old, though its full shape came late. When the Royal Society began publishing the Philosophical Transactions in 1665 under its first secretary, Henry Oldenburg, it was already doing something new: a learned body took institutional responsibility for what appeared under its name, and Oldenburg, as editor, referred certain papers to fellows competent to judge them before they were read out or printed. That was not yet the systematic external refereeing we now mean by "peer review" — that arrived gradually, hardening into standard practice across the sciences over the nineteenth and twentieth centuries, and in many journals only becoming routine within living memory. But the germ was there from the start, and it is worth naming precisely, because the germ is the whole point: a consequential claim should be examined by a disinterested expert before it is granted the authority of publication, not after.

The built-in adversary

What problem did this actually solve? Not dishonesty. The referee was not invented because scientists lied; most did not. It was invented because of a subtler and more universal failure — the author is the last person able to see the flaw in their own work. You cannot referee yourself, not because you are corrupt but because you are inside the claim. The assumption you did not notice you were making, the step that felt obvious because you had already made it a hundred times, the alternative reading you never entertained because your result was too pleasing to interrogate — these are invisible from within precisely to the degree that they are load-bearing. Confidence is not evidence of soundness. Often it is what soundness and self-deception have in common.

So peer review does something that no amount of individual virtue can accomplish: it institutionalizes doubt. It takes the skepticism that a careful thinker is supposed to apply to their own work — and cannot reliably apply, being human — and assigns it to someone else, someone structurally positioned to feel it. Doubt made procedural. The referee is doubt with a name and a deadline, doubt that has to be answered rather than merely felt. And crucially, the doubt comes first. The scrutiny is spent before the claim is granted standing, not afterward when the finding is already in the world doing its work and the cost of being wrong has already been paid by someone.

The genius of peer review was to spend the skepticism before publication, not after the harm — to put the adversary between the claim and its authority, where it is still cheap to be wrong.

Authority is granted, not assumed

This reframes what publication in a serious venue actually means, and the reframing matters. A refereed result does not carry authority because it is true — referees are fallible, and reviewed papers are retracted. It carries authority because it survived something. A disinterested expert tried to find it wanting and could not, or could only in ways the author then addressed. The warrant is not the claim; the warrant is the claim plus the record of what it withstood. That is why "peer-reviewed" functions as a credential at all. It is shorthand for: this did not simply assert itself into standing; it was made to earn standing against a challenge it did not control.

Notice how much of that warrant lives in the fact that the challenge happened and was recorded. A finding that no one contested and a finding that a hostile referee failed to break can be word-for-word identical on the page, and they are not epistemically equal. The second has a history the first lacks. This is the quiet discipline peer review encodes, easy to state and easy to skip: authority over consequential matters should be granted by surviving an adversary, and the surviving should be part of the record. Not the polish of the result, not the eminence of the author, not how sure it sounds — the challenge it withstood. Strip that away and you are back in the world the Royal Society was built to leave behind, where a claim is trusted because of who said it and how well.

This is a different discipline from the one reproducibility encodes, and the difference is worth keeping sharp. Reproducibility — the demand that a claim travel with enough method to be re-run by a stranger who was not there — is about whether the result holds up when repeated. Refereeing is upstream of that: it is the adversarial gate before the claim is published at all, the disinterested reader who tries to break the argument before the world is asked to rely on it. One asks whether the finding can be reproduced; the other asks whether it should be released. A mature system wants both. They fail in different places.

A referee for the decision

Now turn the lens, and the omission is stark. A machine decision is granted authority with no referee whatsoever. It is published to the world — which is to say, acted upon — the instant it is produced, by the very system that produced it, with no independent adversary standing anywhere between the claim and its consequence. The model that reaches the conclusion is the same model that emits it, and there is nothing in the loop whose job is to try to find it wanting first. A person is denied, a transaction is flagged, an application is sorted, and the decision is its own referee. It sounds sure, and sounding sure is treated as though it were the same as having survived a challenge. It is not. It is the exact thing peer review was invented to stop us from mistaking for warrant.

The remedy is not to slow every machine decision to the pace of a journal, and it is not to pretend a referee can read a billion outcomes. It is to build the adversarial check into the decision as a matter of construction, so that a consequential outcome carries the record of having been challenged the way a reviewed paper carries the record of having been read. A decision built to be refereed is examinable — its inputs and the rules it ran under are exposed rather than asserted. It is replayable — a skeptic can re-run it against what was actually known and see whether the conclusion follows. It is contestable — there is a party with standing to attack it and a surface to attack it on. And the adversarial check itself is recorded, so that the difference between a decision that survived scrutiny and one that merely sounds confident is a fact about the record, not a matter of faith. A Decision Receipt built this way earns authority the way a refereed result does; a decision that is its own referee has none of that assurance and all of that confidence.

The fellows of the Royal Society understood something we are on the verge of forgetting at enormous scale: that a claim which has answered to no one is not yet knowledge, however plausible, and that the cheapest place to catch a mistake is before it is granted authority rather than after it has done its work. We are now producing consequential judgments faster than any referee could ever read them, and we have quietly decided that speed excuses us from the discipline that made the judgments trustworthy in the first place. It does not. The adversary was never a courtesy we could drop when we got busy. It was the thing that turned an assertion into a warrant — and a decision that skips it is not faster knowledge. It is confidence with the referee removed.

— Dispatches · Summit Cognitive

---