The risk score at sentencing

Consider a hearing that decides whether a person goes home or is held. Picture it plainly, without drama: a courtroom in the middle of an ordinary week, a defendant who has not yet been to trial or has just been convicted of something, and a judge weighing how much liberty to leave in that person's hands. Into that weighing comes a number. It arrives on a form, alongside the more familiar materials — the charge, the history, the arguments of counsel — and it purports to say how likely this person is to reoffend, or to fail to appear, or to be dangerous. The number was produced by an automated model. How it was produced is proprietary. The defense, asked to respond to it, can respond to almost nothing about it, because there is almost nothing about it they are permitted or able to see.

This is a composite, and I want to be careful with it, because the subject deserves care rather than heat. But the structure it isolates is real, and it is worth naming precisely. The number is not decorative. It is doing work — informing a decision about detention, which is to say a decision about the most concrete thing a state can do to a person short of ending their life. And it is doing that work while remaining, in every respect that matters to a defense, unexaminable.

Evidence that cannot be cross-examined

The right to confront the evidence against you is not a technicality that criminal procedure happens to include. It is close to the whole point. A person accused by the state is entitled to know what is being said against them, to test how it was arrived at, and to put its weaknesses in front of the decider. Cross-examination is the instrument of that entitlement: it is how a claim is made to earn its place, by surviving the pressure of a motivated adversary who is allowed to ask how, and on what basis, and how do you know.

A risk score built on a proprietary model refuses that pressure by construction. Ask how the number was produced and the answer is that the method is protected. Ask what in this person's particular record drove it up rather than down and there is no account that can be interrogated — only the output, presented as a fact about the defendant. The defense cannot test the instrument, because the instrument is sealed. It cannot show the number was reached in error, because it cannot see the path to the number. It cannot even establish what the number would have been had one input been different, because the mapping from inputs to output is exactly what is withheld. What is offered instead is the assurance that the model is sound. But an assurance you are not permitted to test is not evidence you can confront. It is a conclusion you are asked to accept.

So the confrontation right, in this corner of the proceeding, is hollowed while remaining formally intact. The defendant may still cross-examine the arresting officer, still challenge the witnesses, still argue the law. What they cannot do is lay a finger on the one piece of the case that came from a machine — and that piece is weighed all the same.

A number you cannot question is not evidence in a courtroom; it is an oracle, and we abolished trial by oracle for good reasons.

Legitimacy rests on contestable process

It matters to be clear about what is and is not in dispute. The state has the authority to detain. That authority is real, it is lawful, and nothing here contests it. The question is not whether the state may hold a person. The question is what makes its holding legitimate rather than merely permitted — and the answer, in a system like ours, is process. The authority to detain is one thing; the legitimacy of a particular detention is another, and it is earned in the manner of the decision, not asserted by the fact of the power. A criminal justice system rests its claim to authority on due process precisely because raw force does not legitimate itself. What legitimates it is that the person on the receiving end had a real chance to contest the case against them.

An unexaminable model dropped into that process is a foreign object. The entire architecture around it — the rules of evidence, the right to counsel, the burden on the state, the appealability of the outcome — exists to make the state's case contestable, to give the accused a genuine surface on which to fight. The score sits inside that architecture and honors none of it. It cannot be contested, and so it is the one part of the proceeding that operates on a different principle from everything around it: not persuade and withstand challenge, but assert and be believed. A process that has admitted something it cannot examine has not merely added a tool. It has admitted, into the most rights-intensive setting the law knows, a claim that is structurally exempt from the very testing that gives the setting its authority.

This is why the problem is one of legitimacy and not only of accuracy. A score could be accurate on average and still corrode the thing that makes a sentence or a detention legitimate, because legitimacy was never a promise of correctness. It was a promise of contestable process. Substitute a number no one can challenge for a case the defense can fight, and the promise is broken even when the number is right — because the person held was held on a basis they were never allowed to answer.

What admissibility would demand

None of this argues that a court must never use an instrument. It argues that an instrument used against liberty has to meet the standard the setting already sets for everything else, which is that it be examinable enough to contest. Admissibility, in this sense, is not a courtesy. It is the condition under which a claim is allowed to bear weight against a person.

Concretely, a score offered at a bail, sentencing, or parole decision would have to come with the actual basis for it — not the aggregate reputation of the model but the account of this decision: what about this person's record moved the number, made available in a form the defense can read and challenge, so that the defendant can put its weaknesses in front of the judge the way they can with any other evidence. It would have to carry enough of its own operation to be tested, rather than hiding the operation behind trade secret and offering only the output. And it would have to be received as one input into a judgment that a human being owns — a judge who weighs it, can discount it, and remains answerable for the decision — rather than a verdict the human defers to and then attributes elsewhere. A record built for the person affected, addressed to the party with the most at stake, and available while the decision can still be reopened, is what turns a sealed output into something a court can legitimately use. Absent that, the honest description is not that the court considered a risk assessment. It is that the court held a person partly on the word of an instrument neither the defendant nor, in any meaningful sense, the court itself was permitted to examine — and called it evidence.

A score used against a person's liberty must meet the standard of evidence. It does not get to be treated as evidence while being shielded, as a product, from everything that being evidence entails. The trade-secret interest of a vendor is a real interest in a commercial dispute. It is not a reason to hold a human being on a basis they cannot confront. Between those two things the priority is not close, and a system that lets the commercial interest win has misplaced the thing it exists to protect.

The scenario above is illustrative — a composite drawn to show a pattern, not an account of any real person, company, or event.

— Dispatches · Summit Cognitive