What courts know about evidence that ML evaluation forgot

The law of evidence is one of the oldest pieces of working software in the world. It has no compiler and no version control, but it has been patched continuously for roughly four hundred years, by people who learned the hard way what happens when you let the wrong information decide a question that matters to someone's life. It is, at bottom, a single sustained argument about a deceptively narrow problem: not what is true, but what is allowed to settle the matter. Those are not the same question, and the entire discipline exists because someone once mistook them for each other and a person paid for it.

Machine-learning evaluation inherited that same problem and answered it in one line. Does this data improve a metric on a held-out set? If yes, it belongs in the training corpus; if no, it does not. That rule is elegant, it scales, and it is responsible for most of what the field has accomplished. It is also a radical simplification of a question that courts spent centuries refusing to simplify. ML evaluation kept the word predictive and quietly discarded the word admissible — and the gap between those two words is where most of the trouble we are now living through actually lives.

The easy version of this argument is wrong, so let me set it aside. The point is not that courts are slow and good while machine learning is fast and bad. Courts are often slow in ways that hurt people, and machine learning is often the only thing standing between a question and no answer at all. The point is narrower: a tradition built specifically to make consequential decisions contestable accumulated a body of knowledge ML evaluation never had occasion to acquire. The rules of evidence are not obstacles the law failed to optimize away. They are the optimization — what is left after centuries of watching plausible information go wrong.

What the rules are actually for

Consider what a court does before it lets a single fact decide anything. It asks where the fact came from and whether you can trace its custody from origin to courtroom without a gap — chain of custody. It asks whether the thing is what its proponent claims it is, not merely whether it is convincing — authentication. It asks whether the person whose statement is being used against you can be put in a chair and questioned — the right to confront and cross-examine. And it asks, even of evidence that is genuinely relevant, whether its tendency to unfairly prejudice the decision outweighs what it legitimately proves — the rule against unfairly prejudicial evidence. Each of these is a filter, and each filter exists because admitting true-but-tainted information turned out to corrupt outcomes in a way that took generations to diagnose.

Now hold each filter up against how a model is built, and watch what is missing.

Start with provenance. A court will exclude a perfectly accurate piece of physical evidence if nobody can account for where it was between the scene and the trial — not because the gap proves tampering, but because an unaccountable gap means the evidence can no longer be trusted to be contestable. A training corpus is, by construction, a custody gap the size of the internet. The data improved the metric; that is the whole of what we know about it. We typically cannot say where any given example came from, who produced it, under what conditions, or whether it was already the output of another model. The held-out test set measures whether the result is useful. It is structurally incapable of asking whether the inputs were admissible, because admissibility was never one of the things it was built to see.

Then authentication. The evidentiary question is not "does this look right?" but "can you establish that it is what you say it is?" — a question about the identity and integrity of the source, answered before the content is ever weighed. ML evaluation runs the opposite way. An input earns its place by its effect on the output distribution, never by its verifiability. Unverifiable inputs are not rejected; they are simply averaged in, and a sufficiently large average will launder almost anything into something that performs. The metric cannot tell a load-bearing fact from a confident fabrication, because both move the number in the same direction.

An explanation a model gives you is testimony from a witness you are not permitted to cross-examine.

Which brings us to the one that should worry us most: the right to confront. Hearsay is presumptively excluded not because secondhand statements are usually false, but because you cannot interrogate the person who made them — you cannot test their memory, their motive, their perception, their candor under pressure. The confrontation right encodes a hard-won insight: a statement you cannot question is not evidence you can rely on, however true it might happen to be. A model output is hearsay in its purest form. Ask the model to explain itself and it will produce something fluent and responsive, but you are not cross-examining a witness. You are prompting a system to generate a new statement under no obligation to correspond to whatever actually drove the original one. There is no oath, no memory to probe, no inconsistency you can catch and hold it to. The very thing the confrontation right was designed to provide is the thing a model is constitutionally unable to give.

Relevance is not the same as fairness

The fourth principle is the subtlest, and it cuts closest to the metric itself. Courts draw a line between evidence that is relevant and evidence that is admissible, and the gap between them is governed by the rule against unfair prejudice. Evidence can be entirely relevant — it can genuinely make a fact more likely — and still be excluded, because the way it will move the decision is out of proportion to what it actually proves. Relevance is necessary. It was never meant to be sufficient.

A loss function knows only relevance, and it knows it in the most permissive form imaginable. Any correlation that reduces error is, by definition, rewarded. The model has no rule against unfair prejudice and no mechanism for one, because prejudice in the evidentiary sense — a feature that sways the outcome more than it deserves to — is indistinguishable, at the level of the metric, from signal. This is the technical root of every proxy-discrimination story the field has had to apologize for. The forbidden feature was not a bug in the optimization. It was relevant. It improved the metric on the held-out set. The held-out set, having no concept of admissibility, did exactly what it was built to do and waved it through.

Notice that none of these failures are failures of accuracy. A model can be more accurate than any human on a benchmark and still be assembled entirely out of evidence that no court would let near a verdict — untraceable in provenance, unauthenticated at the source, immune to cross-examination, and rewarded for correlations it should never have been allowed to use. Accuracy was the only thing we measured, so accuracy is the only thing we got, and we mistook it for the whole of what makes a decision sound.

Borrowing the knowledge, not the courthouse

The fix is not to put models on trial or slow them to the pace of litigation. It is to recover the distinction the law has been protecting all along — that information which predicts is not the same as information fit to decide — and to build that distinction into how systems are evaluated rather than bolt it on afterward as an apology. That means treating provenance as a property of training data, the authentication of inputs as a precondition, a model's account of itself as testimony to be tested rather than trusted, and refusing to let a metric alone certify that a correlation has earned the right to govern an outcome.

The law of evidence is not a relic that software has outgrown. It is a four-hundred-year record of which true things turned out to be unsafe to decide with, written by people who paid for each entry. ML evaluation began that ledger over from a blank page and is rediscovering the same entries the expensive way — one regulatory action, one reversed decision, one ruined reputation at a time. The knowledge already exists. We simply built a discipline that had no way to ask for it, and then acted surprised that it never arrived.

— Dispatches · Summit Cognitive