The black box and the flight recorder

The phrase has quietly reversed itself, and almost no one has noticed. When an engineer today calls a system a black box, he means something whose inner workings are sealed from view: inputs go in, outputs come out, and what happens between is opaque even to the people who built it. It is a confession of ignorance dressed up as a technical term. But the original black box was the opposite kind of object. It was a device invented for the express purpose of making what happened inside a machine recoverable and readable after the machine had destroyed itself. The name we now reach for to describe inscrutability began its life as the name for legibility. Somewhere between those two meanings sits the whole story of a discipline we already paid for and are now busy forgetting.

Consider how aviation actually became safe, because the answer is not the one intuition offers. It did not become safe because pilots grew more careful, or because manufacturers were exhorted to try harder, or because the public learned to trust the industry. Those things happened, but they are not the mechanism. Aviation became safe because it built an instrument that survived the crash and told the truth about it — and then built a culture around that instrument that treated the record, not anyone's testimony, as the authority. The safety record of modern flight is not a triumph of trust. It is a triumph of reconstruction: the patient, adversarial rebuilding of what actually happened from evidence engineered to outlast the event it documents.

The person usually credited with the idea is David Warren, a researcher at the Aeronautical Research Laboratories in Melbourne, who in the mid-1950s proposed a unit that would continuously capture cockpit voices and instrument readings so that, in the aftermath of an otherwise inexplicable crash, investigators would have something to read besides wreckage. His memo bore the flatly practical title A Device for Assisting Investigation into Aircraft Accidents. The premise was that the most important witness to a disaster is the one who cannot survive it, and that the only way to hear that witness is to have written down, in advance and tamper-resistantly, the inputs and actions leading up to the failure. Around the same device grew the institution that gave it force: no-fault accident investigation, later codified internationally, in which an independent body reconstructs the sequence of events and — crucially — is kept separate from the assigning of blame.

What the recorder actually solved

It is worth being exact about what problem the flight recorder solved, because it is not the obvious one. It did not make aircraft trustworthy. It made them accountable, which is a different and lesser and far more achievable thing. A recorder does not promise that the machine will not fail; it promises that when the machine fails, the failure can be read. The engineering requirements follow from that promise and are severe. The record has to be survivable — it must withstand the very impact and fire that destroyed everything around it, which is why the device is armoured, and why, contrary to its nickname, it is painted high-visibility orange so it can be found in the debris. It has to be tamper-resistant, so that no interested party can quietly revise the account after the fact. And it has to capture the actual inputs, not a summary of them, because a summary is already an interpretation, and interpretation is exactly what the investigation exists to supply.

The second half of the solution was cultural, and it is the half that gets overlooked. A perfect record is useless in a system that punishes honesty, because everyone with knowledge will shade it. So the discipline separated finding-the-truth from assigning-blame: the investigators whose job is to reconstruct what happened do not decide who pays for it. That separation is what lets the record be honest. It is why the reconstruction can trust the data over the narrative, and why participants have reason to let it. The flight recorder and the no-fault investigation are two halves of one invention — a survivable, authoritative record, and an institution built to read it without flinching.

The inversion of the name

Now hold that original meaning against the way we use the term today. We call a machine-learning system a black box to mean that we cannot see why it did what it did — that its reasoning is unavailable, its inputs unrecorded, its behaviour reproducible only by luck. We have taken the name of the one device whose entire purpose was to make a machine legible after failure and pinned it on machines that are illegible by construction. This is not a harmless etymological drift. It is a confession. We adopted the aviation metaphor and discarded the aviation engineering. We kept the word for the thing that made flight answerable and used it to excuse building decision systems that answer for nothing.

We kept the aviation metaphor for the machine we cannot understand, and threw away the invention that let aviation understand its machines.

The comparison is unkind, but it is precise. Most consequential machine-decision systems today have the aircraft and not the recorder. They have the capable, fast, high-consequence machine — the part that acts. What they lack is the tamper-evident capture of the actual inputs, the rules that were in force at the moment of the decision, and enough state to reconstruct the sequence that led to the outcome. And they lack the second half too: the investigative culture that trusts the record over the story, that reconstructs rather than accepts a post-hoc account. When one of these systems fails, there is no orange box in the wreckage. There is a log if you are lucky, an explanation generated after the fact if you are not, and, most often, a shrug.

Building the recorder for decisions

The lesson transfers cleanly, because the underlying problem is identical: a fast, opaque machine takes actions that harm people, and afterward someone needs to know what actually happened. The answer aviation reached is the answer here. Build the recorder first, and treat its survivability and tamper-evidence as design requirements rather than as features to add if there is time. A decision system needs the equivalent of a crash-survivable recorder — a record that captures the inputs as they stood, the rules as they were, and enough state that the decision can be run again and its path reconstructed. This is what a Decision Receipt is for: not a reassuring summary of what the system would like you to believe it did, but a record engineered so that reconstruction can proceed over recollection, and so that the record, not the narrative, is the authority in any later review.

And the cultural half matters as much for decisions as it did for flight. Reconstruction has to be able to trust the record, which means the record has to be built to be trusted by an adversary — provenanced, tamper-evident, admissible in the sense that it could survive a hostile reading. The point of the flight recorder was never to make anyone feel better about the crash. It was to make the crash investigable, so that the next one might not happen. We already know how to do this. We did it for aviation, at real cost, over decades, and the safety record is the proof. That we borrowed the name for the machines we cannot understand, while leaving behind the discipline that made the name mean something, is not an accident of language. It is the whole indictment.

— Dispatches · Summit Cognitive

---