The difference between a log and a record

When something goes wrong, the reflex of a modern institution is to reach for the logs. There are always logs. There are logs of the logs. Terabytes of them, timestamped and indexed, sitting in cold storage against exactly this kind of day. And so it comes as a genuine surprise, each time, to discover that the logs do not answer the question. They contain everything except the thing you need. This is not bad luck, and it is not a gap that more logging would have closed. It is the nature of the object. A log and a record are different kinds of thing, and the difference is the whole subject of this dispatch.

A log is exhaust. It is produced as a byproduct of a system running — a trace emitted so that engineers can debug, so that operations can watch throughput, so that some future forensic effort has raw material to sift. Its defining feature is that it is written without knowing what question it will be asked. It records what was cheap and convenient to record at the point the code happened to be, in the vocabulary of the machine, at whatever granularity someone once configured. It accumulates. Nobody curates it, because curation would require knowing in advance which of the million events mattered, and the entire premise of a log is that you do not.

A record is the opposite premise. A record is built because someone knows a question will be asked later, and builds now the exact artifact that will answer it. It is curated at the moment of creation. It selects, from everything that happened, the things that will need to be accounted for, and it captures them in the form an account requires — not the machine's convenience, but the answer's shape.

A log knows what the machine did. A record knows what someone will need to prove. You can reconstruct one from the other only if you are very lucky, and luck is not a control.

Why you cannot reconstruct a record from logs

The tempting belief is that a record is just a well-organized query over the logs — that everything needed is in there somewhere, and reconstruction is a matter of effort. Sometimes this is true. Usually it is not, and the reasons are instructive. First, logs record events, not the reasons behind them. They will faithfully show that a threshold was applied and an outcome produced, but not which version of the rule was active, not the intent the threshold encoded, not what would have happened had one input differed. The reasoning is not in the exhaust because reasoning was never something the system emitted; it was something it did and discarded.

Second, logs record in the order and vocabulary of the machine, not the decision. To answer was this decision fair you need the evidence that was actually in front of the decision, in the order it was consulted, framed as the decision framed it. The log has the raw reads scattered across services, interleaved with a thousand unrelated events, with no marker saying these, together, in this sequence, were the decision. The edge of the decision — where it began, what it looked at, where it ended — is exactly the thing exhaust does not contain, because drawing that edge is an act of judgment and logs do not judge.

Third, and most quietly, logs rot. They are retained by policy, rotated, downsampled, and eventually deleted, because storing exhaust forever is expensive and nobody made the case that any particular event was worth keeping. A record is retained because it was designated as evidence when it was made. The difference in retention is downstream of the difference in intent. You keep what you curated. You lose what merely accumulated.

Building the record on purpose

A Decision Receipt is a record in the strict sense: an artifact created at the moment of decision, curated to answer the question can this decision be accounted for — carrying the evidence consulted, the rules as they stood, and enough state to replay. It is not a query over logs run later. It is written when the decision is made, by a process that knows the question in advance, in the shape the answer will need. That is what makes it provenance rather than exhaust.

The practical consequence is that you cannot decide after the fact to have kept records. Curation is a design-time act. The institution that runs for years on logs alone and then, on the day of the inquiry, tries to assemble a record from the exhaust, discovers that the material was never the right shape and half of it is gone. The material was abundant. It was simply the wrong kind. Volume was never the missing ingredient; intent was.

So when someone reassures you that everything is logged, hear it precisely for what it is: an assurance about exhaust, not about accountability. The right question is not do you have the data but did you build the record — did anyone, at the moment the decision was made, curate the artifact that would answer for it later? If the answer is a wave toward the log archive, you are being told there is a great deal of material and no record. Those are not the same, and only one of them will still be there, and legible, on the day it matters.

— Dispatches · Summit Cognitive