Version the policy that decided

Your model is versioned. You can name the exact weights that produced any score, roll back to a prior checkpoint, and tell an auditor precisely which model saw which case. Your code is versioned too — every branch, every commit, every deploy accounted for. And then there is the layer between the score and the action: the threshold that separates approve from refuse, the rule that carves out the exception, the flag that routes a case to manual review, the number in a config file that someone raised by two points last Tuesday because the queue was backing up. That layer decides. And in most systems it is the one layer nobody versions.

Call it the decision policy — the thresholds, rules, business logic, and configuration that convert a model's output into an act. It is where the estimate becomes a verdict. It is also, in practice, the most volatile part of the whole system, because it is the part humans reach for when they want to change behavior without retraining anything. A model retrain is a project. Nudging a cutoff is an afternoon. So the policy drifts continuously, through a hundred small untracked edits, while the decisions it produces are logged — if they are logged at all — without any note of which version of the policy was live when each one was made. You can reconstruct the model. You cannot reconstruct the rulebook.

This is the same mistake as instrumenting the model instead of instrumenting the decision, seen from a different angle. The policy is not context around the decision; it is the decision, or the half of it the model does not supply. A record that captures the inputs and the score but treats the governing rules as ambient — current at read time, whatever they happen to be now — has recorded the arithmetic and lost the law.

The rulebook that drifted

Watch how it happens, because it never happens as a decision to leave the policy untracked. It happens as a series of reasonable local moves. A threshold sits in a configuration value, easy to change without a deploy, which is exactly why it was put there. A risk lead raises it because the false-positive rate crept up. A product owner adds an exception for a customer segment. An engineer sets a feature flag to shadow-test a new rule on ten percent of traffic, then forgets to remove it. None of these edits is wrong. But none of them is recorded as a version of the thing that decides, with an identifier, a timestamp, and a note of what changed. The policy is treated as a setting, not as an artifact — and settings have a present value, not a history.

The bill comes due when a decision is questioned. Someone asks why this application was refused in February. You pull the record. It has the inputs, it has the score, it may even name the model version. What it does not have is the policy version — because there was no such thing as a policy version, only a config file whose February state nobody preserved. So you do the natural thing, which is also the wrong thing: you evaluate the February inputs against the policy you have, which is April's. Maybe the threshold moved. Maybe the exception that would have caught this case did not exist yet. You get an answer, cleanly and confidently, and it is an answer to a question nobody asked — what would happen to this case today — dressed up as an account of what happened then. You cannot say which threshold, which exception, which rule set was live at the moment that mattered, so you substitute the current one and call the substitution a reconstruction.

A decision without its policy version is a verdict with no record of which law was in force — and you will be tempted to judge yesterday's call by today's rules, which is how you exonerate or convict the wrong system.

Policy is code that decides

The directive follows from taking the policy seriously as what it is: code that decides, deserving the same discipline as any other code that decides. Put it under version control. Give every version a stable identifier. Stamp every decision record with the identifier of the policy version that produced it. And keep old versions retrievable — not as archived text, but as something you can actually run — so that any past decision can be replayed against the rules that governed it rather than the rules in force now.

Concretely, this means the thresholds and rules stop living as loose configuration that has only a present state and start living as versioned objects with a history. When the risk lead raises the cutoff, that is not an edit to a value; it is a new policy version, identified and dated, with the old one preserved intact. When a decision is made, the record does not point at “the policy” — a moving target — but names the specific version that was live at that instant, frozen into the decision record the way the inputs and the score already are. The Decision Receipt should carry the policy version the way it carries the evidence: as part of the account, not a footnote pointing outward at a document that may have changed.

Do this and reconstruction becomes honest. To ask why the February decision came out as it did, you retrieve February's policy version by its identifier and replay the case against it. The question the replay answers is the right one — given these inputs, this model, and the rules that were actually binding, does the outcome follow? — instead of the misleading one you get by default. If it reproduces the outcome, the record holds. If it diverges, the divergence is the finding, and you can tell whether it came from the model or from a policy that has since moved. Either way you are judging the decision by the law that was in force, which is the only way a judgment of a past decision means anything.

The honest hard parts

Two things are harder than that clean picture admits, and it is worth saying so plainly. The first is that versioning the policy is more than committing a rules file to git. The effective policy — the thing that actually determined the outcome — is often larger than the tidy file that appears to hold the rules. It includes the configuration values read at decision time, the feature flags that were on for that particular case, and sometimes an interaction between the model and its threshold that neither captures alone. A clean rules document can claim one cutoff while a flag quietly overrode it for a slice of traffic. So the discipline is not “version the rules file.” It is: capture whatever actually determined the decision, including the config and the flag state, and bind that whole effective policy to the record. Version the thing that decided, not the thing that documents what you meant to decide.

The second is that keeping old versions runnable — not merely readable — has a real maintenance cost. A policy version is only useful for replay if you can still execute it against inputs, and that means carrying the dependencies, the rule engine, the interpretation logic as they were, not just the parameter values. That is genuine work, and it grows as the system evolves. The honest answer is not to pretend the cost is zero; it is to scope it to what warrants it. For a recommendation nobody will contest, the drift may not be worth chasing. For decisions with consequences for a real person — the ones that get questioned, appealed, litigated — the ability to replay against the governing policy is the difference between an account you can stand behind and a plausible story about today. Pay the cost where the decision is consequential, and know that you are choosing not to pay it elsewhere.

There is a further reason this matters more as systems get more autonomous. A policy that a person edits on a Tuesday at least changes at human speed. A decider that updates itself from its own experience changes faster than anyone edits a config file, and the version that decided a given case may no longer exist by the time the case is questioned. A moving decider without versioning is a system that cannot account for its own past, because its past is continuously overwritten. Versioning the policy is how you keep a fixed point to reason from — a record of which rulebook was in force — in a system built to keep changing the rules.

So when you build the seam where the score becomes an action, do not treat the rules there as a setting with a present value. Treat them as an artifact with a history. Version the policy, identify each version, stamp every decision with the version that produced it, and keep the old versions runnable for the decisions that will be contested. The alternative is not neutrality. It is a system that will always answer the question about yesterday with the rules of today, and never be able to tell you it did.

— Dispatches · Summit Cognitive