The agent that learned from you

Most of the accountability machinery we have built for automated decisions rests on a quiet assumption: that the thing being held to account holds still. You audit a model, certify it, write down what it does, and the document you produce stays true because the model stays the same. Contest a decision six months later and you can go back to the same system, feed it the same inputs, and watch it behave the same way. The system is a fixed object. The record describes a thing that is still there to be described. Nearly every practice we call auditing depends on this and rarely says so, because for most of the history of software it was simply true.

Adaptive and agentic systems break the assumption without announcing that they have. A system that learns online — that updates its weights, its policies, or its retrieved memory from user interactions, from feedback, from its own past outputs — is not a fixed object. It is a process. The version that decided your case last week has been altered by everything that happened since: every correction, every click, every thumbs-up, every conversation it had with someone who was not you. When you come back to contest the decision, the system that made it is gone. Not deleted — evolved. And a decision-maker that has moved on is, for the purposes of accountability, a decision-maker who has disappeared.

The decider that will not hold still

Consider what it means to reconstruct a decision. The whole project of accountable automation is the ability to answer, after the fact, a specific question: on what basis did this system reach this result? To answer it you need the inputs, the rules in force, and the system that combined them. For a static model the last of these is free — the system now is the system then, so you only have to worry about the first two. This is why so much practice treats the model as a given and instruments only the data around it. The given was safe to take for granted.

With a learning system the given evaporates. The model now is not the model then. If it has updated a thousand times since the decision, then auditing the current version tells you about a system that never made the decision you are asking about. It is a sibling, perhaps a close one, but it is not the same witness. You can examine it exhaustively and learn nothing dispositive, because the thing you are examining was not present at the events in question. The reconstruction fails not because the inputs were lost but because the decider was — it kept moving after it acted, and nobody wrote down where it was standing when it did.

This is the distinct accountability problem that adaptive systems introduce, and it is not a variation on the familiar ones. It is not opacity: the system may be perfectly legible at any given instant. It is not missing logs of the inputs: you may have those in full. It is that the identity of the actor is temporal. To hold a learning system accountable for a specific act, you have to be able to name and recover the state it was in at the moment of that act — and current state, however thoroughly documented, is the wrong object.

The drift you cannot trace

It gets worse, because the process by which the system changes is usually even less recorded than the states it passes through. A model that learns from interaction is being reshaped continuously by a stream of signals — corrections from users, reinforcement from outcomes, its own outputs folded back in as training data. In most deployments that stream is treated as ephemeral. The feedback does its work on the weights and then it is gone. What remains is a new version, with no attached account of what pushed it there.

The consequence is that you cannot trace the drift. If the system today decides differently than it did last month, you often cannot say why — cannot point to the feedback that moved it, cannot distinguish learning that improved it from learning that corrupted it. And corruption is not hypothetical. A system that learns from everything learns from adversaries too. It can absorb a bias from a skewed run of interactions. It can be gamed by users who discover that a particular pattern of behavior shifts its outputs in their favor. It can be poisoned by a signal deliberately crafted to move it. If none of the reshaping feedback was recorded, all of these look identical from the outside: the model simply behaves differently now, and you have no way to ask whether it learned something it should never have been allowed to learn.

You cannot cross-examine a witness who has become a different person since the events in question — and a model that learns from everything is exactly that witness.

A witness who has changed can at least be asked what changed them, if they remember. The learning system, by default, does not remember. It carries the effect of its history in its parameters and keeps no account of the causes. So even the modest accountability we extend to a person who has revised their views — tell me what convinced you — is unavailable, because the record of what convinced it was never kept.

Versioning the decider

The response is not to forbid learning; adaptation is often the point, and freezing a system to make it auditable can cost more than it saves. The response is to treat the identity of the decider as something that must be captured, not assumed. If the system will not hold still, then the record has to hold still on its behalf — by pinning, for each consequential act, which version of the system took it, updated on what, so that the specific decider can be named and recovered even after it has moved on.

This extends a principle the series has argued for in the static case. A decision record has always owed its reader the inputs the decision rested on and the rules in force at the time. For a learning system the record owes one thing more: the state of the decider at the moment it decided — a version identifier that resolves to a recoverable configuration, and enough provenance about the updates that produced it to distinguish honest learning from a bias, an exploit, or a poisoned signal. This is what makes replay meaningful for an adaptive system. Replaying the same inputs through today's model tells you what today's model would do; replaying them through the pinned version tells you what actually happened, which is the only question a contest cares about.

None of this requires exposing how the system learns internally, and none of it stops the system from continuing to learn. It requires only that the system, at the moment it acts, records the identity of the version that acted and a traceable account of what shaped it — so that the particular decider can be brought back and examined even after the living system has left it behind. A learning system that cannot do this is not merely hard to audit. It is a decision-maker that erases itself in the act of deciding, and hands you, if you complain, a stranger who was not there.

— Dispatches · Summit Cognitive