The improvement loop needs a version

Continuous improvement can dissolve the object being governed. A model changes. A policy sentence is edited. A knowledge source refreshes. A tool gains a parameter. A grader threshold moves. A prompt is tuned for a new complaint pattern. Each change is small, and the product keeps the same name. When a consequential interaction is reviewed later, no one can identify the system that actually produced it.

OpenAI presents Presence as a production system that adapts as products, policies, user behavior, and models change. Proposed updates are tested against the version in production and can enter a controlled rollout. That comparison requires a stable deployed baseline. Versioning is not clerical metadata around the loop; it is how the loop knows what it is improving.

The version should describe the decision-relevant assembly: model and safety configuration, instructions, policy set, knowledge snapshot or retrieval rules, tools and scopes, escalation routes, graders, thresholds, and feature flags. A single application build identifier is useful only if it can resolve these dependencies exactly.

Version the assembly and the rollout

The rollout state belongs in the record because two users can encounter different systems at the same time. Segment, percentage, region, channel, experiment arm, and activation time can change the applicable behavior. Without rollout evidence, a later reviewer may reproduce the current default and falsely conclude that the original outcome is inexplicable.

Knowledge sources require a different strategy from immutable code. A large corpus may change too often to snapshot completely. Preserve source identifiers, retrieval timestamps, relevant retrieved passages or hashes, indexing version, and access policy. The goal is not to freeze every byte forever; it is to retain the evidence needed to understand the answer that was available then.

If the system can improve without becoming identifiable, accountability declines as capability rises.

Versioning should support comparison, not only lookup. Which component changed between two outcomes? Which policy revision altered escalation? Did a new model cause the gain, or did the grader move? A dependency diff turns continuous improvement from a sequence of anecdotes into an examinable operating history.

Emergency changes still need receipts. A fast rollback, safety patch, or policy correction may bypass ordinary cadence, but it should record who acted, why, what scope changed, what evidence justified urgency, and when the temporary state will receive full review. Speed changes the process; it should not erase the history.

Use semantic identifiers alongside component hashes. A hash can prove which artifact ran, but an operator also needs to know that the release changed the refund ceiling, replaced the language detector, or moved weekend escalations. A concise change manifest connects technical identity to operational meaning and tells reviewers which evaluations, approvals, training, and customer notices may need renewal.

Do not let mutable aliases become the evidence. Names such as latest, approved, production, and current are useful deployment pointers, but they change after the interaction. Resolve them to immutable versions at execution time and preserve the resolution in the receipt. The same rule applies to external APIs and hosted models whose provider-facing name may remain stable while behavior changes underneath it.

Retention should follow consequence and review horizons. Low-risk transient interactions may need only a compact configuration receipt. High-consequence actions may require longer preservation of the relevant evidence, approvals, tool responses, and policy assembly. Define the schedule before an incident, include legal and privacy constraints, and verify that archived dependencies can still be interpreted after the active platform has moved on.

A reproducibility exercise should be part of normal operations rather than reserved for disputes. Select past interactions, resolve every component version, reconstruct the available evidence, and determine whether the behavior can be explained. The goal is not identical generation from a stochastic system; it is a defensible account of the system, authority, information, and rules that bounded the original result. Track missing dependencies until their capture is repaired. Publish the exercise's coverage and failure rate internally.

Operate the boundary

The practical starting point is a named control for the complete decision-relevant assembly and rollout manifest. Write the boundary in terms an operator can evaluate: the initiating principal, permitted purpose, affected resources, allowed consequences, escalation path, expiry condition, and evidence produced. A policy sentence is useful context; the enforced object and its observable state are what make the policy operational.

Test the boundary by reconstructing selected past interactions after model, policy, knowledge, tool, grader, and feature-flag changes using only preserved identifiers and evidence. Preserve the starting state, the agent's route, any intervention, the final effect, and the gaps in observation. Repeat the exercise after changing a model, tool, provider, policy, or data source. A control that passed once should not silently lend its assurance to a materially different system.

The leading signal is the percentage of consequential interactions that resolve to a complete, reproducible assembly with no unknown dependency. Pair it with a consequence measure so teams do not optimize the dashboard while weakening the outcome. Review both on a fixed cadence and after every material incident or migration. When the signal disappears, determine whether the risk disappeared or the instrumentation did.

The release and service owner should own the decision to continue, narrow, pause, or expand the workflow. The owner needs authority over the control and access to its evidence; responsibility without either becomes ceremonial. Record the decision, the evidence cutoff, the residual uncertainty, and the next review date so the claim can age honestly.

— Dispatches · Summit Cognitive