Keep the clock honest

Look at where the timestamps in your decision records come from. Almost certainly the answer is that each service reads its own local wall clock at the moment it writes, and that clock is whatever the operating system happens to believe the time is. That belief drifts. Two services in the same cluster will disagree by tens or hundreds of milliseconds on a good day and by whole seconds when something is wrong. And the clock is settable — by the operator, by a bad config push, by anyone with root on the box that produced the record. You have built an accountability artifact whose most load-bearing field is a number the party being held accountable can quietly change. When a dispute arrives and turns on time, you will discover that the field cannot answer, or worse, that it can be adjusted to give whatever answer is convenient.

This is a build decision, and like most of the ones that matter it is made by default, in the wrong direction, because the default was easy. Reach for the system clock and you get a plausible-looking timestamp for free. What you do not get for free is a timestamp anyone should believe. Treat trustworthy time as an accountability requirement, on the same footing as recording the inputs and the policy: a record that cannot say when, defensibly, is missing a fact it will be asked for.

The clock the operator can set

Start with what actually goes wrong. Machine clocks drift — quartz oscillators run slightly fast or slow, and without correction a server wanders from true time at a rate you can measure in seconds per day. Different services drift differently, so the timestamps in your ledger are not merely imprecise, they are inconsistent with each other: service A's record of an event and service B's record of a later event can carry timestamps in the wrong order. And underneath both problems sits the one that turns a technical nuisance into an accountability failure — the clock is settable. Local time on the machine that writes the record is under the unilateral control of the party operating it. Nothing in a bare wall-clock string distinguishes a time that was read honestly from one that was written to say what the operator wished had happened.

Now stage the dispute. Someone asks whether the disqualifying input arrived before the decision was made or after it — whether the decision rested on that fact or was taken in ignorance of it. Someone asks whether the policy that governs the outcome was actually in force at the instant of the decision, or came into effect an hour later. These are questions about order, and order is exactly what drifting, disagreeing, settable clocks cannot establish. Sequence is not decoration on a decision; sequence is meaning. A fact read before a contradicting fact does different work than the same fact read after — it is available to be weighed, or it is not. A record that cannot establish order cannot reconstruct the decision, because it cannot say what the decision could have known.

A timestamp the operator can set is not a record of when it happened; it is a record of when they would like you to believe it did.

Trusted time and defensible order

The remedy has three moves, in ascending order of what is at stake. First, synchronize and monitor. Run NTP or something better against a disciplined source, and — this is the part usually skipped — measure the skew and alarm on it, so you know when a clock has wandered instead of finding out from a contested record months later. Synchronization does not make time trustworthy against a determined operator, but it makes ordinary drift and disagreement visible and bounded, which is the floor.

Second, for consequential records, take the time from a source outside the interested party's unilateral control. The whole problem with local wall-clock time is that the party the record exists to hold can set it; the answer is to remove that unilateral power. A trusted timestamping authority will attest that a record existed at or before a given moment, and that attestation does not depend on trusting the operator's own clock. Where a full timestamping authority is more than the case warrants, the weaker but real substitute is tamper-evident sequence — monotonic, cryptographically chained sequence numbers such that the operator cannot backdate an entry or slip one in behind another without breaking the chain. You may not be able to prove the exact minute, but you can prove the order, and order is what most disputes actually turn on.

Third, capture the ordering relationships you will need, not just isolated stamps. If the case can hinge on whether the input preceded the decision, then record that the decision was taken against a known, sequenced state of inputs — bind the ordering into the record rather than hoping two independent timestamps can be compared after the fact. Isolated timestamps invite exactly the argument you cannot win: that the clocks disagreed, that one was off, that the comparison is meaningless. A defensible ordering, written down at the time, forecloses it.

The honest limits

Be honest about what is achievable, because the honesty is load-bearing. Perfect global time in a distributed system is genuinely hard. Clock skew is real and never fully eliminated; the practical relativity-of-simultaneity of independent machines means there is no single universal instant you can stamp everything against and trust to the microsecond. Anyone who promises a perfect universal clock is selling something. So do not aim there. Aim at defensible ordering for the events whose order can matter, and trusted time for the slice of records consequential enough to be contested. That is a bounded, buildable target. The fantasy of a perfect clock is not the goal and never was; the goal is a record whose account of when — and above all of what came before what — a skeptic who does not trust you is compelled to accept.

This is why shared trustworthy time is the precondition of any shared record at all — the point of the clocks that agreed. The advance that let scattered parties keep a common account of the world was never a more accurate clock in one hand; it was an agreement about time that no single party could quietly overrule. Your decision ledger needs the same thing at the scale of your own systems. When you next stand up a service that writes a decision record, ask where its sense of time comes from and who can change it. If the answer is the local clock and the operator, you have a ledger that can be edited by the one party it exists to constrain. Synchronize it, source its time from outside your own reach for what matters, and write down the order you will have to defend. Keep the clock honest, because a record you cannot trust to say when is a record that cannot say what happened.

— Dispatches · Summit Cognitive