Make the record fail closed

Find the line in your system where the consequential action commits, and then ask what happens if, at that exact instant, the record store is unreachable. Not degraded, not slow — down. In most systems the honest answer is that the action goes through anyway and the record is simply lost. Nobody decided that on purpose. It is the emergent behavior of two design choices that felt reasonable in isolation: the decision path and the logging path are separate, and the logging write is best-effort. Put those together and you have built a system that will act when it cannot account for the act. You made that choice. You just made it by default.

This is worth naming plainly because the failure hides behind ordinary engineering vocabulary. We call it logging, and logging is the thing you are allowed to drop under pressure — a log line that fails to flush is a nuisance, not an incident. But a record of a consequential decision is not a log line. It is the only evidence that the decision was made on a basis anyone can inspect. When that write fails and the action proceeds, you have not lost a log. You have produced an unaccountable act — a real consequence for a real person with no durable account of why. The directive of this piece is narrow and load-bearing: for the decisions that matter, make the record a precondition of the action, so that no record means no act.

Your logging already fails open

Look at how the two paths are wired, because the coupling is the whole story. The business action — the charge, the denial, the account suspension, the automated escalation — runs on the transactional path, the one with retries and alerts and someone paged when it breaks. The record runs on a different path: an async emit, a fire-and-forget call to a collector, a message dropped on a queue that may or may not be drained. The two are decoupled precisely so that the record cannot slow the action down. That decoupling is the vulnerability. It means the action does not know, and structurally cannot know, whether its own record survived.

Now consider when that record store tends to be unavailable. It is not random. Outages, overload, deploys gone wrong, the incident already in progress — the record store goes down under the same conditions that produce your most contested decisions. So the write fails exactly during the window when the decisions most need accounting, and it fails silently, and the action sails through. You end up with a system whose record is complete on the calm days when nobody needs it and full of holes on the day the postmortem gets written. A best-effort record is not a record that is usually there. It is a record that is guaranteed to be missing at the worst possible moment, by construction.

The record is a precondition, not a side effect

The correction is to treat the record the way you already treat money. You would never let a payment system apply a charge and then attempt, separately and optimistically, to write the ledger entry — and shrug if the ledger write threw. You make the charge and the ledger entry atomic: either both commit or neither does. Do the same here. Where the action and its record can share a transaction, put them in one, so that an act without a durable record is not a rare bug but an impossible state. Where they cannot — the action touches an external system, the record lives in a different store — reach for the same discipline by another mechanism: write the record first and make it durable, then perform the action, and reconcile so that a record without a completed action is the recoverable case and an action without a record cannot occur. The ordering is the point. Commit the account before you commit the consequence.

When you genuinely cannot make them atomic, do not fall back to proceeding blind. Degrade deliberately. That means one of a small number of honest options, chosen in advance: queue the action and block until the record is durable; refuse the high-stakes action outright and return a clean failure the caller must handle; or route the decision to a human who will create the account the machine could not. Each of these is a real cost — added latency, a rejected request, a person pulled in. Pay it knowingly. The alternative you are declining is to let the system act with no way to ever answer for the act, and that is not a cheaper option, only a quieter one.

A system that will act when it cannot record has already ranked the two, and told you it considers the account the expendable half.

The failure that hides itself

Here is the operational trap: a dropped record is the one error that erases its own evidence. A failed charge throws, retries, pages someone. A failed record, on the best-effort path, produces nothing — no exception the action sees, no downstream anomaly, no gap that announces itself. The action succeeded; the metric that matters looks green. So you cannot rely on ordinary error handling to surface this, because the whole pathology is that the error was swallowed upstream. You have to monitor for the absence — reconcile action counts against record counts, alert when a consequential outcome exists with no durable account behind it, and treat any nonzero gap as a real incident rather than acceptable loss. The metric to watch is not how many records you wrote. It is how many actions you took that no record can explain.

The obvious objection is availability, and it deserves an honest answer rather than a dismissal. Fail-closed can mean fail-stopped, and coupling every write to a durable record can turn a record-store blip into a full outage. The answer is that you do not fail closed on everything — you tier by stakes. The vast majority of what a system does is inconsequential and can keep failing open without anyone being wronged; reserve the discipline for the consequential slice, the decisions that impose a cost someone could later be entitled to contest. On that slice, the latency of a durable write and the occasional refusal are not overhead to be optimized away. They are the price of being able to account for what you did, and they are cheap next to the cost of a decision you cannot defend.

Which brings the temptation into focus: the move that creates this whole failure mode is the decision to make the record asynchronous for performance. It is almost always the right call for telemetry and almost always the wrong call for accountability, and the two get conflated because they run through the same logging library. When you make a consequential decision's record async to shave latency, you are not making a performance optimization. You are deciding that under load — exactly when the system is deciding fastest and getting watched hardest — some of those decisions will happen with no record at all, and you are choosing not to know which ones. An unrecorded consequential action is not a logging gap you can backfill. It is an unaccountable act, and you authorized it the moment you made the record best-effort. Go find that line in your system, and make the record a precondition of crossing it.

— Dispatches · Summit Cognitive