Write the threshold down before you tune it

Find out who can change a threshold in your systems, and how, and whether anyone would know. In most organizations the answer is unflattering: a cutoff can be moved by someone close to a dashboard, in a config change that reads like any other, with no record of who decided, what trade-off they were striking, or against which population. The most consequential parameter in a decision system — the line that determines who bears the cost of its errors — is the one your governance does not touch, because it does not look like a policy. It looks like a setting. Govern the setting.

Here is why the line matters more than almost anything else you review. A model only ever estimates; it produces a score and refuses nobody on its own. The threshold is where the estimate becomes a verdict — where everything above a line is treated one way and everything below it another. And where you put that line decides how the system distributes its inevitable errors: how many people it wrongly refuses in order to avoid wrongly approving, or the reverse. That is not a tuning parameter. It is a decision about whom to disadvantage, made once and applied to everyone, and it is exactly the kind of decision a governance function exists to own. Yet it is usually the one that slips past, because it arrives as a number rather than as a proposal.

So the directive, and it is a governance directive, not an engineering one: no operating point ships or moves without a record of the choice it represents, written before the change takes effect. The threshold must be governed as the decision it is — and a decision that is not written down, with its rationale and its date, has not been governed. It has merely happened. Your job is to convert "someone adjusted the cutoff" into "this trade-off was chosen, by this owner, for this reason, on this date, and is due for review on that one." That conversion is the whole of it, and it is almost entirely a matter of refusing to let the line move silently.

Three questions the record must answer

Make the standard concrete, because a governance rule that cannot be checked is decoration. A complete account of an operating point answers three questions, and you should be able to demand all three before approving any change. What is the threshold, stated as the deliberate choice it is and not buried as a constant in a config file? Who set it, and what trade-off does it strike — which errors is it accepting more of in order to accept fewer of which others, and against what population was that trade-off judged? And when was it last examined against the world it is now being applied to? A change that cannot answer those three is not ready to ship, and a governance function that approves it anyway has signed off on a decision it never saw.

The first two questions force the choice into the open. The threshold-setter is usually optimizing for a metric — translating a vague instruction like "keep losses under control" into a specific operating point — and experiences the work as tuning, not as a moral allocation of harm. Asking them to write the trade-off in words is what turns the tuning back into a decision someone can be accountable for. The institution that records "we set this cutoff here, accepting these refusals to avoid those approvals" has produced something an affected party can argue with, and something a regulator can review. That visibility is the point. A buried line cannot be contested, which is precisely why lines stay buried.

A threshold nobody wrote down is a verdict nobody chose — reissued, unexamined, against people it was never calibrated for.

The third question is the one most governance functions forget, and it matters because thresholds drift even when no one touches them. A line set for one population keeps applying as the population shifts, as the upstream model is retuned, as the meaning of a given score moves beneath it. A cutoff that was a defensible trade-off last year can become an indefensible one this year without a single edit — the world moved under a fixed line. So the record needs not just an origin but an expiry: a date by which the operating point must be re-examined against current reality, the way you would re-examine any standing decision whose premises decay. An operating point with no review date is a decision you have agreed to stop thinking about, and stopping is how a reasonable cutoff turns into an indefensible one in the dark.

Make it a gate, not a guideline

The reason to write all this down before tuning, rather than documenting after the fact, is that after the fact is too late to be honest. A rationale reconstructed once a number is already live is a justification, not a decision — assembled to defend a choice that was actually made on instinct at a dashboard. The only record that captures the real reasoning is the one written at the moment of the choice, before the dial moves. So put the requirement in front of the change, not behind it: the operating point and its three answers are part of the change proposal, reviewed and approved as the change, not appended to it once it has shipped.

And make it a gate, not a guideline, because a guideline about thresholds will be observed exactly as often as it is convenient. If a cutoff can be moved without producing the record, it will be — under deadline, under incident pressure, under the ordinary friction of shipping. The control that works is the one wired into the path: the operating point cannot change unless the choice is recorded, the trade-off named, the owner named, the review date set. That is not bureaucracy for its own sake. It is the difference between a system whose most important decision is owned and one whose most important decision is an accident no one has to defend.

So the question to carry into your next review is not whether the model is accurate. It is whether anyone chose, and can defend, and wrote down, the line the model's scores are being measured against — and whether that writing happened before the line moved or after someone went looking for a reason. The model is the part everyone watches. The threshold is the part that decides. Govern the part that decides, and govern it before the dial turns, or you are auditing the estimate and leaving the verdict to whoever last had the dashboard open.

— Dispatches · The Field Manual · Summit Cognitive