The shutoff the system ordered

Consider a household flagged as delinquent. The account has crossed some threshold — a balance past due for enough days, a meter reading that jumped in a way the system reads as non-payment — and the utility's disconnection process does what it was built to do. It schedules the service for shutoff, generates a notice explaining that the shutoff is scheduled, and moves on to the next flagged account. The determination that the household owes what it is said to owe, and the notice informing the household of the consequence, are produced by the same process, in the same pass, on the same evidence. No one has looked. Perhaps the balance is a billing error. Perhaps the meter was misread, or swapped, or reporting a neighbor's draw. Perhaps a payment posted a day after the cutoff and the two records have not yet met. The process cannot tell the difference, because telling the difference was never its job.

What makes this worth stopping on is not that the system might be wrong. Every process is wrong sometimes; that is not news. It is that this particular decision, when it executes, does something to the physical world. It is not a note added to a file. Power stops flowing. Water stops arriving. Heat goes out of a house. A shutoff is a determination that acts — it reaches out of the database and changes the conditions a person is living in — and it does so at a speed and a scale that no human hand could match, which is precisely why utilities automate it and precisely why the automation deserves more scrutiny than the manual version it replaced, not less.

A decision with a physical body

Most of the decisions we worry about in automated systems produce records. A score, a flag, a ranking, an eligibility finding — these are claims about the world that some later step might act on, and the gap between the claim and the action is where review can live. A shutoff collapses that gap. The determination and the consequence are welded together: to decide is to act, and the acting lands on the one place that cannot be abstracted away, which is the house where a person is.

The batch process that orders the shutoff sees fields. It sees a balance, a date, a status code, a meter value. It does not see the oxygen concentrator plugged into the wall in the back bedroom, or the newborn who cannot regulate her own temperature yet, or the fact that it is the first hard freeze of the year. Those facts exist. They are simply not in the schema, and a process can only weigh what the schema carries. This is not a flaw that better data hygiene fixes, because there will always be a fact about a particular household that the fields do not hold and that would have changed the answer if a person had known it. The shutoff process is structurally blind to the very thing that makes a shutoff dangerous.

The meter does not know it is winter; that knowledge was the job of the person the process replaced.

The pause the process removed

There used to be a pause. When shutoffs were carried out by people — a clerk pulling a list, a technician driving to the address — there was friction in the loop, and inside that friction lived a kind of unacknowledged safeguard. The clerk who recognized the address of a family with a sick child, the technician who saw the wheelchair ramp and made a call before killing the power, the supervisor who held a batch because a storm was coming: none of that was written down as policy, and none of it was reliable, but it caught things. Human discretion was doing quiet governance work that no one had named, and because no one had named it, no one thought to preserve it when the process was automated.

That is the real hazard of automating a decision like this. It is not that a machine is colder than a person, though it is. It is that automation removes the hesitation without anyone deciding to remove it. The friction gets optimized away as inefficiency, and the safeguard that was riding inside the friction goes with it, silently, as a side effect nobody chose. The household that would have been caught by a clerk's second look is now caught by nothing, and the absence is invisible because it was never a step in the first place.

Which means the operating choices are the whole game. What balance triggers an automatic shutoff, and after how many days. What protections suspend it — a medical exemption on file, a cold-weather rule, a pending dispute. What human review, if any, gates the action before it fires. These are not settings. They are moral choices wearing operational clothes, and the fact that they live in a configuration file rather than a debated policy does not make them less consequential; it makes them less examined. Automating the decision did not lower the standard it must meet. It raised it — because the human judgment that used to absorb the standard's hard cases has been engineered out, and something now has to be built to replace it deliberately, since it will not be there by accident anymore.

What governance owes the household

Three things follow, and they are demands on the governance of the system, not on its accuracy. First, the household is owed the actual basis and the rule in force, in a form it can contest. Not "your account is past due" but the specific balance, the specific readings, the specific rule the process applied and the version of that rule as it stood when the process applied it. A shutoff notice that states only the conclusion gives the household nothing to argue with; it can protest the outcome but cannot reach the reasoning, and a determination you cannot reach is a determination you cannot rebut.

Second, a decision that is irreversible in effect the moment it executes must not execute without human review. Restoring power the next day does not undo the night. The reversibility that makes automation feel safe — we can always turn it back on — is an accounting fiction when the harm lands in hours that cannot be returned. A physical action this heavy earns, as a hard requirement, a person in the loop before it fires: not to slow everything down, but to stand where the clerk used to stand, on the specific cases the thresholds flag as closest to the line.

Third, the whole determination should leave a record built to justify it — a Decision Receipt that preserves the evidence the process saw, the rule it ran, and the state that produced the outcome, so that when a household says this was wrong there is something to examine rather than a status code and a shrug. The governance point underneath all three is the plain one the composite is meant to expose: a shutoff policy run automatically is still a policy, and a policy is something a person must own and be able to answer for. The machine executes. It does not decide, and it cannot be asked. Someone chose the thresholds, someone chose which exemptions the schema would carry, someone chose whether a human would look before the power went out. Governance is the discipline of making sure that someone can be named — and that the record they left is strong enough to have justified the shutoff in advance, to the household, before the house went dark.

The scenario above is illustrative — a composite drawn to show a pattern, not an account of any real person, company, or event.

— Dispatches · Summit Cognitive