There is a particular kind of tiredness that comes from working alongside a machine that is almost good enough.
You have felt it. The model drafts the memo in seconds, and you spend twenty minutes reading it the way a lawyer reads a contract someone else wrote — not to understand it, but to find the place where it quietly went wrong. The car holds the lane and manages the distance, and you sit with your hands hovering near the wheel, doing nothing, watching for the one moment when doing nothing becomes the wrong thing to have done. The books reconcile themselves, and you scan the reconciliation for the entry that doesn't belong.
This is not the exhaustion of hard work. It is the exhaustion of surrendering the work while keeping the watch. We were promised that automation would take the task off our hands. What it has actually done, for now, is take the doing and leave us the vigilance — which turns out to be the more tiring half.
The comfortable story is that this is temporary. We are in a transition, and transitions, by definition, end. There was a past in which we did all of the work ourselves. There is a present in which the system does ninety per cent and we nudge the remaining ten. And there is a future, close now, in which the system does the whole of the ordinary work and interrupts us only when it meets something it cannot handle — an exception, flagged and handed up, so that we can stay comfortably disengaged until the rare moment we are actually needed. On the other side of the transition, the watch is finally put down. You surrender, and the surrender is safe.
It is a good story. It is the story that sells the software. And it rests on a single assumption that does not survive contact with how the world is actually built: that the loop can close. That there is a version of any consequential system in which the human is, at last, fully outside it.
Follow one decision far enough and you will see that there isn't.
Take the case that ought to be the purest form of surrender — an agent managing your money. It watches the markets, it rebalances the portfolio, it makes the calls a human used to make, and it makes them while you sleep. Suppose it makes a bad one. The loss is real and it is yours, but the decision was the bank's system, so the risk sits with the bank. Banks do not hold risk they can price away, so the bank insures it. The insurer, in turn, does not price complex risk by hand any more; it runs its own models, its own agents, to decide what the bank's agent is likely to cost. So we now have an agent underwriting an agent, and somewhere behind that, a reinsurer with an agent of its own. It is the oldest problem in governance wearing new clothes: who checks the checker. Except now the checkers check each other, all the way down, and the honest question is where the checking finally stops.
Here is where the story quietly wants you to conclude that it stops in a pool — that far enough down the chain, a large enough balance sheet simply absorbs the loss as a statistic, and no human judgment is required at the bottom. It is wrong, and it is wrong because it runs together two things that have to be held apart. A cost can be pooled. It can be spread across a thousand policies, averaged over a decade, funded from a reserve, and in that sense a cost really can be handed to a system and forgotten. But a bad decision does not only produce a cost. It produces a question — who is answerable for this — and answerability is not the kind of thing that pools. You cannot average it across a portfolio. You cannot fund it from a reserve. It does not get smaller when you spread it out. It has to land somewhere, whole, on someone.
And it does. We know it does, because the law has already decided the matter, and it did not decide it in favour of the machine.
The legal system abhors a vacuum. Confronted with a harm and asked to find no one responsible because the responsible party was a piece of software, it declines. It reaches instead for the nearest human it can plausibly hold — and where the nearest human is not obvious, it defines one in advance. This is not a gap the lawyers have failed to close. It is the closing itself. The European Union's AI Act does not grant an autonomous system any standing of its own; it splits the world into providers and deployers and defines each, explicitly, as "a natural or legal person." Aviation regulators treat a drone flying a fully autonomous, pre-programmed path as though the software were merely an advanced steering wheel — the registered human operator answers for wherever it steers, pre-programmed or not. And in the place that matters most to anyone running a regulated business, the United Kingdom's Senior Managers regime requires that a named executive — a specific person, by title — be personally accountable for the behaviour of the algorithms under them. The black-box defence, the plea that the decision was made too fast and too deep in the code for any person to own, is not merely unpersuasive in these frameworks. It is explicitly refused.
So the human at the end of the chain is not a design choice you can engineer away. It is a fixture. And once you see that, the comfortable story doesn't just weaken — it inverts. Because the question stops being whether a human stays in the loop, and becomes what that human is actually there for.
Consider what happened on a road in Tempe, Arizona, in March of 2018.
An experimental autonomous vehicle detected Elaine Herzberg five and a half seconds before it struck and killed her. It had time. What it did not have was the ability to decide what it was looking at. Its perception system classified her first as an unknown object, then as a vehicle, then as a bicycle, and each time it changed its mind it discarded the history it had been building and started again — so that the seconds it had were spent not tracking her but repeatedly meeting her for the first time. Its manufacturer's automatic emergency braking had been deliberately switched off, to keep the ride from feeling jerky; the system was designed to rely on the human behind the wheel to intervene. When the accounting was done, the vehicle's maker faced no criminal charge. The person who did was the safety driver in the seat, who had looked away.
She had looked away, and that was a real failure, and it is also the whole point. The system had been built in a way that all but guaranteed a human would eventually be asked to catch, in a fraction of a second, a mistake the machine had taken five seconds and several changes of mind to make — while the machine's own capacity to stop had been turned off by design. There is a name for the role she was occupying without knowing it. She was the moral crumple zone: the part of the system positioned, like the crumpling front of a car, to absorb the force of a failure it did not cause and could not have prevented. The human in the loop was not there to make the system safe. The human was there so that when the system failed, accountability would have somewhere to come to rest.
The same shape appears in finance with the people taken out entirely. In August of 2012 a market-maker called Knight Capital deployed some code, left one old instruction alive on one server, and watched a retired algorithm wake up and begin buying with no idea how to stop. In forty-five minutes it executed four million trades and lost four hundred and forty million dollars — faster than any human could read the situation, let alone correct it. When the regulator arrived, it could not fine the algorithm and it did not try. It fined the firm, and the fault it named was not the code. It was the failure to supervise: the ignored warnings, the absent kill switch, the forty-five minutes it took human hands to find the machine and turn it off. The accountability did not attach to the decision. It attached to the watching of the decision. Liability had transferred from fault for the act to fault for failing to oversee an act that, by design, was already beyond overseeing.
This is the thing the comfortable story cannot survive. As we automate, accountability does not dissolve into the system and disappear. It does the opposite. It concentrates — onto fewer people, further from the actual decision, with less real ability to shape what the machine does, and less and less practice at doing it themselves. The safety driver, the named executive, the operations team reading warnings they didn't write for a system they couldn't stop: these are not the people who have surrendered the work. They are the people who kept the one part of it that cannot be handed over. Not the doing. The answering.
You do not get to put down the watch on the other side of the transition. There is no other side. There is only a rearrangement of who stands where when the failure arrives — and a quiet, steady movement of the most consequential position, the one where accountability finally lands, toward you.