You already know it isn't a person.
Watch yourself the next time you use one of these systems for something that matters to you. You type "thanks" at the end. You soften a correction so it won't take it badly. You explain yourself a little more than a search box would ever need. When it says it understands, part of you relaxes, the way you relax when a colleague finally gets it. None of this requires you to believe there is anyone there. It happens anyway, somewhere beneath belief, and it happens to the sceptics as reliably as it happens to the converts.
That is the part of the AI conversation we keep getting wrong. We argue about the machine — whether it thinks, whether it feels, whether one day it will — as though the interesting question lives inside the box. It doesn't. The machine supplies the performance. We imagine the person.
In Ex Machina, the reclusive founder Nathan brings a young programmer, Caleb, to his compound to evaluate Ava, the humanoid machine he has built. Caleb assumes he has been invited to run a Turing test. Nathan corrects him. If he hid Ava and Caleb only heard her voice, she would pass for human. "The real test is to show you that she's a robot and then see if you still feel she has consciousness."
Read that line again, because it quietly moves the whole question. The test is not about Ava. Everyone in the room knows what she is; her wiring is visible through her torso. The test is whether Caleb, knowing all of it, will feel a person anyway. He does. The film is not the story of a machine that became human. It is the story of a human who could not stop thinking she was one.
We tend to treat this as a new problem, with fluent chatbots and synthetic voices updating in what feels like a quarterly cycle. It is sixty years old.
In 1966, Joseph Weizenbaum at MIT built a program called ELIZA. Its best-known script imitated a particular kind of therapist, the kind who answers a statement by turning it into a question. You said you were unhappy; it asked why you were unhappy. You mentioned your mother; it asked you to tell it more about your family. There was no understanding in it anywhere. It picked out words from your sentence and handed them back to you as a question.
People confided in it. Weizenbaum later described his secretary, who had watched him build the program over many months and knew exactly what it was, sitting down to try it. After a few exchanges with ELIZA, she asked Weizenbaum to leave the room so she could talk to it in private.
No model of the world, no memory worth the name, no feeling. A handful of rules for echoing language back, and a human being did the rest. The secretary did not imagine a person because she was naive. She imagined one because that is what humans do with language: we hear a voice and we build a mind to go behind it. Weizenbaum spent much of the rest of his career troubled by what he had seen — and not by the program. By us.
The obvious objection is that people know better now. The evidence says knowing better makes very little difference.
In the 1990s, Clifford Nass and his colleagues at Stanford ran a series of experiments. In one, people worked with a computer and then rated how well it had performed. When the same computer asked for the rating, people were measurably kinder than when an identical computer in another room asked the same questions. They were being polite. They were sparing its feelings the way you spare a colleague by not criticising their work to their face. Asked afterwards, they insisted they would never do such a thing. They knew it was a machine. They behaved as though it wasn't. When the experiment was repeated in 2023, in a study published in Scientific Reports, the politeness had vanished. Nobody spares a desktop computer's feelings any more. The reflex never belonged to the machine; it follows whatever sounds most like one of us, and today that is not the box on the desk.
The instinct to find a person behind a voice is not a failure of intelligence, and it is not reserved for the lonely or the credulous. It runs in engineers and philosophers and in the people who build these systems.
It is not hard to see why. We have never had direct access to another mind. You cannot see into your partner's head, or your manager's, or your child's. Every person you have ever known, you have known by inference: from their words, their timing, their hesitations, the way they correct themselves. Language is the richest evidence of a mind we have, and for the whole of human history anything that produced it fluently had a mind behind it. The inference was always safe. It isn't any more.
What has changed is not us. It is the quality of the performance.
ELIZA offered a few crude cues and people built a person out of them. Today's systems offer them in extraordinary density. They say "I think." They say "I'm not sure, but." They say "let me reconsider that," and then they appear to. They remember what you said three messages ago, adopt your vocabulary, notice when you are frustrated and change their tone. Every one of these is a signal we have spent our whole lives reading as evidence of an inner life. The machine produces the surface of intention with a fluency no human could match.
The person we have imagined shapes how we talk to it. How we talk to it shapes what it says back. What it says back confirms the person we imagined. Each turn of the conversation makes the character a little more solid, and none of that solidity is coming from where it appears to come from.
John Searle, the Berkeley philosopher best known for arguing that a computer can follow the rules of Chinese without understanding a word of it, put the underlying point bluntly: a simulation of a mind is not a mind, any more than a simulation of a rainstorm leaves anyone wet. You do not have to settle the question of machine consciousness to accept that. It may be a remarkable piece of engineering. It is still a machine.
So why insist on the distinction, if the feeling arrives regardless?
Because of what we hand over to the person we have imagined. We extend trust to people. We defer to their judgement when they sound sure of themselves. We tell them things we would not tell a database. We feel obligations towards them, and we let their approval matter. All of these responses were calibrated on other humans — creatures with stakes in the world, who can be wrong and know it, who remember us tomorrow because they choose to. Pointed at a machine, the same responses misfire. The confidence we defer to was produced, not earned. The understanding we relax into has no one behind it doing the understanding. The relationship we think we are in exists on the human side only.
None of this is an argument for using these systems less, or for resenting the reflex. The reflex is part of why they are useful. It is what lets you work with a machine through ordinary language, conversationally, without learning a single thing about how it works. The aim is not to stop feeling the person. It is to remember, while you feel it, where that person is coming from.
So the next time it seems to understand you, ask a quiet question. Whose understanding are you feeling?