Why calling it a hallucination makes AI errors harder to fix

We describe AI systems with words borrowed from human minds. They reason, they hallucinate, they understand, they know. Every one of those words imports a claim the evidence doesn’t support, and the confusion runs all the way into policy.

This isn’t pedantry about terminology. The vocabulary decides what people expect, what they trust, and what regulators think they’re regulating.

AI hallucination and the words doing too much work

Term	What it suggests	What is happening
Hallucination	A malfunction, a glitch to be fixed	Normal operation producing an unsupported claim
Reasoning	Deliberate thought	Generating intermediate tokens that improve the answer
Understanding	Grasping meaning	Statistical relationships that predict text well
Knows	Holding a fact	Weights that make certain continuations likely
Learns	Ongoing acquisition	A frozen model; nothing changes after training

Hallucination is the worst offender, because it implies an aberration. The model isn’t failing when it invents a citation. It’s doing exactly what it always does, and this time the likely continuation happened not to be true.

Calling it a hallucination suggests there’s a healthy state to return to. There isn’t. Correct and incorrect output come from the same process.

Why the metaphors stuck

They stuck because they’re useful, which is worth conceding before criticising them.

Saying a model “understands” a question is faster than describing token probabilities, and it predicts behaviour reasonably well most of the time. Shorthand earns its place when it compresses something true.

The trouble is that these particular shorthands compress in a direction that flatters the technology, and nobody chose the vocabulary that would serve users best.

The habit predates the current wave. The paper that introduced the architecture everything runs on was called Attention Is All You Need, and attention there is a precise mathematical operation rather than anything a mind does.

Researchers knew what they meant. The word escaped into general use carrying implications the mathematics never had, which is roughly how all of these terms travelled.

Reasoning is the live case

The word is doing real commercial work right now, and it’s worth examining what sits underneath it.

Google Research showed in Chain-of-Thought Prompting that “prompting a 540B-parameter language model with just eight chain of thought exemplars achieves state of the art accuracy on the GSM8K benchmark”, which is a genuine and measurable result.

DeepSeek’s R1 paper went further, reporting “self-reflection, verification, and dynamic strategy adaptation” emerging from reinforcement learning alone.

Those are strong findings. What they demonstrate is that generating more tokens before answering improves accuracy on checkable problems, which our piece on what thinking tokens buy unpacks.

Whether that constitutes reasoning is a separate question the papers don’t settle, and the marketing treats it as settled.

The benchmark evidence deserves its own caution here. Contamination is widespread enough that researchers now argue benchmarks should be built contamination-resistant by design, so some apparent reasoning gains are memory rather than capability.

Where the vocabulary causes actual harm

The consequences aren’t confined to arguments among researchers.

Users trust confident output because fluent language reads as understanding. That instinct is reasonable with people and misleading with models, and the training makes it worse: OpenAI’s InstructGPT work tuned models on human preference, and preference rewards confident, well-formed answers whether or not they’re right.

Detection is the other case. Watermarking now marks text at generation, but as TechTimes put it, the mark proves processing, not authorship.

Say a document was “written by AI” and you’ve made a claim about authorship that the evidence cannot support. That’s a vocabulary problem with a person on the other end of it, covered further in our guide to why detectors fail.

It reaches into regulation

Rules get written in the same borrowed vocabulary, and that shapes what they can require.

The EU AI Act sorts systems by risk, which is a deliberate move away from describing what models are toward describing what they do, as our piece on what the Act requires sets out.

That framing is more precise than the marketing language, and it’s why the Transparency Code could be enforced quickly. It requires marking output rather than adjudicating whether a system understands anything.

Better words than AI hallucination, where they exist

Some substitutions are easy and cost nothing in readability.

“Unsupported claim” or “fabricated citation” beats hallucination, because it names what happened rather than diagnosing a condition. “The model produced” beats “the model believes”. “Trained on” beats “learned”, since the model isn’t learning anything now.

Reasoning is harder, because there’s no compact alternative to “spends inference compute generating intermediate steps”. Sometimes the accurate phrase is genuinely worse writing, and that’s a real constraint rather than an excuse.

A workable compromise is to use the shorthand once and then be precise where it matters. Say reasoning model in the opening, then describe what it actually does when the distinction changes what a reader should conclude.

The counter-argument

A serious objection says this whole complaint is misplaced, and it’s worth taking properly.

We use mental vocabulary for all sorts of systems without confusion. Thermostats “want” a temperature and nobody is misled. Language is metaphorical throughout, and demanding literalism about AI while accepting it everywhere else is inconsistent.

There’s also a functionalist reply: if a system produces the outputs understanding would produce, insisting it doesn’t really understand may be defending a distinction that does no work.

Both land. The difference is that nobody makes decisions about a person’s job or grade based on what a thermostat wants, and the stakes are what make precision worth the effort here.

What would change this

Interpretability research could settle some of it. If we understood what internal structures produce a given output, we could describe them directly rather than reaching for mental vocabulary.

Scale alone has a similar history. The GPT-3 paper was titled Language Models are Few-Shot Learners, and learner is the same borrowed word, describing a model that does not learn anything after training ends.

Until then, watch which words appear in announcements and which appear in the papers underneath them. The papers say tokens, loss and benchmarks. The announcements say thinks, understands and knows, and the gap between the two is usually where the marketing lives.