One of the most unsettling things about modern AI is that it can state a complete falsehood with the same calm confidence it uses for the truth. It invents a citation, a date, a quote, or a fact that sounds entirely plausible and simply is not real. People call this hallucination, and the instinct is to treat it as a bug. It is closer to an inevitable feature of how these systems work, and understanding why is the first step to using them safely.
A machine that predicts, not knows
A language model does not look up answers in a store of verified facts. It predicts likely text, one piece at a time, based on patterns learned from enormous amounts of writing. When you ask a question, it is not retrieving the answer so much as generating the most plausible-sounding continuation of your words. Usually the most plausible continuation is also the correct one, which is why these tools are useful at all. But when it is not, the model has no internal alarm that fires. Plausible and true are different things, and the system is optimised for the first.
When invention is most likely
Hallucinations are not random; they cluster around predictable situations. Knowing them tells you when to raise your guard:
- Specific facts the model half-remembers — exact figures, dates, page numbers, and names are prime territory for confident errors.
- Obscure or recent topics that were thin or absent in the material the model learned from.
- Questions with a false premise, where the model plays along with your assumption rather than challenging it.
- Requests for sources, which a model may fabricate because it has learned what a citation looks like, not which ones exist.
How to catch it
The defence is not to abandon the tool but to use it like a fast, confident, occasionally unreliable assistant — valuable for drafting and thinking, never the final word on a fact. Ask it to show its reasoning, which makes shaky steps easier to spot. Treat any specific claim you would act on as a lead to verify, not a conclusion to trust. And where it matters, give the model the source material yourself rather than relying on its memory, so it is summarising something real instead of reconstructing it from patterns.
Why this will not simply be fixed
It is tempting to assume the next version will solve this, and systems are indeed getting better at grounding their answers in retrieved documents rather than pure prediction. That genuinely helps. But as long as the core of the technology is generating plausible text, the capacity to generate plausible falsehoods comes along for the ride. The responsible mindset is not to wait for a perfect tool but to build the habit of verification into how you use the imperfect one you have.
Used this way, an AI that occasionally invents things is still enormously useful, because you are supplying the one thing it lacks: a sense of whether the plausible thing is also the true thing. For more on how these systems really behave, see our AI coverage, or browse all stories.
