Why AI invents facts, and what to do about it

A language model does not look anything up. It predicts the next piece of text that best fits everything before it, so an invented citation comes out of the same mechanism that produces a correct one. The model has no internal signal separating the two. That is why the fix is never to ask it to be more careful. The fix is to hand it the source and check the answer against that source.

What the model is actually doing when it answers you

When you ask a question, nothing is retrieved. There is no index being consulted, no record being opened. The model produces text one fragment at a time, each fragment chosen because it fits what came before — the phrasing of your question, the shape of the documents it was trained on, the sentence it has written so far.

A real case number and an invented case number are, to that process, the same kind of object: a plausible continuation. Both look like a case number. Both sit naturally after the words "as established in". The model is not lying, because lying requires knowing the truth and choosing against it. It is producing the most fitting text, and fitting is not the same as true.

This is why the failure feels so strange from the outside. The invented answer arrives in the same tone, at the same speed, with the same steadiness as the correct one. Confidence in the output tells you nothing about accuracy, because confidence was never being measured.

Hallucination, in the research literature, is generated content that is nonsensical or unfaithful to the source material the model was given. The ACM Computing Surveys review of hallucination in natural language generation treats it as a structural property of how these systems generate text, not as a bug in a particular product. That distinction matters for you: it means no vendor is one release away from turning it off.

Why "please be accurate" changes nothing

Adding "do not invent anything" to your prompt feels like a control. It is not one. The instruction becomes more text in the same context, influencing the style of the answer — which is precisely the problem. The answer gets more careful language. It does not get more careful facts.

You can see this yourself in a minute. Ask for something that does not exist: a clause from a regulation that was never written, a ruling on a dispute you invented. A model with no grounding will usually produce something. It will look like the real thing, because looking like the real thing is the whole job it was trained to do.

The three controls that actually work

Give it the source. Paste the contract, the ruling, the accounting standard, the internal policy. When the answer has to come from a document you provided, the model has something to stay faithful to, and you have something to check against. Working from the model's memory is what produces the invented citation; working from a document you supplied is a different task.

Ask for the passage, not the conclusion. Instead of "does this contract allow early termination", ask "quote the clauses that deal with termination, verbatim, and say if none exist". A quotation is verifiable in seconds by searching for the text. A conclusion is not verifiable at all.

Put a human at the point of consequence. Not on every step, which nobody sustains, but at the moment the output leaves your hands: before it goes to the client, before it is filed, before it becomes a decision. NIST's AI Risk Management Framework organises this kind of thinking around a plain question — what is this system being used for, and who gets hurt if it is wrong. A draft email and a filed document are the same technology and completely different risks.

What this means if you are the one who signs

The useful mental shift is small. Treat the model as a fast junior who has read enormous amounts and remembers imprecisely, never as a reference source. You would not file a junior's memo without reading it. You would, though, happily let that junior read forty pages and tell you which three matter — and that is where most of the value is.

The professionals who get burned are not the ones using AI. They are the ones who moved a task into AI without moving the check with it. The check was always there; it was just invisible, buried inside the work of the person doing it by hand.