AI hallucination

An AI hallucination is an answer from a chatbot or other AI tool that sounds certain but is wrong or made up, such as a court case that does not exist or a return policy your shop never had. It reads just as confidently as a correct answer, so you cannot spot it by tone. If AI writes for your customers, someone has to catch these errors.

Why does it happen?

A large language model writes text one token at a time, picking what is likely to come next given its training data. A plausible continuation can be false. The survey by Huang et al. (2023) traces the causes to three stages: the data (errors, and gaps in what the model has seen), the training, and the generation itself, where more randomness, such as a higher temperature, raises the risk. The survey separates two kinds: answers that conflict with facts in the world, and answers that stray from the question or from the text the model was given.

Kalai et al. (OpenAI and Georgia Tech, September 2025) explain why the problem survives training. Most benchmarks score an answer as right or wrong and give no credit for "I don't know", so a model that guesses scores higher than one that admits doubt. Their example: asked for the first author's birthday, with the instruction to answer only if it knew, DeepSeek-V3 gave three different wrong dates in three attempts.

How often does it happen?

There is no single rate: it depends on the model, the task and how you measure. In a preregistered study published in May 2024, researchers at Stanford and Yale (Magesh et al.) found that legal research tools from LexisNexis and Thomson Reuters, which look up case law before they answer, hallucinated between 17% and 33% of the time. That was less than GPT-4 on its own. In OpenAI's system card for o3 and o4-mini (April 2025), o3 hallucinated on 33% of PersonQA questions, which test facts about people, and o4-mini on 48%, against 16% for the older o1. A newer model does not automatically hallucinate less.

What can it cost?

In June 2023 a federal judge in New York fined two lawyers and their firm $5,000 in Mata v. Avianca. They had submitted "non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT", and kept standing by them after the court questioned them.

How do you limit it?

• Let the model answer from your own, current documents with RAG. Huang et al. find that this helps against gaps in the model's knowledge, and that it still fails when the search returns the wrong passages.
• Allow "I don't know". Say so in the prompt, and in your own tests count a wrong answer as worse than no answer, as Kalai et al. propose for benchmarks.
• Show the source with each answer, so the reader can check it.
• Have a person review anything that goes to customers or into official documents (human-in-the-loop), and add guardrails that stop answers outside the tool's scope.

Example

Suppose a software company's support bot is asked about a feature that was removed two years ago. The documentation it was built on covers both old and new versions, so it writes a confident step-by-step guide with menu paths that no longer exist. The customer follows it, fails, and contacts support after all. The measures above address this: search only current documentation, let the bot say "I don't have information about that" when the search finds nothing relevant, and link the source article in every answer.