Cyberpediaby Brain
ENDEFR

AI hallucination: when the model makes things up

A language model predicts the probable word without checking that it is true: it can invent facts, references and code packages.

In 30 seconds

Where it happens
In any text produced by an AI assistant, especially on open-ended questions that call for a long answer and in fields that require contextual expertise2.
What is at stake
The model can invent a fact, a reference, a quotation or the name of a code library, and present it with confidence12.
The reflex
Cross-check every important fact against a trusted external source and have a human review critical information before using it1.
If it is already done
Warn the recipient of the document, correct it with a reliable source and report the error to the person who manages AI tools in the company.

What it is

An AI assistant that writes a summary, a legal memo or a list of references does not consult a database of facts. It produces a plausible string of words. When it is wrong, it sounds just as confident as when it is right, and nothing on screen shows the difference.

This phenomenon is called hallucination. The US NIST also uses the term "confabulation" and notes that people colloquially speak of hallucinations or fabrications2. For an employee, an AI answer is a draft to check, never a source.

Straight answers

Why does an AI make things up?

According to OWASP, models fill gaps in their training data with statistical patterns, without understanding the content. The answers can then look right while having no basis1. NIST describes the same mechanism: models produce outputs that approximate the statistical distribution of their training data, and large language models predict the next word in a sentence2.

Is this a rare flaw?

NIST presents it as a natural result of how generative models are designed2. Two research surveys, one on text generation in general and the other on large language models, describe a closely studied problem. The first covers automatic summarization, dialogue, question answering and machine translation4. The second speaks of plausible but non-factual content and of doubts about the reliability of these models in information retrieval3.

Can an AI also invent its own justifications?

Yes. NIST states that outputs can contain made-up logic or citations that claim to explain the answer. A model sometimes gives reasoning steps even though the answer is wrong2. A well-presented reference therefore proves nothing.

Does the risk come only from the machine?

No. OWASP also names overreliance: users place too much trust in generated content, do not check its accuracy and build false data into important decisions1. NIST notes that the risk appears when users believe the false content, often because of the confident tone of the answer2.

1How it works1 min

A language model does not distinguish what it "knows" from what it completes. NIST specifies that statistical prediction can yield outputs that are accurate and coherent, but also inaccurate or incoherent2. It is mainly a problem for open-ended prompts that call for a long answer, and in fields that require contextual expertise2.

Fabrications take several forms. OWASP lists four. The first is factual inaccuracy: the Air Canada chatbot gave false information to travelers, and the airline was successfully sued1. The second is unsupported claims, with one cited example being ChatGPT fabricating false legal cases, leading to serious difficulties in court1. The third is the illusion of expertise, for example chatbots that present health topics as uncertain when they are not1. The fourth is unsafe code: the model suggests insecure or nonexistent libraries1.

These inventions can be exploited. OWASP describes a scenario in which attackers test coding assistants to find the package names that are often invented, then publish real malicious packages under those names in widely used repositories. Developers who follow the assistant's suggestion include them in their software without knowing it1.

2What you do1 min

The measures below are inspired by OWASP's recommendations1.

  • Cross-check important facts against trusted external sources1.
  • Open every link and every reference cited by the AI. If the page does not exist or does not say what the AI attributes to it, treat the reference as invented. NIST points out that fabricated citations can mislead2.
  • Do not paste an unverified answer into a contract, a legal memo, a financial document or a customer letter. OWASP calls for human review, especially for critical or sensitive information1.
  • Before installing a code package suggested by an assistant, check that it exists and that it is legitimate. OWASP also calls for secure development practices to avoid building in flaws that come from erroneous suggestions1.
  • In a decision chain, keep a trained human to review. OWASP stresses training reviewers, so that they do not blindly rely on AI-produced content1.
3Pitfalls1 min
  • Fluency misleads. A well-written text is not an accurate text, and NIST notes that a confident tone leads people to believe false content2.
  • Specialized fields are the most exposed. OWASP cites law and health as contexts where unsupported claims are particularly damaging1. NIST gives the example of a confabulated medical record summary that could lead to wrong diagnoses2.
  • Asking the AI to proofread itself does not replace an external source. It can justify a wrong answer with reasoning that looks logical2.
  • No attacker is needed for the harm to occur. OWASP describes a company successfully sued after offering a medical diagnosis chatbot without making sure it was reliable. The flaw came from insufficient oversight, not from an attack1.
  • The risk is hard to quantify. NIST writes that it is difficult to estimate the scale and downstream impact of confabulations2. It published its generative AI profile, which helps organizations identify the risks specific to this technology, on 26 July 20245.

Fact sheet

Identifier
CP-271
Wing
4. Artificial intelligence
Brain skill
Verifying AI Outputs & Hallucinations (verify_output)
Brain reflex
air-006
Diagram
G2 · Visual vs Technical Indicators
Languages
de, en, fr
Sources
5 sources checked on 3 November 2026
Sources5 sources
  1. OWASP, “LLM09:2025 Misinformation”. https://genai.owasp.org/llmrisk/llm092025-misinformation/ (accessed 3 November 2026)
  2. NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile”. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf (accessed 3 November 2026)
  3. arXiv, “A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions”. https://arxiv.org/abs/2311.05232 (accessed 3 November 2026)
  4. arXiv, “Survey of Hallucination in Natural Language Generation”. https://arxiv.org/abs/2202.03629 (accessed 3 November 2026)
  5. NIST, “AI Risk Management Framework”. https://www.nist.gov/itl/ai-risk-management-framework (accessed 3 November 2026)

Cite this article

Brain, “AI hallucination: when the model makes things up”, Cyberpedia, 2026. https://startbrain.io/cyberpedia/en/articles/ai-hallucination-invented-facts/

BibTeX 
@misc{brain_cp271_en_2026,
  author = {Brain},
  title = {AI hallucination: when the model makes things up},
  year = {2026},
  howpublished = {Cyberpedia},
  url = {https://startbrain.io/cyberpedia/en/articles/ai-hallucination-invented-facts/},
  note = {CC BY 4.0}
}
CC BY 4.0 licence