---
id: CP-271
slug: ai-hallucination-invented-facts
lang: en
title: "AI hallucination: when the model makes things up"
description: "A language model predicts the probable word without checking that it is true: it can invent facts, references and code packages."
canonical_url: https://startbrain.io/cyberpedia/en/articles/ai-hallucination-invented-facts/
markdown_url: https://startbrain.io/cyberpedia/en/articles/ai-hallucination-invented-facts.md
publisher: Brain (https://startbrain.io)
license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
cite_as: "Brain, \"AI hallucination: when the model makes things up\", Cyberpedia, 2026. https://startbrain.io/cyberpedia/en/articles/ai-hallucination-invented-facts/ (CC BY 4.0)"
wing: 4
theme: verify_output
reflex: air-006
schema: G2
updated: "2026-11-03"
translations:
  en: https://startbrain.io/cyberpedia/en/articles/ai-hallucination-invented-facts/
  fr: https://startbrain.io/cyberpedia/fr/articles/hallucination-ia-faits-inventes/
  de: https://startbrain.io/cyberpedia/de/articles/ki-halluzination-erfundene-fakten/
  es: https://startbrain.io/cyberpedia/es/articles/alucinacion-ia-hechos-inventados/
sources:
  - "n": 1
    title: LLM09:2025 Misinformation
    publisher: OWASP
    url: https://genai.owasp.org/llmrisk/llm092025-misinformation/
    accessed: "2026-11-03"
  - "n": 2
    title: "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile"
    publisher: NIST
    url: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
    accessed: "2026-11-03"
  - "n": 3
    title: "A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions"
    publisher: arXiv
    url: https://arxiv.org/abs/2311.05232
    accessed: "2026-11-03"
  - "n": 4
    title: Survey of Hallucination in Natural Language Generation
    publisher: arXiv
    url: https://arxiv.org/abs/2202.03629
    accessed: "2026-11-03"
  - "n": 5
    title: AI Risk Management Framework
    publisher: NIST
    url: https://www.nist.gov/itl/ai-risk-management-framework
    accessed: "2026-11-03"
---

# AI hallucination: when the model makes things up

An AI assistant that writes a summary, a legal memo or a list of references does not consult a database of facts. It produces a plausible string of words. When it is wrong, it sounds just as confident as when it is right, and nothing on screen shows the difference.

This phenomenon is called hallucination. The US NIST also uses the term "confabulation" and notes that people colloquially speak of hallucinations or fabrications [2]. For an employee, an AI answer is a draft to check, never a source.

## In 30 seconds

- Where it happens: in any text produced by an AI assistant, especially on open-ended questions that call for a long answer and in fields that require contextual expertise [2].
- What is at stake: the model can invent a fact, a reference, a quotation or the name of a code library, and present it with confidence [1][2].
- The reflex: cross-check every important fact against a trusted external source and have a human review critical information before using it [1].
- If it is already done: warn the recipient of the document, correct it with a reliable source and report the error to the person who manages AI tools in the company.

## Straight answers

### Why does an AI make things up?

According to OWASP, models fill gaps in their training data with statistical patterns, without understanding the content. The answers can then look right while having no basis [1]. NIST describes the same mechanism: models produce outputs that approximate the statistical distribution of their training data, and large language models predict the next word in a sentence [2].

### Is this a rare flaw?

NIST presents it as a natural result of how generative models are designed [2]. Two research surveys, one on text generation in general and the other on large language models, describe a closely studied problem. The first covers automatic summarization, dialogue, question answering and machine translation [4]. The second speaks of plausible but non-factual content and of doubts about the reliability of these models in information retrieval [3].

### Can an AI also invent its own justifications?

Yes. NIST states that outputs can contain made-up logic or citations that claim to explain the answer. A model sometimes gives reasoning steps even though the answer is wrong [2]. A well-presented reference therefore proves nothing.

### Does the risk come only from the machine?

No. OWASP also names overreliance: users place too much trust in generated content, do not check its accuracy and build false data into important decisions [1]. NIST notes that the risk appears when users believe the false content, often because of the confident tone of the answer [2].

## How it works

A language model does not distinguish what it "knows" from what it completes. NIST specifies that statistical prediction can yield outputs that are accurate and coherent, but also inaccurate or incoherent [2]. It is mainly a problem for open-ended prompts that call for a long answer, and in fields that require contextual expertise [2].

Fabrications take several forms. OWASP lists four. The first is factual inaccuracy: the Air Canada chatbot gave false information to travelers, and the airline was successfully sued [1]. The second is unsupported claims, with one cited example being ChatGPT fabricating false legal cases, leading to serious difficulties in court [1]. The third is the illusion of expertise, for example chatbots that present health topics as uncertain when they are not [1]. The fourth is unsafe code: the model suggests insecure or nonexistent libraries [1].

These inventions can be exploited. OWASP describes a scenario in which attackers test coding assistants to find the package names that are often invented, then publish real malicious packages under those names in widely used repositories. Developers who follow the assistant's suggestion include them in their software without knowing it [1].

## What you do

The measures below are inspired by OWASP's recommendations [1].

- Cross-check important facts against trusted external sources [1].
- Open every link and every reference cited by the AI. If the page does not exist or does not say what the AI attributes to it, treat the reference as invented. NIST points out that fabricated citations can mislead [2].
- Do not paste an unverified answer into a contract, a legal memo, a financial document or a customer letter. OWASP calls for human review, especially for critical or sensitive information [1].
- Before installing a code package suggested by an assistant, check that it exists and that it is legitimate. OWASP also calls for secure development practices to avoid building in flaws that come from erroneous suggestions [1].
- In a decision chain, keep a trained human to review. OWASP stresses training reviewers, so that they do not blindly rely on AI-produced content [1].

## Pitfalls

- Fluency misleads. A well-written text is not an accurate text, and NIST notes that a confident tone leads people to believe false content [2].
- Specialized fields are the most exposed. OWASP cites law and health as contexts where unsupported claims are particularly damaging [1]. NIST gives the example of a confabulated medical record summary that could lead to wrong diagnoses [2].
- Asking the AI to proofread itself does not replace an external source. It can justify a wrong answer with reasoning that looks logical [2].
- No attacker is needed for the harm to occur. OWASP describes a company successfully sued after offering a medical diagnosis chatbot without making sure it was reliable. The flaw came from insufficient oversight, not from an attack [1].
- The risk is hard to quantify. NIST writes that it is difficult to estimate the scale and downstream impact of confabulations [2]. It published its generative AI profile, which helps organizations identify the risks specific to this technology, on 26 July 2024 [5].

## Sources

1. OWASP, "LLM09:2025 Misinformation". https://genai.owasp.org/llmrisk/llm092025-misinformation/ (accessed 3 November 2026)
2. NIST, "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile". https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf (accessed 3 November 2026)
3. arXiv, "A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions". https://arxiv.org/abs/2311.05232 (accessed 3 November 2026)
4. arXiv, "Survey of Hallucination in Natural Language Generation". https://arxiv.org/abs/2202.03629 (accessed 3 November 2026)
5. NIST, "AI Risk Management Framework". https://www.nist.gov/itl/ai-risk-management-framework (accessed 3 November 2026)

---

Original source: Cyberpedia, the public cybersecurity and AI encyclopedia published by Brain. Canonical version: https://startbrain.io/cyberpedia/en/articles/ai-hallucination-invented-facts/
You may reuse this text freely under the CC BY 4.0 license (https://creativecommons.org/licenses/by/4.0/), provided you credit "Brain, Cyberpedia" with a link to the canonical URL above.