Cyberpediaby Brain
ENDEFR

The lawyer and ChatGPT's fake cases

An AI assistant can invent believable court rulings. Why it happens, and how to check a citation before using it at work.

In 30 seconds

Where it happens
In any text produced by an AI assistant that contains a quotation, a figure or a court ruling.
What the attacker wants
No one. The risk comes from the model itself, which can produce a false answer without any attacker getting involved1.
The reflex
Do not reuse any quotation, figure or reference from an AI without having found it in the original source, then have a person review it before any professional use.
If it is already done
Find the source of each element already used, correct or remove what cannot be verified, and warn the recipients of the document.

What it is

An AI assistant can produce a court ruling that does not exist. OWASP mentions that ChatGPT fabricated fake legal cases, with significant consequences in court1. This risk also applies to an employee who reuses an AI assistant's text as it stands.

The sources cited here do not name the Mata v. Avianca case and do not recount it. This entry therefore uses neither the names of the people, nor the dates, nor the sanction: it keeps to what the sources establish, and explains what an employee can take from it for their own work.

Straight answers

What happened in the Mata v. Avianca case?

Among the sources cited here, only the OWASP page mentions this type of case, without naming the case: ChatGPT is described there as having fabricated fake legal cases, which led to significant difficulties in court1. The details of the proceedings, the date and the sanction do not appear there.

Why does an AI invent references?

A language model does not consult a database of facts. OWASP explains that it fills gaps in its training data with statistical patterns, without understanding the content: the answer can seem right while being unfounded1. NIST describes the same mechanism: these models produce outputs that approximate the statistical distribution of their training data, and large language models predict the next word in a sentence2.

Is this a flaw that will soon be fixed?

The sources do not promise it. A research survey notes that these models can produce plausible but nonfactual content, and that this raises serious questions about their reliability in information retrieval systems3. A theoretical study claims that it is impossible to eliminate hallucination in large language models, and that they will inevitably hallucinate if they are used as general problem solvers4.

What vocabulary should you know?

NIST speaks of "confabulation" for the production of false content presented with confidence, and specifies that everyday language also says "hallucinations" or "fabrications"2. The two words describe the same phenomenon.

Is a text written in a confident tone more reliable?

No. According to NIST, the risk arises when users believe false content, often because of the confident tone of the answer, and then act on that basis2.

1How it works1 min

The model does not look for the truth, it produces the most probable sequence of words. NIST notes that this statistical prediction can give outputs that are accurate and coherent, but also outputs that are inaccurate or inconsistent with each other. It considers this phenomenon particularly marked for open-ended questions calling for long answers, and in domains that require contextual expertise2.

OWASP distinguishes two related problems. The first is hallucination, when the model generates content that looks accurate but is invented. The second is overreliance: users trust the generated text too much and do not verify its accuracy, so that they build false data into important decisions or processes1.

2What you do1 min

The practices below follow the measures from OWASP and NIST, phrased from the employee's point of view.

  • Treat every quotation, figure and reference produced by an AI as unverified. OWASP recommends that users cross-check outputs against trusted external sources1.
  • Find the element in the original source: the text of the ruling, the official document, the study. A reference that you cannot find is not to be used.
  • Cross-check several trusted external sources rather than settling for a single confirmation1.
  • Keep a human validation step before any professional use. OWASP asks for human oversight and fact-checking processes for critical or sensitive information, with reviewers trained so that they do not rely excessively on generated content1.
  • Tell your readers what was produced with an AI. OWASP recommends clearly labeling generated content and informing users of the limits of reliability1.
  • In sensitive domains, such as health or legal proceedings, take extra care: OWASP cites these contexts for unfounded claims1.
3Pitfalls2 min

The confident tone

A fluent, assertive answer proves nothing. NIST points out that confabulations can also take the form of invented reasoning or citations that claim to justify the answer, which can lead people to trust the system wrongly2. A very detailed reference, with a number, a jurisdiction and a date, is therefore not a sign of authenticity.

Trust that settles in

NIST describes automation bias: as the technology seems reliable, people may defer too quickly to its outputs or judge its content better than that of other sources. This bias aggravates the risk of confabulation2. A tool that has answered correctly ten times is not right the eleventh time for that reason.

Asking the AI to check itself

The sources cited here do not directly study the case where the tool is asked to confirm its own answer. They do, however, give elements on the reliability of its outputs. A study on ChatGPT finds self-contradictions, meaning two contradictory sentences in the same context, in 17.7% of the sentences produced. The authors add that a large share of these contradictions, 35.2% for ChatGPT, cannot be verified with online texts5. Their detector, based on prompts given to ChatGPT, reaches about 80% F1 score5. This is useful in research, but it lets errors through. NIST also notes that models sometimes provide reasoning steps even when the answer is wrong2.

Believing the problem will disappear

The theoretical study judges hallucination inevitable4, and the research survey details its current limits3. Human verification therefore remains necessary.

The same errors outside a courtroom

OWASP cites other cases: an Air Canada chatbot gave false information to travelers, and the company was successfully sued1. It also mentions programming assistants that suggest nonexistent or unsafe code libraries1. In a scenario described by OWASP, attackers identify the package names that assistants often invent, publish real malicious packages under those names, and developers include them1.

Fact sheet

Identifier
CP-272
Wing
4. Artificial intelligence
Brain skill
Verifying AI Outputs & Hallucinations (verify_output)
Brain reflex
air-026
Diagram
G2 · Visual vs Technical Indicators
Languages
de, en, fr
Sources
5 sources checked on 3 November 2026
Sources5 sources
  1. OWASP, “LLM09:2025 Misinformation”. https://genai.owasp.org/llmrisk/llm092025-misinformation/ (accessed 3 November 2026)
  2. NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile”. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf (accessed 3 November 2026)
  3. arXiv, “A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions”, 9 November 2023. https://arxiv.org/abs/2311.05232 (accessed 3 November 2026)
  4. arXiv, “Hallucination is Inevitable: An Innate Limitation of Large Language Models”, 22 January 2024. https://arxiv.org/abs/2401.11817 (accessed 3 November 2026)
  5. arXiv, “Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation”, 25 May 2023. https://arxiv.org/abs/2305.15852 (accessed 3 November 2026)

Cite this article

Brain, “The lawyer and ChatGPT's fake cases”, Cyberpedia, 2026. https://startbrain.io/cyberpedia/en/articles/lawyer-fake-cases-chatgpt-mata-avianca/

BibTeX 
@misc{brain_cp272_en_2026,
  author = {Brain},
  title = {The lawyer and ChatGPT's fake cases},
  year = {2026},
  howpublished = {Cyberpedia},
  url = {https://startbrain.io/cyberpedia/en/articles/lawyer-fake-cases-chatgpt-mata-avianca/},
  note = {CC BY 4.0}
}
CC BY 4.0 licence