--- id: CP-272 slug: lawyer-fake-cases-chatgpt-mata-avianca lang: en title: The lawyer and ChatGPT's fake cases description: An AI assistant can invent believable court rulings. Why it happens, and how to check a citation before using it at work. canonical_url: https://startbrain.io/cyberpedia/en/articles/lawyer-fake-cases-chatgpt-mata-avianca/ markdown_url: https://startbrain.io/cyberpedia/en/articles/lawyer-fake-cases-chatgpt-mata-avianca.md publisher: Brain (https://startbrain.io) license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) cite_as: Brain, "The lawyer and ChatGPT's fake cases", Cyberpedia, 2026. https://startbrain.io/cyberpedia/en/articles/lawyer-fake-cases-chatgpt-mata-avianca/ (CC BY 4.0) wing: 4 theme: verify_output reflex: air-026 schema: G2 updated: "2026-11-03" translations: en: https://startbrain.io/cyberpedia/en/articles/lawyer-fake-cases-chatgpt-mata-avianca/ fr: https://startbrain.io/cyberpedia/fr/articles/avocat-faux-arrets-chatgpt-mata-avianca/ de: https://startbrain.io/cyberpedia/de/articles/anwalt-erfundene-urteile-chatgpt-mata-avianca/ es: https://startbrain.io/cyberpedia/es/articles/abogado-sentencias-falsas-chatgpt-mata-avianca/ sources: - "n": 1 title: LLM09:2025 Misinformation publisher: OWASP url: https://genai.owasp.org/llmrisk/llm092025-misinformation/ accessed: "2026-11-03" - "n": 2 title: "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile" publisher: NIST url: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf accessed: "2026-11-03" - "n": 3 title: "A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions" publisher: arXiv url: https://arxiv.org/abs/2311.05232 published: "2023-11-09" accessed: "2026-11-03" - "n": 4 title: "Hallucination is Inevitable: An Innate Limitation of Large Language Models" publisher: arXiv url: https://arxiv.org/abs/2401.11817 published: "2024-01-22" accessed: "2026-11-03" - "n": 5 title: "Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation" publisher: arXiv url: https://arxiv.org/abs/2305.15852 published: "2023-05-25" accessed: "2026-11-03" --- # The lawyer and ChatGPT's fake cases An AI assistant can produce a court ruling that does not exist. OWASP mentions that ChatGPT fabricated fake legal cases, with significant consequences in court [1]. This risk also applies to an employee who reuses an AI assistant's text as it stands. The sources cited here do not name the Mata v. Avianca case and do not recount it. This entry therefore uses neither the names of the people, nor the dates, nor the sanction: it keeps to what the sources establish, and explains what an employee can take from it for their own work. ## In 30 seconds - Where it happens: in any text produced by an AI assistant that contains a quotation, a figure or a court ruling. - What the attacker wants: no one. The risk comes from the model itself, which can produce a false answer without any attacker getting involved [1]. - The reflex: do not reuse any quotation, figure or reference from an AI without having found it in the original source, then have a person review it before any professional use. - If it is already done: find the source of each element already used, correct or remove what cannot be verified, and warn the recipients of the document. ## Straight answers ### What happened in the Mata v. Avianca case? Among the sources cited here, only the OWASP page mentions this type of case, without naming the case: ChatGPT is described there as having fabricated fake legal cases, which led to significant difficulties in court [1]. The details of the proceedings, the date and the sanction do not appear there. ### Why does an AI invent references? A language model does not consult a database of facts. OWASP explains that it fills gaps in its training data with statistical patterns, without understanding the content: the answer can seem right while being unfounded [1]. NIST describes the same mechanism: these models produce outputs that approximate the statistical distribution of their training data, and large language models predict the next word in a sentence [2]. ### Is this a flaw that will soon be fixed? The sources do not promise it. A research survey notes that these models can produce plausible but nonfactual content, and that this raises serious questions about their reliability in information retrieval systems [3]. A theoretical study claims that it is impossible to eliminate hallucination in large language models, and that they will inevitably hallucinate if they are used as general problem solvers [4]. ### What vocabulary should you know? NIST speaks of "confabulation" for the production of false content presented with confidence, and specifies that everyday language also says "hallucinations" or "fabrications" [2]. The two words describe the same phenomenon. ### Is a text written in a confident tone more reliable? No. According to NIST, the risk arises when users believe false content, often because of the confident tone of the answer, and then act on that basis [2]. ## How it works The model does not look for the truth, it produces the most probable sequence of words. NIST notes that this statistical prediction can give outputs that are accurate and coherent, but also outputs that are inaccurate or inconsistent with each other. It considers this phenomenon particularly marked for open-ended questions calling for long answers, and in domains that require contextual expertise [2]. OWASP distinguishes two related problems. The first is hallucination, when the model generates content that looks accurate but is invented. The second is overreliance: users trust the generated text too much and do not verify its accuracy, so that they build false data into important decisions or processes [1]. ## What you do The practices below follow the measures from OWASP and NIST, phrased from the employee's point of view. - Treat every quotation, figure and reference produced by an AI as unverified. OWASP recommends that users cross-check outputs against trusted external sources [1]. - Find the element in the original source: the text of the ruling, the official document, the study. A reference that you cannot find is not to be used. - Cross-check several trusted external sources rather than settling for a single confirmation [1]. - Keep a human validation step before any professional use. OWASP asks for human oversight and fact-checking processes for critical or sensitive information, with reviewers trained so that they do not rely excessively on generated content [1]. - Tell your readers what was produced with an AI. OWASP recommends clearly labeling generated content and informing users of the limits of reliability [1]. - In sensitive domains, such as health or legal proceedings, take extra care: OWASP cites these contexts for unfounded claims [1]. ## Pitfalls ### The confident tone A fluent, assertive answer proves nothing. NIST points out that confabulations can also take the form of invented reasoning or citations that claim to justify the answer, which can lead people to trust the system wrongly [2]. A very detailed reference, with a number, a jurisdiction and a date, is therefore not a sign of authenticity. ### Trust that settles in NIST describes automation bias: as the technology seems reliable, people may defer too quickly to its outputs or judge its content better than that of other sources. This bias aggravates the risk of confabulation [2]. A tool that has answered correctly ten times is not right the eleventh time for that reason. ### Asking the AI to check itself The sources cited here do not directly study the case where the tool is asked to confirm its own answer. They do, however, give elements on the reliability of its outputs. A study on ChatGPT finds self-contradictions, meaning two contradictory sentences in the same context, in 17.7% of the sentences produced. The authors add that a large share of these contradictions, 35.2% for ChatGPT, cannot be verified with online texts [5]. Their detector, based on prompts given to ChatGPT, reaches about 80% F1 score [5]. This is useful in research, but it lets errors through. NIST also notes that models sometimes provide reasoning steps even when the answer is wrong [2]. ### Believing the problem will disappear The theoretical study judges hallucination inevitable [4], and the research survey details its current limits [3]. Human verification therefore remains necessary. ### The same errors outside a courtroom OWASP cites other cases: an Air Canada chatbot gave false information to travelers, and the company was successfully sued [1]. It also mentions programming assistants that suggest nonexistent or unsafe code libraries [1]. In a scenario described by OWASP, attackers identify the package names that assistants often invent, publish real malicious packages under those names, and developers include them [1]. ## Sources 1. OWASP, "LLM09:2025 Misinformation". https://genai.owasp.org/llmrisk/llm092025-misinformation/ (accessed 3 November 2026) 2. NIST, "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile". https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf (accessed 3 November 2026) 3. arXiv, "A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions", 9 November 2023. https://arxiv.org/abs/2311.05232 (accessed 3 November 2026) 4. arXiv, "Hallucination is Inevitable: An Innate Limitation of Large Language Models", 22 January 2024. https://arxiv.org/abs/2401.11817 (accessed 3 November 2026) 5. arXiv, "Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation", 25 May 2023. https://arxiv.org/abs/2305.15852 (accessed 3 November 2026) --- Original source: Cyberpedia, the public cybersecurity and AI encyclopedia published by Brain. Canonical version: https://startbrain.io/cyberpedia/en/articles/lawyer-fake-cases-chatgpt-mata-avianca/ You may reuse this text freely under the CC BY 4.0 license (https://creativecommons.org/licenses/by/4.0/), provided you credit "Brain, Cyberpedia" with a link to the canonical URL above.