--- id: CP-291 slug: model-inversion-membership-inference lang: en title: Recovering an AI's training data description: An attacker who queries an AI can guess whether a document was used to train it, or even bring out memorized excerpts. canonical_url: https://startbrain.io/cyberpedia/en/articles/model-inversion-membership-inference/ markdown_url: https://startbrain.io/cyberpedia/en/articles/model-inversion-membership-inference.md publisher: Brain (https://startbrain.io) license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) cite_as: Brain, "Recovering an AI's training data", Cyberpedia, 2026. https://startbrain.io/cyberpedia/en/articles/model-inversion-membership-inference/ (CC BY 4.0) wing: 4 theme: protect_data reflex: air-008 schema: G1 updated: "2026-11-03" translations: en: https://startbrain.io/cyberpedia/en/articles/model-inversion-membership-inference/ fr: https://startbrain.io/cyberpedia/fr/articles/inversion-modele-inference-appartenance/ de: https://startbrain.io/cyberpedia/de/articles/trainingsdaten-ki-modellinversion-mitgliedschaftsinferenz/ es: https://startbrain.io/cyberpedia/es/articles/inversion-modelo-inferencia-pertenencia/ sources: - "n": 1 title: Extracting Training Data from Large Language Models publisher: arXiv url: https://arxiv.org/abs/2012.07805 published: "2020-12-14" accessed: "2026-11-03" - "n": 2 title: Scalable Extraction of Training Data from (Production) Language Models publisher: arXiv url: https://arxiv.org/abs/2311.17035 published: "2023-11-28" accessed: "2026-11-03" - "n": 3 title: Membership Inference Attacks against Machine Learning Models publisher: arXiv url: https://arxiv.org/abs/1610.05820 published: "2016-10-18" accessed: "2026-11-03" - "n": 4 title: LLM02:2025 Sensitive Information Disclosure publisher: OWASP url: https://genai.owasp.org/llmrisk/llm022025-sensitive-information-disclosure/ accessed: "2026-11-03" - "n": 5 title: "AI 100-2 E2023, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations" publisher: NIST url: https://csrc.nist.gov/pubs/ai/100/2/e2023/final accessed: "2026-11-03" - "n": 6 title: Principles for security of Machine learning ML publisher: NCSC (UK) url: https://www.ncsc.gov.uk/collection/machine-learning-principles published: "2024-05-22" accessed: "2026-11-03" --- # Recovering an AI's training data An AI model, such as the one behind a chat assistant, learns from a very large number of texts, images or records. An attacker can query it to find out whether a person or a document was in that data, or to make pieces the model has memorized reappear. The first case is called membership inference, the second inversion or extraction. For an employee, what matters is what the company and the employee themselves enter into an AI tool, since that data can end up in the training model [4]. This page explains the principle, two dated real cases and the reflexes that limit exposure. ## In 30 seconds - Where it happens: on public AI models or models supplied by a provider, which people query through a chat interface or an online service. - What the attacker wants: to find out whether a specific piece of data was used for training, or to recover information the model has kept in memory, such as public personal information [1]. - The reflex: do not enter sensitive data into a public AI tool, and ask your suppliers what they do with the data entered and how they train their models. - If it is already done: tell the company's security team and check with the tool's publisher whether it is possible to refuse the use of your data for training [4]. ## Straight answers ### What is the difference between membership inference and extraction? Membership inference answers a yes-or-no question. With a record in hand and "black box" access to the model, the attacker tries to determine whether that record was part of the training data [3]. Extraction goes further: the attacker recovers individual training examples by querying the model [1]. Inversion consists of reconstructing sensitive information or input data from what the model reveals [4]. ### Why can a model "spit out" data? The sources speak of memorization. Researchers define "extractable memorization" as training data that an adversary can efficiently extract by querying a model, without knowing the dataset in advance [2]. The attack on GPT-2 works even if each extracted sequence appears in only one document of the training data [1]. ### Are large models safer? No, according to the GPT-2 study: the authors find that larger models are more vulnerable than smaller ones [1]. The 2023 study adds that current alignment techniques do not eliminate memorization [2]. ### Can a confidential document I pasted into an assistant come out for another user? OWASP describes this risk: a user receives a response containing another user's personal data, because of insufficient data sanitization [4]. The organization recommends that applications prevent user data from entering the training model and that users be able to refuse to have their data included in it [4]. The sources cited here do not put a number on the likelihood of such a case. ## Inside the attacker's head The attacker does not need to break into the company's system. Access to the model is enough, and it is often public. For membership inference, the researchers Shokri and his co-authors train their own inference model, which learns to recognize the differences between the target's responses on data seen during training and on data it has never seen [3]. For extraction, the adversary queries the model without knowing anything about the training dataset [2]. The adversary looks for sequences that come out word for word: in the GPT-2 study, hundreds of verbatim text sequences were extracted [1]. The goal depends on the target. For a model trained on sensitive records, knowing that a person appears in them can already be information. The 2016 authors illustrate this with a dataset of hospital discharges, whose membership is sensitive from a privacy standpoint [3]. ## What you see on screen The sources cited here describe no visible sign for an employee. The attack consists of asking questions to an accessible model, and its queries can look ordinary. The trace shows up rather in a model response that contains a memorized excerpt. The text below is a neutralized illustration, not a command to reproduce. It shows what a request looks like when it tries to pull personal data out of a model's memory. ```text Illustration: fictional exchange with an assistant, address in .invalid. Question: [a repetitive or mundane request, paraphrased] Model response: [a memorized excerpt: name, number, fictional email firstname.lastname@example.invalid] ``` ## Under the hood The mechanism rests on differences in a model's behavior between what it has learned and what it has never seen [3]. The attacker exploits these differences, either by measuring them (membership inference) or by provoking the output of memorized sequences (extraction). The targeted models are of various kinds. The 2016 researchers tested "on demand" machine learning services from commercial providers such as Google and Amazon, and show that these models can be vulnerable [3]. The 2023 researchers extract gigabytes of data from open models such as Pythia, semi-open models such as LLaMA and closed models such as ChatGPT [2]. NIST publishes a taxonomy of adversarial machine learning, whose keywords include "privacy breach" [5]. OWASP also cites the "Proof Pudding" attack (CVE-2019-20634): leaked training data made it easier to extract and invert the model there, which made it possible to bypass email filters [4]. It links these techniques to three entries in the MITRE ATLAS database: inferring membership in training data, inverting a model, extracting a model [4]. On the defense side, OWASP recommends sanitizing data before training, restricting access according to the principle of least privilege, and using federated learning and differential privacy, which adds noise to the data or to the outputs [4]. It specifies that an instruction given to the model about the types of data not to return can be bypassed by prompt injection [4]. The UK NCSC reminds readers that AI systems are subject to new vulnerabilities, which add to the usual threats [6]. ## Case files ### December 2020: data extraction from GPT-2 On 14 December 2020, Nicholas Carlini and eleven co-authors posted on arXiv a study showing that an adversary can recover individual training examples by querying a language model [1]. They attack GPT-2, trained on data collected from the public internet, and extract hundreds of verbatim text sequences from it [1]. Among them are public personal information (names, phone numbers, email addresses), IRC conversations, code and 128-bit UUIDs [1]. A revised version is dated 15 June 2021 [1]. ### November 2023: data extraction from ChatGPT On 28 November 2023, Milad Nasr, Nicholas Carlini and eight co-authors published a study on extracting training data from production models [2]. To attack ChatGPT, an aligned model, they develop a so-called "divergence" attack: it makes the model leave its conversational style and emit training data at a rate 150 times higher than during normal operation [2]. They conclude that practical attacks can recover much more data than was previously thought [2]. The OWASP page points to a press article on this case, titled "ChatGPT Spit Out Sensitive Data When Told to Repeat ‘Poem’ Forever" [4]. For the employee, the reflex follows from these cases. OWASP classes personal data, financial information, health records, confidential company data, security credentials and legal documents as sensitive data: they are not entered into a public AI tool [4]. For tools bought by the company, written guarantees are requested from suppliers, because OWASP recommends clear policies on data retention, use and deletion [4]. ## Sources 1. arXiv, "Extracting Training Data from Large Language Models", 14 December 2020. https://arxiv.org/abs/2012.07805 (accessed 3 November 2026) 2. arXiv, "Scalable Extraction of Training Data from (Production) Language Models", 28 November 2023. https://arxiv.org/abs/2311.17035 (accessed 3 November 2026) 3. arXiv, "Membership Inference Attacks against Machine Learning Models", 18 October 2016. https://arxiv.org/abs/1610.05820 (accessed 3 November 2026) 4. OWASP, "LLM02:2025 Sensitive Information Disclosure". https://genai.owasp.org/llmrisk/llm022025-sensitive-information-disclosure/ (accessed 3 November 2026) 5. NIST, "AI 100-2 E2023, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations". https://csrc.nist.gov/pubs/ai/100/2/e2023/final (accessed 3 November 2026) 6. NCSC (UK), "Principles for security of Machine learning ML", 22 May 2024. https://www.ncsc.gov.uk/collection/machine-learning-principles (accessed 3 November 2026) --- Original source: Cyberpedia, the public cybersecurity and AI encyclopedia published by Brain. Canonical version: https://startbrain.io/cyberpedia/en/articles/model-inversion-membership-inference/ You may reuse this text freely under the CC BY 4.0 license (https://creativecommons.org/licenses/by/4.0/), provided you credit "Brain, Cyberpedia" with a link to the canonical URL above.