Recovering an AI's training data

The attacker does not need to break into the company's system. Access to the model is enough, and it is often public. For membership inference, the researchers Shokri and his co-authors train their own inference model, which learns to recognize the differences between the target's responses on data seen during training and on data it has never seen³.

For extraction, the adversary queries the model without knowing anything about the training dataset². The adversary looks for sequences that come out word for word: in the GPT-2 study, hundreds of verbatim text sequences were extracted¹.

The goal depends on the target. For a model trained on sensitive records, knowing that a person appears in them can already be information. The 2016 authors illustrate this with a dataset of hospital discharges, whose membership is sensitive from a privacy standpoint³.

The sources cited here describe no visible sign for an employee. The attack consists of asking questions to an accessible model, and its queries can look ordinary. The trace shows up rather in a model response that contains a memorized excerpt.

The text below is a neutralized illustration, not a command to reproduce. It shows what a request looks like when it tries to pull personal data out of a model's memory.

Illustration: fictional exchange with an assistant, address in .invalid.
Question: [a repetitive or mundane request, paraphrased]
Model response: [a memorized excerpt: name, number, fictional email
  [email protected]]

The mechanism rests on differences in a model's behavior between what it has learned and what it has never seen³. The attacker exploits these differences, either by measuring them (membership inference) or by provoking the output of memorized sequences (extraction).

The targeted models are of various kinds. The 2016 researchers tested "on demand" machine learning services from commercial providers such as Google and Amazon, and show that these models can be vulnerable³. The 2023 researchers extract gigabytes of data from open models such as Pythia, semi-open models such as LLaMA and closed models such as ChatGPT².

NIST publishes a taxonomy of adversarial machine learning, whose keywords include "privacy breach"⁵. OWASP also cites the "Proof Pudding" attack (CVE-2019-20634): leaked training data made it easier to extract and invert the model there, which made it possible to bypass email filters⁴. It links these techniques to three entries in the MITRE ATLAS database: inferring membership in training data, inverting a model, extracting a model⁴.

On the defense side, OWASP recommends sanitizing data before training, restricting access according to the principle of least privilege, and using federated learning and differential privacy, which adds noise to the data or to the outputs⁴. It specifies that an instruction given to the model about the types of data not to return can be bypassed by prompt injection⁴. The UK NCSC reminds readers that AI systems are subject to new vulnerabilities, which add to the usual threats⁶.

December 2020: data extraction from GPT-2

On 14 December 2020, Nicholas Carlini and eleven co-authors posted on arXiv a study showing that an adversary can recover individual training examples by querying a language model¹. They attack GPT-2, trained on data collected from the public internet, and extract hundreds of verbatim text sequences from it¹. Among them are public personal information (names, phone numbers, email addresses), IRC conversations, code and 128-bit UUIDs¹. A revised version is dated 15 June 2021¹.

November 2023: data extraction from ChatGPT

On 28 November 2023, Milad Nasr, Nicholas Carlini and eight co-authors published a study on extracting training data from production models². To attack ChatGPT, an aligned model, they develop a so-called "divergence" attack: it makes the model leave its conversational style and emit training data at a rate 150 times higher than during normal operation². They conclude that practical attacks can recover much more data than was previously thought². The OWASP page points to a press article on this case, titled "ChatGPT Spit Out Sensitive Data When Told to Repeat ‘Poem’ Forever"⁴.

For the employee, the reflex follows from these cases. OWASP classes personal data, financial information, health records, confidential company data, security credentials and legal documents as sensitive data: they are not entered into a public AI tool⁴. For tools bought by the company, written guarantees are requested from suppliers, because OWASP recommends clear policies on data retention, use and deletion⁴.