Direct prompt injection

Researchers Fábio Perez and Ian Ribeiro showed in 2022 that people with little skill but bad intentions can hijack GPT-3 with simple handwritten texts³. They describe two objectives: hijacking the model's goal (goal hijacking) and making it disclose its prompt (prompt leaking)³.

A second type of objective is misuse. A team of researchers described in 2023 results such as unlimited and arbitrary use of the model behind an application, and easy theft of the application's prompt⁴.

Jailbreaking works differently. The authors of the "Jailbroken" study put forward two causes of safety training failure: competing objectives (the model's capabilities conflict with its safety rules) and mismatched generalization (safety does not cover an area where the model nonetheless has capabilities)⁵.

OWASP also cites variants. The attacker can add to their message a string of characters that looks meaningless, or use several languages or an encoding such as Base64 to evade filters¹.

On the user's or the company's side, a successful attack is spotted through its effects. OWASP lists the possible consequences: disclosure of sensitive information, revelation of information about the infrastructure or the system prompts, distorted or biased content, unauthorized access to the model's functions, execution of commands in connected systems¹.

In practice, an assistant that answers off topic, that recites its own instructions, or that produces content the company would never have approved should raise an alarm. OWASP gives a typical scenario: an attacker asks a support chatbot to ignore its instructions, to query private data, and to send emails¹.

Here is an illustration of this type of message, neutralized and paraphrased:

Illustration (paraphrased message, non-functional):
[instruction asking the chatbot to ignore the rules received from the vendor,
then to copy out its internal instructions in full]

For a language model, there are neither "data" nor "instructions", only the next token: the model predicts the most probable token from the text received so far². The vendor's instructions and the sentence typed by the user therefore end up in the same text, with no separation that the model respects².

This point distinguishes prompt injection from SQL injection. In SQL, parameterized queries guarantee that the engine never interprets an input as an instruction². The NCSC writes that prompt injection attacks might never be totally mitigated the way SQL injections are². The approaches under way (detecting attempts, training the model to prioritize instructions, telling the model what is data) layer onto the model a distinction that the technology does not make².

NIST published a taxonomy of adversarial machine learning in March 2025⁶.

One point of caution concerns banned-word filters. The NCSC warns against blocking known phrases such as "ignore previous instructions": there is an infinite number of rewordings that get around this filter².

Prevention

The simple measures retained here come from OWASP and the NCSC.

• Least privilege: give the model the minimum access needed, and handle extended functions in the application's code rather than entrusting them to the model¹.
• Human validation: require a person's approval for privileged operations¹.
• Output filtering: define sensitive categories, detect unauthorized content with semantic filters and string checks, and validate the format of responses with deterministic code¹.
• Role framing: describe the model's role and limits in the system prompt¹. This measure helps but is not enough: OWASP presents all these measures as ways to mitigate the impact¹.
• Testing: run regular penetration tests and simulations, treating the model as an untrusted user¹.
• Monitoring: log enough information to spot suspicious activity, including failed tool and API calls, which can betray the first steps of an attack².

The NCSC adds that a system's security must rest on deterministic safeguards that limit its actions, rather than on the sole hope of preventing malicious content from reaching the model². If the residual risk is intolerable, the use case may not suit a language model².

November 2022: GPT-3 hijacked by simple texts

On November 17, 2022, Fábio Perez and Ian Ribeiro published PromptInject, a framework for analyzing these attacks. They study GPT-3, which they describe as the most widely deployed language model in production, and show that it can be easily misaligned by handwritten inputs³. The two attacks studied are goal hijacking and prompt leaking³.

June 2023: 31 applications out of 36 vulnerable

On June 8, 2023, a team of twelve researchers published a study on real applications that use a language model. After analyzing ten commercial applications, it developed an attack technique named HouYi and applied it to 36 applications: 31 turned out to be susceptible to prompt injection⁴. Ten vendors validated the results, including Notion⁴.

July 2023: protections that give way

On July 5, 2023, Alexander Wei, Nika Haghtalab and Jacob Steinhardt evaluated OpenAI's GPT-4 and Anthropic's Claude v1.3. Their new attacks succeeded on every request in a collection of dangerous requests drawn from the models' red teaming evaluation sets⁵. The authors conclude that vulnerabilities persist despite red teaming and safety training efforts⁵.