Prompt injection

Prompt injection is an attack in which someone slips their own instructions into text that an AI tool reads, so the tool follows those instead of yours. A visitor can type them into your website's chatbot, or hide them in an email, web page or document the AI is asked to handle. The more the tool may do, such as read customer data or send email, the greater the damage.

Where does the name come from?

In September 2022 Riley Goodside showed that GPT-3, asked to translate a text, obeyed a line in that text telling it to ignore its directions. Simon Willison named the attack prompt injection, after SQL injection: in both cases an application pastes untrusted input into its own trusted instructions. A language model reads the result as one text and cannot reliably tell who wrote which part. SQL injection has a standard defence, parameterised queries, which keep data and commands apart. For prompt injection, OWASP wrote in 2025 that it is "unclear if there are fool-proof methods of prevention".

Willison keeps it apart from jailbreaking, which targets the safety filters built into the model itself.

Direct or indirect?

OWASP puts prompt injection first (LLM01) in its 2025 list of risks for LLM applications, and distinguishes two forms:

• Direct: the user types the instructions. In December 2023 a visitor told the ChatGPT-based chatbot of a Chevrolet dealer in California to agree with anything a customer said. It then accepted $1 for a new Tahoe, calling it "a legally binding offer" (AI Incident Database).
• Indirect: the instructions reach the model through content it processes, such as a web page, an email or a file. Greshake et al. demonstrated this in February 2023 against Bing's GPT-4-based chat. The instructions do not have to be visible to people, OWASP notes, as long as the model reads them.

What can an attacker do with it?

That depends on what the application can do, Willison writes: if a chatbot has no access to confidential data or tools, the risk is limited. OWASP lists what can go wrong beyond that: leaked data or system prompts, manipulated answers, unauthorised use of the functions the model can call, and commands run in connected systems. Willison calls the most dangerous mix the lethal trifecta (June 2025): access to private data, exposure to untrusted content and a way to send data out.

EchoLeak in Microsoft 365 Copilot had all three. An attacker's email, written as if addressed to the recipient, got past Microsoft's filters against prompt injection. When a user later asked Copilot a question, it retrieved that email and put internal data into a link or image address that sent it to the attacker's server. Aim Labs reported the flaw in January 2025, Microsoft rated it critical as CVE-2025-32711, and by June the fix was rolled out, Willison reported.

How do you limit it?

OWASP's measures limit the damage:

• Give the model as few rights as possible, and handle sensitive functions in code outside the model.
• Let a person approve risky actions (human-in-the-loop).
• Mark external content as untrusted, and filter input and output (guardrails).
• Test the application with attacks (red teaming).

Model makers also train an instruction hierarchy, in which instructions found in documents or tool output carry no authority. Tests such as IHEval (2025) show that models do not follow it reliably yet.

Example

Suppose a web shop's support bot can look up orders and refund up to €50. A customer writes that he is the owner and that the limit has been lifted. If the limit only exists in the bot's instructions, the model may believe him. If the refund system itself refuses anything above €50, the attack yields at most an odd reply.