Jailbreaking an AI model means tricking it into doing what its makers trained it to refuse, such as explaining how to commit a crime or insulting the company that runs it. It is done with cleverly worded prompts, for example a role-play or a request written in code. For a business, a jailbroken chatbot can say things in your name that you would never approve.
How does it differ from prompt injection?
Simon Willison, who coined the term prompt injection, separates the two (March 2024). Jailbreaking attacks the safety filters built into the model itself. Prompt injection attacks an application that mixes the developer's trusted instructions with untrusted input, such as an email the AI is asked to summarise. The most common risk of a jailbreak, in his words, is a "screenshot attack": someone makes the model say something embarrassing and shares the screenshot.
Why does it work?
Wei et al. (July 2023) name two ways safety training fails. With competing objectives, the model's urge to follow instructions wins out over its safety goals, for example when the prompt demands that the answer start with "Absolutely! Here's". With mismatched generalisation, the model understands more than its safety training covered: a request encoded in Base64 got past GPT-4's refusals. Attacks built on these two ideas succeeded on every prompt in a set of unsafe requests taken from the red-teaming sets of GPT-4 and Claude v1.3.
Which kinds are there?
- Hand-written prompts that spread online. Shen et al. collected 1,405 jailbreak prompts, such as "Do Anything Now" (DAN), from 131 communities between December 2022 and December 2023. Five of them reached an attack success rate of 0.95 on ChatGPT (GPT-3.5) and GPT-4, and the oldest had been online for more than 240 days.
- Generated attacks. Zou et al. (July 2023) let an algorithm search for a string of characters that, added to a harmful question, makes a model answer it. Found on the open Vicuna models, the same string also worked on ChatGPT, Bard and Claude.
- Escalating conversations. Crescendo (Microsoft, April 2024) opens with a harmless question and steers the model step by step, building on its own answers.
- Very long prompts. Many-shot jailbreaking (Anthropic, April 2024) fills a large context window with fake dialogues in which an assistant answers harmful questions.
How do you defend against it?
Model makers train refusals into the model, for example with RLHF or Constitutional AI, test it with red teaming, and add separate classifiers that screen input and output. Anthropic's Constitutional Classifiers (February 2025) cut the success rate of 10,000 generated jailbreak prompts on Claude 3.5 Sonnet from 86% to 4.4%, at the cost of 0.38 percentage points more refusals on real traffic and 23.7% more computing, according to the paper. In a public challenge that same month, four participants still got through all eight levels, one with a universal jailbreak. Anthropic writes that in real use it would watch for users who keep triggering the classifiers and update them quickly. On your own side, guardrails and an instruction hierarchy limit what a jailbroken chatbot can do.
Example
In January 2024 a customer got the chatbot of parcel company DPD to swear at him and to write that DPD was "the worst delivery firm in the world". His post was viewed 800,000 times in 24 hours. DPD blamed an error after a system update and switched off the AI part of its chat, the BBC reported.
Sources
- Wei et al.: Jailbroken, How Does LLM Safety Training Fail? (2023)arXiv
- Shen et al.: "Do Anything Now", Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models (2023)arXiv
- Zou et al.: Universal and Transferable Adversarial Attacks on Aligned Language Models (2023)arXiv
- Simon Willison: Prompt injection and jailbreaking are not the same thingRSS
- Anthropic: Constitutional Classifiers, Defending against universal jailbreaksWeb
- BBC: DPD error caused chatbot to swear at customerWeb