AI red teaming

Red teaming means attacking your own AI system on purpose, before customers or criminals do. Testers try to make a chatbot or AI agent leak data, promise things it may not promise, give harmful answers or take actions it was never allowed to take. What they find gets fixed before launch. For a business, it means finding the weak spots of its chatbot before someone else does.

Where does the term come from?

The term dates from the 1960s in the United States. The RAND Corporation ran Cold War simulations for the military in which the red team played the Soviet Union and the blue team the United States, Wikipedia summarises. In IT security, a red team simulates an attacker against its own organisation. AI red teaming applies the idea to AI models and the products built on them.

How does it work?

People write attacks by hand: jailbreaks, prompt injection, requests for dangerous information, attempts to extract personal data. Anthropic (Ganguli et al., August 2022) had people attack its models and published the 38,961 attacks as a dataset. Its models trained with RLHF became harder to red team as they grew larger.

An AI model can also play the attacker. Researchers at DeepMind (Perez et al., February 2022) let one language model generate test questions for another. They found tens of thousands of offensive replies in a chatbot of 280 billion parameters, as well as phone numbers the chatbot presented as its own and leaked training data.

Microsoft's AI Red Team, which tested more than 100 generative AI products, combines both (Bullwinkel et al., January 2025): automation to cover more ground, people for judgement and context. Its tool for the automated part, PyRIT, is open source.

What have red teams learned?

Microsoft's team lists eight lessons. Three stand out:

• Simple attacks work. "Real attackers don't compute gradients, they prompt engineer", the team quotes: cleverly worded prompts often work as well as mathematically optimised attacks.
• Red teaming is not benchmarking. A benchmark measures known risks; a red team looks for new ones in the context where the system is used.
• The work is never finished. The realistic goal is to make an attack cost more than it yields, through rounds of attacking and repairing that Microsoft calls break-fix cycles.

That cost can be measured. In April 2026 the security company Fabraix proposed ACE, the amount an automated attacker spends before an AI agent calls a forbidden tool. All six budget models it tested were broken, at average costs from $0.23 to $10.21.

Who has to do it?

Model makers red team before a release; OpenAI describes how it works with outside experts in a white paper (January 2025). In the EU, the AI Act has required it since 2 August 2025 for general-purpose AI models with systemic risk, the most capable ones: their makers must conduct and document adversarial testing (Article 55), with red teaming named as an example (Annex XI). OpenAI also names the limits: a test captures one moment while models keep changing, and it takes time and money that smaller organisations may lack.

Example

Suppose a web shop is about to launch a support chatbot. Two staff members spend an afternoon trying to break it: they ask for another customer's order, claim to be the owner, ask for a discount code, paste in a web page with hidden instructions and ask about topics the shop does not cover. Every prompt and answer goes into a spreadsheet with a note on what went wrong. After the fixes, such as a narrower system prompt and a refund limit in the order system itself, they run the same tests again.