AI red teaming is the practice of deliberately attacking an AI model or system with adversarial inputs, scenarios and goals in order to discover harmful, insecure or unintended behavior before release or before real adversaries find it. In AI safety, it is the main empirical method for testing whether safeguards actually hold under pressure, complementing alignment training, policy design and automated benchmarks.
The term comes from military and cybersecurity exercises, where a red team plays the opponent. Applied to artificial intelligence, the scope is broader than security: red teamers probe for dangerous capabilities, policy violations, bias, privacy leaks and failures of agentic systems, and they document each finding so that developers can reproduce, measure and fix it.
How an AI red teaming exercise is run
A red teaming exercise follows a defined scope, threat model and reporting process. Without them, testing turns into random prompting that produces anecdotes rather than evidence. A typical engagement moves through the stages below.
| Stage | Main activity | Output |
|---|---|---|
| Scoping | Define system, harms of interest and rules of engagement | Test plan and threat model |
| Recruiting testers | Combine security, domain and cultural expertise | Diverse red team |
| Probing | Manual and automated adversarial attempts | Raw transcripts and attack logs |
| Triage | Rate severity, likelihood and reproducibility | Prioritized findings |
| Mitigation and retest | Fix, then repeat the successful attacks | Evidence that fixes hold |
The threat model is the anchor. It states who the adversary is, what access they have (a public chat interface, an API, model weights or an agent’s tool environment) and what outcome counts as failure. A red team testing a customer chatbot and one testing an open-weight model face very different attack surfaces.
What red teams look for
Findings are usually grouped into harm categories so that coverage can be tracked and results compared between model versions. The categories below are common across frameworks, although the exact taxonomy varies by organization and regulation.
| Category | Example failure | Why it matters |
|---|---|---|
| Jailbreaks | Role-play prompt bypasses refusal policy | Safeguards fail under rephrasing |
| Prompt injection | Hidden text in a web page redirects an agent | Agents act on untrusted data |
| Dangerous capabilities | Meaningful uplift for weapons or cyberattacks | Severe real-world harm |
| Privacy leakage | Model reveals training data or system prompts | Legal and trust consequences |
| Bias and discrimination | Unequal treatment across demographic groups | Fairness and compliance |
| Misinformation | Confident false claims on health or elections | Public harm at scale |
For agentic systems, the most important questions concern actions rather than words: whether the agent can be tricked into sending data, spending money or modifying files, and whether permission boundaries hold when tool outputs contain adversarial instructions.
Manual, automated and hybrid red teaming
Human red teamers find novel and context-dependent failures that no automated method anticipates, but they are slow and expensive. Automated red teaming uses attacker models, fuzzing and optimization methods to generate thousands of attacks, which gives breadth and repeatability. Mature programs combine both.
| Approach | Strength | Weakness | Typical use |
|---|---|---|---|
| Manual expert testing | Creativity and domain depth | Low volume, costly | High-severity risk areas |
| Crowdsourced testing | Diverse perspectives and languages | Variable quality | Broad policy coverage |
| Automated attacker models | Scale and regression testing | Misses novel strategies | Release gates and monitoring |
| Gradient-based attacks | Finds adversarial suffixes efficiently | Needs model weights | Open-weight model evaluation |
A successful manual attack should be turned into an automated regression test. Otherwise a later model update may silently reintroduce the same weakness.
Red teaming among related safety practices
Red teaming is one part of a safety program, not the whole of it. AI alignment work tries to make the model want the right things; red teaming checks whether it behaves correctly when someone tries hard to make it fail. Benchmarks measure average behavior on fixed sets, whereas red teaming searches for the worst case.
| Practice | Question it answers | Relation to red teaming |
|---|---|---|
| AI alignment | Does the model pursue intended goals | Red teaming tests alignment empirically |
| AI bias audit | Does the model treat groups fairly | Often a dedicated red teaming track |
| Explainable AI | Why did the model produce this output | Helps diagnose red team findings |
| Model card | What are the model’s limits and results | Summarizes red teaming outcomes publicly |
| Penetration testing | Is the surrounding software secure | Covers infrastructure, not model behavior |
Regulators increasingly expect this work. The EU AI Act requires adversarial testing for general-purpose models with systemic risk, and the NIST AI Risk Management Framework describes red teaming as part of measuring risk. Requirements differ by jurisdiction and change over time, so programs should track the rules that apply to them.
Limitations and common mistakes
- Treating a clean result as proof of safety. Red teaming shows the presence of failures, never their absence.
- Testing the model but not the system. Retrieval, tools and plugins introduce attack paths the bare model lacks.
- Narrow tester pools. Testers from one language and culture miss failures that appear elsewhere.
- No severity scale. Without consistent ratings, trivial findings crowd out critical ones.
- Fixing symptoms only. Blocking one exact prompt invites trivial rephrasings.
Red teaming also carries risks for the testers themselves, who may be exposed to disturbing content for long periods. Responsible programs provide wellbeing support, rotation and clear opt-out rules, and they store harmful outputs with restricted access.
Stripped to the essentials
Before a bank opens a new vault, it may hire experts to try to break in. If they succeed, the bank learns where the weak spots are and fixes them before real thieves arrive.
AI red teaming does the same for AI systems. Testers try to trick a model into saying or doing something harmful, such as revealing private data or helping with something dangerous. Every successful trick becomes a lesson that makes the system safer before the public uses it.
Where it is applied
A health insurer prepares to launch an assistant that answers questions about policies and can file claims through a tool. Before launch, a mixed team of testers, including security engineers, clinicians and speakers of five languages spends three weeks attacking it.
They find that a claim form containing hidden instructions can make the assistant attach another customer’s records, and that a role-play prompt in one language extracts medical dosing advice the policy forbids. The developers add strict per-user data scoping for the tool, strengthen multilingual safety training and convert 400 successful attacks into an automated test suite. A retest shows the injection path closed, and the suite now runs on every model update as a release gate.