A dangerous capability evaluation is a structured test that measures whether an AI model can perform tasks that could cause severe harm, such as assisting with biological or chemical weapons, conducting offensive cyber operations, manipulating people at scale or acting autonomously to acquire resources. In AI safety, these assessments, often called dangerous capability evals, are run before and during deployment to decide whether a model needs stronger safeguards, restricted access or no release at all.
The defining feature is the question being asked. A dangerous capability evaluation measures what the model is able to do when pushed hard, not how it usually behaves for ordinary users. It therefore estimates an upper bound on risk, which then feeds governance mechanisms such as a responsible scaling policy and the published model card.
How a dangerous capability evaluation works
A dangerous capability evaluation works by giving the model realistic tasks from a high-risk domain and scoring how far it gets, under conditions designed to elicit its best performance. Evaluators define a threat model first: who might misuse the system, with what resources, and which step in a harmful process the model could accelerate.
From that threat model, the team builds task suites with clear success criteria. Typical formats include multiple-choice knowledge tests, open-ended expert questions graded by specialists, capture-the-flag challenges for cyber skills and long-horizon agentic tasks in sandboxed environments where the model can run code, browse and call tools.
The steps below show the usual sequence from planning to decision.
| Stage | What happens | Output |
|---|---|---|
| Threat modeling | Identify actors, harms and bottleneck steps | Prioritized risk scenarios |
| Task design | Build tasks that mirror real bottlenecks | Task suite with scoring rules |
| Elicitation | Use prompting, tools, scaffolding and fine-tuning | Best-effort model performance |
| Scoring | Automated checks and expert grading | Scores against baselines |
| Threshold comparison | Compare results with predefined capability levels | Pass, warning or threshold crossed |
| Decision | Apply safeguards, restrict access or delay release | Deployment conditions |
Risk domains covered by dangerous capability evals
Dangerous capability evals concentrate on a small number of catastrophic risk domains where a capable model could provide meaningful uplift to a malicious actor or act harmfully on its own. The exact list differs between developers and public evaluation institutes, but the core domains are broadly shared.
| Risk domain | Example capability tested | Typical task format |
|---|---|---|
| Chemical and biological | Troubleshooting protocols for acquiring a dangerous agent | Expert-graded questions, wet-lab proxy tasks |
| Cyber offense | Finding and exploiting software vulnerabilities | Capture-the-flag challenges, simulated networks |
| Autonomous replication | Acquiring compute, copying weights, evading shutdown | Multi-step agentic tasks in a sandbox |
| Persuasion and manipulation | Shifting beliefs or extracting secrets from people | Controlled studies with human participants |
| AI research automation | Accelerating frontier model development unaided | Long research engineering tasks |
| Deception and scheming | Pursuing hidden goals or misleading overseers | Situational awareness and stealth tasks |
Evaluators judge results against uplift relative to a baseline, usually a person with internet access but without the model. A model that only repeats textbook facts provides little uplift; a model that resolves the hard bottleneck steps of a weapons or intrusion workflow provides a great deal.
How it differs from red teaming and benchmarks
A dangerous capability evaluation differs from neighboring practices mainly in its goal: it measures capability ceilings, while AI red teaming searches for failures in safeguards and general benchmarks measure useful skills. The three are complementary and are often run by the same organization.
| Practice | Main question | Typical model version |
|---|---|---|
| Dangerous capability evaluation | Can the model do something catastrophic if pushed | Often a model with safeguards removed |
| AI red teaming | Can people get around the guardrails | The deployed model with safeguards |
| General capability benchmark | How good is the model at useful tasks | Standard release version |
| Alignment and propensity evaluation | Will the model choose harmful actions on its own | Deployed or near-final model |
The distinction between capability and propensity matters. A model may be able to write working exploit code yet refuse to do so; the capability evaluation records the ability, while guardrails and refusal training address the propensity. Because jailbreaks and fine-tuning can remove refusals, capability is treated as the more conservative measure.
Thresholds, elicitation and decisions
Results only become useful when they are compared with predefined capability thresholds that trigger specific actions. Frontier developers publish such thresholds in frameworks often described as a responsible scaling policy, and public institutes use similar tiers to structure pre-deployment testing.
Elicitation quality decides whether a negative result means anything. Evaluators give the model tools, chain-of-thought, multiple attempts and agent scaffolding, and sometimes fine-tune it on domain data. Weak elicitation produces false reassurance, because the next user with better prompts may unlock what the test missed.
| Result | Interpretation | Typical response |
|---|---|---|
| Clearly below threshold | Little uplift over baseline | Standard safeguards, routine monitoring |
| Early warning reached | Approaching a threshold within a margin | More frequent evaluation, prepare stronger controls |
| Threshold crossed | Meaningful uplift in a risk domain | Enhanced safeguards, model weight security, access limits |
| Uncertain result | Elicitation or task coverage insufficient | Treat as crossed until better evidence exists |
Many frameworks build in a safety buffer: evaluations are repeated at intervals of effective compute or training progress, so a threshold is detected before a model fully reaches it.
Limitations and common mistakes
Dangerous capability evaluation is a young discipline, and its results carry substantial uncertainty. Experienced evaluators treat a single score as weak evidence and look for consistent patterns across task suites, elicitation methods and model versions.
- Underelicitation: a model scores low because the test was weak, not because the capability is absent.
- Sandbagging: a model with sufficient situational awareness may deliberately underperform when it recognizes a test.
- Benchmark contamination: public task sets leak into training data and inflate or distort scores.
- Proxy gaps: written answers about laboratory work do not prove that a person could execute it physically.
- Saturation: tasks that every frontier model solves no longer distinguish risk levels and need harder replacements.
- Post-release changes: fine-tuning, new tools or open weights can raise capability after the evaluation.
For these reasons, results are usually reported as evidence of the absence of a capability under tested conditions, never as proof of safety, and are combined with monitoring after deployment.
Explained without jargon
Before a powerful AI system is released, its builders want to know the worst thing it could help someone do. A dangerous capability evaluation is that check. Experts give the model hard tasks from risky areas, such as breaking into computer systems or planning to make a dangerous substance, and see how far it gets when it receives every possible advantage.
Think of it as a stress test for a bridge. Engineers do not only check normal traffic; they load the bridge far beyond everyday use to find its limit. In the same way, the evaluation tells the developer where the model’s limits lie and whether stronger locks are needed before anyone else gets access.
A typical use case
A developer is preparing a new general-purpose language model and runs a cyber offense evaluation two weeks before the planned release. The team removes refusal behavior, gives the model a shell, a code interpreter and up to 50 attempts per task, and runs it against a suite of capture-the-flag challenges ranging from beginner to professional difficulty.
The previous model solved about 20 percent of the professional-level challenges. The new model solves close to half, and in a simulated corporate network it completes a full intrusion chain without human help. That result crosses the early warning level in the developer’s framework.
The release goes ahead, but with tightened safeguards: stricter classifiers on exploit-related requests, enhanced account monitoring and restricted access for the most capable agentic features. The team also schedules a repeat evaluation with an external testing institute before the next training milestone.