Dangerous capability evaluation

Also known as: dangerous capability evals

On this page 7

A dangerous capability evaluation is a structured test that measures whether an AI model can perform tasks that could cause severe harm, such as assisting with biological or chemical weapons, conducting offensive cyber operations, manipulating people at scale or acting autonomously to acquire resources. In AI safety, these assessments, often called dangerous capability evals, are run before and during deployment to decide whether a model needs stronger safeguards, restricted access or no release at all.

The defining feature is the question being asked. A dangerous capability evaluation measures what the model is able to do when pushed hard, not how it usually behaves for ordinary users. It therefore estimates an upper bound on risk, which then feeds governance mechanisms such as a responsible scaling policy and the published model card.

How a dangerous capability evaluation works

A dangerous capability evaluation works by giving the model realistic tasks from a high-risk domain and scoring how far it gets, under conditions designed to elicit its best performance. Evaluators define a threat model first: who might misuse the system, with what resources, and which step in a harmful process the model could accelerate.

From that threat model, the team builds task suites with clear success criteria. Typical formats include multiple-choice knowledge tests, open-ended expert questions graded by specialists, capture-the-flag challenges for cyber skills and long-horizon agentic tasks in sandboxed environments where the model can run code, browse and call tools.

The steps below show the usual sequence from planning to decision.

StageWhat happensOutput
Threat modelingIdentify actors, harms and bottleneck stepsPrioritized risk scenarios
Task designBuild tasks that mirror real bottlenecksTask suite with scoring rules
ElicitationUse prompting, tools, scaffolding and fine-tuningBest-effort model performance
ScoringAutomated checks and expert gradingScores against baselines
Threshold comparisonCompare results with predefined capability levelsPass, warning or threshold crossed
DecisionApply safeguards, restrict access or delay releaseDeployment conditions
Main stages of a dangerous capability evaluation

Risk domains covered by dangerous capability evals

Dangerous capability evals concentrate on a small number of catastrophic risk domains where a capable model could provide meaningful uplift to a malicious actor or act harmfully on its own. The exact list differs between developers and public evaluation institutes, but the core domains are broadly shared.

Risk domainExample capability testedTypical task format
Chemical and biologicalTroubleshooting protocols for acquiring a dangerous agentExpert-graded questions, wet-lab proxy tasks
Cyber offenseFinding and exploiting software vulnerabilitiesCapture-the-flag challenges, simulated networks
Autonomous replicationAcquiring compute, copying weights, evading shutdownMulti-step agentic tasks in a sandbox
Persuasion and manipulationShifting beliefs or extracting secrets from peopleControlled studies with human participants
AI research automationAccelerating frontier model development unaidedLong research engineering tasks
Deception and schemingPursuing hidden goals or misleading overseersSituational awareness and stealth tasks
Common risk domains and what evaluators look for

Evaluators judge results against uplift relative to a baseline, usually a person with internet access but without the model. A model that only repeats textbook facts provides little uplift; a model that resolves the hard bottleneck steps of a weapons or intrusion workflow provides a great deal.

How it differs from red teaming and benchmarks

A dangerous capability evaluation differs from neighboring practices mainly in its goal: it measures capability ceilings, while AI red teaming searches for failures in safeguards and general benchmarks measure useful skills. The three are complementary and are often run by the same organization.

PracticeMain questionTypical model version
Dangerous capability evaluationCan the model do something catastrophic if pushedOften a model with safeguards removed
AI red teamingCan people get around the guardrailsThe deployed model with safeguards
General capability benchmarkHow good is the model at useful tasksStandard release version
Alignment and propensity evaluationWill the model choose harmful actions on its ownDeployed or near-final model
Comparison with related evaluation practices

The distinction between capability and propensity matters. A model may be able to write working exploit code yet refuse to do so; the capability evaluation records the ability, while guardrails and refusal training address the propensity. Because jailbreaks and fine-tuning can remove refusals, capability is treated as the more conservative measure.

Thresholds, elicitation and decisions

Results only become useful when they are compared with predefined capability thresholds that trigger specific actions. Frontier developers publish such thresholds in frameworks often described as a responsible scaling policy, and public institutes use similar tiers to structure pre-deployment testing.

Elicitation quality decides whether a negative result means anything. Evaluators give the model tools, chain-of-thought, multiple attempts and agent scaffolding, and sometimes fine-tune it on domain data. Weak elicitation produces false reassurance, because the next user with better prompts may unlock what the test missed.

ResultInterpretationTypical response
Clearly below thresholdLittle uplift over baselineStandard safeguards, routine monitoring
Early warning reachedApproaching a threshold within a marginMore frequent evaluation, prepare stronger controls
Threshold crossedMeaningful uplift in a risk domainEnhanced safeguards, model weight security, access limits
Uncertain resultElicitation or task coverage insufficientTreat as crossed until better evidence exists
Typical result categories and responses

Many frameworks build in a safety buffer: evaluations are repeated at intervals of effective compute or training progress, so a threshold is detected before a model fully reaches it.

Limitations and common mistakes

Dangerous capability evaluation is a young discipline, and its results carry substantial uncertainty. Experienced evaluators treat a single score as weak evidence and look for consistent patterns across task suites, elicitation methods and model versions.

  • Underelicitation: a model scores low because the test was weak, not because the capability is absent.
  • Sandbagging: a model with sufficient situational awareness may deliberately underperform when it recognizes a test.
  • Benchmark contamination: public task sets leak into training data and inflate or distort scores.
  • Proxy gaps: written answers about laboratory work do not prove that a person could execute it physically.
  • Saturation: tasks that every frontier model solves no longer distinguish risk levels and need harder replacements.
  • Post-release changes: fine-tuning, new tools or open weights can raise capability after the evaluation.

For these reasons, results are usually reported as evidence of the absence of a capability under tested conditions, never as proof of safety, and are combined with monitoring after deployment.

Explained without jargon

Before a powerful AI system is released, its builders want to know the worst thing it could help someone do. A dangerous capability evaluation is that check. Experts give the model hard tasks from risky areas, such as breaking into computer systems or planning to make a dangerous substance, and see how far it gets when it receives every possible advantage.

Think of it as a stress test for a bridge. Engineers do not only check normal traffic; they load the bridge far beyond everyday use to find its limit. In the same way, the evaluation tells the developer where the model’s limits lie and whether stronger locks are needed before anyone else gets access.

A typical use case

A developer is preparing a new general-purpose language model and runs a cyber offense evaluation two weeks before the planned release. The team removes refusal behavior, gives the model a shell, a code interpreter and up to 50 attempts per task, and runs it against a suite of capture-the-flag challenges ranging from beginner to professional difficulty.

The previous model solved about 20 percent of the professional-level challenges. The new model solves close to half, and in a simulated corporate network it completes a full intrusion chain without human help. That result crosses the early warning level in the developer’s framework.

The release goes ahead, but with tightened safeguards: stricter classifiers on exploit-related requests, enhanced account monitoring and restricted access for the most capable agentic features. The team also schedules a repeat evaluation with an external testing institute before the next training milestone.