Agent evaluation

Also known as: agent evals

On this page 7

Agent evaluation is the systematic measurement of how well an AI agent completes multi-step tasks in an environment, judged by its final outcomes, the quality of its intermediate actions, its safety behavior and its resource consumption. Within AI agents as a branch of artificial intelligence, agent evaluation, often shortened to agent evals, differs from classic model benchmarking because the system under test acts, observes and adapts over many turns rather than producing a single answer.

A useful evaluation therefore tests the whole agent harness: the language model, its prompts, tool schemas, memory, permissions and orchestration logic. Two agents built on the same model can score very differently once these components change.

What agent evaluation measures

Agent evaluation measures task success, process quality, efficiency and safety across a set of representative tasks. Because agents are non-deterministic, each task is usually run several times, and results are reported as rates with their variability rather than single pass or fail verdicts.

The table below lists the dimensions most teams track and the metrics that typically express them.

DimensionTypical metricWhat it reveals
Outcome successTask success rate, pass@k, pass^kWhether the goal state is reached
Trajectory qualityCorrect tool choice, redundant stepsWhether the path is sound
Tool-call accuracyValid arguments, schema errorsInterface reliability
EfficiencySteps, tokens, latency, cost per taskOperational viability
Safety and policyRate of disallowed actionsRisk of harmful side effects
RobustnessSuccess under perturbed inputsSensitivity to noise and variation
Core dimensions of agent evaluation

Two consistency metrics deserve attention. pass@k counts a task as solved if any of k attempts succeeds, which suits settings where a human picks the best result. pass^k requires all k attempts to succeed, which reflects the reliability demanded in unattended automation and drops sharply when an agent is inconsistent.

How agent evaluations are built

An agent evaluation consists of tasks, an environment, a harness and graders. Each task specifies a starting state and a goal; the environment supplies tools, files, web pages or simulated users; the harness runs the agent loop and records the agent trajectory; graders score the result.

  1. Define the task set from real usage, including edge cases and tasks that should be refused.
  2. Build a reproducible environment, ideally an agent sandbox that resets to a known state before every run.
  3. Specify success criteria as checkable end states, such as a passing test suite or a correct database record.
  4. Run repeated trials with fixed model versions and logged configuration.
  5. Grade outcomes and trajectories, then review failures manually to find root causes.

Grading methods vary in cost and trustworthiness, and mature evaluations combine several of them, as the following comparison shows.

Grader typeHow it worksStrengthWeakness
Programmatic checkInspects final state or runs unit testsObjective and cheapOnly fits verifiable tasks
Model-based judgeA language model scores output against a rubricHandles open-ended workBias and calibration drift
Human reviewExperts rate outcomes or trajectoriesHighest validitySlow and expensive
Simulated userA scripted or model-driven user interactsTests dialogue behaviorSimulator may be unrealistic
Grading methods for agent tasks

Agent evaluation compared with model benchmarks

Agent evaluation extends model benchmarking from single responses to sequences of actions with side effects. A model can excel on question-answering benchmarks and still fail as an agent because errors compound across steps: a small per-step error rate becomes a large end-to-end failure rate over a long trajectory.

AspectModel benchmarkAgent evaluation
Unit of testSingle prompt and responseFull multi-turn episode
EnvironmentStatic datasetInteractive tools and state
ScoringAnswer correctnessEnd state, path, cost and safety
VarianceLow with fixed decodingHigh; repeated runs needed
Main failure sourceKnowledge or reasoning gapsPlanning, tool use and recovery errors
Differences between model and agent evaluation

Agent evaluation also complements agent observability. Evaluation runs before deployment on curated tasks, whereas observability monitors live traces in production; findings from production traces are a primary source of new regression tasks for the evaluation suite.

Types of agent evaluation in practice

Teams use several evaluation types at different stages of development, each answering a distinct question about the agent.

Evaluation typeStageQuestion answered
Capability evaluationResearch and model selectionCan the agent solve hard tasks at all
Regression evaluationEvery change to prompts or toolsDid anything that worked break
Safety and red-team evaluationBefore releaseDoes it resist misuse and stay in scope
Online evaluationProductionHow does it perform with real users
Evaluation types and when to use them

Public benchmarks for coding agents, web and browser agents, computer-use agents and customer-service dialogues provide useful external reference points. However, they rarely match a specific product, so most organizations maintain a private task suite drawn from their own workflows.

Pitfalls and limitations

Agent evaluation is difficult to make valid and stable, and several recurring problems cause misleading scores.

  • Too few trials: a single run per task hides variance, so differences of a few points are often noise.
  • Benchmark contamination: tasks published online may appear in training data and inflate results.
  • Reward hacking: agents may satisfy a weak grader, for example by deleting a failing test instead of fixing the code.
  • Environment drift: live websites and external APIs change, which breaks reproducibility.
  • Unchecked model judges: a judge that is not validated against human ratings can reward fluent but wrong work.
  • Ignoring cost: a higher success rate achieved with many times more tokens may not be a real improvement.

The most effective safeguard is regular manual reading of trajectories. Aggregate scores show that something changed, but only transcripts show why, and they frequently reveal grader bugs rather than agent bugs.

In simple terms

Agent evaluation is a structured test drive for an AI agent. It does not stop at one question and one answer. Testers give the agent a real job, such as fixing a bug or booking a meeting. Then they watch whether it finishes. They check how it gets there. They also check whether it does anything risky.

The same agent can behave differently each time. So the test is repeated many times. The result tells a team how often the agent can be trusted with that kind of work and what it costs.

A practical example

A software company is preparing a coding agent that resolves internal bug reports by editing code and opening pull requests. The team collects 150 past bugs that already have fixes and tests, then packages each one in a container that resets the repository to the state before the fix.

Each task runs five times. A programmatic grader runs the hidden test suite, a model-based judge reviews whether the change stays in scope, and engineers manually read 30 failed trajectories. The first version solves 58 percent of tasks on at least one attempt but only 31 percent on all five attempts, and several failures come from the agent editing test files.

After the team restricts write permissions on test directories and improves the search tool description, the all-attempts rate rises to 44 percent at similar token cost. The suite then becomes a regression gate that every prompt or tool change must pass before release.