Agent observability

Also known as: agent tracing

On this page 7

Agent observability is the practice of capturing, correlating and analyzing telemetry from AI agents so that engineers can reconstruct what an agent did, why it did it and what it cost at every step of a run. It extends classic software observability to systems in which a language model decides at run time which tools to call, which data to read and when to stop.

In artificial intelligence engineering, the discipline is often called agent tracing, because its central artifact is a trace of the full agent trajectory: model calls, prompts, tool invocations, retrieved documents, handoffs between agents and human approvals. Without it, a nondeterministic multi-step system is effectively a black box in production.

Why agents need dedicated observability

Agents need dedicated observability because their control flow is generated by the model, not written in code. Two runs with the same input can take different paths, call different tools and produce different results, so a stack trace or a request log alone cannot explain a failure.

Agent failures are also often silent. A run can finish with HTTP 200 and a fluent answer while the agent looped five times, ignored a tool error, or grounded its answer in the wrong document. Detecting such semantic failures requires recording the content of each step, not only its status code.

The table below contrasts traditional application monitoring with what agent observability must add.

AspectTraditional serviceAI agent
Execution pathFixed by codeChosen by the model at run time
Failure signalExceptions and error codesOften a plausible but wrong output
Main cost driverCompute timeTokens and external tool calls
Unit of analysisSingle requestMulti-step trajectory or session
Quality checkAssertions and testsEvaluations of reasoning and outcome
Traditional monitoring compared with agent observability

Traces, spans and the agent trajectory

The core data model of agent observability is the distributed trace, borrowed from microservice monitoring. One trace represents one agent run, and each step inside it is a span with a start time, an end time, a parent span and structured attributes.

Spans form a tree that mirrors the agent loop: a root span for the task, child spans for each reasoning turn, and grandchild spans for model calls, tool calls and retrieval. In a multi-agent system, context propagation links the spans of an orchestrator and its subagents into one trace, even across processes.

The OpenTelemetry semantic conventions for generative AI define standard attribute names for model, token counts and operations, which lets traces move between tools without custom mapping. The table lists the span types most systems record.

Span typeKey attributesTypical question it answers
Agent runTask, user, outcome, total costDid the task succeed overall
Model callModel name, prompt, completion, tokensWhy did the agent choose this action
Tool callTool name, arguments, result, errorDid the tool behave as expected
RetrievalQuery, document IDs, scoresWas the context relevant
Handoff or approvalSource, target, decisionWho acted and who authorized it
Common span types in an agent trace

Metrics and evaluations that matter

Traces explain individual runs, while aggregated metrics reveal trends across thousands of runs. Useful metrics combine operational signals with quality signals derived from evaluations, because an agent can be fast and cheap yet wrong.

Quality signals come from online evaluations: automated checks such as schema validation, an LLM-as-judge score, or comparison with a reference answer, attached to spans as scores. Human feedback, such as a thumbs-down or an edited output, is attached in the same way. Thresholds below are illustrative starting points, since acceptable values depend heavily on the task.

MetricWhat it measuresWarning sign
Task success rateShare of runs reaching a correct outcomeDownward drift after a prompt change
Steps per runLength of the trajectoryLong tail suggests loops
Tool error rateFailed or rejected tool callsSudden rise after an API update
Tokens and cost per runInput, output and cached tokensCost growing faster than traffic
End-to-end latencyTime from request to final answerp95 far above the median
Core agent observability metrics

Implementing observability in an agent system

Implementation starts with instrumentation: wrapping model clients, tools and agent framework hooks so that each call emits a span automatically. Many agent frameworks ship such hooks, and OpenTelemetry exporters send the data to a tracing backend or a specialized LLM observability platform.

  1. Assign a stable trace ID per task and propagate it to every subagent, tool and background job.
  2. Record inputs and outputs of each step, with redaction rules for personal and secret data.
  3. Tag runs with prompt, model and tool versions so regressions can be tied to a specific change.
  4. Attach evaluation scores and user feedback to the trace rather than storing them separately.
  5. Define alerts on behavior, such as loop length or tool error rate, not only on uptime.

Collected traces also feed the development cycle. Failed production runs become regression test cases for agent evaluation, and replaying a stored trajectory helps reproduce a bug that depended on a specific tool result.

Risks, costs and common mistakes

Full-content tracing creates its own risks. Prompts and tool results can contain personal data, credentials or confidential documents, so traces must follow the same access control, retention and data protection rules as the source systems.

MistakeConsequenceCorrection
Logging only final answersRoot cause of failures stays hiddenTrace every intermediate step
Storing raw content unredactedPrivacy and security exposureMask sensitive fields at capture
Tracing all traffic at full detailHigh storage costSample routine runs, keep all failures
Monitoring latency without qualityWrong answers go unnoticedAttach evaluation scores to traces
Frequent mistakes and how to correct them

A final caution concerns interpretation. Model reasoning text recorded in a trace is not a guaranteed explanation of the model’s internal computation; it is useful evidence, but conclusions should be confirmed by the actual tool calls and outcomes.

In plain words

An AI agent works through a task in many small steps: it thinks, looks something up, uses a tool, and decides what to do next. Agent observability is like a flight recorder for those steps. It writes down each move, how long it took, what it cost and what came back.

When something goes wrong, the team does not have to guess. It can replay the recording and see exactly where the agent went off course. The key takeaway is simple: you cannot fix or trust an agent whose steps you cannot see.

A real-world example

An online retailer runs a customer support agent that can look up orders, check shipping status and issue refunds of up to 100 dollars. After a prompt update, complaints rise, but error dashboards show nothing unusual because every request still succeeds technically.

The engineering team opens traces for the complaint sessions and filters by the new prompt version. The traces show that the agent now calls the shipping tool with the order number in the wrong field, receives an empty result, and then tells customers that their parcel is lost. Average steps per run had risen from four to seven, and the tool span reported a validation warning that the agent ignored.

The team fixes the tool schema description, adds an alert on empty tool results, and turns the failing sessions into regression tests. Within a day, the task success rate returned to its previous level, and the trace data showed cost per conversation falling by roughly a third.