Agent observability is the practice of capturing, correlating and analyzing telemetry from AI agents so that engineers can reconstruct what an agent did, why it did it and what it cost at every step of a run. It extends classic software observability to systems in which a language model decides at run time which tools to call, which data to read and when to stop.
In artificial intelligence engineering, the discipline is often called agent tracing, because its central artifact is a trace of the full agent trajectory: model calls, prompts, tool invocations, retrieved documents, handoffs between agents and human approvals. Without it, a nondeterministic multi-step system is effectively a black box in production.
Why agents need dedicated observability
Agents need dedicated observability because their control flow is generated by the model, not written in code. Two runs with the same input can take different paths, call different tools and produce different results, so a stack trace or a request log alone cannot explain a failure.
Agent failures are also often silent. A run can finish with HTTP 200 and a fluent answer while the agent looped five times, ignored a tool error, or grounded its answer in the wrong document. Detecting such semantic failures requires recording the content of each step, not only its status code.
The table below contrasts traditional application monitoring with what agent observability must add.
| Aspect | Traditional service | AI agent |
|---|---|---|
| Execution path | Fixed by code | Chosen by the model at run time |
| Failure signal | Exceptions and error codes | Often a plausible but wrong output |
| Main cost driver | Compute time | Tokens and external tool calls |
| Unit of analysis | Single request | Multi-step trajectory or session |
| Quality check | Assertions and tests | Evaluations of reasoning and outcome |
Traces, spans and the agent trajectory
The core data model of agent observability is the distributed trace, borrowed from microservice monitoring. One trace represents one agent run, and each step inside it is a span with a start time, an end time, a parent span and structured attributes.
Spans form a tree that mirrors the agent loop: a root span for the task, child spans for each reasoning turn, and grandchild spans for model calls, tool calls and retrieval. In a multi-agent system, context propagation links the spans of an orchestrator and its subagents into one trace, even across processes.
The OpenTelemetry semantic conventions for generative AI define standard attribute names for model, token counts and operations, which lets traces move between tools without custom mapping. The table lists the span types most systems record.
| Span type | Key attributes | Typical question it answers |
|---|---|---|
| Agent run | Task, user, outcome, total cost | Did the task succeed overall |
| Model call | Model name, prompt, completion, tokens | Why did the agent choose this action |
| Tool call | Tool name, arguments, result, error | Did the tool behave as expected |
| Retrieval | Query, document IDs, scores | Was the context relevant |
| Handoff or approval | Source, target, decision | Who acted and who authorized it |
Metrics and evaluations that matter
Traces explain individual runs, while aggregated metrics reveal trends across thousands of runs. Useful metrics combine operational signals with quality signals derived from evaluations, because an agent can be fast and cheap yet wrong.
Quality signals come from online evaluations: automated checks such as schema validation, an LLM-as-judge score, or comparison with a reference answer, attached to spans as scores. Human feedback, such as a thumbs-down or an edited output, is attached in the same way. Thresholds below are illustrative starting points, since acceptable values depend heavily on the task.
| Metric | What it measures | Warning sign |
|---|---|---|
| Task success rate | Share of runs reaching a correct outcome | Downward drift after a prompt change |
| Steps per run | Length of the trajectory | Long tail suggests loops |
| Tool error rate | Failed or rejected tool calls | Sudden rise after an API update |
| Tokens and cost per run | Input, output and cached tokens | Cost growing faster than traffic |
| End-to-end latency | Time from request to final answer | p95 far above the median |
Implementing observability in an agent system
Implementation starts with instrumentation: wrapping model clients, tools and agent framework hooks so that each call emits a span automatically. Many agent frameworks ship such hooks, and OpenTelemetry exporters send the data to a tracing backend or a specialized LLM observability platform.
- Assign a stable trace ID per task and propagate it to every subagent, tool and background job.
- Record inputs and outputs of each step, with redaction rules for personal and secret data.
- Tag runs with prompt, model and tool versions so regressions can be tied to a specific change.
- Attach evaluation scores and user feedback to the trace rather than storing them separately.
- Define alerts on behavior, such as loop length or tool error rate, not only on uptime.
Collected traces also feed the development cycle. Failed production runs become regression test cases for agent evaluation, and replaying a stored trajectory helps reproduce a bug that depended on a specific tool result.
Risks, costs and common mistakes
Full-content tracing creates its own risks. Prompts and tool results can contain personal data, credentials or confidential documents, so traces must follow the same access control, retention and data protection rules as the source systems.
| Mistake | Consequence | Correction |
|---|---|---|
| Logging only final answers | Root cause of failures stays hidden | Trace every intermediate step |
| Storing raw content unredacted | Privacy and security exposure | Mask sensitive fields at capture |
| Tracing all traffic at full detail | High storage cost | Sample routine runs, keep all failures |
| Monitoring latency without quality | Wrong answers go unnoticed | Attach evaluation scores to traces |
A final caution concerns interpretation. Model reasoning text recorded in a trace is not a guaranteed explanation of the model’s internal computation; it is useful evidence, but conclusions should be confirmed by the actual tool calls and outcomes.
In plain words
An AI agent works through a task in many small steps: it thinks, looks something up, uses a tool, and decides what to do next. Agent observability is like a flight recorder for those steps. It writes down each move, how long it took, what it cost and what came back.
When something goes wrong, the team does not have to guess. It can replay the recording and see exactly where the agent went off course. The key takeaway is simple: you cannot fix or trust an agent whose steps you cannot see.
A real-world example
An online retailer runs a customer support agent that can look up orders, check shipping status and issue refunds of up to 100 dollars. After a prompt update, complaints rise, but error dashboards show nothing unusual because every request still succeeds technically.
The engineering team opens traces for the complaint sessions and filters by the new prompt version. The traces show that the agent now calls the shipping tool with the order number in the wrong field, receives an empty result, and then tells customers that their parcel is lost. Average steps per run had risen from four to seven, and the tool span reported a validation warning that the agent ignored.
The team fixes the tool schema description, adds an alert on empty tool results, and turns the failing sessions into regression tests. Within a day, the task success rate returned to its previous level, and the trace data showed cost per conversation falling by roughly a third.