LLM observability is the practice of collecting, correlating and analyzing telemetry from applications built on large language models, so that teams can explain why a given request produced a given answer, at what cost and with what latency. In AI infrastructure, it extends classic application monitoring with signals specific to language models: prompts, completions, token counts, retrieval context, tool calls and quality evaluations.
Traditional monitoring answers whether a service is up and fast. LLM observability must also answer whether the output was correct, safe and useful, because a model can return a well-formed response in 800 milliseconds that is still a hallucination. It therefore combines distributed tracing, metrics, logs and automated evaluation in one workflow.
How LLM observability captures a request
LLM observability records each user interaction as a trace made of nested spans. A span represents one step, such as prompt assembly, a vector search, a model call, a tool invocation or a guardrail check, and carries timing, inputs, outputs and metadata. The trace shows the full causal chain from the user question to the final answer.
Instrumentation usually happens through SDK wrappers around model clients and agent frameworks, or through an AI gateway that sees every call. The OpenTelemetry project defines semantic conventions for generative AI spans, which lets teams export LLM telemetry into the same backends as the rest of their services.
The table below lists the span types that a typical retrieval-augmented application emits.
| Span type | Recorded data | Question it answers |
|---|---|---|
| Prompt construction | Template version, variables, final prompt | What exactly did the model receive |
| Retrieval | Query, retrieved chunks, similarity scores | Was the right context found |
| Model call | Model name, parameters, tokens, latency | How fast and how costly was generation |
| Tool call | Tool name, arguments, result, errors | Did the agent act correctly |
| Guardrail check | Policy, verdict, flagged content | Was unsafe output blocked |
| User feedback | Rating, comment, edit | Did the answer satisfy the user |
Key metrics for LLM applications
LLM observability tracks three families of metrics: performance, cost and quality. Performance metrics such as time to first token and time per output token describe user-perceived speed, while cost depends almost entirely on input and output token volume multiplied by provider prices or internal GPU cost.
The following table summarizes the core metrics with typical reference ranges for interactive chat applications.
| Metric | Typical target range | What a deviation signals |
|---|---|---|
| Time to first token | Under 1 s for chat | Queueing, long prompts or overloaded inference server |
| Time per output token | Roughly 10-50 ms | Decoding bottleneck or large model choice |
| Tokens per request | Depends on use case | Prompt bloat or runaway agent loops |
| Cost per request | Set by product budget | Expensive model routing or retries |
| Error and timeout rate | Well below 1% | Provider limits, rate limiting or failing tools |
| Evaluation score | Defined per task | Quality regression after a prompt or model change |
Averages hide problems, so teams watch p95 and p99 latency and cost per user segment. A small share of very long conversations often accounts for a disproportionate share of spend.
Evaluating output quality in production
Quality is the hardest signal to observe because there is rarely a single correct answer. LLM observability platforms attach online evaluations to sampled traces, scoring dimensions such as faithfulness to retrieved context, relevance, toxicity, format validity and refusal rate.
- Deterministic checks validate JSON schemas, length limits, forbidden strings and citation presence.
- LLM-as-a-judge uses a second model with a rubric to grade answers at scale, with known biases toward longer outputs.
- Human review queues collect expert labels on a sample to calibrate automated judges.
- Implicit feedback such as regenerations, copy actions and abandoned sessions indicates dissatisfaction.
Automated judges must themselves be validated against human labels, otherwise a team can optimize toward a metric that does not reflect real user value. Production traces that fail evaluation become regression test cases for the offline evaluation suite.
LLM observability compared with related practices
LLM observability overlaps with several neighboring disciplines but focuses on per-request explanation of generative behavior. The comparison below clarifies the boundaries.
| Practice | Primary focus | Typical signals |
|---|---|---|
| Application performance monitoring | Service health and latency | Request rates, errors, CPU, memory |
| Model monitoring | Statistical behavior of predictive models | Feature drift, prediction distribution, accuracy |
| LLM observability | Explaining individual generative requests | Traces, prompts, tokens, evaluations |
| LLMOps | Full lifecycle of LLM applications | Deployment, versioning, evaluation, governance |
| Experiment tracking | Offline training and test runs | Hyperparameters, datasets, benchmark scores |
In practice, LLM observability is the production feedback loop of LLMOps: it surfaces failures that then drive prompt changes, model routing decisions and new evaluation datasets.
Implementing LLM observability responsibly
A practical rollout starts with tracing every model call and attaching prompt and model versions, then adds cost dashboards, latency alerts and sampled evaluations. The decision table below covers common design choices.
| Decision | Recommended practice | Reason |
|---|---|---|
| Payload logging | Log full prompts with redaction | Debugging requires the exact input |
| Sampling rate | Trace all calls, evaluate a sample | Evaluation costs tokens and time |
| Version tagging | Tag prompt, model and index versions | Regressions must be attributable |
| Data retention | Short default, longer for flagged traces | Limits privacy and storage exposure |
| Alerting | Alert on cost, latency and score drops | Quality failures are silent otherwise |
Prompts and completions often contain personal or confidential data, so observability pipelines need redaction, access control and retention limits consistent with privacy law and internal policy. Storing raw conversations without these controls turns a debugging tool into a security liability.
Limitations and common pitfalls
LLM observability explains behavior after the fact but does not make a model reliable by itself. Common pitfalls include collecting traces nobody reviews, trusting an uncalibrated judge model and ignoring non-determinism, which means the same prompt may produce different answers on replay.
Telemetry volume is another constraint: a busy agent can emit dozens of spans per request, and storing full payloads at scale becomes expensive in storage and query cost. Teams should define in advance which questions the data must answer and size retention to those questions.
A plain-language summary
LLM observability is a way of watching what an AI assistant does behind the scenes. For every question a user asks, it records what the system looked up, what it sent to the model, what came back, how long it took and how much it cost.
It works like a flight recorder for AI answers. When something goes wrong, such as a slow reply or a wrong answer, the team can replay the steps and see exactly where the problem started.
An example from practice
A software company runs a support assistant that answers customer questions from its documentation. After a weekly update, the number of thumbs-down ratings rises from about 4% to 11%, but latency and error dashboards look normal.
Engineers filter traces with negative feedback and see that the retrieval span now returns outdated pages, because a new search index version was deployed without the latest documentation. The faithfulness evaluations confirm that answers match the retrieved text, so the model is not at fault. The team rebuilds the index, tags each trace with the index version and adds an alert on evaluation scores. Negative ratings return to the previous level within two days, and the investigation takes hours instead of a week of guessing.