LLM observability

On this page 8

LLM observability is the practice of collecting, correlating and analyzing telemetry from applications built on large language models, so that teams can explain why a given request produced a given answer, at what cost and with what latency. In AI infrastructure, it extends classic application monitoring with signals specific to language models: prompts, completions, token counts, retrieval context, tool calls and quality evaluations.

Traditional monitoring answers whether a service is up and fast. LLM observability must also answer whether the output was correct, safe and useful, because a model can return a well-formed response in 800 milliseconds that is still a hallucination. It therefore combines distributed tracing, metrics, logs and automated evaluation in one workflow.

How LLM observability captures a request

LLM observability records each user interaction as a trace made of nested spans. A span represents one step, such as prompt assembly, a vector search, a model call, a tool invocation or a guardrail check, and carries timing, inputs, outputs and metadata. The trace shows the full causal chain from the user question to the final answer.

Instrumentation usually happens through SDK wrappers around model clients and agent frameworks, or through an AI gateway that sees every call. The OpenTelemetry project defines semantic conventions for generative AI spans, which lets teams export LLM telemetry into the same backends as the rest of their services.

The table below lists the span types that a typical retrieval-augmented application emits.

Span typeRecorded dataQuestion it answers
Prompt constructionTemplate version, variables, final promptWhat exactly did the model receive
RetrievalQuery, retrieved chunks, similarity scoresWas the right context found
Model callModel name, parameters, tokens, latencyHow fast and how costly was generation
Tool callTool name, arguments, result, errorsDid the agent act correctly
Guardrail checkPolicy, verdict, flagged contentWas unsafe output blocked
User feedbackRating, comment, editDid the answer satisfy the user
Typical spans in a traced LLM request

Key metrics for LLM applications

LLM observability tracks three families of metrics: performance, cost and quality. Performance metrics such as time to first token and time per output token describe user-perceived speed, while cost depends almost entirely on input and output token volume multiplied by provider prices or internal GPU cost.

The following table summarizes the core metrics with typical reference ranges for interactive chat applications.

MetricTypical target rangeWhat a deviation signals
Time to first tokenUnder 1 s for chatQueueing, long prompts or overloaded inference server
Time per output tokenRoughly 10-50 msDecoding bottleneck or large model choice
Tokens per requestDepends on use casePrompt bloat or runaway agent loops
Cost per requestSet by product budgetExpensive model routing or retries
Error and timeout rateWell below 1%Provider limits, rate limiting or failing tools
Evaluation scoreDefined per taskQuality regression after a prompt or model change
Core metrics and what they reveal

Averages hide problems, so teams watch p95 and p99 latency and cost per user segment. A small share of very long conversations often accounts for a disproportionate share of spend.

Evaluating output quality in production

Quality is the hardest signal to observe because there is rarely a single correct answer. LLM observability platforms attach online evaluations to sampled traces, scoring dimensions such as faithfulness to retrieved context, relevance, toxicity, format validity and refusal rate.

  • Deterministic checks validate JSON schemas, length limits, forbidden strings and citation presence.
  • LLM-as-a-judge uses a second model with a rubric to grade answers at scale, with known biases toward longer outputs.
  • Human review queues collect expert labels on a sample to calibrate automated judges.
  • Implicit feedback such as regenerations, copy actions and abandoned sessions indicates dissatisfaction.

Automated judges must themselves be validated against human labels, otherwise a team can optimize toward a metric that does not reflect real user value. Production traces that fail evaluation become regression test cases for the offline evaluation suite.

LLM observability overlaps with several neighboring disciplines but focuses on per-request explanation of generative behavior. The comparison below clarifies the boundaries.

PracticePrimary focusTypical signals
Application performance monitoringService health and latencyRequest rates, errors, CPU, memory
Model monitoringStatistical behavior of predictive modelsFeature drift, prediction distribution, accuracy
LLM observabilityExplaining individual generative requestsTraces, prompts, tokens, evaluations
LLMOpsFull lifecycle of LLM applicationsDeployment, versioning, evaluation, governance
Experiment trackingOffline training and test runsHyperparameters, datasets, benchmark scores
How LLM observability differs from adjacent practices

In practice, LLM observability is the production feedback loop of LLMOps: it surfaces failures that then drive prompt changes, model routing decisions and new evaluation datasets.

Implementing LLM observability responsibly

A practical rollout starts with tracing every model call and attaching prompt and model versions, then adds cost dashboards, latency alerts and sampled evaluations. The decision table below covers common design choices.

DecisionRecommended practiceReason
Payload loggingLog full prompts with redactionDebugging requires the exact input
Sampling rateTrace all calls, evaluate a sampleEvaluation costs tokens and time
Version taggingTag prompt, model and index versionsRegressions must be attributable
Data retentionShort default, longer for flagged tracesLimits privacy and storage exposure
AlertingAlert on cost, latency and score dropsQuality failures are silent otherwise
Design decisions and recommended practice

Prompts and completions often contain personal or confidential data, so observability pipelines need redaction, access control and retention limits consistent with privacy law and internal policy. Storing raw conversations without these controls turns a debugging tool into a security liability.

Limitations and common pitfalls

LLM observability explains behavior after the fact but does not make a model reliable by itself. Common pitfalls include collecting traces nobody reviews, trusting an uncalibrated judge model and ignoring non-determinism, which means the same prompt may produce different answers on replay.

Telemetry volume is another constraint: a busy agent can emit dozens of spans per request, and storing full payloads at scale becomes expensive in storage and query cost. Teams should define in advance which questions the data must answer and size retention to those questions.

A plain-language summary

LLM observability is a way of watching what an AI assistant does behind the scenes. For every question a user asks, it records what the system looked up, what it sent to the model, what came back, how long it took and how much it cost.

It works like a flight recorder for AI answers. When something goes wrong, such as a slow reply or a wrong answer, the team can replay the steps and see exactly where the problem started.

An example from practice

A software company runs a support assistant that answers customer questions from its documentation. After a weekly update, the number of thumbs-down ratings rises from about 4% to 11%, but latency and error dashboards look normal.

Engineers filter traces with negative feedback and see that the retrieval span now returns outdated pages, because a new search index version was deployed without the latest documentation. The faithfulness evaluations confirm that answers match the retrieved text, so the model is not at fault. The team rebuilds the index, tags each trace with the index version and adds an alert on evaluation scores. Negative ratings return to the previous level within two days, and the investigation takes hours instead of a week of guessing.