Langfuse

On this page 7

Langfuse is an open source LLM engineering platform that records, analyzes and evaluates what applications built on large language models actually do in development and production. Among AI frameworks & tools, it belongs to the LLM observability category: it does not build or run models itself, but it captures every model call, retrieval step and tool invocation as a trace that engineers can inspect, score and compare.

Langfuse combines four functions that teams otherwise assemble from separate tools: tracing, prompt management, evaluation and cost and latency analytics. Its core is released under the MIT license and can be used as a managed cloud service or self-hosted on the team’s own infrastructure.

How Langfuse traces LLM applications

Langfuse traces an application by recording each request as a tree of observations that mirrors the program’s execution. A trace represents one end-to-end request, such as a user question to a chatbot, and nested observations capture the individual steps inside it.

Instrumentation happens through Python and JavaScript/TypeScript SDKs, decorators that wrap existing functions, and integrations with libraries such as LangChain, LlamaIndex, LiteLLM, the OpenAI SDK and the Vercel AI SDK. Recent SDK versions are built on OpenTelemetry, so traces from any OpenTelemetry-compatible instrumentation can be ingested. Data is sent asynchronously in batches, which keeps the overhead on the application’s response time low.

The table below shows the main data objects Langfuse records and what each one captures.

ObjectWhat it representsTypical content
TraceOne end-to-end requestInput, output, user ID, tags
SpanA unit of work inside a traceRetrieval step, tool call, function duration
GenerationA single model callPrompt, completion, model name, tokens, cost
SessionA group of related tracesA multi-turn conversation
ScoreA quality judgment attached to a traceNumeric, boolean or categorical value
Core data objects in Langfuse

Prompt management and evaluation workflows

Beyond tracing, Langfuse acts as a central prompt registry. Prompts are stored with version numbers and labels such as production or staging, fetched by the application at runtime and cached client-side, so a team can change a prompt without redeploying code and can roll back to an earlier version when quality drops.

Evaluation connects these prompts to measurable quality. Langfuse supports several scoring methods, and mature teams usually combine them because each covers a different blind spot.

MethodHow it worksBest suited for
LLM-as-a-judgeA model grades outputs against a rubricScalable checks of relevance or tone
Human annotationReviewers label traces in queuesGround truth and judge calibration
User feedbackEnd users rate answers in the productReal-world satisfaction signals
Custom scores via APICode computes metrics and posts themDeterministic checks such as format validity
Evaluation methods supported by Langfuse

Datasets and experiments close the loop. Engineers collect representative inputs, often copied from production traces, run a new prompt or model version against the dataset and compare the scored results side by side before release. This turns prompt changes from guesswork into regression testing for LLM behavior.

Architecture and self-hosting

A self-hosted Langfuse deployment consists of a web and API server, a background worker and several data stores. Current versions use PostgreSQL for transactional data, ClickHouse for high-volume trace analytics, Redis for queues and caching, and S3-compatible object storage for raw event payloads.

Small teams can start with Docker Compose on one machine, while production installations typically run on Kubernetes with managed databases. The choice between the cloud service and self-hosting usually depends on data residency, because traces contain full prompts and outputs that may include personal or confidential data.

The following decision table summarizes when each option fits.

SituationRecommended optionReason
Prototype or small teamManaged cloudNo infrastructure to operate
Strict data residency rulesSelf-hostedTraces never leave own infrastructure
Millions of traces per daySelf-hosted on Kubernetes or cloud enterprise planNeeds scaled ClickHouse and workers
Local experimentationDocker ComposeStarts on a laptop in minutes
Choosing a Langfuse deployment

Langfuse is most often compared with LangSmith, the observability platform from the LangChain team, and with experiment trackers such as MLflow and Weights & Biases. The key distinction is that Langfuse is framework-agnostic and open source, while it focuses on LLM applications rather than classic model training.

ToolPrimary focusLicensing and hosting
LangfuseLLM tracing, prompts, evaluationMIT core, cloud or self-hosted
LangSmithLLM tracing tied to LangChain ecosystemProprietary, cloud or enterprise self-hosting
MLflowExperiment tracking and model registry, with newer LLM tracingOpen source, self-hosted or managed
Weights & BiasesTraining runs, sweeps, LLM evaluationProprietary, cloud or dedicated
Langfuse and neighboring tools

Langfuse is also not an LLM gateway. A proxy such as LiteLLM routes requests to model providers, while Langfuse observes them; the two are frequently used together, with the gateway forwarding call data to Langfuse.

Limitations and practical pitfalls

Langfuse makes problems visible, but it does not fix them, and its value depends on disciplined instrumentation. Several pitfalls appear repeatedly in real deployments.

  • Sensitive data in traces: prompts and outputs are stored in full unless teams apply masking functions before sending them.
  • Uncalibrated judges: LLM-as-a-judge scores drift and share model biases, so they need periodic comparison with human labels.
  • Missing context: traces without user IDs, session IDs or release tags are hard to filter and compare.
  • Lost events in short-lived processes: serverless functions must flush the SDK queue before they exit.
  • Self-hosting effort: operating ClickHouse at scale requires real database expertise.

Cost figures shown in Langfuse are estimates based on token counts and configured model prices, so they should be reconciled with provider invoices rather than treated as billing records.

In plain words

Langfuse is like a flight recorder for AI applications. A chatbot answers a question. Langfuse writes down what it was asked and which documents it read. It also notes the reply, the time it took and the cost.

Developers use this record to find out why an answer went wrong. They also test whether a new instruction works better than the old one. And they keep costs under control. Because it is open source, a company can run it on its own servers and keep the data private.

A real-world example

A software company runs a customer support assistant that answers questions from its help center using retrieval-augmented generation. Users begin to complain that answers about billing are vague, but the logs show only that requests succeeded. The team adds the Langfuse Python SDK, which takes about a day, and starts recording every retrieval and generation step.

Within a week, the traces show that billing questions retrieve outdated articles in about one case out of five, and that a long system prompt adds roughly 2,000 tokens to every request. The team fixes the index, shortens the prompt and builds a dataset of 300 real billing questions with an LLM-as-a-judge score for correctness, calibrated against 50 human-labeled answers. The result is a measurable rise in correctness on the dataset, lower cost per answer and a regression test that runs before every prompt change.