LangSmith

On this page 7

LangSmith is a platform for tracing, evaluating and monitoring applications built on large language models, developed by the company behind LangChain. Among AI frameworks & tools, it belongs to the LLM observability and evaluation category: it does not run models itself, but records what an application sends to models and tools, measures the quality of the results and helps teams find the cause of failures.

Although it originated alongside LangChain and LangGraph, LangSmith is framework-agnostic. Applications written with plain provider SDKs or other frameworks can send data to it through its Python and TypeScript libraries or through OpenTelemetry, which makes it a general tool for the development lifecycle of chatbots, retrieval systems and agents.

How LangSmith records traces

LangSmith captures the behavior of an application as a trace, a tree of nested runs that represents one end-to-end request. Each run stores the inputs, outputs, timing, token counts, cost estimates, errors and metadata of a single step, such as a model call, a retriever query or a tool invocation.

Instrumentation is usually light. Developers set environment variables such as LANGSMITH_TRACING and LANGSMITH_API_KEY, wrap a provider client, or add the @traceable decorator to their own functions. The SDK sends trace data asynchronously in the background, so tracing adds little latency to the user-facing request.

The core data objects and their purposes are listed in the following table.

ObjectWhat it containsTypical use
RunOne step with inputs, outputs and timingInspecting a single model or tool call
TraceA tree of runs for one requestFollowing an agent from prompt to answer
ThreadRelated traces from one conversationReviewing multi-turn chat sessions
ProjectA collection of traces from one applicationSeparating production from staging data
DatasetReference examples with inputs and expected outputsRepeatable offline evaluation
ExperimentResults of running an app version on a datasetComparing prompts, models or code changes
Core LangSmith data objects

Evaluation methods in LangSmith

LangSmith supports systematic evaluation so that teams can judge changes by measured quality instead of by a few manual tries. The standard workflow is to build a dataset, run the application against every example, and score each output with one or more evaluators.

Evaluators differ in cost, reliability and the kind of quality they can measure, as the next table shows.

Evaluator typeHow it scoresStrengthWeakness
Code-based checkDeterministic rule such as exact match or valid JSONFast, cheap and reproducibleCannot judge nuance or style
LLM-as-judgeA model grades output against criteriaScales to open-ended answersCan be biased or inconsistent
Human annotationReviewers score items in annotation queuesHighest trust for subtle casesSlow and costly
Pairwise comparisonTwo outputs are ranked against each otherSensitive to small improvementsNeeds two versions to compare
Types of evaluators and when they fit

Evaluation also runs in two modes. Offline evaluation tests a candidate version on a curated dataset before release, while online evaluation scores samples of live production traces automatically. A mature setup uses both and feeds interesting production traces back into datasets, so the test set grows with real user behavior.

LangSmith is one of several LLMOps platforms, and the choice between them often depends on licensing, hosting and existing infrastructure. Its strongest advantage is tight integration with LangChain and LangGraph, where traces of chains and agent graphs appear with little configuration.

ToolMain focusLicensing and hostingBest fit
LangSmithTracing, evaluation and monitoring for LLM appsCommercial cloud, self-hosting on enterprise plansTeams using LangChain or LangGraph
LangfuseOpen source LLM observabilityOpen source, self-hosted or cloudTeams needing full data control
MLflowGeneral machine learning lifecycleOpen sourceOrganizations with existing ML pipelines
Weights & BiasesExperiment tracking with LLM toolingCommercial cloud and on-premisesResearch and training-heavy teams
LangSmith and neighboring tools

LangSmith should also be distinguished from LangChain itself. LangChain is a library for building applications, whereas LangSmith is a service for observing and testing them; neither requires the other, and many production users adopt LangSmith while writing their orchestration code by hand.

Using LangSmith across the development lifecycle

LangSmith is most valuable when it accompanies an application from prototype to production. Beyond tracing and evaluation, it offers a prompt playground for testing prompt variants against models, a prompt registry with versioning, dashboards for latency, error rate and cost, and alerts on threshold breaches.

  • During prototyping: trace every run to understand how chains and agents actually behave.
  • Before release: run experiments on a dataset and block changes that lower key scores.
  • In production: monitor latency, cost and feedback, and sample traces for online evaluation.
  • After incidents: locate the failing step in the trace tree and add the case to a regression dataset.

Many teams connect experiments to continuous integration, so that a pull request triggers an evaluation run and reviewers see score changes next to code changes. This turns prompt and model updates into testable engineering changes rather than informal edits.

Limitations and pitfalls of LangSmith

LangSmith improves visibility but does not guarantee quality. Its scores are only as good as the datasets and evaluator criteria behind them, and a poorly designed LLM judge can reward fluent but wrong answers. Privacy obligations also apply, because traces may contain personal data or confidential documents.

PitfallConsequenceRemedy
Tracing sensitive inputs unmaskedPersonal data stored in a third-party serviceUse input and output masking or self-hosting
Tiny or unrepresentative datasetsScores do not predict production qualityGrow datasets from real traces
Unvalidated LLM judgesMisleading quality signalsCalibrate judges against human labels
Tracing every request at full volumeHigh storage cost and noiseApply sampling and retention policies
Common pitfalls and remedies

Teams should also budget for trace volume and retention, since pricing and storage scale with the number of traces. Setting a sampling rate for high-traffic endpoints keeps costs predictable without losing the signal needed for debugging.

In simple terms

LangSmith is like a flight recorder for AI applications. Every time a chatbot or AI agent answers someone, LangSmith writes down each step it took: what it asked the AI model, which documents it looked up and what came back.

When something goes wrong, developers can replay that record and see exactly where the mistake happened. They can also run the application on a fixed set of test questions and compare scores before and after a change, much like a teacher grading the same exam twice.

A practical example

An online retailer runs a customer support assistant that answers questions about orders and returns using a retrieval system over policy documents. After a prompt change, complaints about wrong return deadlines rise. The engineering team opens LangSmith and filters production traces with negative user feedback.

The traces show that the retriever often returns an archived policy page from the previous year. The team removes the stale page, adds 60 problematic questions to a dataset and runs an experiment comparing the old and new configurations with a correctness judge and an exact-date check.

The new configuration raises the correctness score on the dataset from about 78 percent to 93 percent, and the team adds the experiment to its release pipeline. Within two weeks, complaints about return deadlines drop back to their earlier level.