Distributed tracing is the observability technique that follows a single request across every service it touches and records each unit of work as a timed span, linked into one trace by a shared identifier. In observability for cloud-native systems, it answers the question that metrics and logs answer poorly: where exactly a slow or failed request spent its time.
A monolith can be profiled in one process, but a request in a microservice architecture may pass through a gateway, a dozen services, queues and databases. Distributed tracing reconstructs that path as a causal timeline, so engineers can see the critical path, the failing hop and the dependencies between components.
How distributed tracing works
Each traced request receives a trace ID, typically a 128-bit random value, when it enters the system. Every operation along the way creates a span with its own span ID, a parent span ID, a start time, a duration, a status and descriptive attributes. The parent links form a tree that observability tools render as a waterfall chart.
The identifiers must cross process boundaries, which is called context propagation. Services pass them in request headers, most commonly the W3C Trace Context traceparent header, or in message metadata for queues and event streams. A single service that drops the header breaks the trace into disconnected fragments.
Spans are produced by instrumentation, either automatic agents and libraries that wrap frameworks and clients, or manual code around business operations. They are exported, usually in batches, to a collector that processes them and forwards them to a tracing backend for storage and query.
| Element | What it contains | Purpose |
|---|---|---|
| Trace | All spans sharing one trace ID | Represents one end-to-end request |
| Span | Name, timing, status, attributes, events | Measures one operation |
| Span context | Trace ID, span ID, sampling flags | Travels between services |
| Span link | Reference to a span in another trace | Connects batch and fan-in work |
| Baggage | Key-value pairs propagated downstream | Carries request-scoped context |
Sampling strategies and their trade-offs
Recording every span of every request is expensive at scale, so tracing systems use sampling. The central trade-off is between cost and completeness: low sampling rates save storage and network traffic but may miss the rare request that matters.
| Strategy | Decision point | Strength | Weakness |
|---|---|---|---|
| Head-based sampling | At trace start, often 1-10% | Simple and cheap | Misses rare errors and outliers |
| Tail-based sampling | After the trace completes | Keeps errors and slow traces | Collector must buffer whole traces |
| Rate-limited sampling | Fixed traces per second | Predictable cost | Uneven coverage across services |
| Full capture | Every request | Complete data | Costly beyond low traffic |
Tail-based sampling is popular for production systems because rules such as “keep all traces with errors or latency above 2 seconds” preserve the interesting cases. Its cost is a stateful collector tier that must hold spans for several seconds until each trace is complete.
Distributed tracing compared with metrics and logs
Distributed tracing is one of the three main telemetry signals, alongside metrics and logs, and each answers a different question. Mature setups correlate them, for example by writing the trace ID into every log line and attaching example trace IDs to metric data points.
| Signal | Answers the question | Typical cost profile |
|---|---|---|
| Traces | Where did this request spend time or fail | High per request, controlled by sampling |
| Metrics | Is the system healthy and how much is happening | Low, aggregated numbers |
| Logs | What exactly happened inside one component | Medium to high, grows with verbosity |
OpenTelemetry has become the common vendor-neutral standard for producing all three signals, with shared APIs, SDKs and the OTLP protocol. Using it decouples instrumentation from the tracing backend, so a backend can be replaced without re-instrumenting applications.
Implementing tracing in practice
A practical rollout starts with automatic instrumentation of entry points, HTTP and gRPC clients, database drivers and message clients, which yields useful traces within days. Manual spans are then added around business operations, such as payment authorization, where automatic tools see only generic calls.
| Symptom | Likely cause | Fix |
|---|---|---|
| Traces end at a queue | Context not propagated in message headers | Inject and extract context in producers and consumers |
| Many root spans per request | A proxy strips trace headers | Allow trace headers through gateways |
| Spans with negative gaps | Clock skew between hosts | Synchronize clocks, rely on parent timing |
| Exploding storage costs | High-cardinality attributes or full capture | Limit attributes, apply tail sampling |
| Sensitive data in spans | Raw request bodies recorded | Redact attributes in the collector |
Semantic conventions, the standardized attribute names for HTTP, databases and messaging, keep traces queryable across teams. Consistent service names and environment attributes matter as much as the spans themselves.
Limitations and common misconceptions
Distributed tracing shows where time is spent, not why. A long database span reveals a slow query, but the query plan, lock contention or resource saturation must still be investigated with database metrics, logs or profiling.
- Tracing is not free: instrumentation adds small overhead, and export and storage add real cost.
- Partial instrumentation produces misleading gaps that look like slow network time.
- Sampled data is not suitable for exact counting; use metrics for rates and totals.
- Asynchronous workflows need span links rather than strict parent-child trees.
- Traces can carry personal data if attributes are not reviewed.
Put simply
Picture a parcel that passes through several sorting centers before it reaches a home. A tracking number lets anyone see each stop and how long the parcel waited there.
Distributed tracing gives every request in a software system a tracking number. Each service it visits adds a timed entry. When something is slow or broken, engineers can see exactly which stop caused the delay instead of guessing.
In practice
An online retailer notices that checkout latency at the 99th percentile has risen from 800 milliseconds to 3 seconds, while average latency looks normal. Metrics show the problem but not its location, so the team queries traces that tail-based sampling kept because they exceeded 2 seconds.
The waterfall view shows that the checkout service calls the inventory service, which in turn calls a pricing service once per item in the cart. For carts with more than 30 items, these sequential calls add up to over 2 seconds. The team replaces them with one batched pricing request, and the 99th percentile latency drops back below one second within a day of the fix.