Traces are the observability signal that records the end-to-end path of a single request or transaction through a system as a tree of timed operations called spans. Each trace shows which services and components took part, in what order, how long each step took and where an error occurred.
Traces form one of the three classic telemetry signals alongside metrics and logs. Metrics answer how much and how often, logs record discrete events, and traces explain why one particular request was slow or failed by connecting work done across processes, hosts and cloud services into one causal record.
How a trace is structured
A trace is identified by a trace ID, a 16-byte value shared by every span belonging to the same request. Each span represents one unit of work, such as an incoming HTTP request, a database query or a message publish, and has its own 8-byte span ID plus the ID of its parent span. The parent references turn a flat list of spans into a tree, with the root span representing the whole operation.
| Field | Content | Purpose |
|---|---|---|
| Trace ID | Identifier shared by the whole request | Groups spans into one trace |
| Span ID and parent ID | Own identifier and caller identifier | Builds the call tree |
| Name and kind | Operation name; server, client, producer, consumer or internal | Describes the role of the step |
| Start time and duration | Timestamps with microsecond or nanosecond precision | Shows latency and overlap |
| Attributes | Key-value pairs such as HTTP route or database system | Enables filtering and grouping |
| Events | Timestamped notes inside the span | Records exceptions or milestones |
| Status | Unset, ok or error | Marks failed operations |
Spans may also carry links to spans in other traces. Links describe relationships that are not strict parent-child calls, for example a batch job that processes messages produced by many separate requests.
Tracing tools usually display a trace as a waterfall view, with one horizontal bar per span placed on a shared timeline. Reading it means finding the critical path: the chain of spans that determines total duration. Work running in parallel off that path does not slow the request, so optimizing it rarely helps, while a single long span on the critical path often explains most of the latency. Gaps between a parent span and its children are equally informative, because they usually reveal uninstrumented work, queuing or time spent waiting for a lock.
Context propagation between services
A trace spans multiple services only if each service passes the trace context to the next one. The dominant format is the W3C Trace Context standard, which sends a traceparent header containing the version, trace ID, parent span ID and sampling flags, plus an optional tracestate header for vendor-specific data.
Instrumentation libraries, most commonly from OpenTelemetry, inject this context into outgoing HTTP requests, gRPC calls and message headers, and extract it on the receiving side. A single service that drops the header, such as a proxy or a queue consumer without instrumentation, splits one request into several disconnected traces.
Traces compared with metrics and logs
Each telemetry signal answers a different question, and mature observability practice correlates them rather than choosing one. The table summarizes their differences.
| Signal | Unit of data | Best question answered | Cost driver |
|---|---|---|---|
| Traces | Tree of spans per request | Where did this request spend time or fail | Number of spans stored |
| Metrics | Aggregated numeric time series | Is the system healthy right now | Number of label combinations |
| Logs | Individual event records | What exactly happened at this moment | Volume of text ingested |
Correlation works by sharing identifiers. When logs include the trace ID and span ID, an engineer can jump from a slow trace to the exact log lines it produced. Metrics can carry exemplars, sample trace IDs attached to histogram buckets, which connect a latency spike to concrete requests.
Sampling and storage choices
Recording every span of every request is expensive at scale, so tracing systems apply sampling. The choice of strategy decides which traces survive and therefore which problems remain visible.
| Strategy | Decision point | Advantage | Drawback |
|---|---|---|---|
| Head-based sampling | At the root span, before work starts | Simple and cheap | May discard rare errors |
| Tail-based sampling | After the whole trace completes | Keeps errors and slow requests | Requires buffering in a collector |
| Rate-limited sampling | Fixed traces per second | Predictable cost | Low-traffic paths may vanish |
| Full capture | Every request kept | Complete visibility | High storage and processing cost |
Head-based rates of 1-10% are common for high-traffic services, while tail-based policies often keep all traces with errors and all traces above a latency threshold, plus a small random share for baseline comparison.
Common pitfalls with traces
Traces deliver value only when instrumentation is consistent across the whole request path. The most frequent problems are the following:
- Broken context propagation through queues, proxies or custom thread pools.
- High-cardinality span names, such as names containing user IDs, which make aggregation impossible.
- Sensitive data in attributes, for example full SQL with personal values or authorization headers.
- Clock skew between hosts, which makes child spans appear to start before their parents.
- Over-instrumentation that creates thousands of tiny spans and inflates cost without insight.
Following semantic conventions for attribute names, such as standard keys for HTTP methods and database systems, keeps traces comparable across languages and teams and allows backends to build service maps automatically.
In simple terms
A trace is a detailed travel log for one request. When a customer clicks a button, the request may pass through a website, several internal services and a database. The trace records each stop, the order of the stops and how long each one took.
It works like tracking a parcel. Instead of only knowing that delivery was late, the team sees exactly which depot held it up, which makes the cause of a delay or error much easier to find.
A practical example
An online store notices that checkout latency at the 99th percentile has risen from about 800 ms to over 4 seconds, while average latency looks normal. Metrics confirm the problem but do not explain it. Engineers open traces kept by a tail-based sampling rule for checkout requests slower than two seconds.
The slow traces share one pattern: a call from the pricing service to a discount database produces dozens of sequential query spans, one per cart item, instead of a single batched query. The issue appears only for large carts. After the team rewrites the lookup as one query, the traces show a single short span and checkout latency returns to normal.