Metrics are numeric measurements of a system recorded over time, such as request rate, error count, latency or memory use, stored as time series identified by a name and a set of labels. In observability they form one of the three core telemetry signals alongside logs and traces, and they are the signal best suited to dashboards, alerting and trend analysis.
A metric deliberately discards individual event detail in exchange for compactness. Instead of recording every request, a service records how many requests occurred and how long they took in aggregate, which makes metrics cheap to store, fast to query and ideal for answering the question of whether something is wrong right now in cloud and distributed systems.
How metrics are structured and collected
Each metric value is a sample: a timestamp and a number belonging to one time series. A series is uniquely identified by the metric name plus its label set, so http_requests_total{method="GET", status="500"} and the same name with status="200" are two separate series.
Collection follows one of two models. In the pull model, popularized by Prometheus, a server scrapes an HTTP endpoint on each target at a fixed interval, commonly 15 to 60 seconds. In the push model, used by OpenTelemetry exporters and many agents, the application or a collector sends data to a backend. Pull makes target health visible automatically, while push suits short-lived jobs and serverless functions that may vanish before a scrape.
Metric types and what each measures
Choosing the correct metric type determines which questions the data can answer later. The four instrument types below are common to Prometheus and OpenTelemetry, although naming differs slightly.
| Type | Behavior | Typical example | Usual query |
|---|---|---|---|
| Counter | Only increases, resets on restart | Total requests served | Rate per second |
| Gauge | Goes up and down | Memory in use, queue depth | Current value or average |
| Histogram | Counts observations in buckets | Request latency distribution | Percentiles across instances |
| Summary | Computes quantiles on the client | Latency percentiles of one process | Read precomputed quantile |
Counters are almost never read raw. A rate converts them into a useful figure: if a counter rises from 12,000 to 15,000 over a 300-second window, the service handled 3,000 requests in 300 seconds, or 10 requests per second. Rate functions also detect counter resets after restarts.
Histograms are preferred over summaries in distributed systems because bucket counts can be summed across instances before a percentile is estimated. Percentiles from summaries cannot be averaged meaningfully: the mean of ten per-pod p99 values is not the p99 of the service.
Metrics compared with logs and traces
Metrics, logs and traces answer different questions and work best together. Metrics reveal that something changed, logs explain what happened in one component, and traces show where time was spent across services.
| Property | Metrics | Logs | Traces |
|---|---|---|---|
| Unit of data | Numeric sample in a series | Timestamped event record | Tree of spans for one request |
| Detail level | Aggregated | Individual event | Individual request path |
| Storage cost | Low and predictable | High, grows with traffic | High, usually sampled |
| Best for | Alerting and trends | Root-cause detail | Latency across services |
| Main risk | High label cardinality | Volume and noise | Sampling gaps |
Modern systems link the signals with exemplars, which attach a trace identifier to a specific metric sample, allowing an engineer to jump from a latency spike on a dashboard directly to a representative trace.
Cardinality and the cost of metrics
Cardinality is the number of unique time series a metric produces, and it is the main driver of metrics cost and performance. It equals the product of the distinct values of every label.
| Label added | Distinct values | Resulting series |
|---|---|---|
| method | 5 | 5 |
| status code | 10 | 50 |
| endpoint | 40 | 2,000 |
| pod | 50 | 100,000 |
| user ID (anti-pattern) | 1,000,000 | 100 billion potential series |
At a 15-second scrape interval, each series produces 5,760 samples per day. The 100,000 series above therefore generate 576 million samples daily; with typical time-series compression of roughly 1 to 2 bytes per sample, that is about 0.6 to 1.2 GB per day before replication. Unbounded labels such as user IDs, request IDs or full URLs belong in logs or traces, never in metric labels.
Choosing which metrics to collect
Effective metric sets follow established methods rather than collecting everything available. Each method targets a different class of component.
| Method | Applies to | Signals measured |
|---|---|---|
| RED method | Request-driven services | Rate, errors, duration |
| USE method | Resources such as CPU, disks, pools | Utilization, saturation, errors |
| Four golden signals | User-facing systems | Latency, traffic, errors, saturation |
| SLI-based selection | Services with reliability targets | Ratios of good events to valid events |
Metrics that feed service level indicators deserve the most care, because they drive the error budget and therefore release and on-call decisions. They should measure user experience as closely as possible, for example the share of requests completed in under 300 ms, rather than host-level CPU.
Common mistakes with metrics
- Alerting on causes instead of symptoms: high CPU may be harmless, while rising error rates always affect users.
- Averaging latency: means hide the slow tail; use percentiles from histograms instead.
- Poor histogram buckets: buckets that miss the real latency range make percentile estimates meaningless.
- Inconsistent naming and units: mixing milliseconds and seconds or vague names breaks queries; include the base unit in the name.
- Retaining full resolution forever: downsampling older data to 5-minute or 1-hour resolution cuts storage while preserving trends.
A plain-language summary
Metrics are like the gauges on a car dashboard. The speedometer, fuel gauge and temperature needle do not tell you every detail of the engine, but a quick glance shows whether everything is normal.
Software systems use metrics the same way. They count things such as visitors, errors and waiting time, and they record those numbers every few seconds. When a number moves outside its usual range, an alert warns the team before most customers notice a problem.
An example from practice
An online booking service runs 30 instances of its checkout API in a Kubernetes cluster. The team defines a service level objective that 99.5% of checkout requests succeed within 500 ms, measured over 30 days, and records request counts and a latency histogram with labels for endpoint and status code only.
One evening the error-ratio metric climbs from 0.1% to 4% within ten minutes, and a burn-rate alert pages the on-call engineer. The dashboard shows that only requests to the payment endpoint are failing. An exemplar on the latency histogram links to a trace that points to timeouts in a third-party payment gateway.
The engineer enables a fallback payment provider, and the error ratio returns to normal within 15 minutes. The metrics detected the incident in minutes and narrowed it to one endpoint before anyone opened a log file.