Logs are timestamped records of discrete events that applications, operating systems and infrastructure components emit while they run, such as a request being served, an error being thrown or a user signing in. Each log entry captures what happened, when, where and in what context, usually as a line of text or a structured object.
In observability for cloud and distributed systems, logs are one of the three classic telemetry signals, alongside metrics and traces. Metrics show that something is wrong, traces show where in a request path it happens, and logs supply the detailed evidence of why. Their richness makes them indispensable for debugging and auditing, and also makes them the most expensive signal to store at scale.
How logs are produced and collected
A log pipeline has four stages: emission, collection, processing and storage. Code writes entries through a logging library; in containers the convention is to write to standard output, where the runtime captures the stream. An agent or collector on each node reads the stream, enriches entries with metadata such as host, pod and region, and ships them to a central backend for indexing and search.
The table shows the components of a typical cloud log pipeline.
| Stage | Typical tools | Main concern |
|---|---|---|
| Emission | Language logging libraries, syslog, journald | Format, level and context fields |
| Collection | Fluent Bit, Vector, OpenTelemetry Collector | Reliable delivery and buffering |
| Processing | Parsers, filters, redaction rules | Structure, sampling and privacy |
| Storage and query | Elasticsearch or OpenSearch, Loki, cloud log services | Retention, indexing cost, search speed |
OpenTelemetry defines a common log data model and lets collectors attach trace and span identifiers to log records, which links each entry to the distributed trace of the request that produced it.
Log levels and structured logging
Log levels classify entries by severity so that operators can filter noise and control volume. Most libraries follow a scale derived from syslog, and production systems commonly run at INFO, enabling DEBUG only temporarily.
| Level | Meaning | Typical production setting |
|---|---|---|
| TRACE and DEBUG | Detailed internal state for developers | Disabled or sampled |
| INFO | Normal significant events | Enabled |
| WARN | Unexpected but handled condition | Enabled, reviewed |
| ERROR | Failed operation needing attention | Enabled, often alerted |
| FATAL or CRITICAL | Process cannot continue | Enabled, always alerted |
Structured logging writes each entry as key-value data, typically JSON, instead of free text. A structured entry such as {"level":"error","order_id":"8812","latency_ms":2300} can be filtered and aggregated reliably, whereas free text needs fragile regular expressions. For cloud systems, structured logs with consistent field names are the single most valuable logging practice.
Logs compared with metrics and traces
Each telemetry signal answers a different question, and mature teams use all three together rather than forcing one signal to do everything.
| Aspect | Logs | Metrics | Traces |
|---|---|---|---|
| Unit | Individual event | Numeric time series | Request path made of spans |
| Question answered | What exactly happened | How much and how often | Where time was spent |
| Cardinality tolerance | High, any field | Low to moderate labels | High, per request |
| Storage cost | Highest per signal | Lowest | Moderate, usually sampled |
| Best for | Debugging, audit, forensics | Alerting, dashboards, SLOs | Latency analysis across services |
Logs can be turned into metrics by counting matching entries, but alerting on log queries is slower and costlier than alerting on native metrics, so frequent signals such as error rates are better exported as metrics directly.
Retention, cost and compliance
Log volume grows with traffic, and indexing and retention dominate cost. Teams therefore separate logs by value: hot storage for recent searchable data, cheaper object storage for archives, and sampling or dropping for repetitive low-value entries such as health checks.
Retention periods follow purpose. Operational debugging usually needs days to a few weeks, security and audit logs often need one year or longer, and regulated sectors may require multiple years. Logs frequently contain personal data, so privacy rules such as data minimization apply to them, and secrets must never be written in the first place.
Common logging mistakes
- Logging secrets and personal data: tokens, passwords and full card numbers end up in widely accessible systems.
- Unstructured messages: inconsistent free text makes queries slow and unreliable.
- No correlation identifier: without a request or trace ID, events from one transaction cannot be joined across services.
- DEBUG left on in production: volume and cost multiply while signal drowns in noise.
- Logging only on failure paths: missing context about what preceded an error prevents root-cause analysis.
- Clock skew: unsynchronized hosts produce misordered timelines; use NTP and UTC timestamps.
A plain-language summary
Logs are a diary that software keeps about itself. Every time something notable happens, such as a customer placing an order or a payment failing, the program writes a short dated note.
When something goes wrong, engineers read these notes to reconstruct exactly what happened and in what order, much like investigators reading a ship’s logbook after a storm. Good diaries are tidy, consistent and never record anyone’s passwords.
An example from practice
An online travel agency receives complaints that some bookings fail at the payment step, yet its dashboards show an error rate below one percent. An engineer filters the structured logs for level=error and service=checkout over the last six hours and groups the results by the payment_provider field.
The query shows that nearly all failures share one provider and one currency, with a timeout message after 30 seconds. The trace identifier in each entry leads to the corresponding traces, which confirm that a currency conversion call is waiting on an unreachable endpoint. The team routes that currency to a backup service within the hour, and adds a metric for provider timeouts so the problem triggers an alert next time instead of waiting for customer complaints.