The ELK stack is a combination of three open-source tools, Elasticsearch, Logstash and Kibana, used to collect, process, index, search and visualize logs and other machine data from many servers in one place. In monitoring & logs, it is one of the most widely deployed platforms for centralized log aggregation, troubleshooting, security analysis and operational dashboards.
The name is an acronym of its original components. After lightweight shippers called Beats were added, the vendor renamed the family the Elastic Stack, but ELK remains the common name for the architecture. Teams use it to replace the habit of logging into individual servers and reading files with grep by a single searchable store covering every host, container and application.
How the ELK stack processes log data
Log data flows through the stack in a pipeline of four stages: shipping, processing, indexing and visualization. A shipper on each server reads log files or the systemd journal and forwards new lines. Logstash or an ingest pipeline then parses raw text into structured fields, Elasticsearch indexes those fields for fast search, and Kibana presents them in queries and dashboards.
| Component | Role | Key characteristics |
|---|---|---|
| Beats and Elastic Agent | Collect and ship data from hosts | Lightweight, Filebeat for logs, Metricbeat for metrics |
| Logstash | Parse, enrich and route events | Input, filter and output plugins, runs on the JVM |
| Elasticsearch | Store, index and search documents | Distributed, built on Apache Lucene, JSON over HTTP |
| Kibana | Query, visualize and manage | Discover view, dashboards, alerting, KQL queries |
Logstash pipelines are defined in three blocks. Inputs receive data from Beats, syslog, Kafka or files. Filters such as grok, dissect, date and geoip turn an access-log line into fields like client IP, status code and response time. Outputs send the result to Elasticsearch or other destinations. For simple parsing, Elasticsearch ingest pipelines can replace Logstash entirely.
How Elasticsearch stores and searches logs
Elasticsearch stores each log event as a JSON document in an index. Every text field is analyzed into an inverted index, which maps terms to the documents containing them, so a search across billions of lines returns in milliseconds to seconds. Numeric and keyword fields support aggregations such as counts per status code or percentiles of latency.
Indexes are split into primary shards distributed across nodes, and each primary can have replica shards for redundancy and read throughput. Logs normally go into time-based indexes or data streams, for example one backing index per day, so old data can be deleted or moved as a whole. The main sizing parameters are summarized below.
| Parameter | Common guidance | Why it matters |
|---|---|---|
| JVM heap | About half of RAM, below about 31 GB | Keeps compressed object pointers efficient |
| Shard size | Roughly 10 to 50 GB | Too small wastes memory, too large slows recovery |
| Replicas | 1 for most log indexes | Survives loss of one node |
| Master-eligible nodes | 3 dedicated in larger clusters | Stable cluster coordination |
| Retention | Days to months by data tier | Drives total storage cost |
Index lifecycle management automates retention by moving indexes through hot, warm, cold and frozen tiers: recent data on fast SSD nodes, older data on cheaper storage, and finally deletion. Without such a policy, log clusters tend to grow until disks reach their watermarks and indexing stops.
ELK compared with other logging platforms
Several platforms solve the same problem with different trade-offs in indexing strategy, licensing and operating cost. The comparison below places ELK among its common alternatives.
| Platform | Indexing approach | Licensing and model | Typical strength |
|---|---|---|---|
| ELK stack | Full-text index on all fields | Source-available and AGPL options, self-managed or hosted | Powerful search and analytics |
| OpenSearch | Full-text index, forked from Elasticsearch | Apache 2.0 | Open license, similar features |
| Splunk | Indexes events, schema at search time | Commercial, priced by volume | Mature enterprise and security features |
| Grafana Loki | Indexes labels only, not log text | AGPL | Low storage cost, pairs with Prometheus |
| Graylog | Uses an OpenSearch or Elasticsearch backend | Open core | Simpler log-focused interface |
The licensing history matters for planning. In 2021 Elasticsearch and Kibana moved from the Apache 2.0 license to source-available licenses, which led to the community fork OpenSearch; an AGPL option was later added back. Organizations offering the software as a service should check the license terms carefully.
Designing a reliable ELK deployment
A production deployment needs more than three installed packages. The decisions below have the largest effect on reliability and cost.
- Use structured logging where possible, so applications emit JSON and parsing becomes trivial and robust.
- Add a buffer such as Kafka or a persistent Logstash queue between shippers and Elasticsearch to absorb spikes and outages.
- Define explicit mappings and templates so a field is never indexed as text in one index and as a number in another.
- Secure the cluster with TLS, authentication and role-based access, because logs often contain personal data and secrets.
- Monitor the monitoring system, including indexing latency, rejected requests, heap usage and disk watermarks.
Teams that do not want to operate the cluster themselves often use a managed Elasticsearch or OpenSearch service, which removes node management but not the need to design indexes, retention and parsing.
Common pitfalls with the ELK stack
Most ELK problems come from uncontrolled growth and missing data hygiene rather than from software bugs. The table pairs typical symptoms with their causes.
| Symptom | Likely cause | Remedy |
|---|---|---|
| Cluster turns red or read-only | Disk flood-stage watermark reached | Lifecycle policies, more capacity |
| High heap use and slow queries | Too many small shards | Fewer, larger shards; rollover by size |
| Mapping conflicts and dropped fields | Dynamic mapping of inconsistent data | Index templates, field naming standard |
| Unparsed messages in Kibana | Grok pattern does not match new format | Structured logs, pipeline tests |
| Lost events during outages | No buffer between shippers and cluster | Persistent queues or Kafka |
Costs also grow with retention and verbosity. Logging at debug level in production or storing every field as full text can multiply storage needs several times, so log levels and field mappings deserve the same review as application code.
The simple explanation
Every server keeps a diary of what it does. With dozens of servers, the diaries are scattered and hard to read together. The ELK stack collects all the diaries into one library. It sorts every entry by date, server and type, and adds a search box and charts on top.
So when something breaks, an engineer does not open fifty files. One question is enough, such as all errors from the checkout service in the last hour. The answer arrives in seconds. The key benefit is seeing the whole system in one place.
How it is used in practice
An e-commerce company runs 40 web servers, a payment service and a Kubernetes cluster. Filebeat on every node ships Nginx access logs and application logs in JSON to a three-node Elasticsearch cluster, with data streams rolling over daily and an index lifecycle policy keeping 14 days on SSD and 90 days on cheaper disks.
One evening, customers report failed checkouts. An engineer filters Kibana to HTTP 502 responses in the last 30 minutes and sees that all of them come from three servers that received a new configuration. A dashboard of response times confirms the pattern, the change is rolled back, and the incident is resolved in about 15 minutes instead of hours of searching through individual log files.