Elasticsearch is a distributed search and analytics engine built on the Apache Lucene library that stores JSON documents and makes them searchable in near real time through an HTTP-based REST API. Among database systems, it occupies the niche of a document-oriented search store: it excels at full-text relevance ranking, filtering and aggregations over large volumes of semi-structured data, rather than at transactional record keeping.
Organizations use it for site and product search, log and metrics analysis, security event monitoring and, increasingly, vector similarity search. It is usually deployed beside a primary database rather than as a replacement for one, because it trades strict consistency and transactions for speed, scale and relevance.
How Elasticsearch indexes and finds documents
The core data structure in Elasticsearch is the inverted index, inherited from Lucene. Instead of mapping each document to its words, it maps every term to the list of documents that contain it, so a query for a word becomes a fast lookup rather than a scan of every record.
Before text reaches the index, an analyzer processes it. A typical analyzer splits text into tokens, lowercases them, removes stop words and may reduce words to their stems, so that a search for “running” can match “runs”. Structured fields such as numbers, dates and keywords are stored in separate structures, including doc values for sorting and aggregations and BKD trees for numeric and geographic ranges.
New documents are written to an in-memory buffer and a transaction log, then made searchable when a refresh creates a new Lucene segment, by default about once per second. This is why the engine is described as near real time: a document is durable immediately but becomes visible to searches only after the next refresh.
Relevance is calculated with the BM25 scoring model by default, which weighs how often a term appears in a document against how common it is across the index. The building blocks behind these mechanisms are summarized in the table below.
| Component | Role | Practical effect |
|---|---|---|
| Index | Logical collection of documents | Unit of querying and settings |
| Mapping | Schema defining field types and analyzers | Determines how fields are searched |
| Shard | Independent Lucene index holding part of the data | Enables horizontal scaling |
| Replica | Copy of a primary shard on another node | Provides failover and read capacity |
| Segment | Immutable file set inside a shard | Merged in the background over time |
Cluster architecture and scaling
An Elasticsearch deployment is a cluster of nodes that share data through shards. Each index is divided into a fixed number of primary shards at creation, and each primary can have any number of replicas. A query fans out to one copy of every relevant shard, and a coordinating node merges the partial results into the final response.
Nodes take on specialized node roles, which lets operators separate cluster management from heavy data work and place older data on cheaper hardware.
| Node role | Responsibility | Typical sizing concern |
|---|---|---|
| Master-eligible | Maintains cluster state and shard allocation | Stability; three nodes avoid split decisions |
| Data hot | Stores recent, frequently queried data | Fast storage and CPU |
| Data warm or cold | Stores older, rarely queried data | Capacity at low cost |
| Ingest | Runs pipelines that transform documents | CPU for parsing |
| Coordinating only | Routes requests and merges results | Memory for large aggregations |
Shard sizing is one of the most consequential design decisions. A widely used guideline keeps shards between roughly 10 and 50 GB, because very small shards waste heap memory on overhead while very large ones slow down recovery and rebalancing. Index lifecycle management automates rollover to new indices and the movement of aging data across tiers.
Elasticsearch compared with other database systems
Elasticsearch overlaps with several neighbors in the database systems landscape, yet its design goals differ. The comparison below shows where it fits relative to commonly paired technologies.
| System | Primary model | Strength | Weakness relative to Elasticsearch |
|---|---|---|---|
| Elasticsearch | Distributed document search | Full-text relevance and aggregations | Not applicable |
| PostgreSQL | Relational, ACID | Transactions and joins | Simpler full-text ranking at large scale |
| MongoDB | Document store | Flexible operational records | Less advanced relevance tuning |
| ClickHouse | Columnar analytics | Fast aggregations on structured data | Weaker free-text search |
| Redis | In-memory key-value | Sub-millisecond lookups | Limited querying and dataset size |
The key distinction is consistency and transactions. Elasticsearch offers per-document atomicity and optimistic concurrency control through sequence numbers, but it has no multi-document transactions and no relational joins in the classical sense. For that reason, most architectures treat a relational or document database as the system of record and synchronize data into Elasticsearch for search.
Typical use cases and query capabilities
Queries are written in the Query DSL, a JSON language that combines full-text clauses, exact filters and boolean logic. Filters do not affect scoring and can be cached, so separating filter context from query context is a standard performance practice. Aggregations compute counts, averages, percentiles and histograms over the matching documents in the same request.
- Application search: product catalogs with typo tolerance, synonyms, facets and autocomplete.
- Observability: centralized logs, metrics and traces, often collected through Beats or Logstash and visualized in Kibana.
- Security analytics: correlation of events for threat detection in a SIEM workflow.
- Semantic search: dense vector fields queried with approximate nearest neighbor algorithms such as HNSW, frequently combined with keyword scoring in hybrid search.
Besides the Query DSL, the engine also accepts an SQL interface and the piped query language ES|QL for analysts who prefer those styles. Its licensing changed in 2021, which led to the OpenSearch fork; since 2024, Elasticsearch has again been available under an open source license option alongside its other licenses.
Limitations and operational pitfalls
Elasticsearch rewards careful design and punishes neglect. Most production incidents trace back to a small set of recurring operational mistakes, listed below with their usual remedies.
| Symptom | Likely cause | Fix |
|---|---|---|
| Heap pressure and long garbage collection | Too many small shards | Consolidate indices and use rollover |
| Mapping explosion | Dynamic mapping of arbitrary keys | Define explicit mappings or flattened fields |
| Slow deep pagination | Large from and size values | Use search_after with a point in time |
| Data loss after node failure | Indices without replicas | Keep at least one replica |
| Inconsistent search results after writes | Refresh interval not yet elapsed | Accept delay or refresh selectively |
Changing the type of an existing field requires reindexing the data into a new index, so mappings deserve attention up front. Operators should also remember that snapshots, not replicas, are the backup mechanism: a replica faithfully copies an accidental deletion within moments.
Put simply
Think of the index at the back of a thick textbook. Instead of reading every page to find a word, a reader looks it up and jumps straight to the right pages. Elasticsearch builds that kind of index for huge collections of documents, then spreads the work across many computers so answers come back in a fraction of a second.
It is excellent at finding and counting things quickly, such as matching products or error messages. It is not designed to be the main ledger of a business, so companies usually keep their official records in another database and copy them into Elasticsearch for searching.
In practice
An online marketplace with about 12 million product listings stores orders and inventory in PostgreSQL. Its original search used SQL pattern matching, which returned results in two to four seconds, ignored spelling mistakes and ranked items arbitrarily.
The engineering team deployed a six-node Elasticsearch cluster with six primary shards and one replica each, and a change-data-capture pipeline that streams updates from PostgreSQL within a few seconds. The mapping defines product names with a language analyzer, brands as keyword fields for facets and a synonym list that links terms such as “sofa” and “couch”.
After launch, the median search response fell below 100 milliseconds, misspelled queries began returning useful results and category facets updated instantly. PostgreSQL remained the system of record, so a cluster outage would affect search but never order data.