Elasticsearch

On this page 7

Elasticsearch is a distributed search and analytics engine built on the Apache Lucene library that stores JSON documents and makes them searchable in near real time through an HTTP-based REST API. Among database systems, it occupies the niche of a document-oriented search store: it excels at full-text relevance ranking, filtering and aggregations over large volumes of semi-structured data, rather than at transactional record keeping.

Organizations use it for site and product search, log and metrics analysis, security event monitoring and, increasingly, vector similarity search. It is usually deployed beside a primary database rather than as a replacement for one, because it trades strict consistency and transactions for speed, scale and relevance.

How Elasticsearch indexes and finds documents

The core data structure in Elasticsearch is the inverted index, inherited from Lucene. Instead of mapping each document to its words, it maps every term to the list of documents that contain it, so a query for a word becomes a fast lookup rather than a scan of every record.

Before text reaches the index, an analyzer processes it. A typical analyzer splits text into tokens, lowercases them, removes stop words and may reduce words to their stems, so that a search for “running” can match “runs”. Structured fields such as numbers, dates and keywords are stored in separate structures, including doc values for sorting and aggregations and BKD trees for numeric and geographic ranges.

New documents are written to an in-memory buffer and a transaction log, then made searchable when a refresh creates a new Lucene segment, by default about once per second. This is why the engine is described as near real time: a document is durable immediately but becomes visible to searches only after the next refresh.

Relevance is calculated with the BM25 scoring model by default, which weighs how often a term appears in a document against how common it is across the index. The building blocks behind these mechanisms are summarized in the table below.

ComponentRolePractical effect
IndexLogical collection of documentsUnit of querying and settings
MappingSchema defining field types and analyzersDetermines how fields are searched
ShardIndependent Lucene index holding part of the dataEnables horizontal scaling
ReplicaCopy of a primary shard on another nodeProvides failover and read capacity
SegmentImmutable file set inside a shardMerged in the background over time
Core building blocks of Elasticsearch

Cluster architecture and scaling

An Elasticsearch deployment is a cluster of nodes that share data through shards. Each index is divided into a fixed number of primary shards at creation, and each primary can have any number of replicas. A query fans out to one copy of every relevant shard, and a coordinating node merges the partial results into the final response.

Nodes take on specialized node roles, which lets operators separate cluster management from heavy data work and place older data on cheaper hardware.

Node roleResponsibilityTypical sizing concern
Master-eligibleMaintains cluster state and shard allocationStability; three nodes avoid split decisions
Data hotStores recent, frequently queried dataFast storage and CPU
Data warm or coldStores older, rarely queried dataCapacity at low cost
IngestRuns pipelines that transform documentsCPU for parsing
Coordinating onlyRoutes requests and merges resultsMemory for large aggregations
Common node roles in a cluster

Shard sizing is one of the most consequential design decisions. A widely used guideline keeps shards between roughly 10 and 50 GB, because very small shards waste heap memory on overhead while very large ones slow down recovery and rebalancing. Index lifecycle management automates rollover to new indices and the movement of aging data across tiers.

Elasticsearch compared with other database systems

Elasticsearch overlaps with several neighbors in the database systems landscape, yet its design goals differ. The comparison below shows where it fits relative to commonly paired technologies.

SystemPrimary modelStrengthWeakness relative to Elasticsearch
ElasticsearchDistributed document searchFull-text relevance and aggregationsNot applicable
PostgreSQLRelational, ACIDTransactions and joinsSimpler full-text ranking at large scale
MongoDBDocument storeFlexible operational recordsLess advanced relevance tuning
ClickHouseColumnar analyticsFast aggregations on structured dataWeaker free-text search
RedisIn-memory key-valueSub-millisecond lookupsLimited querying and dataset size
Elasticsearch and related systems

The key distinction is consistency and transactions. Elasticsearch offers per-document atomicity and optimistic concurrency control through sequence numbers, but it has no multi-document transactions and no relational joins in the classical sense. For that reason, most architectures treat a relational or document database as the system of record and synchronize data into Elasticsearch for search.

Typical use cases and query capabilities

Queries are written in the Query DSL, a JSON language that combines full-text clauses, exact filters and boolean logic. Filters do not affect scoring and can be cached, so separating filter context from query context is a standard performance practice. Aggregations compute counts, averages, percentiles and histograms over the matching documents in the same request.

  • Application search: product catalogs with typo tolerance, synonyms, facets and autocomplete.
  • Observability: centralized logs, metrics and traces, often collected through Beats or Logstash and visualized in Kibana.
  • Security analytics: correlation of events for threat detection in a SIEM workflow.
  • Semantic search: dense vector fields queried with approximate nearest neighbor algorithms such as HNSW, frequently combined with keyword scoring in hybrid search.

Besides the Query DSL, the engine also accepts an SQL interface and the piped query language ES|QL for analysts who prefer those styles. Its licensing changed in 2021, which led to the OpenSearch fork; since 2024, Elasticsearch has again been available under an open source license option alongside its other licenses.

Limitations and operational pitfalls

Elasticsearch rewards careful design and punishes neglect. Most production incidents trace back to a small set of recurring operational mistakes, listed below with their usual remedies.

SymptomLikely causeFix
Heap pressure and long garbage collectionToo many small shardsConsolidate indices and use rollover
Mapping explosionDynamic mapping of arbitrary keysDefine explicit mappings or flattened fields
Slow deep paginationLarge from and size valuesUse search_after with a point in time
Data loss after node failureIndices without replicasKeep at least one replica
Inconsistent search results after writesRefresh interval not yet elapsedAccept delay or refresh selectively
Common problems and their fixes

Changing the type of an existing field requires reindexing the data into a new index, so mappings deserve attention up front. Operators should also remember that snapshots, not replicas, are the backup mechanism: a replica faithfully copies an accidental deletion within moments.

Put simply

Think of the index at the back of a thick textbook. Instead of reading every page to find a word, a reader looks it up and jumps straight to the right pages. Elasticsearch builds that kind of index for huge collections of documents, then spreads the work across many computers so answers come back in a fraction of a second.

It is excellent at finding and counting things quickly, such as matching products or error messages. It is not designed to be the main ledger of a business, so companies usually keep their official records in another database and copy them into Elasticsearch for searching.

In practice

An online marketplace with about 12 million product listings stores orders and inventory in PostgreSQL. Its original search used SQL pattern matching, which returned results in two to four seconds, ignored spelling mistakes and ranked items arbitrarily.

The engineering team deployed a six-node Elasticsearch cluster with six primary shards and one replica each, and a change-data-capture pipeline that streams updates from PostgreSQL within a few seconds. The mapping defines product names with a language analyzer, brands as keyword fields for facets and a synonym list that links terms such as “sofa” and “couch”.

After launch, the median search response fell below 100 milliseconds, misspelled queries began returning useful results and category facets updated instantly. PostgreSQL remained the system of record, so a cluster outage would affect search but never order data.