Time-series database

On this page 7

A time-series database (TSDB) is a database designed to store, compress and query sequences of timestamped measurements, such as server metrics, sensor readings, energy consumption or market prices. Among database models, it specializes in data where time is the primary axis: records arrive roughly in time order, are rarely updated, and are queried mostly as ranges and aggregates over time windows.

A time series is usually identified by a metric name plus a set of tags or labels, for example cpu_usage with host and region, and consists of a stream of points, each a timestamp and one or more values. A TSDB organizes storage, indexing and retention around that shape, which lets it ingest hundreds of thousands to millions of points per second on suitable hardware and keep years of history at a fraction of the size a general-purpose table would need.

How a time-series database stores data

TSDBs exploit the regularity of time-series data. Writes are mostly append-only in time order, so the engine can buffer recent points in memory, write them out in immutable time-partitioned blocks, and drop whole blocks when they expire instead of deleting individual rows.

Compression is the second pillar. Consecutive timestamps are stored with delta-of-delta encoding, which reduces regular intervals to a few bits, and floating-point values with XOR-based encoding that exploits small changes between neighbors. On typical monitoring data, such schemes reach around one to two bytes per point, compared with tens of bytes for a naive row layout; actual ratios depend strongly on how noisy the values are.

TechniqueWhat it doesBenefit
Time partitioningSplits data into chunks by time rangeFast range scans and cheap expiry
Columnar layoutStores each field separatelyBetter compression and aggregation speed
Delta and XOR encodingStores differences instead of raw valuesStorage reduced by an order of magnitude or more
Inverted index on labelsMaps tag values to seriesFast selection of matching series
Write buffer or head blockHolds recent points in memoryHigh ingestion throughput
Core storage techniques in time-series databases

Retention, downsampling and query patterns

Time-series data loses detail value as it ages. TSDBs therefore support retention policies that delete raw data after a set period and downsampling that keeps lower-resolution summaries for longer.

Data tierResolutionKept for
Raw pointsEvery 10-15 seconds7-30 days
First rollup1-minute or 5-minute averages, minimum and maximum3-12 months
Long-term rollup1-hour aggregatesSeveral years
A typical tiered retention plan

Queries in a TSDB are dominated by time-window aggregation: average per minute, rate of increase of a counter, percentiles over the last hour, or the last known value per device. Query languages such as PromQL, InfluxQL and SQL with time_bucket functions provide these operations directly, along with gap filling and interpolation for irregular data.

Cardinality as the main scaling limit

The number of distinct series, called cardinality, is the parameter that most often limits a time-series database. Each unique combination of metric name and label values creates a new series with its own index entries and memory overhead.

A label with unbounded values, such as a user identifier, request identifier or full URL, can turn a few thousand series into millions within hours, exhausting memory and slowing every query. Labels should describe dimensions with a bounded set of values, such as region, host or status code, while unbounded identifiers belong in logs or traces rather than in metric labels.

Time-series databases compared with other database models

A TSDB is one of several specialized database models, and time-stamped data can also live in general-purpose systems. The comparison shows where the specialization pays off.

ModelData shapeStrength for time-stamped data
Time-series databaseSeries of timestamped points with labelsIngestion, compression, window aggregation
Relational databaseNormalized tablesJoins and transactions; heavy at billions of points
Document databaseJSON-like documentsFlexible event payloads; weaker compression
Graph databaseNodes and edgesRelationships, not measurements over time
Vector databaseHigh-dimensional embeddingsSimilarity search, not time windows
Time-series database and neighboring database models

Implementations follow two main paths. Purpose-built engines such as Prometheus, InfluxDB and VictoriaMetrics use custom storage and query languages. Extensions of relational databases, such as TimescaleDB on PostgreSQL, add time partitioning and compression while keeping full SQL and joins with business tables. Column stores such as ClickHouse are also widely used for time-stamped analytics.

Choosing and operating a time-series database

The right choice depends on what the data is for and how it is queried. The decision table summarizes common situations.

SituationSuitable approachReason
Infrastructure and application metricsPull-based metrics TSDBNative alerting and label queries
Industrial sensors with business contextRelational database with time-series extensionSQL joins with assets and orders
Ad hoc analytics on billions of eventsColumnar analytical databaseFast scans with flexible queries
Small volume, a few seriesOrdinary relational table with an index on timeNo extra system to run
Choosing a storage approach for time-series data
  • Out-of-order and late data may be rejected or handled slowly by engines tuned for in-order writes.
  • Updates and deletes of single points are expensive or unsupported in many TSDBs.
  • Clock skew between sources corrupts ordering; synchronize clocks with NTP.
  • Counter resets after a restart must be handled with rate functions, not raw differences.

The idea in brief

A time-series database is a diary for measurements. Each entry says what was measured, when and what the value was. Examples are the temperature in a room every minute or the load on a server every ten seconds.

Because the entries always arrive in time order, the database can pack them very tightly and answer questions like what the average was each hour last week. It can also throw away old detail automatically and keep only the summaries.

A practical scenario

A wind farm operator monitors 200 turbines, each reporting about 50 measurements such as rotor speed, blade pitch, vibration and power output every second. That adds up to roughly 10,000 points per second and well over 800 million points per day, which overwhelmed the plain relational table first used for storage.

The operator moved the data to a PostgreSQL database with a time-series extension, partitioned into daily chunks and compressed after two days. Raw data is kept for 30 days, one-minute averages for two years. Storage per day fell by more than 90%, dashboards showing a month of vibration trends load in seconds, and maintenance engineers still join the measurements with the asset and service tables in ordinary SQL to spot turbines that need inspection.