Real-time analytics

On this page 7

Real-time analytics is the practice of collecting, processing and querying data within seconds or milliseconds of its creation, so that the results describe the current state of a business or system rather than yesterday’s snapshot. In analytics within the Data discipline, it sits at the fast end of a spectrum that runs from overnight batch reporting through near-real-time refreshes to continuous stream processing.

The defining property is low end-to-end latency: the time between an event occurring and that event influencing a chart, an alert or an automated action. Fraud scoring, logistics tracking, live operational dashboards and dynamic pricing all depend on it, and each tolerates a different delay.

How real-time analytics pipelines process events

A real-time analytics pipeline moves each event through ingestion, stream processing, storage and serving without waiting for a scheduled batch. Applications, sensors or databases emit events, a message broker such as Apache Kafka or Amazon Kinesis buffers them durably, and a stream processor such as Apache Flink or Spark Structured Streaming filters, enriches and aggregates them as they arrive.

Aggregations over unbounded streams require windowing, which groups events into finite slices. Tumbling windows are fixed and non-overlapping, sliding windows overlap to produce smoother metrics, and session windows close after a gap of inactivity. Change data capture (CDC) tools extend the same model to operational databases by turning every row change into an event.

The table below lists the main pipeline stages and the latency each typically contributes.

StageTypical componentsLatency contribution
Event captureSDKs, sensors, CDC connectorsMilliseconds
Message transportKafka, Kinesis, Pulsar5-100 ms
Stream processingFlink, Spark Streaming, ksqlDBMilliseconds to seconds
Analytical storageApache Druid, ClickHouse, Apache PinotSub-second ingestion
Serving and visualizationDashboards, APIs, alerting rulesRefresh interval dependent
Stages of a typical real-time analytics pipeline

End-to-end latency is the sum of every stage, so a dashboard that refreshes only every 60 seconds cannot deliver sub-second insight even if the stream processor is fast.

Latency tiers and freshness requirements

The phrase “real time” covers several distinct latency tiers, and naming the tier precisely prevents costly overengineering. Hard real time in the control-systems sense, where a missed deadline is a failure, rarely applies to analytics; most business cases fall into soft real time or near real time.

TierTypical freshnessRepresentative use
Streaming decisioningUnder 100 msCard fraud scoring at authorization
Operational real time1-10 secondsLive order and fleet monitoring
Near real time1-15 minutesMarketing campaign pacing
Intraday batchHourlyInventory replenishment reports
Daily batch24 hoursFinancial close and executive reporting
Common latency tiers in analytics

Two further measures matter alongside latency. Data freshness describes how old the newest record in a query result is, while query latency describes how quickly the system answers once asked. A system can answer in 50 ms yet show data that is ten minutes stale, which is why service-level objectives should state both freshness and response time.

Real-time analytics differs from batch analytics mainly in when computation happens: continuously as data arrives rather than periodically over accumulated data. Classic business intelligence and OLAP cubes were built for batch loads, and descriptive analytics has traditionally summarized completed periods.

ApproachComputation timingMain strengthMain weakness
Real-time analyticsContinuous, per eventImmediate reactionHigher operational cost
Batch analyticsScheduled jobsSimple, complete, cheapStale results
Micro-batch processingSmall intervals of secondsEasier fault toleranceAdded fixed delay
Event trackingAt collection timeRaw behavioral detailNo analysis by itself
Real-time analytics versus neighboring approaches

Architects often combine approaches. The Lambda architecture runs a batch layer for accuracy and a speed layer for freshness, which forces teams to keep two implementations of each metric in step. The Kappa architecture keeps a single streaming path and rebuilds historical results by rereading the retained event log from the beginning, which simplifies logic but demands a long-retention, replayable event store.

Designing and choosing a real-time analytics solution

The right design starts from the decision the data must support and the cost of delay for that decision, not from a preferred technology. If no person or system will act within minutes, a near-real-time refresh is usually sufficient and far cheaper to operate.

SituationRecommended approachReason
Automated action per eventStream processing with in-memory stateDecision must precede the next event
Interactive drill-down on fresh dataReal-time OLAP storeSub-second queries on streaming ingest
Hourly business reviewScheduled batch or micro-batchFreshness beyond needs adds cost
Alerting on thresholdsStreaming rules engineDetects breaches as they occur
Decision guide for real-time analytics designs

Correctness controls deserve as much attention as speed. Event time versus processing time must be distinguished, because mobile devices and network delays deliver events out of order. Watermarks tell the processor how long to wait for late events, and exactly-once semantics prevent duplicated counts after failures. Clear KPI definitions, shared with the batch layer, keep live and historical figures consistent.

Limitations, risks and common mistakes

Real-time analytics introduces operational complexity and cost that batch systems avoid, and many failures come from underestimating that burden. Continuous pipelines need monitoring of consumer lag, schema evolution and state size around the clock.

SymptomLikely causeRemedy
Live totals differ from daily reportsLate events dropped or definitions differAlign logic and tune watermarks
Counts jump after a restartAt-least-once delivery duplicatesIdempotent writes or exactly-once sinks
Dashboards lag during peaksGrowing consumer lagAdd partitions and processing capacity
Alerts fire constantlyThresholds ignore normal varianceUse baselines and anomaly detection
Frequent problems in real-time analytics
  • Fresh is not the same as accurate: early figures are often provisional and revised as late data arrives.
  • Watching numbers move without a defined response creates distraction rather than value.
  • Short windows amplify noise, so statistical significance is harder to reach than in daily aggregates.

The simple explanation

Most reports look backward: they tell you what happened yesterday or last month. Real-time analytics tells you what is happening now, usually within a few seconds, by analyzing information the moment it is created.

Think of the difference between reading a newspaper and watching a live traffic map. The newspaper is useful but describes the past. The live map lets you change your route before you hit the jam. The trade-off is that keeping the live map running costs more effort, so it pays off only when acting quickly actually matters.

How it is used in practice

A regional food delivery company processes about 4,000 orders per hour at dinner peak. Every order, courier location ping and restaurant status change flows into a message broker, and a stream processor computes estimated delivery times per district in 30-second tumbling windows. The results land in a real-time OLAP store that powers the operations dashboard.

When average preparation time at restaurants in one district rises above 25 minutes, a rule triggers an alert and the dispatch system automatically widens the courier search radius. Over one quarter, the operations team reduced orders delivered more than 15 minutes late by roughly a fifth, while finance continued to use the nightly batch warehouse for revenue reporting, reconciled against the live figures each morning.