Windowing · stream processing

Also known as: tumbling window · sliding window

On this page 8

Windowing in stream processing is the technique of dividing an unbounded stream of events into finite groups, called windows, so that operations such as counts, sums, averages and joins can produce results. Because streaming data never ends, an aggregate like “orders per minute” only makes sense once the stream is sliced into bounded intervals, and the window definition decides which events belong to each result.

Windows are usually defined by time or by event count, and the most common shapes are the tumbling window and the sliding window, with session windows used for activity-based grouping. Engines such as Apache Flink, Kafka Streams and Apache Beam implement windowing together with event time, watermarks and triggers, which together determine when a window is complete and how late data is handled.

How windowing turns a stream into finite results

A windowed operator assigns every incoming event to one or more windows, keeps per-window state, and emits a result when the window is considered complete. The assignment uses a timestamp or a position in the stream, and the state is usually an incremental aggregate, such as a running sum, rather than the full list of events.

Each window is typically identified by its key and its time bounds. A stream of payments keyed by merchant, for example, produces a separate five-minute window for every merchant. Windows are commonly half-open intervals, so an event stamped exactly 10:05:00 belongs to the window starting at 10:05, not to the one ending there.

The lifecycle of a window has a few distinct steps, summarized in the following table.

StepWhat the engine doesKey concept
AssignmentMaps each event to its window or windowsWindow assigner
AccumulationUpdates per-window state as events arriveIncremental aggregation
TriggeringDecides when to emit a resultTrigger and watermark
EmissionSends the aggregate downstreamOutput mode
CleanupDiscards state after allowed latenessState retention
Lifecycle of a window in a stream processor

Types of windows

The window type defines how intervals are placed along the timeline and whether they overlap. Choosing the wrong type changes the meaning of the result, not only its performance.

Window typeShapeOverlapTypical use
Tumbling windowFixed size, back to backNone; each event in one windowPer-minute counts, hourly billing totals
Sliding windowFixed size, advancing by a smaller slideYes; events in several windowsMoving averages, rolling error rates
Session windowVariable size, closed by an inactivity gapNone per keyUser visits, device activity bursts
Count windowFixed number of eventsOptionalBatches of N readings from a sensor
Global windowOne window for the whole streamNot applicableCustom triggers on unbounded state
Main window types in stream processing

A sliding window of 10 minutes with a 1-minute slide places each event in ten windows, so it multiplies state and output by roughly the ratio of size to slide. Some engines call this a hopping window and reserve “sliding” for windows that move with every event; terminology differs between products, so the exact definition should be checked in each engine’s documentation.

Event time, processing time and watermarks

Windowing is only meaningful once the time domain is chosen. Event time is the timestamp at which something happened at the source, while processing time is the clock of the machine handling the event. Network delays, retries and offline devices make the two diverge by seconds or even hours.

Event-time windows give correct and reproducible results but must cope with out-of-order data. A watermark is the engine’s assertion that no events older than a given timestamp are expected; when the watermark passes the end of a window, the window can fire. Events that arrive after that point are late data, handled by an allowed-lateness period, a side output or simply dropped.

The table below compares the two time domains.

PropertyEvent-time windowsProcessing-time windows
Timestamp sourceEmbedded in each eventClock of the operator
Result on replayDeterministicChanges with load and timing
Out-of-order eventsPlaced in the correct windowCounted in whatever window is open
LatencyWaits for the watermarkLowest possible
ComplexityNeeds watermarks and lateness rulesSimple to configure
Event-time and processing-time windows

Triggers and output modes

A trigger decides when a window emits results, which lets designers trade latency against completeness. The default trigger fires once when the watermark passes the window end, but practical pipelines often add early and late firings.

  • Early firings emit speculative partial results, for example every 30 seconds, before the window closes.
  • On-time firing emits the result considered complete when the watermark crosses the window end.
  • Late firings emit corrected results when late events arrive within the allowed lateness.
  • Accumulating mode sends the full updated value each time, while discarding mode sends only the change since the last firing.

Downstream systems must understand which mode is in use. A database sink that adds every emission to a total will double count results from an accumulating trigger, whereas an upsert keyed by window handles both modes correctly.

Choosing window size and type

The right window follows from the business question and the tolerance for delay, not from engine defaults. The decision table below covers common situations.

RequirementRecommended windowReason
Non-overlapping totals for reportsTumbling, event timeEach event counted exactly once
Smooth trend for alertingSliding with a short slideReacts quickly without jagged jumps
Measure a user visitSession with a 30-minute gapFollows real activity, not the clock
Lowest latency dashboardTumbling, processing timeNo waiting for watermarks
Mobile devices that sync offlineEvent time with long allowed latenessLate events still reach the right window
Selecting a windowing strategy

Window size directly drives state size and memory use. A one-hour sliding window with a one-second slide keeps 3,600 overlapping windows per key, which can exhaust memory; engines mitigate this with pane-based aggregation, sharing partial results between overlapping windows.

Common pitfalls in windowed pipelines

Many streaming defects are windowing defects in disguise. Typical problems include the following:

  • Watermarks that advance too fast drop legitimate late events and undercount totals.
  • Watermarks that stall, often because one idle partition receives no events, keep every window open and let state grow without limit.
  • Mixing time zones in daily windows produces totals that disagree with business reports; daily windows should be aligned explicitly to the intended zone.
  • Ignoring retractions from late firings leaves dashboards with stale or duplicated values.

A non-technical view

Think of a river of events that never stops flowing, such as clicks on a website. You cannot ask how many clicks there are in total, because the number keeps growing forever. Instead, you place buckets along the river and ask how many clicks fell into each bucket.

Windowing is the rule for placing those buckets. You can use one bucket per minute, buckets that overlap to show a moving trend, or a bucket per visitor that closes when the visitor goes quiet. The hard part is knowing when a bucket is full, because some events arrive late, like letters delayed in the post.

A concrete example

A payment processor monitors card transactions to detect fraud bursts. The streaming team computes, for every card, the number and total value of transactions in a sliding window of 10 minutes advancing every 30 seconds, using event time taken from the payment terminal.

Terminals on poor mobile connections sometimes deliver events up to two minutes late, so the pipeline uses a watermark that trails the newest timestamp by 90 seconds and allows a further five minutes of lateness. Early firings every 30 seconds feed a risk model, and corrected late results update the same record by window key.

When a stolen card is used at five shops within six minutes, the window count crosses the threshold of four transactions and the card is blocked within about 40 seconds of the fifth purchase. Because results are computed in event time, a replay of the same day for auditing produces exactly the same alerts.