Watermark · stream processing

Also known as: event-time watermark

On this page 7

A watermark in stream processing is a monotonically advancing timestamp that an engine carries alongside the data to declare that it expects no more events with an event time earlier than that value. In streaming data systems, the watermark is the mechanism that lets a pipeline decide when a time window is complete enough to emit a result, even though events arrive out of order.

The concept is also called an event-time watermark. It has nothing to do with the visible or invisible marks embedded in images and documents; here it is purely a progress signal for event time inside engines such as Apache Flink, Apache Beam runners and Spark Structured Streaming.

Why stream processing needs watermarks

Watermarks exist because event time and processing time diverge. A sensor reading taken at 10:00:00 may reach the cluster at 10:00:02, or at 10:07 if a mobile device was offline. An engine that groups events by when they happened, rather than by when they arrived, needs a rule for deciding that a window such as 10:00 to 10:05 will receive no further data.

Without such a rule, the system faces an impossible choice: wait forever and never produce output, or close windows by wall-clock time and silently produce incorrect aggregates. The watermark is the compromise. When the watermark passes the end of a window, the engine fires the window and treats its result as final, or at least as final as the configured lateness policy allows.

The table below separates the time notions that a watermark connects.

ConceptWhat it measuresRelation to the watermark
Event timeWhen the event occurred at the sourceThe watermark is expressed in event time
Processing timeWhen the operator handles the eventDrives how fast the watermark advances
Ingestion timeWhen the event entered the platformA simpler substitute with weaker guarantees
WatermarkEstimated completeness of event timeTriggers windows and timers
Time concepts in stream processing

How a watermark is generated and propagated

A watermark is usually generated at or near the source. The most common strategy is bounded out-of-orderness: the engine tracks the highest event time seen so far and subtracts a fixed tolerance. With a tolerance of 10 seconds and a maximum observed timestamp of 12:00:30, the watermark becomes 12:00:20, meaning anything older is considered late.

Downstream, each operator receives watermarks from all of its input channels and forwards the minimum across inputs. This rule guarantees correctness: an operator joining two partitions cannot claim progress beyond its slowest input. The consequence is that one idle or lagging partition can hold back the watermark for the entire job.

Engines offer several generation strategies, summarized in the following table.

StrategyHow it worksTrade-off
Bounded out-of-ordernessMaximum event time minus a fixed delaySimple, but the delay is a guess
Ascending timestampsWatermark equals latest event timeZero latency, any disorder becomes late data
Source-providedBroker or connector supplies progressAccurate when the source knows completeness
Heuristic or adaptiveDelay tuned from observed latenessBetter fit, harder to reason about
Idle-source timeoutExcludes silent partitions after a timeoutUnblocks progress, risks dropping late data
Common watermark generation strategies

Watermarks, windowing and late data

Watermarks matter most in windowing. Tumbling, sliding and session windows are all defined in event time, and the watermark decides when each window’s state can be evaluated and garbage-collected. Without watermarks, a streaming aggregation would have to keep state for every window indefinitely, causing unbounded memory growth.

Events that arrive after the watermark has passed their window are late events. Engines handle them through explicit policies, and choosing among them is a product decision as much as a technical one.

PolicyBehaviorSuitable when
DropLate events are discardedApproximate metrics are acceptable
Allowed latenessWindow state is kept and results are updatedCorrections downstream are possible
Side outputLate events go to a separate streamLate data needs auditing or reprocessing
Larger watermark delayFewer events are considered lateCompleteness outweighs latency
Options for handling late events

Allowed lateness means a window can fire more than once: an early result when the watermark passes, then updated results as stragglers arrive. Sinks must then accept upserts or retractions, which ties watermark design closely to exactly-once semantics and the capabilities of the target system.

Choosing the watermark delay

The watermark delay embodies the central trade-off between latency and completeness. A short delay produces results quickly but classifies more events as late; a long delay captures nearly all events but postpones every result and keeps more state in memory.

Practitioners set the delay from measured lateness distributions, not intuition. Recording the difference between event time and arrival time for a few days, then choosing a value near the 99th or 99.9th percentile, is a sound starting point. The table shows typical ranges by source type; actual values depend heavily on network conditions and client behavior.

Source typeTypical disorderCommon delay range
Server logs in one data centerSmall and predictable1 to 10 seconds
IoT sensors on cellular linksBursty, network-dependent30 seconds to several minutes
Mobile apps with offline modeHours in edge casesMinutes plus a late-data path
Change data captureMostly ordered per tableSeconds
Typical watermark delays by data source

Common misconceptions and failure modes

Watermarks are a heuristic, not a guarantee, except where the source can prove completeness. Several recurring problems follow from misunderstanding this.

  • Stalled watermark: an idle partition with no events freezes progress and no windows ever fire.
  • Clock skew at the source produces future timestamps that push the watermark ahead and make valid data late.
  • Confusing watermark with processing time leads to windows that close by wall clock and miss events.
  • Delay set too small silently drops data, so late-event counters should always be monitored.
  • Delay set too large inflates state size and checkpoint duration.

Robust pipelines expose watermark lag, the gap between current processing time and the watermark, as a core metric. A steadily growing lag signals backpressure, a stuck source or a misconfigured timestamp extractor.

In simple terms

Think of a teacher collecting exam papers that students mail in. Letters arrive out of order and some are delayed. At some point the teacher must say: all papers dated before Friday have probably arrived, so it is time to grade that batch.

A watermark is that statement for a computer. It lets the system stop waiting and produce an answer, while accepting that a few very late letters may need special handling. Waiting longer gives more complete results; waiting less gives faster ones.

A practical example

A ride-hailing platform computes the number of trip requests per city in one-minute tumbling windows to adjust surge pricing. Requests come from phones on mobile networks, and an analysis of a week of traffic shows that 99.5 percent of events arrive within 8 seconds of their event time, while a small tail arrives minutes later.

The engineers configure a bounded out-of-orderness watermark of 10 seconds, an idle-source timeout of 30 seconds for quiet partitions, and a side output for late events. Each window fires roughly 10 seconds after it ends, which is fast enough for pricing decisions. Late requests flow into a nightly batch job that corrects the historical reports, so the real-time view stays responsive and the archived figures remain accurate.