A watermark in stream processing is a monotonically advancing timestamp that an engine carries alongside the data to declare that it expects no more events with an event time earlier than that value. In streaming data systems, the watermark is the mechanism that lets a pipeline decide when a time window is complete enough to emit a result, even though events arrive out of order.
The concept is also called an event-time watermark. It has nothing to do with the visible or invisible marks embedded in images and documents; here it is purely a progress signal for event time inside engines such as Apache Flink, Apache Beam runners and Spark Structured Streaming.
Why stream processing needs watermarks
Watermarks exist because event time and processing time diverge. A sensor reading taken at 10:00:00 may reach the cluster at 10:00:02, or at 10:07 if a mobile device was offline. An engine that groups events by when they happened, rather than by when they arrived, needs a rule for deciding that a window such as 10:00 to 10:05 will receive no further data.
Without such a rule, the system faces an impossible choice: wait forever and never produce output, or close windows by wall-clock time and silently produce incorrect aggregates. The watermark is the compromise. When the watermark passes the end of a window, the engine fires the window and treats its result as final, or at least as final as the configured lateness policy allows.
The table below separates the time notions that a watermark connects.
| Concept | What it measures | Relation to the watermark |
|---|---|---|
| Event time | When the event occurred at the source | The watermark is expressed in event time |
| Processing time | When the operator handles the event | Drives how fast the watermark advances |
| Ingestion time | When the event entered the platform | A simpler substitute with weaker guarantees |
| Watermark | Estimated completeness of event time | Triggers windows and timers |
How a watermark is generated and propagated
A watermark is usually generated at or near the source. The most common strategy is bounded out-of-orderness: the engine tracks the highest event time seen so far and subtracts a fixed tolerance. With a tolerance of 10 seconds and a maximum observed timestamp of 12:00:30, the watermark becomes 12:00:20, meaning anything older is considered late.
Downstream, each operator receives watermarks from all of its input channels and forwards the minimum across inputs. This rule guarantees correctness: an operator joining two partitions cannot claim progress beyond its slowest input. The consequence is that one idle or lagging partition can hold back the watermark for the entire job.
Engines offer several generation strategies, summarized in the following table.
| Strategy | How it works | Trade-off |
|---|---|---|
| Bounded out-of-orderness | Maximum event time minus a fixed delay | Simple, but the delay is a guess |
| Ascending timestamps | Watermark equals latest event time | Zero latency, any disorder becomes late data |
| Source-provided | Broker or connector supplies progress | Accurate when the source knows completeness |
| Heuristic or adaptive | Delay tuned from observed lateness | Better fit, harder to reason about |
| Idle-source timeout | Excludes silent partitions after a timeout | Unblocks progress, risks dropping late data |
Watermarks, windowing and late data
Watermarks matter most in windowing. Tumbling, sliding and session windows are all defined in event time, and the watermark decides when each window’s state can be evaluated and garbage-collected. Without watermarks, a streaming aggregation would have to keep state for every window indefinitely, causing unbounded memory growth.
Events that arrive after the watermark has passed their window are late events. Engines handle them through explicit policies, and choosing among them is a product decision as much as a technical one.
| Policy | Behavior | Suitable when |
|---|---|---|
| Drop | Late events are discarded | Approximate metrics are acceptable |
| Allowed lateness | Window state is kept and results are updated | Corrections downstream are possible |
| Side output | Late events go to a separate stream | Late data needs auditing or reprocessing |
| Larger watermark delay | Fewer events are considered late | Completeness outweighs latency |
Allowed lateness means a window can fire more than once: an early result when the watermark passes, then updated results as stragglers arrive. Sinks must then accept upserts or retractions, which ties watermark design closely to exactly-once semantics and the capabilities of the target system.
Choosing the watermark delay
The watermark delay embodies the central trade-off between latency and completeness. A short delay produces results quickly but classifies more events as late; a long delay captures nearly all events but postpones every result and keeps more state in memory.
Practitioners set the delay from measured lateness distributions, not intuition. Recording the difference between event time and arrival time for a few days, then choosing a value near the 99th or 99.9th percentile, is a sound starting point. The table shows typical ranges by source type; actual values depend heavily on network conditions and client behavior.
| Source type | Typical disorder | Common delay range |
|---|---|---|
| Server logs in one data center | Small and predictable | 1 to 10 seconds |
| IoT sensors on cellular links | Bursty, network-dependent | 30 seconds to several minutes |
| Mobile apps with offline mode | Hours in edge cases | Minutes plus a late-data path |
| Change data capture | Mostly ordered per table | Seconds |
Common misconceptions and failure modes
Watermarks are a heuristic, not a guarantee, except where the source can prove completeness. Several recurring problems follow from misunderstanding this.
- Stalled watermark: an idle partition with no events freezes progress and no windows ever fire.
- Clock skew at the source produces future timestamps that push the watermark ahead and make valid data late.
- Confusing watermark with processing time leads to windows that close by wall clock and miss events.
- Delay set too small silently drops data, so late-event counters should always be monitored.
- Delay set too large inflates state size and checkpoint duration.
Robust pipelines expose watermark lag, the gap between current processing time and the watermark, as a core metric. A steadily growing lag signals backpressure, a stuck source or a misconfigured timestamp extractor.
In simple terms
Think of a teacher collecting exam papers that students mail in. Letters arrive out of order and some are delayed. At some point the teacher must say: all papers dated before Friday have probably arrived, so it is time to grade that batch.
A watermark is that statement for a computer. It lets the system stop waiting and produce an answer, while accepting that a few very late letters may need special handling. Waiting longer gives more complete results; waiting less gives faster ones.
A practical example
A ride-hailing platform computes the number of trip requests per city in one-minute tumbling windows to adjust surge pricing. Requests come from phones on mobile networks, and an analysis of a week of traffic shows that 99.5 percent of events arrive within 8 seconds of their event time, while a small tail arrives minutes later.
The engineers configure a bounded out-of-orderness watermark of 10 seconds, an idle-source timeout of 30 seconds for quiet partitions, and a side output for late events. Each window fires roughly 10 seconds after it ends, which is fast enough for pricing decisions. Late requests flow into a nightly batch job that corrects the historical reports, so the real-time view stays responsive and the archived figures remain accurate.