Stream processing is a computing model that processes unbounded sequences of events continuously as they arrive, updating results within milliseconds to seconds instead of waiting for a complete dataset. A stream processor reads events from a log or message broker, applies filters, transformations, joins and aggregations, keeps intermediate state, and emits results downstream without a defined end.
In streaming data architectures, stream processing is the computation layer that sits on top of event transport systems such as Kafka, Apache Pulsar or Amazon Kinesis. Engines such as Apache Flink, Kafka Streams and Spark Structured Streaming implement it. Its defining challenges are time, state and correctness under failure, because events can arrive late, out of order or more than once.
How stream processing works
A streaming job is a directed graph of operators: sources read from partitions of an event stream, intermediate operators transform and aggregate, and sinks write to databases, topics or alerting systems. Each operator runs as many parallel instances, and events are routed between them by key so that all events for one customer or device reach the same instance.
Operators are either stateless, such as filtering or parsing one event at a time, or stateful, such as counting events per user, joining two streams or detecting a sequence of actions. Stateful operators keep their state in memory or in an embedded store, and the engine periodically writes checkpoints of that state together with the input offsets, so the job can resume exactly where it stopped after a crash.
The table below lists the main operator types and the state each one needs.
| Operator | Purpose | State required |
|---|---|---|
| Map and filter | Transform or drop individual events | None |
| Keyed aggregation | Running counts, sums or averages per key | One value per key |
| Windowed aggregation | Totals per time window | One value per key and open window |
| Stream-table join | Enrich events with reference data | Copy of the table |
| Stream-stream join | Match related events from two streams | Buffered events within a time bound |
| Pattern detection | Recognize sequences of events | Partial matches per key |
Event time, watermarks and windowing
Stream processing distinguishes event time, when something actually happened, from processing time, when the system sees it. A mobile app that was offline may deliver events minutes or hours late, so results based on processing time can assign them to the wrong period. Correct analytics almost always require event time.
To decide when a time period is complete, engines use watermarks: a moving assertion that no more events older than a given timestamp are expected. A window closes and emits its result when the watermark passes its end. The allowed lateness setting keeps windows open a little longer, and events arriving after that are dropped or sent to a side output.
| Window type | Definition | Typical use |
|---|---|---|
| Tumbling window | Fixed size, non-overlapping, for example 1 minute | Per-minute counts and metrics |
| Sliding (hopping) window | Fixed size advancing by a smaller step | Moving averages, rolling 5-minute rates |
| Session window | Closes after a gap of inactivity, for example 30 minutes | User sessions and visits |
| Global window with triggers | One window, results emitted on custom conditions | Running totals, count-based output |
Watermark configuration is a direct trade-off between latency and completeness. A watermark that lags event time by 5 seconds produces fast but potentially incomplete results; one that lags by 10 minutes captures more late data but delays every window by that amount.
Delivery guarantees and fault tolerance
Because machines fail and networks retry, every streaming system must define what happens to an event during a failure. These processing guarantees describe the effect of each event on the final results, not how many times it physically travels through the network.
| Guarantee | Behavior on failure | Cost and requirements |
|---|---|---|
| At-most-once | Events may be lost, never duplicated | Cheapest, no replay needed |
| At-least-once delivery | No loss, but duplicates possible | Replayable source, idempotent or tolerant sinks |
| Exactly-once semantics | Each event affects state and output once | Checkpoints plus transactional or idempotent sinks |
Exactly-once semantics inside the engine are achieved by consistent checkpoints and source replay. End-to-end exactly-once additionally requires a sink that supports transactions or idempotent writes; otherwise duplicates reappear at the output after a recovery. Many production systems choose at-least-once with idempotent upserts as a simpler alternative.
Another core mechanism is backpressure. When a downstream operator or sink cannot keep up, the engine slows reading from the source instead of buffering without limit, and the growing consumer lag on the broker becomes the key health metric.
When to choose stream processing
Stream processing is justified when the value of a result decays quickly with time. When a daily or hourly result is sufficient, batch processing remains simpler and cheaper to operate.
| Requirement | Recommended approach | Reason |
|---|---|---|
| Block a fraudulent payment before approval | Stream processing | Decision needed within hundreds of milliseconds |
| Alert on equipment sensor anomalies | Stream processing | Delay increases damage |
| Live operational dashboard | Stream or micro-batch | Seconds-level freshness is enough |
| Monthly financial reporting | Batch processing | Completeness and reconciliation matter most |
| Training a model on historical data | Batch processing | Bounded data, no latency need |
Common pitfalls in stream processing
- Using processing time for business metrics, which shifts late events into the wrong period.
- Unbounded state growth from keys that never expire, eventually exhausting memory; state time-to-live settings prevent it.
- Hot keys that send a large share of events to one parallel instance and create a bottleneck.
- Non-idempotent sinks that turn replay after a failure into duplicated records.
- Schema changes without compatibility rules, which break consumers mid-stream.
- Underestimating operations: upgrades, rescaling and state migration require more care than rerunning a batch job.
Explained without jargon
Stream processing means handling data the moment it arrives. Think of a supermarket cashier who scans each item as it moves along the belt. The total is always up to date. Nobody waits for the end of the day.
Computers use this to react quickly, for example to stop a suspicious card payment within a second. The hard part is keeping the answer correct when some information arrives late or twice.
A typical use case
A payment processor handles several thousand card transactions per second. Each authorization request is published to an event stream, and a Flink job keyed by card number maintains state such as the number of transactions and the distinct countries seen in a 10-minute sliding window.
When a card shows purchases in two distant countries within minutes, or more than a set number of attempts in a short window, the job emits a risk signal that the authorization service reads before approving the payment. Checkpoints every few seconds ensure that a node failure does not reset the counters. The result is that suspicious transactions are flagged within about a second, instead of being found the next morning by a batch report.