An event stream is an unbounded, time-ordered sequence of immutable event records, each describing something that happened, such as a payment, a click, a sensor reading or a status change. In streaming data, it is the fundamental data structure: producers append events continuously, and consumers read and process them as they arrive rather than waiting for a finished dataset.
Because events are facts that are never updated in place, a stream can be replayed, consumed by many independent applications and used to rebuild state at any point in time. Platforms such as Apache Kafka store event streams durably as partitioned, append-only logs.
How an event stream is structured
Each event is a small record with a key, a value, a timestamp and optional headers. The key identifies the entity the event concerns, such as an account or device, and determines which partition stores it. Within one partition, events keep a strict order and receive a sequential offset.
A stream is split into partitions so that it can scale horizontally across servers and consumers. Ordering is guaranteed per partition, not across the whole stream, which is why events that must stay in order, such as all updates to one order, share the same key.
| Element | Role | Example |
|---|---|---|
| Key | Groups related events and selects the partition | Customer ID 48213 |
| Value | Payload describing what happened | Order placed, total 59.90 |
| Timestamp | When the event occurred or was recorded | 2026-11-09T17:40:48Z |
| Offset | Position within the partition | 1,204,377 |
| Headers | Metadata such as schema version or trace ID | schema-version 3 |
Event streams compared with related concepts
Event streams are often discussed together with message queues and batch files. The differences concern retention, replay and the number of consumers.
| Concept | Retention | Consumers | Typical purpose |
|---|---|---|---|
| Event stream | Retained for a period or indefinitely | Many independent readers | Shared history of facts |
| Message queue | Deleted after acknowledgment | One worker per message | Distributing tasks |
| Message broker | Depends on the product | Routes to subscribers | Decoupling producers and consumers |
| Batch file | Stored as a finished dataset | Read after completion | Periodic bulk processing |
| Change data capture feed | Retained like a stream | Many readers | Database changes as events |
Because consumers track their own offsets, a new application can start from the beginning of the retained history and catch up, while existing consumers continue unaffected. This replayability is the decisive advantage of a log-based stream over a classic queue.
Time, ordering and delivery guarantees
Events carry two notions of time. Event time is when the event actually happened, while processing time is when a system handles it. Network delays and offline devices make events arrive late and out of order, so correct aggregations over time windows must use event time and watermarks that estimate how complete the data is.
Delivery semantics determine how failures affect results. Most platforms offer at least at-least-once delivery, which can produce duplicates, and some support exactly-once processing within their own ecosystem through idempotent producers and transactions.
| Guarantee | Behavior on failure | Consequence for consumers |
|---|---|---|
| At-most-once | Events may be lost | Acceptable only for noncritical metrics |
| At-least-once | Events may be repeated | Consumers must deduplicate or be idempotent |
| Exactly-once | Each event affects results once | Higher overhead, limited to supported systems |
Designing and operating event streams
Good stream design starts with well-defined event schemas, managed in a schema registry with compatibility rules so that producers can evolve events without breaking consumers. Events should describe business facts in the past tense, such as order placed, rather than commands.
| Parameter | Typical range | Trade-off |
|---|---|---|
| Partition count | A few to several hundred per stream | More parallelism, more overhead |
| Retention period | Hours to indefinite | Replay capability versus storage cost |
| Replication factor | Usually 3 | Durability versus storage and network |
| End-to-end latency | Milliseconds to seconds | Batching improves throughput but adds delay |
| Consumer lag | Ideally near zero | Growing lag signals insufficient capacity |
When consumers cannot keep up, backpressure mechanisms slow producers or buffer data, and monitoring consumer lag shows the problem early. Choosing the partition key carefully prevents hot partitions, where one very active key overloads a single consumer.
Common pitfalls
- Assuming global ordering across partitions, which leads to race conditions in consumers.
- Ignoring duplicates under at-least-once delivery, which inflates counts and payments.
- Windowing by processing time, which assigns late events to the wrong period.
- Putting large payloads in events, instead of storing big objects elsewhere and referencing them.
- Changing event schemas without compatibility checks, which breaks downstream consumers.
The idea in brief
An event stream is a running list of everything that happens, written down the moment it happens. Each entry says what occurred, to whom and when, and entries are only ever added at the end, never changed.
It resembles a ship’s logbook. Anyone can read it from the start to understand the whole journey, or follow the newest lines to react to events as they occur. Many different readers can use the same logbook at the same time without getting in each other’s way.
A practical scenario
A card payment processor publishes every authorization attempt to an event stream keyed by card identifier, with 24 partitions and seven days of retention. Peaks reach several thousand events per second.
Three independent consumers read the same stream. A fraud detection service scores each event within about 200 milliseconds and blocks suspicious cards, a ledger service updates balances idempotently using the event identifier, and an analytics job aggregates approvals by merchant in one-minute event-time windows.
When a bug in the fraud model is fixed, the team replays the previous three days of events through the corrected model to identify missed fraud, without affecting the ledger or analytics consumers that continue reading in real time.