Exactly-once semantics is the processing guarantee that every event affects the final state and outputs exactly one time, even when brokers, producers or processing tasks fail and work is retried. In streaming data systems it is the strongest of the three classic delivery guarantees, above at-most-once and at-least-once, and it is often called exactly-once delivery, although that name is slightly misleading.
The guarantee does not mean that a message physically crosses the network only once. Networks lose acknowledgments, so retries and duplicates are unavoidable. Exactly-once semantics is achieved by combining at-least-once transmission with deduplication or atomic commits, so that repeated attempts leave no visible trace in state, aggregates or downstream records.
How exactly-once semantics is achieved
Exactly-once semantics rests on a simple idea: the system may repeat work, but the effects of repeated work must be invisible. Engineers build this from three mechanisms that usually appear together: idempotent writes, transactional commits and consistent checkpoints of state and input position.
An idempotent producer attaches a producer identifier and a sequence number to each record, so the broker can discard a retried record it has already stored. A transactional producer goes further and writes records to several partitions together with the consumer offsets in one atomic transaction; consumers configured to read only committed data never see aborted writes. A stream processor then takes periodic checkpoints that capture operator state and source offsets at a consistent cut, and on failure it rewinds both to the last checkpoint.
The table below summarizes the building blocks and the failure each one neutralizes.
| Mechanism | What it does | Failure it neutralizes |
|---|---|---|
| Idempotent producer | Deduplicates retried records by sequence number | Lost acknowledgment from the broker |
| Transactional writes | Commits output records and offsets atomically | Crash between writing output and recording progress |
| Read-committed isolation | Hides records from aborted transactions | Consumers reading partial results |
| Distributed checkpoints | Snapshots state and source positions consistently | Task crash that loses in-memory state |
| Two-phase commit sink | Pre-commits output, finalizes after checkpoint | External system receiving uncommitted data |
| Idempotent sink keys | Upserts by a deterministic key | Replayed output after recovery |
Delivery guarantees compared
Exactly-once semantics is best understood against its two weaker neighbors. At-most-once never retries, so data can be lost; at-least-once delivery always retries, so data can be duplicated. Exactly-once sits on top of at-least-once and removes the duplicates.
| Guarantee | Loss possible | Duplicates possible | Typical cost |
|---|---|---|---|
| At-most-once | Yes | No | Lowest latency and overhead |
| At-least-once | No | Yes | Retries and replay after failure |
| Exactly-once | No | No visible duplicates | Coordination, commit latency, more state |
A frequent source of confusion is the difference between exactly-once processing and exactly-once delivery to an external system. A stream processor such as Apache Flink can guarantee that its internal state reflects each event once, yet a sink that sends emails or calls a payment API can still repeat an action after recovery. End-to-end exactly-once requires every hop, including the source and the sink, to participate in the protocol.
Ordering and time are separate concerns. Exactly-once semantics says nothing about whether events arrive in event time order or how a watermark advances; a pipeline can be exactly-once and still produce wrong window results if late data is handled poorly.
Exactly-once semantics in common platforms
Major streaming platforms implement exactly-once semantics differently, and the scope of the guarantee varies by platform. Knowing that scope is essential before relying on it.
| System | Core mechanism | Scope of the guarantee |
|---|---|---|
| Kafka | Idempotent and transactional producers, read-committed consumers | Read-process-write cycles within Kafka |
| Kafka Streams | Processing guarantee set to exactly-once with transactions | State stores and output topics in the same cluster |
| Apache Flink | Aligned or unaligned checkpoints plus two-phase commit sinks | End-to-end with transactional or idempotent sinks |
| Apache Pulsar | Broker deduplication and transactions | Produce and acknowledge across topics |
| Amazon Kinesis | At-least-once delivery from the stream | Exactly-once only with idempotent consumers or sinks |
In Kafka the relevant producer settings are enable.idempotence=true and a transactional.id, and consumers use isolation.level=read_committed. In Flink, exactly-once checkpointing combined with a transactional sink commits output only when a checkpoint completes, so output visibility is tied to the checkpoint interval, commonly between a few seconds and a few minutes.
Costs and design choices
Exactly-once semantics is not free, and the main trade-off is latency and throughput against correctness. Transactions add coordinator round trips, and read-committed consumers must wait until a transaction commits before they see its records.
The checkpoint or commit interval is the central tuning knob. A short interval lowers end-to-end latency and reduces replay after failure but increases coordination overhead; a long interval improves throughput but delays output and enlarges the amount of work repeated after a crash. Under heavy backpressure, aligned checkpoints can stall, which is why unaligned checkpoints exist.
| Workload | Recommended guarantee | Reason |
|---|---|---|
| Financial ledgers and billing | Exactly-once end to end | Double counting has direct monetary cost |
| Aggregated metrics and counters | Exactly-once or idempotent upserts | Duplicates inflate totals silently |
| Log shipping and search indexing | At-least-once with keyed upserts | Rewriting the same document is harmless |
| High-volume telemetry sampling | At-most-once or at-least-once | Small loss or duplication is tolerable |
| Notifications and external API calls | At-least-once with idempotency keys | External side effects cannot be rolled back |
A practical rule is to prefer idempotent sinks keyed by a stable event identifier whenever the target system supports upserts. They deliver the same observable result as transactions with less coordination and survive replays from any source.
Pitfalls and misconceptions
Most failures of exactly-once semantics come from assumptions about the boundary of the guarantee, not from bugs in the platform. The following points recur in production incidents.
- Side effects outside the transaction such as emails, HTTP calls or writes to a non-transactional database are repeated after recovery unless they carry idempotency keys.
- Non-deterministic processing, for example using wall-clock processing time or random values, can produce different output on replay, which breaks the illusion of a single execution.
- Transaction timeouts that are shorter than the checkpoint interval cause aborted transactions and apparent data loss in downstream readers.
- Consumers left on read-uncommitted isolation see records from aborted transactions and therefore observe duplicates.
- Zombie instances of a restarted task can keep writing unless producer fencing through a stable transactional identifier is in place.
- Upstream duplicates created by the original data source are outside the guarantee and still require deduplication by business key.
Put simply
Imagine a cashier who must scan every item, but the scanner sometimes beeps without confirming. The cashier scans again to be safe. Exactly-once semantics is the register that recognizes the repeated scan and charges only once.
In a data pipeline, messages are resent whenever something might have failed. Exactly-once semantics does not stop the resending. It makes sure that each event counts once in the final result, so totals, balances and reports stay correct after crashes.
In practice
A payment provider streams card transactions through a broker into a processing job that maintains running balances per merchant and writes settlement records to an output topic. One night a processing node crashes after updating balances for a batch of about 40,000 transactions but before recording its progress.
Because the job runs with checkpoint-based exactly-once processing and a transactional sink, the restarted job rolls state and input offsets back to the last completed checkpoint, replays the batch and commits the results in a single transaction. The settlement service reads only committed records, so it never sees the partial output. No merchant is credited twice, and the reconciliation report the next morning matches the card network totals without manual correction.