Kafka, formally Apache Kafka, is an open-source distributed event streaming platform that stores streams of records in durable, partitioned and replicated logs and lets many applications write to and read from them independently. In streaming data architectures, Kafka acts as the central backbone that decouples the systems producing events from the systems processing them.
Kafka was created at LinkedIn, released as open source in 2011 and became a top-level Apache Software Foundation project in 2012. Unlike a traditional message broker that deletes a message once it is delivered, Kafka retains records for a configured period regardless of whether they have been read, so consumers can replay history, recover from failures and join late without data loss.
How Kafka stores and delivers events
Kafka works by appending records to an ordered, immutable log and letting each consumer track its own position in that log. Every record has an optional key, a value, a timestamp and optional headers, and it receives a sequential number called an offset within its partition.
| Component | Role | Key property |
|---|---|---|
| Broker | Server that stores partitions and serves requests | A cluster typically runs three or more |
| Topic | Named stream of related records | Logical category, such as orders |
| Partition | Ordered, append-only log within a topic | Unit of parallelism and ordering |
| Producer | Client that writes records to topics | Chooses partition, often by key hash |
| Consumer group | Set of consumers sharing a topic’s work | Each partition read by one member |
| Offset | Position of a record in a partition | Committed by consumers to track progress |
Ordering is guaranteed only within a single partition, not across a whole topic. Producers therefore send all records with the same key, such as one customer ID, to the same partition, which preserves per-entity order while spreading the total load across many partitions and brokers.
Cluster metadata was historically managed by Apache ZooKeeper. Newer Kafka versions use KRaft, a built-in consensus protocol based on Raft, and Kafka 4.0 removed ZooKeeper support entirely, which simplifies operations to a single system.
Replication, durability and delivery guarantees
Kafka achieves durability by replicating each partition across several brokers. One replica acts as the leader and handles writes; followers copy its log, and those that are fully caught up form the in-sync replica set. If the leader fails, an in-sync follower takes over.
Durability and latency are tuned with a few settings. The table shows the most consequential ones and the trade-off each controls.
| Setting | Common value | Effect |
|---|---|---|
| replication.factor | 3 | Copies of each partition across brokers |
| min.insync.replicas | 2 | Replicas that must confirm a write |
| acks on the producer | all | Write succeeds only after in-sync replicas confirm |
| enable.idempotence | true | Prevents duplicates from producer retries |
| retention.ms | 604800000 (7 days) | How long records stay before deletion |
With a replication factor of 3 and min.insync.replicas of 2, a topic tolerates the loss of one broker without losing acknowledged writes or rejecting new ones. Delivery semantics depend on when consumers commit offsets: committing after processing gives at-least-once delivery, while idempotent producers combined with transactions enable exactly-once processing between Kafka topics.
Kafka compared with other messaging systems
Kafka is often grouped with message brokers, but its log-based design gives it different strengths. The comparison below contrasts it with a classic queue-based broker, such as RabbitMQ, and with a managed cloud stream, such as Amazon Kinesis.
| Aspect | Kafka | Queue-based broker | Managed cloud stream |
|---|---|---|---|
| Storage model | Retained, replayable log | Messages removed after acknowledgment | Retained log with fixed retention limits |
| Consumer model | Pull, each group tracks offsets | Broker pushes to consumers | Pull per shard |
| Replay | Native, by resetting offsets | Limited or not supported | Supported within retention window |
| Routing flexibility | Basic, by topic and partition | Rich exchanges and routing keys | Basic, by shard key |
| Operations | Self-managed or managed service | Self-managed or managed service | Fully managed only |
The practical rule is that Kafka fits high-throughput event streams that several consumers read independently, while a queue-based broker fits task distribution with complex routing and per-message acknowledgment. Kafka can sustain very high throughput, often hundreds of megabytes per second per cluster, because it writes sequentially to disk and relies on batching and the operating system page cache.
The Kafka ecosystem and typical uses
Kafka is surrounded by components for integration and stream processing, which turn a storage layer into a complete event platform.
- Kafka Connect moves data between Kafka and external systems, such as databases, object storage and search indexes, through reusable connectors.
- Kafka Streams is a client library for stateful stream processing, including joins, aggregations and windowing by event time.
- Schema registries store Avro, Protobuf or JSON Schema definitions so producers and consumers evolve record formats compatibly.
- Change data capture tools publish every database insert, update and delete as an event stream.
Typical uses include activity tracking, log and metrics aggregation, event-driven microservices, real-time fraud detection and feeding data lakes. In each case, the defining benefit is the same: producers write once, and any number of consumers process the same event stream at their own pace.
Operational pitfalls and limitations
Kafka is reliable when sized and configured deliberately, but several common operational mistakes cause lag, data loss or uneven load.
| Symptom | Likely cause | Fix |
|---|---|---|
| Growing consumer lag | Too few partitions or slow processing | Add partitions and consumers, optimize handlers |
| One broker overloaded | Skewed keys concentrate traffic | Choose higher-cardinality keys |
| Lost writes after failure | acks set to 1 with few replicas | Use acks all and min.insync.replicas 2 |
| Duplicate processing | Offsets committed after side effects fail | Idempotent consumers or transactions |
| Frequent rebalances | Processing exceeds poll interval | Raise max.poll.interval.ms or batch smaller |
Partition count deserves particular care. Adding partitions later changes key-to-partition mapping, which breaks per-key ordering for existing keys, so topics should be created with enough headroom. Kafka also does not provide backpressure toward producers in the way a bounded queue does; slow consumers simply fall behind, and monitoring consumer lag is the essential signal.
The idea in brief
Kafka is software that lets many computer systems share a continuous flow of events, such as clicks, payments or sensor readings.
Imagine a long notebook where every event is written on a new line and never erased for a set time. Any team can read the notebook from wherever it stopped. One team may read live. Another may reread last week. Nobody has to wait for anyone else, and nothing is lost when a reader takes a break. Kafka is that shared notebook for data, copied onto several machines so it survives failures.
A practical scenario
A payment provider processes about 20,000 card transactions per second at peak. Each authorization is published to a Kafka topic named payments with 48 partitions, keyed by card number so that all events for one card stay in order. The topic uses a replication factor of 3, min.insync.replicas of 2 and producer acks set to all.
Three consumer groups read the same stream independently. A fraud-scoring service evaluates each transaction within milliseconds, a ledger service writes balances, and a Kafka Connect sink loads events into a data warehouse every few minutes. When the warehouse load failed for two hours during maintenance, its connector simply resumed from its last committed offset, and no transaction was lost or duplicated in the ledger.