A consumer group is a named set of cooperating consumer instances that divide the partitions of a topic among themselves, so that each record is delivered to exactly one member of the group while the group as a whole reads the entire stream. In streaming data platforms such as Kafka, the consumer group is the basic unit of horizontal scaling and fault tolerance on the reading side of an event stream.
Several independent consumer groups can read the same topic at the same time, and each keeps its own position in the log. This combination turns a single stream into both a work queue within a group and a publish-subscribe channel across groups, which is why the concept sits at the center of most stream processing architectures.
How a consumer group divides the work
A consumer group works by giving each partition to one consumer at a time. A topic is split into partitions, which are ordered, append-only logs. When consumers join a group under the same group ID, a coordinator on the broker side assigns every partition to exactly one member. A member may own several partitions, but no partition is read by two members of the same group simultaneously.
This rule has a direct consequence: the number of partitions caps the parallelism of a group. With 12 partitions, up to 12 consumers can work in parallel; a thirteenth instance joins the group but stays idle as a standby. Ordering is guaranteed only within a partition, so records that must stay in sequence, such as all events for one customer, are routed to the same partition by a message key.
The table below shows how the ratio between consumers and partitions shapes the behavior of a group.
| Situation | Assignment result | Practical effect |
|---|---|---|
| Fewer consumers than partitions | Each consumer owns several partitions | Full coverage, uneven load is possible |
| Equal numbers | One partition per consumer | Maximum parallelism for the topic |
| More consumers than partitions | Surplus consumers receive nothing | Idle instances act only as failover capacity |
| Single consumer | All partitions on one instance | Simple, but no scaling and slow recovery |
Offsets, commits and delivery guarantees
A consumer group records its progress as a committed offset per partition, which is the position of the next record the group should read. In Kafka these commits are stored in an internal compacted topic, so the position survives restarts and is shared by whichever member owns the partition next. Each group keeps independent offsets, which is why a new group can replay a topic from the beginning without disturbing existing readers.
The moment of the commit decides the delivery guarantee. Committing after processing gives at-least-once delivery, because a crash between processing and commit causes the same records to be read again. Committing before processing gives at-most-once delivery and risks silent loss. Exactly-once semantics require that output writes and offset commits happen in one atomic transaction, or that the sink is idempotent.
| Commit strategy | Guarantee | Risk on failure |
|---|---|---|
| Automatic periodic commit | Usually at-least-once | Duplicates or gaps around the commit interval |
| Manual commit after processing | At-least-once | Reprocessing of the last uncommitted batch |
| Commit before processing | At-most-once | Records lost if the consumer crashes |
| Transactional commit with output | Exactly-once within the platform | Higher latency and configuration effort |
Automatic commits, typically every 5 seconds by default, are convenient but obscure what was actually processed. Teams with strict correctness needs disable them and commit explicitly after a batch has been written to its destination.
Rebalancing and membership
A rebalance is the process by which a consumer group redistributes partitions whenever its membership or the topic changes. It is triggered when a consumer joins, leaves cleanly, crashes, or misses its heartbeat deadline, and when partitions are added to a subscribed topic. During a classic rebalance, members stop consuming, give up their partitions and receive a new assignment.
Liveness is judged by two timers. The session timeout, commonly between 10 and 45 seconds, detects a dead process through missing heartbeats. The maximum poll interval, often 5 minutes, detects a live process that is stuck in slow processing and has not asked for new records. Exceeding either removes the member and starts a rebalance.
Assignment strategies differ in how much disruption they cause, as the following comparison summarizes.
| Strategy | How partitions move | Best suited for |
|---|---|---|
| Range or round-robin (eager) | All partitions revoked, then reassigned | Small, stable groups |
| Sticky assignment | Keeps previous ownership where possible | Consumers with local caches or state |
| Cooperative incremental | Only moved partitions are paused | Large groups that scale often |
| Static membership | Restarted instance keeps its identity | Rolling deployments without churn |
Consumer groups compared with related models
The consumer group sits between the classic message queue and the pure publish-subscribe topic. Within one group it behaves like a queue that spreads work across workers; across groups it behaves like publish-subscribe, because every group receives every record. Unlike a traditional message broker that deletes a message after acknowledgment, a log-based system retains records for a configured period, so consumption does not remove data.
| Model | Who receives a record | Ordering and replay |
|---|---|---|
| Kafka consumer group | One member per group, every group | Per-partition order, replay by offset reset |
| Queue with competing consumers | One consumer in total | Weak ordering, no replay after acknowledgment |
| Apache Pulsar subscription | Depends on subscription type | Exclusive, shared, failover or key-shared modes |
| Amazon Kinesis consumer application | One worker per shard lease | Per-shard order, replay within retention |
Apache Pulsar makes the trade-off explicit: a shared subscription distributes individual messages across consumers without partition limits but gives up strict ordering, while a key-shared subscription preserves order per key. Amazon Kinesis applications achieve a similar effect with shard leases tracked in an external table.
Sizing, monitoring and common mistakes
The single most important health metric of a consumer group is consumer lag, the difference between the latest offset in a partition and the group’s committed offset. Lag that grows steadily means the group cannot keep up with producers; lag that spikes and recovers usually points to rebalances or slow downstream systems. Lag should be watched per partition, because an average hides one stuck partition.
- Too few partitions: parallelism cannot grow later without repartitioning, which changes key-to-partition mapping.
- Slow processing inside the poll loop: exceeding the poll interval causes repeated rebalances, known as a rebalance storm.
- Shared group IDs by accident: two unrelated applications using one ID silently split the data between them.
- Non-idempotent sinks with at-least-once delivery: replayed records create duplicate side effects such as double charges.
- Hot keys: one very active key overloads a single partition and its consumer while others stay idle.
A practical rule is to provision more partitions than the current consumer count, often two to three times, so the group can scale out without changing the topic. Backpressure is handled naturally, because each consumer pulls records at its own pace instead of having them pushed.
In plain words
Imagine a large pile of incoming letters sorted into numbered trays. A consumer group is a team of clerks who agree that each tray belongs to one clerk at a time, so no letter is handled twice and none is forgotten. If a clerk goes home, the trays are handed to colleagues; if a new clerk arrives, some trays are passed over.
Another team in a different department can read copies of the same letters for its own purpose, keeping its own bookmark in each tray. That is the key idea: sharing work inside a team, full copies for every team.
A real-world example
An online retailer publishes every order event to a topic with 24 partitions, keyed by customer ID. A billing service runs as one consumer group with six instances, each owning four partitions, and commits offsets only after the invoice is stored. A separate analytics group reads the same topic with three instances and feeds a reporting database.
During a seasonal sale, billing lag climbs to several minutes. The team scales the billing group to 12 instances; cooperative rebalancing moves two partitions to each new instance without pausing the rest, and lag returns to under a second within minutes. Because invoices are written with an idempotent key, the few records replayed during the rebalance cause no duplicate charges, and the analytics group is unaffected throughout.