Publish-subscribe, often shortened to pub/sub, is a messaging pattern in which producers publish messages to a named topic without knowing who will read them, and every subscriber registered on that topic receives its own copy. In streaming data, it is the basic distribution model behind event-driven pipelines, because it decouples senders from receivers in identity, in time and in scale.
The pattern is implemented by a message broker or a distributed log such as Kafka, Apache Pulsar or Amazon Kinesis. The broker accepts each published event, stores or forwards it, and fans it out to all interested subscribers, so a new consumer can be added without changing a single line of producer code.
How publish-subscribe routes messages
Publish-subscribe works by placing an intermediary between producers and consumers. A producer sends an event to a topic, the broker records it, and the broker then delivers the event to each active subscription on that topic. Producers never address consumers directly, which is the essential difference from point-to-point messaging.
Brokers use one of two delivery styles. In the push model, the broker sends messages to subscribers as they arrive and must slow down when a subscriber falls behind. In the pull model, used by log-based systems, each subscriber reads at its own pace and tracks its position with an offset, which makes replay and recovery straightforward.
The following table lists the main components of a publish-subscribe system and the role each one plays.
| Component | Role | Typical detail |
|---|---|---|
| Producer | Publishes events to a topic | Chooses a key that decides the partition |
| Topic | Named channel for one kind of event | Split into partitions for parallelism |
| Broker | Stores and distributes events | Replicates data across several nodes |
| Subscription | Registered interest in a topic | Durable or temporary |
| Consumer | Reads and processes events | Commits offsets or sends acknowledgments |
Delivery guarantees and ordering
A publish-subscribe system is defined as much by its delivery guarantee as by its routing. The guarantee states how many times a subscriber may see a message when networks fail, consumers crash or brokers restart, and it determines how much defensive logic the consumer must contain.
The table below compares the three common guarantees and what they demand from the application.
| Guarantee | Possible outcome | What the consumer must do |
|---|---|---|
| At-most-once | Messages may be lost, never repeated | Accept occasional gaps |
| At-least-once delivery | No loss, duplicates possible | Make processing idempotent |
| Exactly-once semantics | Each effect applied once | Use transactions or deduplication end to end |
Ordering is usually guaranteed only within a partition, not across a whole topic. Events that share a key, such as one customer identifier, land in the same partition and keep their relative order, while events with different keys may interleave. Designs that need global ordering must accept a single partition and therefore limited throughput.
Retention is a further parameter. Classic queue-style brokers delete a message once every subscription has acknowledged it, while log-based brokers keep events for a configured retention period, commonly from hours to several weeks, or indefinitely with compaction or tiered storage. Long retention allows a new subscriber to replay history from the beginning.
Publish-subscribe compared with related patterns
Publish-subscribe is one of several ways to move data between services, and it is often confused with message queuing. The decisive question is whether each message goes to one receiver or to all interested receivers.
| Pattern | Who receives a message | Typical use |
|---|---|---|
| Publish-subscribe | Every subscription gets a copy | Event distribution to many services |
| Point-to-point queue | Exactly one consumer from a pool | Work distribution and task processing |
| Request-response | One addressed service, which replies | Synchronous queries and commands |
| Event stream with consumer groups | One copy per group, shared inside it | Combines fan-out with load balancing |
The last row shows how modern platforms merge both ideas. A consumer group behaves like one logical subscriber: the topic fans out to every group, but within a group the partitions are divided among members, so each event is processed once per group. Adding a group adds a new independent reader; adding members to a group adds processing capacity.
Designing topics and subscriptions
Good publish-subscribe design starts with topic granularity. A topic should carry one kind of event with one schema, such as order placed or sensor reading, so that subscribers can rely on a stable contract. Very coarse topics force consumers to filter out irrelevant data, while very fine topics multiply operational overhead.
The following decision table summarizes common design choices and the reasoning behind them.
| Situation | Recommended choice | Reason |
|---|---|---|
| Per-entity order matters | Partition by entity key | Keeps related events in sequence |
| Schemas change over time | Use a schema registry with compatibility rules | Prevents breaking existing subscribers |
| Consumers need history | Choose log-based retention | Supports replay and late subscribers |
| Very high fan-out with small messages | Use a lightweight push broker | Reduces per-subscriber storage cost |
| Bursty producers | Size partitions for peak load | Avoids lag and backpressure at peaks |
Partition count deserves particular care, because it sets the upper limit of parallel consumers in one group. Increasing it later is possible on most platforms but can reshuffle key placement and break per-key ordering during the transition, so teams usually provision for expected growth from the start.
Limitations and common mistakes
The loose coupling that makes publish-subscribe attractive also hides problems. Producers cannot see whether anyone consumes their events, so broken or slow subscribers often go unnoticed until consumer lag grows large. Monitoring lag per subscription is therefore a basic operational requirement.
- Assuming exactly-once by default: most systems deliver at least once, so consumers must tolerate duplicates.
- Expecting topic-wide ordering: order holds per partition, not across the whole topic.
- Using a hot key: one very active key overloads a single partition while others stay idle.
- Publishing unversioned schemas: an incompatible change silently breaks downstream consumers.
- Treating the broker as a database: retention and query features are limited compared with real storage.
A further subtlety is dual writes. A service that updates its database and then publishes an event can fail between the two steps, leaving systems inconsistent. The transactional outbox pattern or change data capture solves this by deriving events from the committed database change.
The short version
Publish-subscribe works like a newsletter with many readers. A writer sends each issue to the newsletter service, not to individual people, and everyone who signed up receives a copy. The writer does not need to know who the readers are, and new readers can sign up at any time.
In computer systems, this means one application can announce that something happened, and many other applications can react in their own way. The sender stays simple, and new uses can be added without touching it.
Where it makes a difference
Consider an online retailer whose checkout service publishes an order placed event to a single topic with 12 partitions, keyed by order identifier. Four independent consumer groups subscribe: inventory reservation, payment capture, email confirmation and a stream processing job that updates a sales dashboard every few seconds.
When the marketing team later wants a fraud-scoring service, it simply registers a fifth subscription and replays the last seven days of retained events to warm up its model. The checkout service is not redeployed or changed at all. Because delivery is at least once, every consumer checks the order identifier before acting, so a duplicate event never charges a customer twice. The result is a checkout path that stays fast and unchanged while the number of downstream uses keeps growing.