A poison message is a message in a queue or event stream that cannot be processed successfully no matter how often it is retried, typically because of malformed content, an unexpected schema or a business rule it violates. In reliability engineering for automation, the term describes the risk that one such message is redelivered endlessly, blocking a queue, consuming worker capacity or crashing consumers in a loop.
Handling poison messages is a core part of building reliable automated workflows. The usual remedy is to count delivery attempts, stop retrying after a limit and move the message to a dead letter queue where it can be inspected without disrupting normal traffic.
How a poison message disrupts processing
A poison message disrupts processing because most messaging systems use at-least-once delivery: when a consumer fails or does not acknowledge a message, the broker makes it available again. If the failure is caused by the message itself, every new attempt fails in the same way, creating a redelivery loop.
The damage depends on how the queue delivers messages. In a strictly ordered queue or a partition of an event log, the poison message sits at the head and blocks every message behind it. In a competing-consumers queue, other messages continue, but the poison message keeps consuming retries, log space and CPU, and it may repeatedly crash a worker process.
Typical causes are listed in the following table, together with the symptom they usually produce.
| Cause | Example | Typical symptom |
|---|---|---|
| Malformed payload | Truncated or invalid JSON | Parser exception on every attempt |
| Schema mismatch | Producer added a required field the consumer rejects | Validation error after a deployment |
| Invalid business data | Order references a deleted customer | Constraint violation in the target system |
| Oversized message | Attachment exceeds consumer memory or API limits | Out-of-memory crash or rejected call |
| Unsupported version or encoding | Old event version, wrong character set | Deserialization failure |
| Consumer bug | Edge case such as a null value or a leap day | Same exception for one specific record |
Detecting poison messages
Poison messages are detected by tracking how many times each message has been delivered and comparing that count with a threshold. Most brokers provide a delivery counter, and the consumer or the broker moves the message aside when the counter exceeds the configured maximum.
The mechanisms differ by platform, as the table below shows.
| Platform | Detection mechanism | Typical setting |
|---|---|---|
| Amazon SQS | Receive count with a redrive policy | maxReceiveCount, often 3 to 10 |
| Azure Service Bus | Delivery count per message | MaxDeliveryCount, default 10 |
| RabbitMQ | Delivery limit on quorum queues and dead-letter exchange | delivery-limit set per queue |
| Apache Kafka | No broker counter; handled in consumer code | Retry topics and a dead letter topic |
| IBM MQ | Backout count with a backout queue | Backout threshold on the queue |
Detection must distinguish permanent failures from transient ones. A timeout from a busy database is transient and deserves retries with exponential backoff and jitter; a schema violation is permanent and should be routed aside after the first or second attempt instead of wasting the full retry budget.
Poison message, dead letter queue and retry policy
A poison message is the problem, while the dead letter queue and retry policy are parts of the remedy. The terms are often used together and sometimes confused, so the following comparison clarifies their roles.
| Concept | What it is | Role with poison messages |
|---|---|---|
| Poison message | A message that always fails processing | The fault that must be isolated |
| Retry policy | Rules for how often and when to retry | Limits attempts before giving up |
| Exponential backoff | Increasing wait time between retries | Prevents tight failure loops |
| Dead letter queue | Separate queue for undeliverable messages | Holds the poison message for inspection |
| Idempotency | Safe repetition of the same operation | Prevents duplicate effects during retries |
| Error workflow | Automated handling path for failures | Alerts staff and starts remediation |
A message in a dead letter queue is not automatically a poison message. It may have expired, exceeded a size limit or been rejected by routing rules. Classifying dead-lettered messages by reason helps teams decide whether to fix data, fix code or simply replay.
Designing workflows that tolerate poison messages
Resilient workflows treat poison messages as expected events, not emergencies. The design goal is that one bad message never stops healthy traffic and never disappears without a trace.
- Validate early at the consumer boundary and reject clearly invalid messages without retries.
- Set a finite retry limit with backoff, typically three to five attempts for permanent-looking errors.
- Preserve the original message and its metadata, including headers, error text and attempt count.
- Alert on dead letter growth rather than on each individual failure.
- Provide a replay path so fixed messages can be resubmitted once the cause is corrected.
- Use non-blocking retry topics in ordered streams so one partition is not frozen.
The following table maps typical responses to the root cause once a poison message has been isolated.
| Root cause | Recommended action | Replay |
|---|---|---|
| Consumer bug | Fix and deploy the consumer | Replay after deployment |
| Bad source data | Correct data in the source system | Replay or discard if resent |
| Schema change | Add backward-compatible handling | Replay all affected messages |
| Truly invalid request | Notify the sender or business owner | Discard with a recorded reason |
Common mistakes
The most damaging mistake is unlimited retries, which turns a single malformed record into a permanent load on the system. Other frequent errors include catching exceptions and silently acknowledging the message, which loses data without any record, and retrying immediately without delay, which floods logs and can trigger rate limits on downstream APIs.
Ordering requirements deserve special care. When strict order matters, for example for account balance updates, skipping a poison message may create inconsistent state, so the workflow may need to pause that entity’s stream while allowing other entities to continue.
A plain-language summary
A poison message is a task that can never be completed, for example a form with missing or garbled information. An automated system keeps trying to process it, fails and tries again, over and over.
The sensible solution is the same one an office would use. After a few failed tries, the problem item is put in a separate tray for a person to check. The rest of the work then continues normally.
An example from practice
An online retailer sends order events to a queue that feeds its warehouse system. After a checkout update, one order arrives with a quantity field written as text instead of a number, and the warehouse consumer throws an exception on every attempt.
Because the queue is configured with a maximum of five deliveries, the broker moves the order to a dead letter queue after the fifth failure. An alert notifies the integration team, which fixes the producer, corrects the record and replays it.
Thousands of other orders continue shipping during the incident, and the faulty order is fulfilled within an hour instead of blocking the warehouse feed for the entire day.