A retry policy is a set of rules that determines whether, how many times and at what intervals a failed operation is attempted again before the system gives up. In automation reliability, it is the mechanism that turns temporary failures, such as network timeouts, rate limits or brief service outages, into successful runs without human intervention.
A complete retry policy specifies which errors are retryable, the maximum number of attempts, the delay between attempts and what happens after the final failure. Applied carelessly, retries multiply load on a struggling service and duplicate side effects, so a good policy is always paired with idempotency, backoff and a dead letter destination.
How a retry policy works
A retry policy works by classifying each failure and deciding the next step: retry after a delay, fail immediately or escalate. When a step in a workflow or job fails, the runtime checks the error against the policy, increments the attempt counter and schedules the next attempt if allowed.
Classification is the most important decision. Transient errors, such as connection resets, timeouts and HTTP 429, 502, 503 or 504 responses, are usually worth retrying. Permanent errors, such as HTTP 400 validation failures, 401 or 403 authorization failures and 404 for a missing resource, will fail again identically and should not be retried.
| Error type | Examples | Retry? |
|---|---|---|
| Network failure | Timeout, connection reset, DNS error | Yes, with backoff |
| Rate limiting | HTTP 429, quota exceeded | Yes, honoring Retry-After if provided |
| Server unavailable | HTTP 502, 503, 504 | Yes, with backoff |
| Invalid request | HTTP 400, schema validation error | No, fix the input |
| Authorization failure | HTTP 401, 403 | No, unless a token refresh resolves it |
| Internal server error | HTTP 500 | Limited retries, then escalate |
After the last allowed attempt, the policy defines a terminal action: mark the run as failed, move the item to a dead letter queue, notify an operator or trigger a compensating workflow that undoes earlier steps.
Retry policy parameters
A retry policy is defined by a small set of parameters that control persistence and pacing. Their values should reflect how long a dependency is typically unavailable and how quickly the result is needed.
| Parameter | Typical values | Purpose |
|---|---|---|
| Maximum attempts | 3 to 10 for calls, more for queued jobs | Limits wasted work |
| Initial delay | 100 ms to a few seconds | First pause after failure |
| Backoff multiplier | 2 is the most common | Growth of delay per attempt |
| Maximum delay | 30 seconds to several minutes | Caps a single wait |
| Jitter | Random 0 to 100 percent of the delay | Spreads retries from many clients |
| Overall deadline | Seconds for user requests, hours for jobs | Stops retrying when the result is useless |
Jitter is essential at scale. Without it, thousands of clients that failed at the same moment retry at the same moment, producing synchronized waves of traffic that can keep a recovering service down. Randomizing each delay spreads the load evenly over time.
Retry strategies compared
Retry strategies differ mainly in how the waiting time changes between attempts. Exponential backoff with jitter is the usual default for calls to shared services, while simpler strategies fit local or low-volume cases.
| Strategy | Delay pattern | Suitable for | Risk |
|---|---|---|---|
| Immediate retry | No delay | Rare, brief glitches | Hammers a failing service |
| Fixed interval | Same delay each time | Low-volume internal jobs | Synchronized retries |
| Linear backoff | Delay grows by a constant | Moderate load | Slow to back off under heavy failure |
| Exponential backoff | Delay doubles each attempt | Remote APIs and shared services | Long waits without a cap |
| Exponential with jitter | Doubling plus randomness | Large fleets of clients | Slightly less predictable timing |
A retry policy also interacts with a circuit breaker. While retries handle individual failures, a circuit breaker stops calls entirely after a failure threshold, giving the dependency time to recover and preventing retry storms across the whole system.
Retries, idempotency and dead letter queues
A retry policy is only safe when the operation being retried is idempotent, meaning that repeating it has the same effect as performing it once. A timeout does not tell the caller whether the server completed the work, so a blind retry of a payment or an email can duplicate the side effect.
Common techniques include idempotency keys sent with each request, upserts instead of inserts and checks for prior completion before acting. When retries are exhausted, a dead letter queue keeps the failed item with its error details so that it can be inspected, fixed and replayed instead of being lost.
Common retry policy mistakes
Poor retry policies cause outages as often as they prevent them. The most damaging mistakes involve retrying the wrong errors, retrying at several layers and retrying without limits.
| Mistake | Consequence | Better practice |
|---|---|---|
| Retrying permanent errors | Wasted attempts and delayed alerts | Classify errors explicitly |
| Retries at every layer | Attempts multiply, such as 3 x 3 x 3 = 27 | Retry at one chosen layer |
| No maximum attempts | Endless loops and growing load | Set attempts and a deadline |
| No jitter | Synchronized retry storms | Add randomized delay |
| Silent final failure | Lost work nobody notices | Dead letter queue and alerting |
Stripped to the essentials
A retry policy is the rule for what to do when something does not work the first time. It is like calling a friend whose line is busy: you wait a little, try again, wait a bit longer, and after a few tries you send a message instead.
The important part is knowing when trying again makes sense. A busy line is worth another try; a wrong phone number is not, no matter how often you dial.
Where it is applied
A subscription company runs a nightly workflow that charges renewing customers through a payment provider. Occasionally the provider returns HTTP 503 for a few minutes, and previously the whole batch failed and had to be restarted by hand.
The team defines a retry policy: up to six attempts with exponential backoff starting at two seconds, capped at five minutes, with full jitter. Each charge request carries an idempotency key based on the invoice number, so a retried request cannot charge a customer twice. Card declines are classified as permanent and are not retried.
Items that still fail go to a dead letter queue reviewed each morning. Since the change, short provider outages resolve themselves without manual work, and no duplicate charges have occurred.