A job retry is the automatic re-execution of a failed background job by a scheduler or job queue, after a defined delay and within a limited number of attempts. In scheduling, it is the mechanism that lets a system recover from transient failures such as a network timeout, a locked database row or a briefly unavailable API without human intervention.
A retry is more than running the job again. The scheduler must detect the failure, record the attempt, decide whether the error is worth retrying, calculate when the next attempt should start and, after the last attempt, move the job to a final failed state where people can inspect it. Each of these decisions affects reliability, cost and data correctness.
How a job retry cycle works
In a typical setup, a worker takes a job from the job queue, executes it and reports the result. If the job raises an error, exceeds its time limit or the worker disappears, the scheduler marks the attempt as failed and evaluates the retry rules attached to the job type.
When another attempt is allowed, the job is usually not run immediately. It is stored as a delayed job with a future run time, and an ordinary worker picks it up when that time arrives. Crashed workers are handled through leases or visibility timeouts: if a worker does not confirm completion within the lease, the job becomes visible again and is retried implicitly.
Because a job may fail after doing part of its work, most queues provide at-least-once execution. The same job can therefore run more than once, which is why retried jobs must be designed to be idempotent, producing the same outcome when repeated.
Retry parameters and typical values
Every retry configuration answers a few questions: how often, how long to wait, for which errors and what happens at the end. The table lists the core parameters found in most schedulers and job frameworks.
| Parameter | Typical range | What it controls |
|---|---|---|
| Maximum attempts | 3 to 25 | How many times the job runs before final failure |
| Initial delay | A few seconds to a few minutes | The wait before the first retry |
| Backoff strategy | Fixed, linear or exponential | How the delay grows between attempts |
| Maximum delay | Minutes to hours | The cap that keeps waits reasonable |
| Jitter | Random spread of 10 to 100 percent | Prevents many jobs retrying at the same moment |
| Job timeout | Seconds to hours, per job type | When a hanging attempt counts as failed |
| Retryable errors | A list of error classes | Which failures qualify for another attempt |
Defaults differ widely between tools, so relying on them without checking is risky. The following examples show how far apart common defaults are.
| Tool | Default behavior | Practical note |
|---|---|---|
| Sidekiq | 25 retries with growing delays over about three weeks | Suitable for long outages, too slow for urgent jobs |
| Celery | Up to 3 retries when a task calls retry, 180 seconds apart | Retries are opt-in inside the task code |
| Hangfire | 10 automatic attempts with increasing delays | Configured per job with an attribute |
| Kubernetes Job | Backoff limit of 6, delays from 10 seconds capped at 6 minutes | Counts failed pods, not application errors |
| Amazon SQS | Redelivery after the visibility timeout, 30 seconds by default | A redrive policy sets the maximum receive count |
Which failures deserve a retry
The most important retry decision is classifying the error. Retrying a transient failure usually succeeds, while retrying a permanent failure only wastes resources and delays the moment when someone fixes the real problem.
| Failure type | Examples | Recommended response |
|---|---|---|
| Transient network error | Connection reset, DNS timeout | Retry with backoff |
| Overloaded dependency | HTTP 503, HTTP 429 with a wait hint | Retry after the indicated or computed delay |
| Worker crash or eviction | Lost lease, killed container | Implicit retry through redelivery |
| Invalid input | Validation error, HTTP 400 | Fail immediately and report |
| Missing resource | Record deleted, HTTP 404 | Fail or skip, depending on business rules |
| Authentication failure | Expired credentials, HTTP 401 | Fail fast and alert an operator |
Job retry compared with related scheduling concepts
Job retry is often discussed together with neighboring mechanisms. Distinguishing them avoids duplicated or conflicting retry logic at different layers of a system.
- Retry policy is the reusable rule set that defines attempts, delays and error classes; a job retry is the execution of such a policy for one specific job.
- In-process retry repeats a single HTTP call inside a running job, while a job retry restarts the whole job later, possibly on another worker.
- Scheduled re-run starts a recurring job at its next planned time regardless of failure; a retry is triggered by the failure itself.
- Dead letter handling receives jobs whose retries are exhausted, so it marks the end of the retry path, not an alternative to it.
Common mistakes with job retries
Retries hide failures when they work and amplify failures when they are misconfigured. Several patterns appear repeatedly in production incidents.
- Duplicate side effects: a job that sends an email or charges a card before failing repeats that action on every attempt unless it uses an idempotency key.
- Retry storms: thousands of jobs retrying on the same schedule overload a recovering service; exponential backoff with jitter spreads the load.
- Multiplied retries: three client retries inside a job with five job retries mean up to fifteen calls per original request.
- Stale work: a job retried days later may act on outdated data, so time-sensitive jobs need a maximum age after which they expire.
- Silent exhaustion: jobs that reach the final failed state without an alert are effectively lost; monitor the failed set.
The simple explanation
Imagine calling a shop that is briefly busy. Instead of giving up, you wait a few minutes and try again, waiting a little longer each time, and after several tries you write down the problem for someone to look at. A job retry does exactly this for background tasks: if a task fails for a temporary reason, the system tries it again later.
The tricky part is knowing when to stop. Some problems fix themselves, while others never will, such as a wrong address. Good retry settings retry the temporary problems and report the permanent ones quickly.
How it is used in practice
An e-commerce platform generates shipping labels through a carrier API in a background job after each order. During peak season the carrier occasionally returns HTTP 503 errors for a few minutes. The team configures the label job with eight attempts, exponential backoff starting at 15 seconds, a cap of 10 minutes and full jitter, and marks validation errors such as an invalid postal code as non-retryable.
Each job passes the order number as an idempotency key to the carrier, so a retried request never creates a second label. During a short carrier outage, hundreds of jobs fail and then succeed within half an hour without anyone intervening. Orders with bad addresses fail at the first attempt and appear on a review dashboard, where customer service corrects them the same day.