A delayed job is a unit of background work that is placed in a job queue together with a future execution time, so that no worker may run it before that moment arrives. In scheduling within automation, it is the standard way to say do this later, exactly once, without keeping a process or a user request waiting.
Delayed jobs sit between two neighboring ideas. A task scheduler fires work on a recurring calendar, while an ordinary job in a job queue runs as soon as a worker is free. A delayed job is one-off and time-shifted: it is created by an event, such as a sign-up or a failed payment, and becomes eligible for execution only after its delay expires.
How a delayed job moves through the queue
A delayed job follows a fixed lifecycle: it is enqueued with a payload and a run-at timestamp, waits in a holding area, becomes ready when the timestamp passes, and is then claimed and executed by a worker. The holding area is the part that distinguishes it from an immediate job.
Implementations store pending jobs in a structure ordered by time. Redis-based systems typically use a sorted set keyed by the run-at timestamp, and a poller moves due entries into the ready list. Database-backed queues add an indexed run_at column and select rows where run_at is in the past, often with row locking such as SELECT … FOR UPDATE SKIP LOCKED so that two workers never claim the same job. Message brokers achieve the same effect with per-message delays, delayed-message plugins or dead-letter routing with a time to live.
The following table lists the states a delayed job passes through and what the system guarantees in each one.
| State | What happens | Typical guarantee |
|---|---|---|
| Scheduled | Job stored with payload and run-at time | Persisted, not visible to workers |
| Ready | Run-at time passed, job moved to ready list | Eligible for the next free worker |
| Claimed | Worker locks or reserves the job | Visibility timeout or lock prevents double claim |
| Completed | Handler finished and acknowledged | Job removed or archived |
| Failed | Handler raised an error or timed out | Job retry policy or dead-letter queue applies |
Key parameters of a delayed job
Every delayed job is defined by a small set of parameters that decide when, how reliably and how often it runs. Most of them are set at enqueue time and cannot change afterwards without canceling and re-creating the job.
| Parameter | Typical range | What it controls |
|---|---|---|
| Delay or run-at time | Seconds to several weeks | Earliest moment the job may start |
| Polling interval | 100 ms to 60 s | How late a due job can be picked up |
| Visibility timeout | 30 s to 12 hours | When an unacknowledged job returns to the queue |
| Maximum attempts | 1 to about 25 | How many times a failing job is retried |
| Idempotency key | Unique string per job | Whether duplicates can be detected and skipped |
| Priority | Small integer scale | Order among jobs that are ready at the same time |
Two parameters deserve particular attention. The polling interval sets the lateness floor: a job due at 10:00:00 with a 5-second poller may start at 10:00:05, and later still if all workers are busy. A delay is therefore always a not-before time, never an exact start time. Some managed queues also cap the delay at a short maximum, around 15 minutes in several cloud offerings, which forces longer waits into a database or a scheduler.
Delayed job compared with related scheduling concepts
A delayed job is best understood by contrast with the other members of the scheduling family. The difference lies in what triggers the work and how often it recurs.
| Concept | Trigger | Recurrence | Typical use |
|---|---|---|---|
| Delayed job | Event plus a delay | Once | Reminder email 24 hours after sign-up |
| Immediate queued job | Event | Once, as soon as possible | Resize an uploaded image |
| Scheduled task (cron) | Calendar expression | Repeating | Nightly report at 02:00 |
| Job retry with backoff | Failure of a previous attempt | Bounded repeats | Call a payment API again after an error |
| Timer in a workflow engine | Step inside a long-running process | Once per process instance | Escalate an approval after three days |
In practice these mechanisms overlap. A job retry is usually implemented as a delayed job with an exponentially growing delay, and many task schedulers simply enqueue delayed jobs when their time arrives. The useful distinction is intent: a delayed job expresses a business rule about time, while a retry expresses recovery from failure.
Designing delayed jobs that behave correctly
A reliable delayed job is idempotent, small and re-validated at execution time. Because the world changes between enqueue and execution, the handler should re-read current state instead of trusting the payload. A reminder to complete a purchase must first check whether the purchase already happened.
- Store identifiers, not snapshots: pass an order ID and load the order when the job runs.
- Make handlers idempotent so that an at-least-once queue can safely deliver the same job twice.
- Use UTC timestamps for run-at values to avoid daylight saving shifts.
- Keep a cancellation path, either by deleting the job or by a status flag the handler checks.
- Monitor queue lag, the gap between run-at time and actual start, as the main health metric.
The choice of backend follows from delay length and volume. The decision table below summarizes common situations.
| Situation | Recommended approach | Reason |
|---|---|---|
| Short delays, high volume | In-memory broker with sorted-set scheduling | Low latency and cheap polling |
| Delays of days or weeks | Database table with indexed run-at column | Durable storage and easy cancellation |
| Strict audit requirements | Transactional outbox in the application database | Job creation commits with the business change |
| Multi-step business processes | Workflow engine with durable timers | State and timers managed together |
Common pitfalls with delayed jobs
Most production problems with delayed jobs come from treating the delay as a promise or from forgetting that the job outlives the code that created it.
- Stale payloads: the job acts on data that changed during the delay.
- Code drift: a job enqueued last month runs against a new handler signature; versioned payloads prevent crashes.
- Thundering herd: thousands of jobs share the same run-at time; adding a small random jitter spreads the load.
- Lost jobs on restart: an in-memory delay without persistence disappears when the process stops.
- Duplicate execution: a visibility timeout shorter than the handler runtime lets a second worker start the same job.
The thundering herd problem is easy to overlook. Scheduling every renewal reminder for exactly 09:00 creates a spike that can exhaust workers and downstream APIs, while a jitter of a few minutes produces the same user experience with a flat load curve.
A non-technical view
A delayed job is a note to the system that says do this later. Instead of doing a task right away, the system writes it down with a time on it and puts it in a waiting tray. When the time comes, the next available helper picks it up and does it.
Think of a letter you hand to the post office and ask them to deliver on a certain date. You do not stand at the counter until then. The post office keeps it safe and sends it when the day arrives, perhaps a little after the exact hour, but never before.
A concrete example
An online course platform wants to encourage students who sign up but never start a lesson. When a student registers, the application enqueues a delayed job with the student ID and a run-at time 48 hours later. The job is stored in a database-backed queue, and a pool of four workers polls for due jobs every five seconds.
Two days later a worker claims the job, loads the student record and checks progress. If the student has already completed a lesson, the handler exits without sending anything. Otherwise it sends a short reminder. Because the handler uses the student ID as an idempotency key, a duplicate delivery after a worker crash never produces a second email, and the team sees a steady trickle of reminders instead of a daily spike.