A worker is a long-running background process that takes jobs from a job queue or task scheduler, executes them and reports the outcome. In scheduling and task orchestration, workers are the execution layer: the scheduler decides what should run and when, the queue holds the pending work, and workers do the actual processing outside the request path of the user-facing application.
Workers make systems responsive and scalable. A web application can accept an upload in milliseconds and leave resizing, emailing or report generation to workers, and operators can add or remove workers to match load without changing the application code.
How a worker processes jobs
A worker runs a fetch, execute and acknowledge loop. It connects to a broker such as a message queue, a database table or a key-value store, reserves a job, runs the matching handler with the job’s payload and then confirms the result so the job is not delivered again.
- Poll or subscribe: the worker waits for new jobs, either by polling or through a push from the broker.
- Reserve: the job is locked or hidden from other workers for a visibility timeout or lease.
- Execute: the handler code runs with the job’s arguments, often within a time limit.
- Acknowledge: on success the job is deleted or marked complete.
- Handle failure: on error the job is released for a job retry, delayed or moved to a dead letter queue.
The acknowledgement model determines delivery guarantees. With acknowledgement after execution, a crash during processing causes the job to be redelivered, which gives at-least-once delivery and requires idempotent handlers. Acknowledging before execution avoids duplicates but risks losing jobs.
Concurrency models and key parameters
A single worker process can execute one job at a time or many in parallel. The right concurrency model depends on whether jobs are limited by CPU or spend most of their time waiting for networks and disks.
| Model | How parallelism is achieved | Best suited for |
|---|---|---|
| Process-based | Several operating system processes per host | CPU-bound work, isolation from crashes |
| Thread-based | Multiple threads in one process | Mixed workloads with shared memory |
| Asynchronous event loop | Many coroutines on one thread | I/O-bound calls to APIs and databases |
| Container per job | A fresh container or function for each job | Strong isolation and bursty load |
Beyond the model, a handful of tunable parameters decide throughput, latency and resilience. Typical values below are starting points that are adjusted after measuring real workloads.
| Parameter | Typical range | Effect of increasing it |
|---|---|---|
| Concurrency per worker | 1 to 50 for CPU work, up to hundreds for I/O | Higher throughput, more memory and contention |
| Prefetch count | 1 to 100 jobs | Less broker latency, less fair distribution |
| Visibility timeout or lease | 30 seconds to several hours | Fewer duplicate deliveries, slower recovery after crashes |
| Job time limit | Seconds to hours | Long jobs allowed, stuck jobs detected later |
| Maximum jobs before restart | Hundreds to thousands | Fewer restarts, more risk from memory leaks |
| Retry count | 3 to 10 attempts with backoff | More resilience, longer delay before failure |
A frequent rule is that the visibility timeout must exceed the longest expected job duration. Otherwise a slow but healthy job is handed to a second worker while the first is still running.
Worker compared with related scheduling concepts
The worker is often confused with the components around it. Each has a distinct responsibility, and keeping them separate allows each part to scale and fail independently.
| Component | Responsibility | Example question it answers |
|---|---|---|
| Worker | Executes jobs | How is this job processed? |
| Job queue | Stores and delivers pending jobs | What work is waiting? |
| Task scheduler | Creates jobs at defined times | When should this job start? |
| Delayed job | A job held back until a future time | When may this job run at the earliest? |
| Job retry | Policy for re-running failed jobs | What happens after a failure? |
In some systems the roles merge: a cron-style scheduler may run jobs directly in its own process. That works for a few light tasks but couples timing to execution, so a slow job can delay every later schedule.
Scaling and operating workers
Workers are usually stateless and horizontally scalable. Capacity grows by starting more worker instances, and autoscaling based on queue depth or oldest job age is more accurate than scaling on CPU alone, because I/O-bound workers can be overloaded with low CPU use.
Operational practice also requires graceful shutdown. When a deployment stops a worker, it should finish or release its current jobs before exiting, since abrupt termination leads to redelivery and, with non-idempotent handlers, duplicate side effects. Separate queues per priority or job type prevent long reports from blocking urgent notifications.
Monitoring focuses on queue length, processing time, failure rate and retries. A steady rise in queue length signals too few workers, while a rise in retries usually points to a failing dependency rather than to the workers themselves.
Common mistakes with workers
Most worker incidents stem from assumptions about exactly-once execution or from shared resources that do not scale with the number of workers.
- Non-idempotent handlers: a redelivered payment or email job runs twice.
- Large payloads in the queue: store files elsewhere and pass only identifiers.
- Unbounded concurrency: dozens of workers overwhelm a database connection pool or a rate-limited API.
- Silent failures: exceptions caught and ignored make jobs look successful.
- No dead letter queue: poison messages are retried forever and block throughput.
Stripped to the essentials
Picture a restaurant. Waiters take orders and pin them on a rail. Cooks in the kitchen pick up each order, prepare it and mark it done. A worker is the cook in this picture.
The application takes requests quickly and leaves the slow work on the rail, which is the queue. Workers handle that work in the background. When the kitchen gets busy, you add more cooks, not more waiters.
Where it is applied
A photo-sharing service receives thousands of uploads each hour. Resizing each image into five formats takes up to two seconds, which would make uploads feel slow if done in the web request. The service stores the original, places a resize job with the image identifier in a queue and returns immediately.
A pool of process-based workers, each with a concurrency of four, picks up the jobs. The visibility timeout is set to five minutes, well above the longest resize, and failed jobs retry three times with backoff before landing in a dead letter queue. During evening peaks, autoscaling adds workers when the oldest job is older than 30 seconds. Uploads stay fast for users, and thumbnails usually appear within seconds.