Job priority

On this page 7

Job priority is a value attached to a background job that tells the scheduler which work to run first when demand exceeds available capacity. In scheduling for automation systems, priority determines the order in which jobs leave a queue and reach a worker, so that time-critical tasks are not delayed by bulk or routine processing.

Priority only matters under contention. When workers are idle, every job starts immediately regardless of its level; the setting becomes decisive during peaks, backlogs and outages, which is exactly when getting the order wrong is most expensive.

How schedulers apply job priority

A scheduler applies job priority at the moment it selects the next job for a free worker. The two dominant implementations are a single queue ordered by a numeric priority field and a set of separate queues per priority level that workers poll in a defined order.

With a numeric priority field, the queue behaves like a priority heap: the job with the highest or lowest value, depending on the convention, is dequeued first, and ties are broken by enqueue time. With multiple named queues, a worker checks the critical queue, then the default queue, then the low queue, either strictly or with weights.

The following table shows how widely used tools express priority, because conventions differ and are a frequent source of configuration mistakes.

SystemPriority modelDirection
Celery with RabbitMQNumeric message priority 0 to 255, often capped at 10Higher value runs first
SidekiqNamed queues with weights or strict orderListed or heavier queue first
Linux nice valueInteger from -20 to 19Lower value runs first
Kubernetes PriorityClass32-bit integer per pod classHigher value is scheduled first
Database-backed queuesInteger column in the job tableDefined by the query ordering
Priority conventions in common job systems

Because the direction is not universal, teams should document the convention once and validate it in tests. Inverting it silently pushes the most urgent jobs to the back of the queue.

Strict versus weighted priority

The choice between strict and weighted priority decides whether low-priority work can ever be delayed indefinitely. Strict ordering always serves the higher level first, while weighted ordering gives each level a proportional share of worker attention.

StrategyBehavior under loadMain riskTypical use
Strict priorityLower queues wait until higher ones are emptyStarvation of low jobsSmall number of truly urgent jobs
Weighted priorityEach queue served in proportion to its weightUrgent jobs still share capacityMixed workloads with steady traffic
Priority with agingWaiting jobs gain priority over timeMore complex tuningLong backlogs that must eventually clear
Dedicated worker poolsEach level has reserved workersIdle capacity at quiet timesHard latency targets for one class
Priority scheduling strategies

Starvation is the defining failure of strict priority. If a steady stream of high-priority jobs never lets the queue empty, reports, cleanups and backfills at the lowest level never run. Aging, which raises the effective priority of a job the longer it waits, is the classic countermeasure.

Job priority controls ordering, not capacity, timing or failure handling, and it works best combined with neighboring controls rather than as a substitute for them.

  • A concurrency limit caps how many jobs of a type run at once; priority decides which ones fill those slots.
  • A delayed job is not eligible until a set time; priority applies only after that time has passed.
  • A job dependency blocks a job until its prerequisites finish, regardless of its priority level.
  • A job timeout stops a job that runs too long, which keeps a stuck high-priority job from occupying a worker forever.

A frequent design error is using priority to solve a capacity problem. If the queue is permanently backlogged, raising priorities only reshuffles who waits; adding workers or reducing work is the real fix.

Designing priority levels in practice

Effective designs use few, clearly defined levels, typically three to five, each tied to a business consequence rather than to the opinion of the team that wrote the job. Too many levels make the ordering hard to reason about and invite priority inflation, where every team marks its jobs as urgent.

The decision table below maps common job categories to sensible levels.

Job typeSuggested levelReason
Password reset or login emailCriticalA user is actively waiting
Payment webhook processingHighDelays affect revenue and reconciliation
Thumbnail and search indexingDefaultVisible but tolerant of seconds of delay
Nightly reports and exportsLowDeadline is hours away
Historical backfillLowestShould use only spare capacity
Assigning priority to typical jobs

Teams should also measure queue latency per level, meaning the time between enqueue and start. A rising latency in the critical level is an early warning that the priority scheme or the worker pool no longer matches the load.

Common mistakes with job priority

Most problems with job priority come from misconfiguration and organizational drift rather than from scheduler bugs. The table lists frequent symptoms and their remedies.

SymptomCauseFix
Urgent jobs run lastInverted priority directionVerify convention with an automated test
Reports never finishStarvation under strict priorityAdd weights, aging or a reserved worker
Priority has no visible effectBroker prefetch fills workers earlyLower prefetch to one or a few jobs
Most jobs marked criticalPriority inflation across teamsTie levels to documented criteria
Priority problems and fixes

Prefetching deserves special attention. Many brokers deliver several messages to a worker in advance for throughput, and once a low-priority job sits in a worker buffer, a newly arrived critical job cannot overtake it. Setting a low prefetch count is often required for priority to work as intended.

The idea in brief

Job priority is like the triage desk in a hospital emergency room. Patients do not always see a doctor in arrival order; the most urgent cases go first, and minor injuries wait.

A computer system does the same with its background work. When there is more work than workers, the scheduler looks at each job’s priority and picks the most important one. Good priority rules keep urgent work fast while still making sure the less urgent jobs eventually get done.

A practical scenario

An online retailer runs all background work through one job queue served by 20 workers. During a holiday sale, a nightly catalog export of 50,000 products starts at the same moment as a surge of orders. Order confirmation emails begin arriving up to 15 minutes late, and customers contact support asking whether their orders went through.

The engineering team splits the work into three queues: critical for confirmations and password resets, default for order processing, and low for exports and backfills. They configure weighted polling at 6, 3 and 1, reduce broker prefetch to one job and reserve four workers exclusively for the critical queue. At the next sale, confirmation emails leave within seconds, while the catalog export still finishes before morning using the remaining capacity.