Sidekiq

On this page 7

Sidekiq is an open-source background job processing framework for Ruby that stores jobs in Redis and executes them concurrently on multiple threads inside a single process. In scheduling, it is the component that takes work out of the web request cycle, runs it later or at a specific time, and retries failed jobs automatically.

Within the wider field of Automation, Sidekiq is one of the most widely used job queue systems in the Ruby on Rails ecosystem. It is distributed as a free open-source edition plus the commercial Sidekiq Pro and Sidekiq Enterprise editions, which add reliability, batching and periodic scheduling features.

How Sidekiq processes jobs

Sidekiq works on a simple producer and consumer model. Application code, the producer, calls a method such as perform_async on a job class; Sidekiq serializes the class name and arguments to JSON and pushes the payload onto a named list in Redis. Separate Sidekiq server processes, the consumers, pull payloads from those lists and call the job’s perform method.

Each server process runs a configurable pool of worker threads. Because Ruby threads share memory, one process with ten threads uses far less RAM than ten separate single-threaded processes. The trade-off is that job code must be thread-safe, and CPU-heavy work is limited by the Global VM Lock in the standard MRI interpreter, so Sidekiq shines mainly on I/O-bound tasks such as API calls, email delivery and database writes.

The main moving parts of a Sidekiq deployment are summarized below.

ComponentRoleWhere it lives
ClientSerializes and enqueues jobsInside the application process
RedisStores queues, schedules, retries and statisticsSeparate data store
Server processFetches jobs and manages worker threadsDedicated process or container
Worker threadExecutes one job at a timeInside a server process
Scheduler pollerMoves due delayed jobs into their queuesInside each server process
Web UIShows queues, retries and dead jobsMounted in the web application
Core components of a Sidekiq setup

Since version 7, job classes include Sidekiq::Job, which replaced the older Sidekiq::Worker name, and the same version introduced capsules that let one process run separate thread pools for different sets of queues.

Scheduling, retries and job lifecycle

Sidekiq supports three timing modes: immediate execution, delayed execution and recurring execution. Delayed jobs created with perform_in or perform_at are stored in a Redis sorted set scored by timestamp, and a poller in each server process periodically moves due jobs into their target queue. Delivery is therefore approximate, typically within seconds of the requested time rather than to the millisecond.

Recurring work is not part of the open-source core. Teams either use the periodic jobs feature of Sidekiq Enterprise, which accepts cron expressions, or a community extension that adds cron-style scheduling. The table compares the timing options.

ModeTypical callTypical use
Immediateperform_asyncSend a confirmation email after signup
Delayedperform_in(1.hour)Remind a user about an abandoned cart
At a set timeperform_at(timestamp)Publish a post at a planned moment
RecurringCron expression in Enterprise or an extensionNightly report or hourly cleanup
Timing options available with Sidekiq

When a job raises an exception, Sidekiq places it in the retry set and tries again with exponential backoff. By default a job is retried up to 25 times over roughly three weeks; after the last attempt it moves to the dead set, where it stays for a limited period and can be retried manually from the Web UI. Retry counts can be lowered per job class, and setting retries to zero sends failures straight to the dead set or discards them.

Configuration parameters that matter

A small number of settings determine most of Sidekiq’s behavior in production. Getting them right is more important than any code-level optimization.

SettingTypical valueWhat it affects
Concurrency5 by default, often 10 to 25Threads per process and database pool size
Queues and weightscritical, default, lowOrder in which queues are polled
Retry count25 by defaultHow long a failing job keeps trying
Redis connection poolConcurrency plus a few spareAvoids threads waiting for Redis
Shutdown timeoutAbout 25 secondsTime for jobs to finish during deploys
Key Sidekiq settings and their effects

Queue priority can be strict or weighted. Strict ordering always drains the first queue before the next, which can starve low-priority queues; weighted ordering samples queues in proportion to their weights. A common pattern is to run separate processes for critical queues so that slow jobs elsewhere cannot delay them.

Sidekiq compared with other job processors

Sidekiq is one of several job queue tools, and its position is defined by its threaded Ruby execution and Redis storage. The comparison below places it next to frequently evaluated alternatives.

ToolLanguageBacking storeConcurrency model
SidekiqRubyRedisThreads per process
ResqueRubyRedisForked process per job
Delayed JobRubyRelational databaseSingle-threaded processes
Solid QueueRubyRelational databaseThreads and processes
CeleryPythonBroker such as RabbitMQ or RedisProcesses, threads or green threads
Sidekiq and related job processing tools

Database-backed queues avoid running Redis and allow enqueuing inside the same database transaction as the business data, while Sidekiq usually offers higher throughput and lower latency. Rails applications can also use Sidekiq through the Active Job abstraction, at the cost of some Sidekiq-specific options.

Reliability limits and common mistakes

The open-source edition fetches jobs in a way that can lose in-flight jobs if a process crashes abruptly, for example after an out-of-memory kill. Sidekiq Pro adds a reliable fetch strategy that keeps in-progress jobs recoverable. Regardless of edition, Sidekiq guarantees at-least-once execution, so jobs may run more than once.

  • Write idempotent jobs: a retried payment or email job must check whether its work was already done.
  • Pass identifiers, not objects: arguments must be simple JSON types, and passing a record ID avoids stale data.
  • Match the database pool to concurrency: too few connections cause timeout errors under load.
  • Configure Redis eviction correctly: a policy that evicts keys can silently delete queued jobs.
  • Avoid enqueuing before a transaction commits: the job may run before the data it needs exists.

Put simply

Sidekiq is like a to-do list with a team of helpers behind a web application. When the application has a slow chore, such as sending an email or creating a report, it writes the chore on the list and immediately gets back to serving the user.

The helpers pick chores from the list and do them in the background, several at once. Chores can also be scheduled for later. If a chore fails, the helpers try again automatically after waiting a little longer each time.

In practice

An online ticketing service built on Rails sells tickets for a concert, and 40,000 orders arrive within one hour. Generating a PDF ticket and sending a confirmation email takes about two seconds per order, which would make checkout unbearably slow if done during the web request.

Instead, the checkout code enqueues a Sidekiq job with only the order ID. Four server processes with 20 threads each work through the queue, while a separate process handles a critical queue for payment webhooks. A reminder job is scheduled with perform_at for 24 hours before the show. When the email provider briefly returns errors, the affected jobs enter the retry set and succeed a few minutes later.

The result is a checkout response under one second and every confirmation delivered without manual intervention.