Sidekiq: Background Jobs in Ruby

Sidekiq is a Ruby background job processor that allows you to move time-consuming tasks out of your request-response cycle. Instead of making users wait for tasks like sending emails, processing images, or calling external APIs, you queue them for background processing. This guide covers everything you need to know to integrate Sidekiq into your Ruby applications.

Why Background Jobs Matter

Web applications often need to perform tasks that take seconds or even minutes to complete. If you handle these synchronously, users experience slow response times and your server threads get blocked. Background jobs solve this by moving these tasks to a separate process that runs independently of the web request.

Sidekiq uses Redis as a message broker to store job data. It processes jobs using worker threads instead of forked processes, making it incredibly efficient. A single Sidekiq process can handle hundreds of jobs per second with minimal memory overhead.

Installing Sidekiq

Add Sidekiq to your Gemfile:

gem 'sidekiq'

Then install the gem:

bundle install

Sidekiq requires Redis to be running. You can start Redis locally or use a hosted service:

redis-server

Creating Your First Worker

Workers are Ruby classes that define the work to be done in the background. Each worker has a perform method that Sidekiq calls when processing the job:

class HardWorker
  include Sidekiq::Worker

  def perform(id, name)
    # Do something expensive
    puts "Processing #{name} with ID #{id}"
    sleep(5)  # Simulate expensive operation
  end
end

The include Sidekiq::Worker module adds the perform_async class method to your worker. This method enqueues the job for background processing:

# Enqueue a job to be processed asynchronously
HardWorker.perform_async(42, "Alice")

# You can also pass additional arguments
HardWorker.perform_async(42, "Alice", { priority: "high" })

Job Execution Options

Sidekiq provides several options to control how jobs execute. You can schedule jobs to run in the future or repeat them at intervals:

# Execute in 10 minutes from now
HardWorker.perform_in(600, 42, "Bob")

# Execute at a specific time
HardWorker.perform_at(1.hour.from_now, 42, "Charlie")

# Repeat every hour, starting now
HardWorker.perform_async(42, "Diana")
# Then use sidekiq-scheduler or sidekiq-cron for recurring jobs

Configuring Sidekiq

Sidekiq configuration goes in an initializer. Here’s a typical configuration for a Rails application:

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.redis = {
    url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1'),
    size: ENV.fetch('SIDEKIQ_CONCURRENCY', 25).to_i
  }
end

Sidekiq.configure_client do |config|
  config.redis = {
    url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1')
  }
end

The concurrency setting controls how many worker threads run simultaneously. Each thread can process one job at a time. Start with 25 threads and adjust based on your workload and server resources.

Queue Management

Sidekiq supports multiple queues with different priorities. By default, jobs go to the “default” queue, but you can specify different queues:

class EmailWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'mailers'

  def perform(user_id, subject, body)
    # Send email
  end
end

Configure queue priorities in your Sidekiq configuration:

# config/sidekiq.yml
:concurrency: 25
:queues:
  - [critical, 3]
  - [default, 2]
  - [mailers, 1]
  - [low, 1]

The numbers indicate priority weight—higher numbers mean higher priority. Jobs in the “critical” queue get processed before “default”, which gets processed before “low”.

Error Handling and Retries

Sidekiq automatically retries failed jobs with exponential backoff. Configure retry options in your worker:

class SensitiveWorker
  include Sidekiq::Worker
  sidekiq_options retry: 5  # Maximum 5 retries
  sidekiq_retry_in do |count|
    10 * (count + 1)  # 10s, 20s, 30s, 40s, 50s
  end

  def perform(data)
    # Work that might fail
  end
end

For jobs that should never retry, use sidekiq_options retry: false:

class NoRetryWorker
  include Sidekiq::Worker
  sidekiq_options retry: false

  def perform(id)
    # Don't retry on failure
  end
end

You can also move failed jobs to a dead queue after all retries are exhausted using the dead queue feature.

Testing Workers

Testing Sidekiq workers requires a different approach than regular unit tests. Use the sidekiq/testing helper to process jobs synchronously:

require 'sidekiq/testing'

describe HardWorker do
  it 'performs the job' do
    HardWorker.perform_async(42, "test")
    
    # Process jobs synchronously
    HardWorker.drain
    
    # Assert the job ran
    expect(HardWorker.jobs.size).to eq(0)
  end
end

For integration tests that actually run jobs asynchronously, use Sidekiq::Testing.inline!:

Sidekiq::Testing.inline!

Starting Sidekiq

Start the Sidekiq process from your application root:

bundle exec sidekiq -C config/sidekiq.yml

In production, use a process manager like systemd or Supervisor to keep Sidekiq running:

bundle exec sidekiq -d -P /tmp/sidekiq.pid -C config/sidekiq.yml

The -d flag daemonizes the process and -P specifies the PID file location.

When to Use Sidekiq

Sidekiq excels when you need to process many jobs quickly, handle long-running tasks without blocking web requests, or scale background processing across multiple workers. It’s particularly well-suited for sending emails, processing file uploads, calling third-party APIs, generating reports, and scheduled maintenance tasks.

When Not to Use Sidekiq

Avoid Sidekiq for very simple tasks that complete in milliseconds—you’ll add unnecessary complexity. If you only need to run jobs occasionally, a simpler solution might suffice. Also, Sidekiq requires Redis, so you need to budget for that infrastructure overhead.