Sidekiq and Redis in Ruby-on-Rails

Handle background processing and job queuing

What is Sidekiq

• Background job processing in Rails applications, designed to help manage worker processes asynchronously
• Queue up web requests as a worker process (a worker is a unit that runs concurrent code), and handle a large number of jobs concurrently by using multiple threads.

Why Sidekiq

• Particularly useful for offloading time-consuming tasks, such as sending emails, processing images, or performing background computations
• Used to improve the responsiveness, and prevent blocking the main app, it can process multiple jobs concurrently and schedule jobs to run at a later time

Configuration

Create an initializer file for Sidekiq (e.g., config/initializers/sidekiq.rb):

Sidekiq.configure_server do |config|
  config.redis = { url: ENV["REDIS_URL"] || "redis://localhost:6379/0" }
end

Sidekiq.configure_client do |config|
  config.redis = { url: ENV["REDIS_URL"] || "redis://localhost:6379/0" }
end

Redis

What is Redis

• Key value pairs that use in-memory for storing data as a database (cache) to give high-performance, single-threaded
• When you enqueue a job in Sidekiq, Redis is used as the storage backend to manage job queues.

Redis Connection

config.redis - where the connection of sidekiq and redis is happening

$redis = Redis.new(url: ENV["REDIS_URL"] || "redis://localhost:6379/0")

Job Lifecycle

Enqueuing Jobs → Processing Jobs → Retries

Creating a Worker

Create a worker class that defines the background job. Workers are classes that include the Sidekiq::Worker module and a perform method

# app/workers/my_worker.rb
class MyWorker
  include Sidekiq::Worker

  def perform(*args)
    # Perform your background job here
    puts "Performing job with arguments: #{args}"
  end
end

Sidekiq Job Enqueue

In Sidekiq, jobs are enqueued using the perform_async method, which is part of the Sidekiq API provided in Sidekiq::Worker module

# app/controllers/my_controller.rb
class MyController < ApplicationController
  def create
    # Enqueue a job
    MyWorker.perform_async('arg1', 'arg2')

    render plain: "Job has been enqueued"
  end
end

For example, we call the cron controller to perform_sync the worker, then introduce a worker with a cronjob to pull data from an external API and store it in our database.

Sidekiq Job Process

Retrieve jobs from the Redis queue, and execute the perform method in the worker class

Sidekiq Job Retries

Sidekiq has built-in support for handling job failures and automatic retries. By default, Sidekiq will automatically retry a failed job.

There is an option for how many times you can retry, where you can customize retries by using sidekiq_options, searching “sidekiq_options retry”

Running Sidekiq

Sidekiq API provides access to real-time information about workers, queues, and jobs. The API allows you to perform actions like clearing queues, deleting specific jobs, and retrieving job details.

bundle exec sidekiq is a command used to start the Sidekiq background job processing system in a Ruby on Rails application. Sidekiq is a popular gem for handling background jobs in Ruby applications.

# Access Local Dashboard
# config/routes.rb
http://localhost:{port}/sidekiq
http://localhost:{port}/sidekiq/queues

More Information

Sidekiq

Sidekiq and Active Job both are related to background job processing in Rails applications, both are designed to be processed asynchronously. This is particularly useful for offloading time-consuming tasks, such as sending emails, processing images, or performing background computations, to improve the responsiveness of web applications and prevent them from blocking the main application.

Sidekiq is a Ruby library that helps manage worker processes in Rails applications.

Active Job is a framework for declaring jobs and making them run on a variety of queuing backends.

Redis

Currently, Sidekiq pro has a mechanism to reduce whenever Redis is down, it will save lost jobs in RAM that allows storing the last 1000 pushing jobs on local memory in case the connection between the Sidekiq client and Redis server is lost.

In case Redis is down, when the Sidekiq client adds a job, the job will be added to a client queue. After Redis is back, when the Sidekiq client adds a new job, it also adds all jobs in the client queue to Redis. This avoids job loss. Extended pushing jobs will be lost if there are more than 1000 pushing jobs.

To know if a job is unique or not, Sidekiq-unique-job needs to check the Redis queue. When Redis is down, the sidekiq-unique-job throws exceptions. However, if the Redis is down, unique jobs will be lost.

The Schedule Queue on Sidekiq is based on a nonrelational database (redis) which has to look through the whole list to find what to run (which is the issue moving everything to a cronjob would be solving for scalability). So, storing the execution time in a relational database where a filtering select statement shouldn’t take long.

Overall, redis uses memory for storing data to give high performance. As long as the memory is big enough, we will perform well.

Recap

• A job is a unit of work in your Ruby application
• A queue is a list of jobs that are ready to execute right now
• A process is a Sidekiq process with one or more threads for executing jobs.
• Queue is one part of Sidekiq “processes” that are running
• Queue up web requests as a worker process