Sidekiq

Sidekiq is a popular background job processing library for Ruby applications, and it is used in Mastodon for various purposes:

• Federated content delivery
• Push notifications
• Media processing
• Scheduled tasks

By using Sidekiq, Mastodon can handle multiple tasks concurrently and efficiently, improving the performance and responsiveness of the platform. This is crucial for providing a good user experience, especially considering the federated and decentralized nature of Mastodon.

In the vmst.io environment, Sidekiq processes nearly two million tasks per day.

Sidekiq runs on the DigitalOcean managed Kubernetes platform which automatically scales the number of required resources.

Tuning

Bootstrapping a Mastodon server involves 30% community management, 20% general server management, and 50% determining what has angered Sidekiq.

By default, deploying Mastodon from its source generates a single Sidekiq service with 25 threads. Each active thread can potentially establish a connection to the PostgreSQL database or perform other tasks, such as sending email notifications, fetching remote images, or notifying other servers of user posts. This behavior is somewhat unpredictable.

Two values must be managed for Sidekiq:

• The threads assigned to the service, which can be configured in the .service file or the docker-compose.yml file, depending on the deployment type.
• The DB_POOL variable, which sets the maximum number of open connections the service can have to the database (bearing in mind that not every thread will open a connection).

It is best practice for these values to be the same for each Sidekiq service. By default, there is only one Sidekiq service.

25 threads may suffice for an server with a few hundred active users or a larger server under light load. However, a single popular toot going viral can quickly cause queues to back up and timelines to stop updating until the backlog is processed.

Moreover, when one server struggles, it can cause delays in other servers within the federation, leading to "red light" errors and other issues.

One solution could be to add more threads to your service, but this may result in additional database connections. PostgreSQL has limits based on the deployment size, which can be addressed by using PgBouncer as part of the deployment.

It might be tempting to increase the number of threads in your Sidekiq service file from 25 to 50, 100, etc. However, this is not the correct approach. Sidekiq does not utilize more than 25-30 threads generated by a single service; you will simply consume CPU cycles and waste potential database connections.

The solution is to spawn more Sidekiq workers by creating additional systemd services or Docker containers dedicated to Sidekiq. Each service should be limited to the queue(s) you want them to process in the order of their priority.

Example mastodon-sidekiq-push-pull.service configuration file:

...
Environment="RAILS_ENV=production"
Environment="DB_POOL=25"
Environment="MALLOC_ARENA_MAX=2"
Environment="LD_PRELOAD=libjemalloc.so"
ExecStart=/root/.rbenv/shims/bundle exec sidekiq -c 25 -q push -q pull
...

As configured this service fill will only process the push and the pull queues shown with the -q options, in the order listed. It also has the maximum of open database connections of 25 set in DB_POOL with 25 threads set in the -c option.

An explanation for the purpose of each queue can be found on docs.joinmastodon.org.

Queue Design