Sidekiq is a popular background job processing library for Ruby applications, and it is used in Mastodon for various purposes:
- Federated content delivery
- Push notifications
- Media processing
- Scheduled tasks
By using Sidekiq, Mastodon can handle multiple tasks concurrently and efficiently, improving the performance and responsiveness of the platform. This is crucial for providing a good user experience, especially considering the federated and decentralized nature of Mastodon.
In the vmst.io environment, Sidekiq processes nearly two million tasks per day.
Sidekiq runs on the Digital Ocean managed Kubernetes platform which automatically scales the number of required resources.
Bootstrapping a Mastodon instance involves 30% community management, 20% general server management, and 50% determining what has angered Sidekiq.
By default, deploying Mastodon from its source generates a single Sidekiq service with 25 threads. Each active thread can potentially establish a connection to the PostgreSQL database or perform other tasks, such as sending email notifications, fetching remote images, or notifying other servers of user posts. This behavior is somewhat unpredictable.
Two values must be managed for Sidekiq:
- The threads assigned to the service, which can be configured in the
.servicefile or the
docker-compose.ymlfile, depending on the deployment type.
DB_POOLvariable, which sets the maximum number of open connections the service can have to the database (bearing in mind that not every thread will open a connection).
It is best practice for these values to be the same for each Sidekiq instance. By default, there is only one Sidekiq instance.
25 threads may suffice for an instance with a few hundred active users or a larger instance under light load. However, a single popular toot going viral can quickly cause queues to back up and timelines to stop updating until the backlog is processed.
Moreover, when one instance struggles, it can cause delays in other instances within the federation, leading to "red light" errors and other issues.
One solution could be to add more threads to your service, but this may result in additional database connections. PostgreSQL has limits based on the deployment size, which can be addressed by using pgBouncer as part of the deployment.
It might be tempting to increase the number of threads in your Sidekiq service file from 25 to 50, 100, etc. However, this is not the correct approach. Sidekiq does not utilize more than 25-30 threads generated by a single service; you will simply consume CPU cycles and waste potential database connections.
The solution is to spawn more Sidekiq workers by creating additional
systemd services or Docker containers dedicated to Sidekiq.
Each service should be limited to the queue(s) you want them to process in the order of their priority.
mastodon-sidekiq-push-pull.service configuration file:
... Environment="RAILS_ENV=production" Environment="DB_POOL=25" Environment="MALLOC_ARENA_MAX=2" Environment="LD_PRELOAD=libjemalloc.so" ExecStart=/root/.rbenv/shims/bundle exec sidekiq -c 25 -q push -q pull ...
As configured this service fill will only process the push and the pull queues shown with the
-q options, in the order listed.
It also has the maximum of open database connections of 25 set in
DB_POOL with 25 threads set in the
An explanation for the purpose of each queue can be found on docs.joinmastodon.org.