Handling Millions of Discord Messages: CactusFire & Memphis.dev

This article was co-written together with LH the founder of CactusFire.

About Cactusfire

CactusFire is a multifunctional Discord bot that has grown steadily over the years and currently boasts over 250,000 servers and more than 25 million users. The bot has a wide range of features, including moderation tools, music playback, server management, and more.
CactusFire’s growth and feature-rich nature have made it an invaluable asset for Discord communities seeking a comprehensive and reliable bot.

The use case

The team was faced with a challenge regarding the CactusFire gateway. Previously, the bot processed all messages sent by Discord, resulting in the need to handle over 20 million events daily. Initially, a straightforward gateway was used to process the events as raw data and transmit them via WS. However, this approach proved unsustainable in the long term.

The requirements

As the project grew and more services were added, a lot of inter-service communication needed to occur.
It was important for us that the chosen solution will have the following capabilities:

• A central point where messages can be exchanged between services
• Load balancing
• Message persistence
• Handling high levels of traffic without becoming overwhelmed
• Good performance
• Easy deployment

Choosing Memphis

Using a message broker can help to make large projects more scalable, resilient, and easier to manage as they grow and become more complex, and it answers all our requirements above. We were exploring different MQ/MB options, including Kafka (which never really worked for us) and RabbitMQ (with a difficult deployment). However, after carefully evaluating our use case needs, we found that Memphis was the best choice for the following reasons:

First, Memphis, as an event processing platform, is highly scalable and can handle large volumes of real-time data with very low latency. This is crucial for the Cactusfire use case, as we needed to ensure that the system was able to process large amounts of data efficiently and without delay.

Secondly, Memphis offers great flexibility regarding how messages can be sent and received. This means that we could easily customize the format and content of messages according to the application’s needs, allowing us to work more efficiently.

Another good feature is the “dead-letter” feature that ensures messages are not lost if a service goes down, allowing them to be processed once the service comes back online.

Additionally, we liked to have a visual panel where we could see in real-time the use we were giving it.

Last but not least, we got fast support from the technical team on their discord server.

In summary, Cactusfire is steadily establishing itself as the de facto standard among Discord bots, continuously growing with each passing day. while there are many MQ/MB options available, I chose Memphis because it is highly scalable, flexible, and easy to use and deploy, making it perfect for our use case.