Bitnami RabbitMQ cluster: searching for the maximum performance

Published on March 23, 2018
Raquel Campuzano Godoy

Raquel Campuzano Godoy

Passionated about IT, fencing, and books. Content Manager @bitnami.

Share this article

RabbitMQ is a well-known open source message broker. We can define it as a general purpose messaging solution for enterprises, originally designed to implement the Advanced Message Queuing Protocol (AMQP) standard, but it now incorporating a wide variety of messaging protocols.

It is considered one of the two most popular open source solutions, the other being Kafka, for sharing communication between different applications. While Apache Kafka excels in real-time event storage and sourcing, RabbitMQ is perfect for application communication, both synchronous and asynchronous.

If your current RabbitMQ deployment is suffering from performance issues as a result of to the increasing number of messages your server needs to process, you should take the leap from single to multi-tier virtual machines.

Bitnami has configured a default three-node RabbitMQ cluster to cover your scalability needs. Deploy it either on the Google Cloud Platform or the Microsoft Azure.

The clustering approach: a safe bet to ensure maximum performance of RabbitMQ

There are four ways to make the RabbitMQ cluster distributed: clustering, Queue mirroring, and using the Federation or the Shovel plugins. The Bitnami Multi-tier solution for RabbitMQ was created by adopting the clustering approach. It is recommended for scenarios where you have machines located in a single data center which need to prioritize the high performance of the cluster over high availability.

By default, Bitnami approach implies having the content of a queue on a single node, although this data is reachable from any of the cluster nodes. To avoid a decline in cluster performance, Bitnami has decided not to include the queue mirroring option in the current configuration of its solution.

When the environment is more complex, and you have more than one broker in different administrative domains, (that, for example, requires different users, data centers, or versions of RabbitMQ or Erlang) the recommendation is to connect all of them using the Federation or the Shovel plugin, which works at a lower level.

Benefits of clustering in the Bitnami Multi-Tier solution for RabbitMQ

The Bitnami RabbitMQ cluster is composed of multiple nodes that constitute a single logical broker. The solution is configured with three nodes by default, but the number of nodes can be changed at deployment time. The image below shows, in the RabbitMQ Management Panel, the state of a six-node RabbitMQ broker. A message queue has been created in node-0:

cluster status

A RabbitMQ broker (also known just as "cluster") is a logical pooling of several Erlang nodes. Each of them runs the RabbitMQ application, and they share users, virtualhosts, queues, exchanges, bindings, and runtime parameters.

The benefits of this solution can be attributed to the way of how is the cluster is configured (including the specific tweaks made by the Bitnami team):

  • Simple horizontal scalability
  • High performance
  • Complete visibility of the messages queues at any node

Simple horizontal scalability

In the case of RabbitMQ, when your server is overwhelmed by receiving a high number of requests per second, the amount of messages in the queue makes the solution unable process all of them without decreasing response time. Worse than that, messages could be even lost. So if you want to respond to those requests quickly, it's essential to balance excess loads across different servers.

To easily respond to increased demand, you can simply add more nodes (learn how to add nodes to RabbitMQ clusters running on the Google Cloud Platform or on Microsoft Azure). With just three to seven nodes, you can achieve excellent cluster performance.

In RabbitMQ, there are two types of nodes: disk or RAM nodes. RAM nodes store internal database tables in volatile RAM memory. In almost all cases, having disk nodes is the preferable option, so as to achieve data persistence. For that reason, Bitnami configures the whole broker with disk nodes with their disk free limit set to mem_relative, 1.0.

In any case, you can change the node type by stopping the node and executing the rabbitmqctl change_cluster_node_type ram command. Take a look at our detailed instructions for Google Cloud Platform and Microsoft Azure.

High performance

Bitnami decided to pick the clustering approach because it ensures proficient performance and increases the throughput. For that reason, the Bitnami RabbitMQ cluster doesn't use mirror queues, locating the queues in a single node instead.

Mirror queues can reduce the speed of the cluster and make the system difficult to manage because they generate a lot of traffic when copying multiple queues across all the nodes.

Complete visibility of the message queues at any node

Message queues reside on one node by default but are visible and reachable from all nodes. That way, it's possible to see the queues on all nodes if a client connects to any node (even when the queues are not located in the node you are connected).

In the Federation/Shovel approach, a client connected to a broker will only see the queues stored in that broker. You can also start or stop the nodes as needed; you just need to ensure that the node can contact another cluster member at the time of shutdown.

You can see the status of all nodes in two ways:

  • Accessing the RabbitMQ Management Panel though the opened port 15672 by browsing to http://SERVER-IP:15672/.
  • Accessing the main server host through SSH and executing the following command using the RabbitMQ CLI:

    $ sudo rabbitmqctl cluster_status

In both cases, you will see a list of cluster nodes and their current status.

Authentication and network configuration in the Bitnami RabbitMQ cluster

The authentication method consists of a username and password. If you deploy the solution on the Google Cloud platform, that password will be randomly assigned. You can change it at any time by following these instructions.

As many other Bitnami solutions, RabbitMQ is only accessible from the same network due to security reasons. Thus, ports 5672 and 15672 (RabbitMQ Management Panel, this port is only enabled in the main node) are opened but only for access from the internal network.

To connect to RabbitMQ from a different machine in the same network, you can use the following command:

$ rabbitmqadmin -H SERVER_IP -u USER_NAME -p PASSWORD list vhost

We discourage connecting to the cluster from a different network; instead, we strongly recommend you peer both virtual networks to secure the connections between the two instances. The only port opened for remote connections is port 22 (SSH).

If you don't know yet if the Bitnami RabbitMQ cluster can address your needs, why not give it a try? Launch it now in your preferred cloud platform without any extra charge!