As an organization, we have been working with RabbitMQ for a long time and have seen more configuration mistakes than most. We know how to configure for optimal performance and how to get the most stable cluster. With this series, we are going to share all this knowledge so you can follow the best practices with your own RabbitMQ platform. Please read Part 1 for general best practice and dos and don’ts tips for RabbtitMQ.
Keep your queue short (if possible)
To get optimal performance, keep queues as short as possible. Longer queues require more processing overhead. We recommend that queues always stay around 0 for optimal performance.
Set a queue max-length if needed
If your application often gets hit by spikes of messages we recommend setting a max-length on the queue. This keeps the queue short by discarding messages from the head of the queue so that it never gets larger than the max-length setting.
Remove the policy for lazy queues
CloudAMQP enables lazy queues by default. Lazy queues are queues where the messages are automatically stored to disk, thereby minimizing the RAM usage, but extending the throughput time. Messages are only loaded into memory when they are needed.
Use transient messages
Persistent messages are written to disk as soon as they reach the queue, which affects throughput. Use transient messages for the fastest throughput.
Use multiple queues and consumers
Queues are single-threaded in RabbitMQ, and one queue can handle up to about 50 thousand messages. You will achieve better throughput on a multi-core system if you have multiple queues and consumers and if you have as many queues as cores on the underlying node(s).
The RabbitMQ management interface collects and calculates metrics for every queue in the cluster. This might slow down the server if you have thousands upon thousands of active queues and consumers. The CPU and RAM usage may also be affected negatively if you have too many queues.
Split your queues over different cores
Queue performance is limited to one CPU core. You will, therefore, get better performance if you split your queues into different cores, and into different nodes, if you have a RabbitMQ cluster.
RabbitMQ queues are bound to the node where they were first declared. Even if you create a cluster of RabbitMQ brokers, all messages routed to a specific queue will go to the node where that queue lives. You can manually split your queues evenly between nodes, but; the downside is remembering where your queue is located.
We recommend two plugins that will help you if you have multiple nodes or a single node cluster with multiple cores:
Consistent hash exchange plugin
The consistent hash exchange plugin allows you to use an exchange to load-balance messages between queues. Messages sent to the exchange are consistently and equally distributed across many queues, based on the routing key of the message. The plugin creates a hash of the routing key and spreads the messages out between queues that have a binding to that exchange. It could quickly become problematically to do this manually, without adding too much information about numbers of queues and their bindings into the publisher.
The consistent hash exchange plugin can be used if you need to get the maximum use of many cores in your cluster. Note that it’s important to consume from all queues. Read more about the consistent hash exchange plugin.
RabbitMQ sharding The RabbitMQ sharding plugin does the partitioning of queues automatically; i.e., once you define an exchange as sharded, then the supporting queues are automatically created on every cluster node and messages are sharded accordingly. RabbitMQ sharding shows one queue to the consumer, but in reality, it consists of many queues running in the background. The RabbitMQ sharding plugin gives you a centralized place where you can send your messages, plus load balancing across many nodes, by adding queues to the other nodes in the cluster. Read more about RabbitMQ Sharding.
Disable manual acks and publish confirms
Acknowledgment and publish confirms have a performance impact; for the fastest possible throughput, manual acks should be disabled.
Avoid multiple nodes (HA)
One node will give you the highest throughput compared to an HA cluster setup. Messages and queues are not mirrored to other nodes.
Disable plugins you are not using
Some plugins might be great, but they also consume a lot of CPU or may use a high amount of RAM. Because of this, they are not recommended for a production server. Disable plugins that are not in use. Use the control panel in CloudAMQP to enable - or disable - plugins.
