We are happy to announce that RabbitMQ 3.12 is now available on CloudAMQP. This version of RabbitMQ comes with some critical changes in certain user-facing areas and, more prominently, the introduction of native MQTT support in RabbitMQ.
This article will mostly focus on the Native MQTT support in this RabbitMQ release. Additionally, we will also briefly cover some feature highlights, namely:
- Quorum queue performance improvements
- Classic queue memory usage and performance improvements
Native MQTT support in RabbitMQ
AMQP 0.9.1 is RabbitMQ’s core protocol. To support other protocols like MQTT, RabbitMQ accepts messages from the MQTT plugin (in this case). It then forwards these messages to its core protocol, AMQP 0.9.1.
In other words, RabbitMQ had no native support for MQTT; it only proxies MQTT messages via its core protocol. While this approach provides a simple way to extend RabbitMQ beyond AMQP, it has some performance limitations.
Fortunately, this has changed drastically in RabbitMQ 3.12. In this RabbitMQ version, the MQTT and Web MQTT plugins had been rewritten to add native support for the MQTT protocol in RabbitMQ.
With this improvement, the MQTT and Web MQTT plugins parse MQTT messages and send them directly to the appropriate queues as opposed to forwarding the messages to queues via the AMQP 0.9.1 protocol. This, in turn, had led to some substantial performance improvements.
For example:
- A reduction of up to 95% in memory usage and hundreds of GBs with multiple connections.
- Additionally, end-to-end latency has improved by a remarkable 50% - 70%.
- Lastly, there has been a 30% - 40% increase in throughput.
Let’s explore what is responsible for this radical performance improvement.
In the previous versions of RabbitMQ, before 3.12, where the MQTT plugins proxy via the AMQP 0.9.1 protocol, each incoming MQTT connection did spin up a total of 22 Erlang processes. The image below demonstrates this proxying and the number of Erlang processes created - each blue dot represents an Erlang process.
Figure 1 - RabbitMQ MQTT Proxy
RabbitMQ 3.12, on the other hand, only creates 1 Erlang process for each incoming MQTT connection. The image below demonstrates how the MQTT plugins in RabbitMQ 3.12 publish messages directly to a queue without any proxying as well as the single Erlang process created.
Figure 2 - RabbitMQ Native MQTT
While Erlang processes are lightweight and by extension, memory efficient, typically, an MQTT workload involves a large number of Internet of Things (IoT) devices that periodically transmit data to an MQTT broker.
For example, let’s take the case of an IoT setup with 2 million incoming MQTT connections. In a situation like that, it makes a big scalability difference whether RabbitMQ creates 44 million Erlang processes (up to RabbitMQ 3.11) or just 2 million Erlang processes (RabbitMQ 3.12).
In plain terms, the performance improvement in 3.12 comes directly from the reduced requirement for process memory and the work done in routing messages to the appropriate queues.
In addition to the introduction of native support for MQTT in RabbitMQ 3.12, a number of other improvements had been made. Furthermore, RabbitMQ 3.12 also features many internal API improvements in preparation for RabbitMQ 4.0 with Khepri.
We will quickly go over two of these improvements in the subsequent section.
Other improvements in RabbitMQ 3.12
Here, we will quickly review other enhancements made in RabbitMQ 3.12
Quorum queue performance improvements
Before RabbitMQ 3.12, Quorum Queues tended to exhibit a degrading throughput under sustained publisher load without consumers. In situations like that, the queues grow really large, which usually results in a quickly deteriorating throughput.
However, this issue is now resolved in RabbitMQ 3.12. As a result, the Quorum Queues can now sustain higher throughput even with larger backlogs. Additionally, they offer higher average throughput, particularly when the Single-Active Consumer feature is used.
This release also features a reduced memory footprint of quorum queues.
Classic queue memory improvements
RabbitMQ 3.12 also features changes that have significantly reduced the memory footprint and improved the predictability of memory usage, resulting in better throughput for classic queues.
The reduced memory footprint of classic queues is majorly due to one change: in RabbitMQ 3.12, messages published to classic queues will mostly be saved on disk, and only a small part will be stored in memory. The number of messages in memory depends on how quickly they are consumed.
This improvement has particularly benefited classic queues that tend to have longer backlogs. Previously, classic queues with many messages waiting to be processed would experience decreased performance due to high memory usage.
However, this new improvement allows RabbitMQ to handle such queues more efficiently, ensuring better overall performance.
How to upgrade to RabbitMQ 3.12
Upgrading to RabbitMQ 3.12 is a simple process if you already have a CloudAMQP cluster. You can upgrade from the Nodes view in the CloudAMQP Console or via the API.
It is also advisable to keep your Erlang version up to date while upgrading your RabbitMQ version. However, note that for upgrades to RabbitMQ versions > 3.9, the Erlang upgrade will be handled automatically on CloudAMQP. So, you don't need to worry about that.
You can read our blog post on Erlang and RabbitMQ upgrades for more information.
Note to CloudAMQP users
Before upgrading, keep this in mind:
The message timestamp and routing node stamp plugins have been deprecated in 3.12.0 in favor of built in functionality. During the upgrade to 3.12, if any of the two plugins are enabled, necessary configuration will be automatically added to keep the functionality. However, it is not yet possible to enable or disable these new features on CloudAMQP after the upgrade or for newly created 3.12 clusters. We are working on implementing the necessary changes for full support.
Conclusion
RabbitMQ 3.12 has seen a number of improvements in certain user-facing areas and even its internal APIs in preparation for RabbitMQ 4.0 with Khepri. However, the most notable of these changes is native MQTT support, which inherently makes RabbitMQ and MQTT broker - opening doors for a diverse range of possibilities for IOT with RabbitMQ.
Do you want to test out MQTT in RabbitMQ? Upgrade your instance now from the console.
You do not have a Cloud account? You can easily signup on CloudAMQP, create a RabbitMQ instance and conveniently test out MQTT without all the hassle of installing and setting up your own RabbitMQ instance.
For any suggestions, questions, or feedback, get in touch with us at contact@cloudamqp.com
