The wait is over! The next major version for RabbitMQ is now available on CloudAMQP. 4.0 brings several changes and improvements to RabbitMQ. For example quorum queue message priority and the removal of classic queue mirroring are just a few of the enhancements. We are looking forward to the steady adoption of Quorum Queues and Streams. In this blogpost we will look into what this version brings...
Classic Queues undergo big changes in 4.0
Queue storage version 1 has been removed in 4.0. However, queues can still be migrated from v1 to v2. Queue storage version 2 is more efficient than v1. Storage version 2 was originally introduced in version 3.10.0, and we have recommended its use ever since.
HA
Classic queue mirroring is not available in version 4.0. Classic queues are now a non-replicated queue type. Users will be unable to create a high availability policy going forward. HA will be removed from classic queues to upgrade to 4.0 as well as new clusters to prepare for 4.0. If you rely on classic queue mirroring today, it is best to migrate to quorum queues. For migrating from classic to quorum queues check our blog post.
Khepri: fully supported but not enforced
RabbitMQ's switch from Mnesia DB to Khepri for metadata persistence has been discussed as a 4.0 happening. However, even if Khepri is fully supported in 4.0, Mnesia DB is not totally replaced just yet.
The real-time database Mnesia, written in Erlang, will continue to be used in 4.0. The introduction of Khepri as the default metadata store will have to wait until version 4.1. Mnesia is projected to be removed in version 4.2. Although, it is fully supported in version 4.0, testing before it is enforced in later versions is still encouraged by the RabbitMQ core team.
The key differences between these two is that Khepri is a distributed database store that uses the Raft consensus algorithm, designed for RabbitMQ by RabbitMQ. Mnesia, on the other hand is a general purpose internal database used by RabbitMQ. Want to learn more about Khepri? Check out our blog.
New exchange type: Local Random Exchange
New exchange types are rare, but we've written extensively about the existing ones to help you set up your system effectively. With 4.0, a new exchange type is introduced: Local random exchange.
With the random local exchange, a message is always delivered to a local queue. A queue residing on a single node. With multiple queues bound to the exchange, a random queue is picked for the message to be delivered. This new exchange type is Intended to be used with exclusive queues.
Quorum queues version 4
Safe just got safer. Quorum queues in RabbitMQ replicates data over different nodes using the Raft consensus algorithm. With this release, Quorum Queues are upgraded to version 4. A more behind-the-scenes upgrade improves Quorum queues with a reliability update. Most users will just enjoy a more stable Quorum queue.
With this version, message priority has been made available for Quorum queues. It’s important to note that it works differently than message priority does in classic queues. In Quorum Queues, priority messages are delivered in a 2:1 ratio to normal messages, allowing normal and priority consumption. This means that normal messages won’t get left behind in the queue. Normal message consumption in Quorum Queues follow a first in first out pattern.
In classic queues, messages with high-priority are always prioritized first, potentially leaving normal messages in the queue indefinitely. The updated priority system for Quorum Queues has two levels: normal and high - which matches the AMQP 1.0 specification for message priority. Any value greater than 4 is a high priority. Publishers that do not specific message priority are defaulted to 4. The image below illustrates the 2:1 ratio for priority and normal messages consumption.
Figure 1 - Quorum Queue priority messages
Delivery limit default to 20
Quorum queues now have a default redelivery limit of 20 in 4.0. In previous versions it was infinite, leading to unnecessary resource consumption. By default, messages that reach this limit will be dropped from the queue. A dead letter exchange should be used to retain these messages. The limit can be overwritten with a policy argument delivery-limit.
AMQP version 1.0 native support
A key feature of the 4.0 release is native support for AMQP 1.0. If you are familiar with RabbitMQ, you may know that AMQP 1.0 has been available through the use of a plugin for quite some time. However, this approach introduced additional overhead compared to the more commonly used AMQP 0.9.1. Now, with 4.0 native AMQP 1.0 is seamlessly integrated, eliminating the overhead of the plugin. Lets look into the details…
AMQP 1.0 is more efficient in v4.0. The streamlined approach reduces overhead by no longer proxying messages through the AMQP 1.0 plugin, which uses a 0.9.1 client.
The image below illustrates how messages are delivered via the AMQP 1.0 protocol with the plugin. Messages are proxied from 1.0 to 0.9.1 through the plugin. Erlang processes are used by the plugin (represented by “P”).
Figure 2 - RabbitMQ 3.13 AMQP 1.0 Plugin
Native AMQP 1.0 uses a single Erlang process per session, as opposed to 15 Erlang processes per session in 3.13. This illustration depicts Erlang processes for 4.0 using only 1 Erlang process. This reduction is due to the fact that messages are no longer proxied through a 0.9.1 client in the plugin. Illustration depicting reduced Erlang processes for 4.0.
Figure 3 - RabbitMQ 4.0 Native AMQP 1.0
How to upgrade
4.0.3 will be available for creating new clusters. Ensuring customers are able to test 4.0.
New feature summary
- Classic Queue mirroring removed
- Quorum Queue message priority is now available
- Quorum Queue redelivery limit defaulted to 20
- New exchange type: random local exchange
- Khepri metadata store supported but not enforced
- Native AMQP 1.0 support
Note to CloudAMQP customers
If you have a RabbitMQ cluster with us at CloudAMQP, here are some things to keep in mind.
- Khepri needs to be enabled on the cluster by our support team. It is recommended to test it before using it in production. Ready for Khepri? Follow our guide.
- Scaling down multi-node clusters to a single node with Classic Queues will result in some Classic Queues dropped. Remember to migrate away from classic mirrored queues or Migrate to quorum queues.
Breaking changes
- As mentioned earlier, HA is not available. This can cause unexpected behaviors with multi-node clusters and plan changes.
- For 4.0, cluster metrics will undergo some changes. Some metrics will be modified, while some will be dropped completely. Check the integration documentation for details.
- With this launch there will be some alarms and metrics integrations that may not work as expected. For dedicated clusters the Netsplit and Connection flow alarm will not be available.
Conclusion
RabbitMQ 4.0 is the next major version and features many enhancements that raise the bar for message brokers. The removal of classic queue mirroring will be the most abrupt change for most users. We have battle tested RabbitMQ 4.0, but it is always good to test before upgrading.
Ready to test native AMQP 1.0 and Khepri? Create a 4.0 cluster from the CloudAMQP console.
Need a CloudAMQP account? You can easily signup on CloudAMQP, and create a RabbitMQ instance to test 4.0.
For any suggestions, questions, or feedback, get in touch with us at contact@cloudamqp.com
