What’s new in RabbitMQ 3.10?

Skip this intro and take me to the how-to change version at CloudAMQP guide

Classic Queue Version 2 with new queue index and message store implementations

The RabbitMQ team has made improvements to the index and message store implementation for classic queues. These improvements are called Classic Queue Version 2 or CQv2 for short. For replicated queues across multiple nodes, we still recommend Quorum Queues.

The benefits of using Classic Queue v2 (CQv2) compared to v1 (CQv1) are:

• Increased throughput for workloads where consumers keep up with producers
• Less memory usage
• Less variable memory usage
• More efficient consumer delivery

How do I enable Classic Queue version 2?

CQv2 can be enabled via a policy, or as a queue argument:

1. Policy:
   
2. Queue argument:

How do I migrate a classic queue version 1 to version 2?

In order to take advantage of the new features of Classic Queue v2, you need to change the queue version to 2 via a policy as described above. After only a few seconds, queues with up to a million messages will be migrated. The migration requires some server resources so here are a few tips:

• If your queues have transient messages, and/or a lot of messages, converting a queue from v1 to v2 can be memory intensive. To avoid running out of memory we do recommend that you apply a lazy policy to the queue before converting queues from v1 to v2.
• Deleting the policy will downgrade the queue back to v1. We do not recommend downgrading v2 queues to v1 as this showed to be more memory intensive and time-consuming.
• Make sure your clients are setting the x-queue-version queue argument to 2 when declaring queues.

Performance test Classic Queue Version 1 versus Version 2

The following test scenarios are used to discover how a CloudAMQP instance performs on Classic Queues version 1 versus version 2. We have performed these tests on a single node Awesome Ape plan in AWS using:

• m6g.2xlarge with 8 CPUs and 32GB memory
• 100GB gp3 disk with 3000 IOPS and 125MB/s
• RabbitMQ Perftest tool
• RabbitMQ flow control disabled

Scenario 1: High throughput

Let's see how Classic Queue version 1 and version 2 perform in an optimal scenario to get the maximum throughput. Messages are just 1 byte and 100 bytes in size and the number of queues and producers are the same as the number of cores, which is 8.

Classic queues version 1

Perftest command:

bin/runjava com.rabbitmq.perf.PerfTest --queue-pattern 'cqv1-1-%d'
--queue-pattern-from 1 --queue-pattern-to 8 --size 1 --producers 8
--consumers 16 -z 300 -f persistent -c 50000 -H $URL

1-byte messages:

100-byte messages:

Classic queues version 2

Perftest command:

bin/runjava com.rabbitmq.perf.PerfTest --queue-pattern 'cqv2-1-%d'
--queue-pattern-from 1 --queue-pattern-to 8 --size 1 --producers 8
--consumers 16 -z 300 -f persistent -c 50000 --queue-args x-queue-version=2 -H $URL

1-byte messages:

100-byte messages:

Summary scenario 1

With very small messages and a limited number of queues and connections, Classic Queue version 2 performs 20% faster than version 1 with the same amount of memory usage. In general, the larger the message size, the less throughput (msgs/s).

The first peak is the number of messages delivered in a v1 queue, and peak 2 for a version 2 queue with a 1-byte message size:

Scenario 2: 100 Queues, 100 producers, 200 consumers

In this scenario, we can see how Classic Queues version 1 performs with an increased number of queues and connections, compared to version 2.

Classic queues version 1

Perftest command:

bin/runjava com.rabbitmq.perf.PerfTest --queue-pattern 'cqv1-2-%d'
--queue-pattern-from 1 --queue-pattern-to 100 --size 1 --producers 100
--consumers 200 -z 300 -f persistent -c 50000 -H $URL

1-byte messages: 20600 msg/s

Classic queues version 2

Perftest command:

bin/runjava com.rabbitmq.perf.PerfTest --queue-pattern 'cqv2-2-%d'
--queue-pattern-from 1 --queue-pattern-to 100 --size 1 --producers 100
--consumers 200 -z 300 -f persistent -c 50000
--queue-args x-queue-version=2 -H $URL

1-byte messages: 53800 msg/s

Summary scenario 2

Consumer delivery is much faster with Classic Queues version 2 than version 1 in a scenario where there are many queues and connections. Publishing and consuming rates vary a lot more when using more queues and connections. Even though it is using the same amount of memory, version 2 is around 160% faster than version 1. In general, having many queues and connections takes up more memory if we compare scenario 1 with scenario 2.

The above test scenarios show that Classic Queue version 2 is outperforming version 1 in both cases.

Quorum Queues messages move to disk

Previously you had to set x-max-in-memory-length or x-max-in-memory-bytes to control the memory usage of a quorum queue, but with RabbitMQ 3.10 messages are automatically moved to disk as quickly as possible, ignoring those two settings. This will make throughput and latency less variable.

See more in-depth investigation on x-max-in-memory-length and x-max-in-memory-bytes in our RabbitMQ Summit 2021 talk, from time 8:01:

Message TTL for Quorum Queues

Message TTL has long been supported for classic queues, and with RabbitMQ 3.10 it’s also supported in quorum queues. Specify the message TTL as before, by setting the expiration field when sending a basic.publish.

Dead lettering strategies for Quorum Queues

You can now use two different strategies when using dead lettering in Quorum Queues, ‘at-most-once’ or ‘at-least-once’:

at-most-once is the same behavior as classic and quorum queues in RabbitMQ 3.9, meaning messages can get lost during transit between queues. This is the default strategy.

at-least-once is introduced in RabbitMQ 3.10 and guarantees that a message has been transferred between the queues. Dead lettered messages will be consumed from the original queue and published to the target queue using publisher confirms to make sure they are delivered. To use this strategy you need to opt in.

How to enable ‘at-least-once’ Dead Lettering for Quorum Queues?

You need to opt-in to use at-least-once Dead Lettering for quorum queues, as at-most-once is the default. Here are the policies that need to be applied:

• Set ‘dead-letter-strategy’ = ‘at-least-once’
• Set ‘overflow’ = ‘reject-publish’ (default is ‘drop-head’).
• Configure a dead letter exchange.

It can also be set via queue arguments, just start them with ‘x-’

How to upgrade to RabbitMQ 3.10