What you need to know about Classic Queues in RabbitMQ

Note: This blog is especially useful for users running RabbitMQ versions earlier than 4.0 who are still using features like lazy queues, queue mirroring, or Classic Queues version 1.

Classic Queues background

Classic Queue was the only queue type in RabbitMQ, when it launched in 2007. At that time disk access was significantly slower. As a result, Classic Queues were initially designed to buffer most of their data in memory.

As storage technology advanced, the need to avoid disk writes declined. Consequently, with version 3.6.0, RabbitMQ introduced the lazy mode in Classic Queues. In this mode, Classic Queues store most messages on disk to conserve memory—especially useful for handling long queues.

Regarding the term “mature”; RabbitMQ has simply been on the market for a longer time then Kafka (2007 vs 2011, respectively). Both RabbitMQ and Kafka are “mature”, which means they both are considered to be reliable and scalable messaging systems.

Then came version 3.10 when the Classic Queue rethinking process started - CQv2 was introduced and improved in every minor version culminating in 4.0 where CQv1 was removed. Additionally, starting from RabbitMQ 3.12.0, all Classic Queues default to a behaviour that’s intimately similar to what the lazy mode provides.

It is worth noting that Classic Queues also support queue mirroring for added redundancy.

Let’s look at the different features we’ve mentioned in the previous paragraphs one at a time, discussing what they are about and practical tips to help you use them correctly.

Classic Queues V1 & V2

As stated earlier, In RabbitMQ versions prior to 4.0, Classic Queues come in two flavours. Generally, both queue versions store persistent and transient messages on disk — formally called the persistence layer.

Two components make up the persistence layer: the queue index and the message store. The queue index tracks the position of each message within a queue and records whether it has been delivered and acknowledged. Each queue has its own dedicated queue index.

The message store, on the other hand, persists all messages across queues within a vhost. It holds the message bodies and metadata like properties or headers. Each vhost technically has two message stores—one for transient and one for persistent messages—but they are generally combined and referred to as "the message store.”

While both CQV1 and CQV2 adopt the structure of the persistence layer described above in their implementation, they differ in certain ways. Here is one key difference…

CQV1 vs CQV2

In Classic Queues Version 1, messages are either embedded in the queue index or persisted in the shared message store, depending on their size, including headers. By default, messages smaller than 4 KB (configurable) are stored directly in the queue index; larger messages are sent to the shared message store.

In version 2 of Classic Queues, instead of embedding messages in the queue index, RabbitMQ introduces a per-queue message store. Smaller messages are typically written to this store, while larger messages go to the shared message store.

Practical tips

• CQV2 is the only Classic Queue implementation in RabbitMQ versions ≥ 4.0. If you upgrade to 4.0 with CQV1, the queues would be converted to CQV2 under the hood.
• CQV2 provide superior performance, lower and more stable memory usage — this has further improved significantly in RabbitMQ versions ≥ 3.12.0
• If you're using a version below 4.0, switch to CQV2 for better performance. In 4.0+, CQV2 is the default, so no action is needed.

Migration from V1 → V2

If you currently work with CQV1 and would like to migrate, you can read our guide on migrating from CQV1 to CQV2.

Lazy Queues

In older versions of RabbitMQ, Classic Queues could be set to lazy mode, where all messages were stored on disk. Starting with RabbitMQ 3.12.0, this setting is no longer needed as both Classic Queue versions now default to a similar behaviour:

• Generally, messages are written to disk. However, a small number of messages are kept in memory for quick delivery to consumers. The number depends on how fast the consumers process them.
• If a message can be delivered and acknowledged by a consumer before it's written to disk, it won't be stored on disk at all.

Practical tips

• When using CQV1 in older versions of RabbitMQ, enable lazy mode to prevent excessive memory usage. However, be aware that this can result in higher latency in these older versions, since messages are first written to disk.
• For a better experience, we highly recommend upgrading to RabbitMQ 3.12.0 or later, where ‘lazy’ behaviour is the default, so to speak. The latency is also a lot lower in the more recent versions.
• Again, simply migrate to CQV2.

Mirrored Classic Queues

Mirrored Queues have been the de facto way of getting high availability and data redundancy in RabbitMQ for many years. But what are they?

Mirrored Classic Queues operate with one leader queue and several mirror queues. All reads and writes are handled by the leader, which replicates every action (write, read, ack, nack, etc.) to the mirrors. The leader sends a confirmation to the publisher only after all live mirrors receive the message. If the leader fails, one of the mirrors is promoted to leader, ensuring the queue remains available without data loss.

Even though mirrored queues have been in use for many years, the feature still has some serious design flaws that has made it a less than ideal choice. Thus, you need to…

Drop those Mirrored Queues now, before someone gets hurt!

But what is wrong with mirrored classic queues anyway?

• Performance is slower than it should be because messages are replicated using a very inefficient algorithm.
• Furthermore, there are issues with Classic Queues’s data synchronisation model:
  • The basic problem is that when a broker goes offline and comes back again, any data it had in mirrors gets discarded. Now that the mirror is back online but empty, the administrators have a decision to make: to synchronise the mirror or not. "Synchronise" means replicate the current messages from the leader to the mirror.
  • But here is the catch: Synchronisation is blocking, causing the whole queue to become unavailable.
• Above all, these queues are supported in RabbitMQ versions ≥ 4.0. Thus, you can upgrade to the latest RabbitMQ version with queue mirroring enabled.

Practical tips

• For high availability and redundancy, migrate to Quorum Queues now!
• Also consider Stream Queues in RabbitMQ - they are replicated and also offer Kafka-like features in RabbitMQ.

Conclusion

Classic Queues have come a long way since their inception, but as RabbitMQ continues to evolve, it's crucial to stay updated and make the most of the latest features. Thus, if your use-case requires Classic Queues, always use CQV2. Furthermore, if you need high availability and data redundancy, do not use Mirrored Queues, use Quorum Queues instead.

Need help migrating from CQV1 → CQV2 or from Mirrored Queues to Quorum Queues? Feel free to get in touch with our support team - we’d be happy to help. Thank you for reading!