FAQ: When and how to use the RabbitMQ Dead Letter Exchange

This article examines when and how to use the RabbitMQ dead letter exchange by covering:

• The dead letter exchange
• Dead letter queues
• Strategies for dealing with dropped messages

Dead letter handling is a key part of almost any messaging system, and RabbitMQ is no different.

When is a message considered dead?

There are three identified situations where a message becomes undeliverable after reaching RabbitMQ:

• A message is negatively acknowledged by the consumer
• The TTL of a message expires
• The queue reaches capacity

By default, the broker drops these messages. Publishing is successful, however, the RabbitMQ consumer never handles or has a change to handle the message successfully.

Please note: Messages that are dropped by the exchange, messages that are not routed need to be handled by the alternate exchange.

Why use a dead letter exchange

Queues attached to a dead letter exchange collect dropped messages, with the next steps determined by you. In other words - it's up to you to decide how to handle messages in the dead letter queue. When implemented correctly, information is almost never lost.

Attach a dead letter exchange when you know that you have messages that might be nacked, but still needs to be handled. When you can’t lose messages with an expiring TTL or when the queue might reach its capacity.

Setting up a RabbitMQ dead letter exchange

Dead letter exchanges are no different than other exchanges:

Simply specify the exchange normally and declare it as a backup for a queue:

channel.exchange_declare("dlx_exchange", "direct")
channel.queue_declare("test_queue", arguments={
  "x-dead-letter-exchange": "dlx_exchange", "x-dead-letter-routing-key": "dlx_key"})

The x-dead-letter-exchange parameter tells the test_queue to use the dlx_exchange for dead messages. Notice how the exchange is not dedicated to a single queue.

Creating and Binding RabbitMQ Dead Letter Queues

Just as with a dead letter exchange, a dead letter queue is a regular queue in RabbitMQ; it is just attached to the exchange.

Create the queue normally and attach it to the exchange:

channel.queue_declare("dead_letter_queue")

If desired, specify a new routing key:

channel.queue_bind("dead_letter_queue", "dlx_exchange", "dlx_key")

The x-dead-letter-routing-key changes the routing key from the one used in the original message to dlx_key. This allows you to redirect dead letter messages from multiple queues to the same dead letter queue.

Retrying dead letters

You can create a redelivery mechanism using dead letter queues by attaching the original exchange to the dead letter queue. However, it is possible to create an endless cycle of redelivered messages which could clog dead letter queues if left unchecked.

Create a separate dead letter queue that stores messages before pushing them to an exchange or routing them back to the original queue.

To further improve the system, include an incremented property in the message body indicating the number of times the message was received. This requires handling dead letters in a separate consumer but allows you to eventually drop messages or push them to storage.

Mixing with a delayed exchange

To further improve dead letter handling, a RabbitMQ plugin exists to specify a delay in an exchange. The RabbitMQ Delayed Message Plugin works with RabbitMQ 3.8 or later.

A delayed message exchange introduces the x-delayed-message type passed at creation time:

channel.exchange_declare(exchange="dlx_exchange",
                         exchange_type="x-delayed-message",
                         arguments={"x-delayed-type": "direct"})

channel.queue_declare("test_queue",
                      arguments={"x-dead-letter-exchange": "dlx_exchange",
                      "x-dead-letter-routing-key": "dlx_key"})

channel.basic_publish(exchange="dlx_exchange",
                       routing_key="test_route",
                       properties=pika.BasicProperties(
                           headers={"x-delay": 2500}
                       ),
                       body='Hello World!')

Specify the x-delay header in messages, using successive delay times to avoid overburdening RabbitMQ. This lets you simplify the process of dead letter handling.

Catching orphaned messages in RabbitMQ

Whether caused by a negative acknowledgment, expiration of the time to live, or exceeded queue length, losing information is detrimental for many applications. Knowing when and how to use the RabbitMQ dead letter exchange avoids losing undelivered messages.

Try testing this concept in CloudAMQP where you have fine-grained control over RabbitMQ queues and exchanges.