FAQ: What is the RabbitMQ Federation plugin

We will explore how this concept works throughout this RabbitMQ federation plugin tutorial.

What is RabbitMQ federation?

Federation allows for the transfer of messages between different RabbitMQ clusters. There is no concern for users and virtual hosts or versions of RabbitMQ.

RabbitMQ ships with the Federation Plugin by default and offers support for:

• Federated Queues
• Federated Exchanges

Both structures serve different purposes. One acts as a load balancer, while the other helps grow your cluster.

When to use a Federated Queue or Exchange

There are numerous reasons to use a federated queue or exchange that are not related to creating a massive application.

Use federation when:

• Dealing with messaging in high-latency situations
• Spreading messages across multiple isolated hosts
• Ensuring high availability
• Upgrading a cluster
• Building a large-scale application

To a degree, clustering will increase performance, but be cautious when increasing the number of attached machines and consumers in an extremely large application scenario, as there are hardware limitations regarding the number of connections created.

You may also need to replicate brokers. Isolation is a key component of application security and is naturally enforced when performing a blue-green upgrade. It can make life difficult when trying to send messages to all consumers without federation.

Enabling the RabbitMQ Federation Plugin

RabbitMQ supports federation out of the box. The broker ships with the plugin ready to install.

Enable the federation plugin on all target brokers by running the following on the command line:

rabbitmq-plugins enable rabbitmq_federation
rabbitmq-plugins enable rabbitmq_federation_management

The management plugin lets you control federation from the UI. You are now ready to connect your brokers.

Federated Queues

Federated queues are one of two types of data structures offered by the federation plugin. Federated queues allow you to spread messages among multiple brokers or clusters. A remote, federated queue, retrieves messages when there are no more locally produced messages to consume.

This structure allows you to:

• Transfer load from one cluster to another during periods of heavy usage
• Move messages across clusters during an upgrade

This type of queue acts like a cross-broker or cross-cluster consumer. It can act as an intelligent load balancer.

A Federated Queue Example

Consider a globally-available mapping company, Direction Quest, serving tens of millions of messages. They use RabbitMQ to process images, standardize addresses, update databases, and serve requests.

To reduce latency, the developers create clusters in different regions. However, the clusters serving heavily populated cities become overloaded during the holidays. Federated queues in an area with a smaller population take on some of the load to keep the application up-to-date and running smoothly.

Creating a Federated Queue

Creating a federated queue requires slightly more configuration than when creating a normal queue. You must set a policy to tell the broker how to route the message.

Continuing with the mapping application example, we want to route messages to a queue that handles incoming geolocation requests from a mobile application. Assume that the devices connect to a web server that pushes to RabbitMQ. DirectionQuest wants to federate the cluster serving the northern plains in the US with the one serving California.

The first step after enabling the plugin is to create a queue to receive messages on the downstream broker (the downstream broker is the broker that should receive the transferred messages from the upstream broker):

Next, tell the downstream broker how to access the upstream broker:

We used the dq virtual host and named the policy demo.queue_federation.

Then, generate a user policy specifying which queues to federate on the downstream broker:

Check that your queue is federated:

The federated queue in the northern plains states, the least populated part of the United States, now processes messages from California when no messages are available locally.

Federated Exchanges

Federated exchange differs from the federated queues. Exchanges replicate the flow of messages from one location to another. Using this structure, an exchange on the remote host connects to a queue on the upstream server to which all messages are published.

This data structure allows for:

• Broadcasting messages to an extremely large number of connected consumers
• Maintaining a hot standby for high availability
• Migration

It is also possible to use the shovel plugin for migration. A shovel can run on an entirely separate node and allows you to move messages between brokers with loosely related purposes and structures.

The federated exchange works well when scaling out brokers or striving for 100 percent uptime. It can help when nearing connection limits for your hardware by giving you access to another cluster.

Federated Exchange Use Case

Direction Quest has thousands of commercial customers such as Pizza Palace, who rely heavily on giving delivery drivers accurate directions without any downtime. Delays could cost these businesses tens of thousands of dollars.

Direction Quest creates a standby cluster whose messages mirror those of a working cluster to meet the need for availability. They use a federated exchange to do so.

The company also wants to push updates to every mapping application at once. They set up a series of federated exchanges to do so.

Creating a Federated Exchange

There is no need to change the way you set up exchanges for a federated exchange. As with the federated queue, you need to specify policies to handle messages. We use the same upstream link and user policy as when we created the queue.

Declare two fanout exchanges to broadcast Direction Quest’s map updates on each broker. Name each federation.map_updates:

There is no requirement to use the fanout exchange. However, Direction Quest wants to broadcast messages to every consumer.

Attach a queue downstream for testing:

Bind the queue with our exchange:

Publish a message upstream and ensure it reaches the downstream server:

Publish a message upstream and ensure it reaches the downstream server:

The direction request repeats these steps using the command line and automated scripts for dozens of clusters across the globe. They use the top-level cluster as a source to reach every commercial consumer as well as the software running their popular website.

Federation Cautions

There are several details to keep in mind when dealing with queue federation. Certain limitations may cause problems if you don’t account for them.

Each queue applies arguments separately. Also, it is possible to create cyclical graphs when setting up exchanges. Plan your topology carefully to avoid routing messages in a never-ending loop. Set max-hops appropriately.

Using Federated Queues and Exchanges in RabbitMQ

The RabbitMQ federation plugin places load balancing and high availability within reach. Using federation allows you to push updates, perform upgrades and broadcast across clusters.

You can try out federation via CloudAMQP, using the same method from this article or over the command line.