This article is a living document that is continually updated. Last updated January 2022.
Queue federation connects an upstream queue to transfer messages to the downstream queue when there are consumers that have capacity on the downstream queue. It's perfect to use when migrating between two clusters. Consumers and publishers can be moved in any order and the messages won't be duplicated (which is the case if you do exchange federation). Instead, messages are transferred to the new cluster when your consumers are there. The federated queue will only retrieve messages when it has run out of messages locally, it has consumers that need messages, and the upstream queue has "spare" messages that are not being consumed.
Let us start by defining the concept of upstream and downstream servers. Upstream servers are the servers towards messages are published initially. Downstream servers are where the messages get forwarded to. The downstream cluster can be thought of as subscribing to messages from the upstream cluster.
Queue Federation example setup
In this article we have our cluster set up in Amazon, in US-East-1 (in the image marked with a star). We need to migrate to a cluster in EU-West-1 (in the image marked with a moon). The servers in US-East-1 will be defined as the upstream servers of the servers in EU-West-1.
The old cluster in US-East-1 has some queues that should be federated, the metric-queues (metric.cpu and metric.memory).
-
Set up the new cluster
-
Start by setting up the new cluster. (In this case a
new cluster in EU-West-1).
It is possible to copy all configuration settings (queues, exhcanges, enabled plugins etc.) from an old CloudAMQP cluster to a new one by selecting "copy settings (optional)" in step 3 of 4, as in the image below.
-
Start by setting up the new cluster. (In this case a
new cluster in EU-West-1).
-
Create a policy that matches the queues that you would like to federate.
A policy is a pattern that queue names are matched against. The "pattern" argument is a regular expression used to match queue (or exchange) names. In this case, we tell the policy to federate all queues whose name begin with "metric.".
-
Navigate to 'Admin' -> 'Policies' and press 'Add / update a policy'
to create the policy.
A policy can apply to an upstream set or a single upstream of exchanges and/or queues. In this example we apply to all upstreams, federation-upstream-set is set to all.
NOTE: Policies are matched every time an exchange
or queue is created.
-
Navigate to 'Admin' -> 'Policies' and press 'Add / update a policy'
to create the policy.
-
Set up the federation to the upstream server.
- Open the management interface for the downstream server (EU-West-1) and go to the 'Admin'->'Federation Upstreams' screen and press 'Add a new upstream'. Fill in all information needed, the URI should be the URI of the upstream server.
-
Leave expiry time and TTL blank. Leave this blank
means that it will stay "forever".
-
Set up the queues that should be federated.
-
The queues can be created as you always create your queues
(i.e manually from the management interface or from the application code).
The queues should be marked as federated if the policy is applied, like in the image below.
If the queues are not marked as federated, check that the regexp is working.
-
The queues can be created as you always create your queues
(i.e manually from the management interface or from the application code).
The queues should be marked as federated if the policy is applied, like in the image below.
If the queues are not marked as federated, check that the regexp is working.
-
Everything is now set up.
It is time to move the publisher OR the consumer to the new cluster. You can move the publisher and/or the consumer in any order. The federated queue will only retrieve messages when it has run out of messages locally when it has consumers that need messages, or when the upstream queue has "spare" messages that are not being consumed.
-
Verify that messages are consumed
Verify that the messages published to the upstream server are consumed by the downstream server when the consumers are moved to the downstream server.
Note: when using queue federation, the original routing key of the messages will be removed as they are used to publish towards the downstream cluster. If your consumers rely on this information we recommend using shovels instead.
Please email us at contact@cloudamqp.com if you have any suggestions, questions or feedback.
