Hosting RabbitMQ 101
To us at CloudAMQP, the managers of the world’s largest fleet of RabbitMQ clusters, spinning up a RabbitMQ server on a virtual machine is pretty easy. The challenge comes when a server with queue synchronization problems needs to be fixed at 4 a.m. or a seamless upgrade of outdated clusters. It can also be a major challenge to automate the recovery of a 5-node Quorum Queue cluster that's been abused by a buggy client library.
The complex yet fun part is to collaborate with the customer and increase the performance of their cluster based on a very specific use case. After our years of experience with RabbitMQ, we know a thing or two about hosting. We’ve written this guide to help you decide on a RabbitMQ provider.
These are the things we will take a deeper look into:
- Pricing - What’s included in the price?
- Support - How can you ensure a good level of support?
- Plugins - Are plugins supported and easy to enable?
- Feature support - Are your required features enabled?
- Performance and benchmarking - What should you look for in regards to stability and performance.
- Alarms and notifications - Is it possible to set up alarms and monitor the state of your server?
- Security and Compliance - Firewall, PrivateLink, VPC, and certificates. What’s supported by the RabbitMQ hosting provider?
- RabbitMQ and Erlang Version - Is the latest version enabled in a timely manner? Is the new version validated by the provider?
- Metrics and logs integrations - Are any third-party logs and metrics integrations available?
- Backups - Does the service provider enable any types of backups?
- Best practice guidelines and the RabbitMQ community - Does the service provide best practice guidelines in any way?
Pricing
Pay attention to what is included in the cost when finding a good RabbitMQ hosting provider. Some providers include all traffic and disk costs as well as premium support in the price, so at the end of the month, your bill will be no more than the price you signed up for. Other providers may charge extra for all traffic or have a separate cost for support options, giving you a fluctuating bill. Another hidden cost to consider is the hours needed for maintenance, configuration, and monitoring by your own team.
Questions to consider when evaluating the pricing level:
-
First of all, ask the provider what’s included in the price.
- Do you need to pay for additional costs, like traffic costs?
- Are you able to scale up or down when needed and in an easy way?
- At the end of the month, will you only be charged for the actual time an instance has been used? Or will you be charged for the entire month?
- What payment options are available i.e. card, invoice, etc?
- Is it possible to use PO payments?
- Does the provider offer monthly, quarterly, or annual payment options?
Support
When it comes to maintenance, is it up to you to keep track of the server state and solve issues that may occur?
No matter your level of experience with RabbitMQ, the age of your hardware, or vigilance in configuration and keeping up with new releases, failures may occur. The level of support can impact the ability to get assistance when your service suffers from issues you cannot solve yourself. If your server has a problem or goes down, a reliable support team should be able to notify you and assist you in solving the issue without message loss and as little downtime as possible. Oftentimes this includes allocating more RAM to the instance or increasing disk IOPS. At the same time, the support team should communicate with you to explain the situation and advise on how to prevent similar situations going forward.
In our experience, this maintenance and monitoring are both time-consuming and expensive, which is something to keep in mind when choosing a provider for your system. You should also check if this level of support is included in the price of your RabbitMQ provider.
Questions to consider when evaluating the support level:
- Ask the provider if you are supported by a real person or by a chatbot in urgent situations?
- Is support included at the base level or is there an extra fee associated with it?
- Does the provider have skilled support team members spread across the globe for true 24/7/365 coverage?
- Does the support team share best practices for using and running RabbitMQ with the wider community and engaging in the RabbitMQ ecosystem?
- Does the support team have experience and specialized software for recovering messages in tricky situations such as after a netsplit or when a queue index can’t be rebuilt?
- What's the SLA (Service Level Agreement) level?
- How long is the response time for common and urgent requests?
Plugins
One of the main reasons that RabbitMQ is the most deployed message broker is the ability to tweak the broker based on the use case. A RabbitMQ broker offers a lot of built-in features like routing, clustering, and persistence. Still, enabling RabbitMQ plugins can enhance your broker’s default capabilities.
When determining what service is best for hosting, look for the ability to enable and disable plugins for RabbitMQ. Additionally, be sure to determine how easy or difficult it is to enable a plugin with your potential hosting provider.
Questions to consider when evaluating plugins:
- Are the plugins you need supported by the provider?
- Can you upload your own custom plugins?
- Is it easy to enable and disable plugins?
- Is the host running the latest version of the plugin?
- Look at common plugins and consider if you need any of them in the future.
Feature support
Make sure that the RabbitMQ hosting provider supports all the features that you think should exist in the default RabbitMQ setup.
Quorum queues
Quorum Queues was released with RabbitMQ 3.8 and is considered to be one of the most important features in RabbitMQ's recent history because it replaces the unreliable "mirrored queues”.
RabbitMQs website details this by saying: “In many cases, quorum queues would be a superior option to classic queue mirroring. Readers are encouraged to get familiar with quorum queues and consider them instead of classic mirrored queues.”
The evidence is overwhelming that Quorum Queues is a desirable feature, so make sure that your hosting provider offers support for it.
Questions to consider in regards to feature support:
- Are all features you intend to use actually supported?
-
How many nodes do you need in your cluster?
- Does the service provider offer clusters larger than 2 nodes?
Performance and Benchmarking
Without question, two main goals to keep in mind are stability and performance.
When it comes to stability, make sure that your RabbitMQ hosting service takes proactive actions if your RabbitMQ cluster has issues or goes down. A good provider actively monitors your instance to make sure it’s running smoothly.
Look for a provider that promises a certain level of performance when it comes to expected message throughput. Make sure you run an array of performance tests to prove it can handle peak loads. Here are a couple of parameters to test, but keep in mind that, depending on your use case, the ones you choose may vary:
- transient and persistent messages
- publisher confirms and consumers acks
- small and big messages
- classic queues and quorum queues
- multi nodes and single node
- a large number of connections
- a large number of queues
Ultimately you want to check how resources such as CPU, memory, disk, and network are coping when under pressure. Make sure that your provider is running the latest hardware available for the best performance and price.
One thing to note when running performance tests is that some of them might need to run for an extended period of time (not just a minute but rather an hour or more) in order to get a proper reading rather than one that shows what your instance is capable of in a burst situation.
For performance testing, there is a great application maintained by the RabbitMQ team called rabbitmq-perf-test that you can download and run (Java required). The best practice is to run it on a machine in the same data center where your RabbitMQ is located.
Questions to consider in regards to performance and benchmarking:
- Ask your provider if they impose any limits that might affect performance, such as bandwidth or I/O.
- Ask for help with benchmarking for your specific case.
Alarms and notifications
When looking for a RabbitMQ hosting provider, be sure to ask if alarms can be set up easily. It is useful for you to receive accurate alerts based on performance anomalies in your application by activating queue alarms, consumer alarm, CPU alarms, and memory alarms.
At the very least, services should have basic alarms that can be sent via email or other communication services.
Questions to consider regarding alarms:
- Is it important for you to send alarms straight to integrated services like PagerDuty and Slack? If so, ask your provider if it is possible or not with the potential hosting provider.
- What type of alarms are supported? Alarms that could help are;
- When the number of messages in a queue reaches a specific threshold.
- When the number of consumers for a queue is more or less then a specific given number.
- When the number of connections or channels is growing at a rapid speed (due to channel or connection leaks).
- Is it possible to set up alarms for a specific queue, or alarms that monitor many queues?
Security and Compliance
At the very least, you should check for support for TLS (SSL). All servers should be configured with server certificates signed by a CA.
Another thing to look for is support for custom certificates or TLS client authentication. Check which TLS versions and ciphers they support, and if you're client(s) are able to use them. Older Java clients don't support GCM ciphers, for instance.
Firewall
A firewall lets you restrict access to your cluster by closing ports and/or specify which IP range should have access. Do you need to be able to set up different ports for different ranges? For example, only your server can access port 5762 (AMQP), but the management interface is still open to all. If the hosting provider has support for VPC peering, you can use the firewall to close all access to the cluster from the internet.
Your RabbitMQ hosting provider should allow you to configure a firewall for your cluster directly from the UI and allow you to specify an IP range and decide which ports should be opened for that range.
PrivateLink
If you are considering AWS as a host, remember that PrivateLink provides private connections from the VPC directly to the cluster, i.e. the data never leaves the AWS network. It is good if your RabbitMQ hosting provider encourages encrypted connections and has support for AWS PrivateLink. It should also be easy to enable AWS PrivateLink from UI and API.
Compliance Frameworks
With an ever-changing threat landscape for cybersecurity, the most common concern among prospective customers has been focused on the security and privacy of their data, and for a good reason. When relying on third parties to host your data, it’s important to understand how your data will be protected, so check the host’s compliance with industry best practices and standards.
There are a lot of various standards and regulations. As a customer, it’s your job to decide which of those are most important to your cluster, for example:
- Make sure your vendor follows general and overall security measures by verifying an overarching security standard such as SOC 2 Type 2, or ISO27001.
- If you handle data that belongs to individuals in the EU, your hosting provider must be compliant with the GDPR.
- If you handle health data that belongs to individuals in the US, your hosting provider must be compliant with HIPAA.
Some frameworks to look for are listed below:
- SOC 2 Type 2: An auditing procedure developed by the American Institute of CPAs (AICPA), which ensures that a business is handling customer data securely and in a manner that protects the organization and the privacy of its customers.
- GDPR: The General Data Protection Regulation (GDPR) is an EU regulation on data security and privacy related to personal data. GDPR applies to all organizations operating within the EU as well as to non-EU organizations with customers in the EU).
- HIPAA: The Health Insurance Portability & Accountability (HIPAA) is an American act that requires the protection and confidential handling of protected health information.
Questions to consider in regards to security:
- Does the provider have support for PrivateLink?
- Are you able to restrict access to your cluster through a firewall?
- Does your provider need to be certified by a specific security standard, such as SOC 2 Type 2, or ISO27001?
RabbitMQ and Erlang Version
New versions of RabbitMQ and Erlang are released on a regular basis. Make sure that your new provider verifies these releases based on how they affect the customers, and roll them out in a timely yet safe manner. If a RabbitMQ host provider is supporting an old version, you may miss out on crucial features or important security updates.
Questions to consider in regards to RabbitMQ version
- Make sure that the service provider verifies new releases based on how they affect the customers, and roll them out in a timely yet safe manner.
Metrics and logs integrations
Third-party logs and metrics are extremely useful to be able to determine the health of your servers and RabbitMQ brokers at any given time. When looking for a RabbitMQ host, you should ask yourself two things: First, is it important for you to receive monitoring information along with an overview of how the state of your system? Second, is it important to get logs and metrics into the same place as your other systems? Some examples of common and well used third-party logs and metrics services are listed below:
Questions to consider when evaluating metrics and log integrations:
- Which logs and metrics services are you using for your other services today, and do you want to integrate your RabbitMQ logs and metrics into this service?
- Does the RabbitMQ service provider have support for integration to your third-party logs or metrics service?
Backups
As with any system, backups are important. It doesn’t make much sense to backup the complete message queue, since the data could be flowing at an enormous speed, instead, ensure that the data is safe on disk once it has arrived at the broker.
Be certain to backup your RabbitMQ definitions including users, vhosts, queues, exchanges, bindings, and runtime parameters. If needed, your definitions should easily be available to download and import via the RabbitMQ Management HTTP API.
Questions to consider in regards to backups:
- Is it possible to back up any data with your service provider?
Best practice guidelines and the RabbitMQ Community
Take a look at how often your potential RabbitMQ hosting providers lead workshops, discussions, publications, and other training sessions. A good collaboration within the community and encouragement for new developers to try message queues can indicate a high level of expertise.
Questions to consider in regards to best practice guidelines:
- Does your service provider have a depth of knowledge and experience in RabbitMQ? Be sure that your RabbitMQ hosting provider at least has a user message board to exchange ideas, best practices, and strategies with others, if nothing else.
- Does the provider have any type of diagnostic tool available that can advise you as a customer on common anti-patterns and best practices?
Summary
In short, look for an expert RabbitMQ hosting provider that offers both support and monitoring, integrated tools, and options to enable plugins, and more expertise and experience, where all of the mentioned criteria should be included in the price.
Feel free to add questions or comments, make suggestions, and provide feedback, both on our product and on this article.
