Smoke signals, carrier pigeons, courier companies, emails - we as human beings have always sought ways to communicate, even when there have been distances in between us. The same need exists in the computer world - a method to send, receive, and create the message itself.
Just like humans, some demand quick answers, such as when you eagerly tap someone on the shoulder, and sometimes you send an email and wait for the answer. Sometimes you do not expect an answer at all.
Software has the same need for communication. Of course, there are different ways to communicate, even for IT systems. One of these ways is to connect two systems with each other via a queue, much like when sending emails. Both systems only connect to the queue, or their mailbox, but never really to each other.
Let's dive into the world of message queueing.
What is a message queue?
The actual definition of a message queue is a component for inter-process communication that passes control, content or messages to another component.
To illustrate message queues another way, think of it as ordering at your favorite restaurant. A front-line worker takes your order for your favorite meal. The front-line worker then makes a request to the kitchen for your food. The kitchen receives the order and makes the food. In this scenario, an instruction was received and action was performed that triggered an event – order taken, request triggered, an action performed.
While the kitchen is making your food, the front-line worker doesn’t wait until it appears, they keep taking orders from other customers. The kitchen continues to operate and process through orders. Each point has its own responsibilities and doesn’t wait for the other to finish what they’re doing, and neither is constrained by time.
Message queue use cases
Now you might think, “where on earth would a message queue fit in my architecture”? The simple and quick answer is when:
- you have “timeout errors” due to too many requests at the same time.
- you need a decoupled way to communicate between or within your application.
- you are polling a data store too often and you want this data store to be available to answer qualified queries instead
- you need to scale up and down during peak hours
In other words, any time a task is not part of a basic user transaction, and/or the results doesn’t impact a user response, that’s a perfect job for a message queue.
Message queues can also be used for more advanced scenarios. For example, AMQP based message queues like RabbitMQ can be configured to route and distribute messages according to rules and alternative processes. Processing systems can take their time to process through the messages. When the business and workload grow and the need to scale up the system is obvious, adding more consumers to work on the queues is fast and easy.
We will now explain cases where a message queue can be beneficial. Simply put, two obvious cases can be used as examples of when message queues really shine:
- For long-running processes and background jobs
- As the middleman in between microservices
Long-running processes and background jobs
When requests take a significant amount of time, it is the perfect scenario to incorporate a message queue.
Imagine a web service that handles multiple requests per second and cannot under any circumstances lose one. Plus the requests are handled through time-consuming processes, but the system cannot afford to be bogged down. Some real-life examples could include:
- Images Scaling
- Sending large/many emails
- Search engine indexing
- File scanning
- Video encoding
- Delivering notifications
- PDF processing
- Calculations
The middleman in between microservices
For communication and integration within and between applications, i.e. as the middleman between microservices, a message queue is also useful. Think of a system that needs to notify another part of the system to start to work on a task or when there are a lot of requests coming in at the same time, as in the following scenarios:
- Order handling (Order placed, update order status, send an order, payment, etc.)
- Food delivery service (Place an order, prepare an order, deliver food)
- Any web service that needs to handle multiple requests
Before we dive into some interesting short stories of how successful companies have been using a message queue in their architecture, I will share a video explaining message queueing:
Image scaling
Let’s start with a use case where companies need to scale images.
In real estate, brokers need large quantities of photos to effectively market a property and catch a buyer’s eye. Through the platform, once a broker adds a new image of the property, the task of scaling it to a user-friendly size is given to RabbitMQ. The message queue holds on to the image scaling task until the consumer grabs it, scales it, and the image is published to the website in the new, efficient size.
But why would you use a message queue in this scenario? Simply because many images might be added at the same time, which might cause timeout errors in the scaling part of the system.
Hemnet is a property listing platform that moved from a fully on-premise s olution to a new cloud-based solution in less than a year. The entire story of Hemnet is available in another blog that goes into detail on how RabbitMQ has been deployed as a pipeline for image scaling, among other things.
From monolith to microservices
One of the fastest growing digital parking services, Parkster, has broken down its monolithic architecture into faster, decoupled microservices. Like Netflix, Parkster initially began life as a monolith to prove its business model. They quickly realized that this type of system - where the entire application is built into a single unit, with one codebase and one system - isn’t efficient or effective.
A few years into building the company, Parkster began breaking up the monolith into multiple small microservices communicating via message queues. The full story about Parkster and CloudAMQP deserves its own article!
File Scanning
Softonic, the software and app discovery portal, has more than 100 million users every month and delivers over 2 million downloads every single day. The portal has a constant flow of events and commands between the services it offers, and RabbitMQ is the message queue of choice, contributing to the fast, and effective architecture Softonic is known for.
Users upload files to the Softonic platform, where they are scanned for viruses, and information about the file is collected before being distributed to other users. The user is notified immediately after the upload is successful. The file information, in the form of new binary data, is located on a dedicated service, and a notification is sent to the message queue that there is data ready for the other services. Other services handle this data, which is persistent and added to the website at the end of the process.
In the case of the Softonic platform, the microservice architecture featuring RabbitMQ allows the web servers to respond quickly to requests instead of being forced to perform resource-heavy processes on the spot, which is bad for the user experience.
Read about File Scanning at Softonic.
PDF processing
A web application allows users to upload information to a website. The site will handle this information and generate a PDF and email it back to the user. Handling the information, generating the PDF, and sending the email will in this example case take several seconds and that is one of the reasons why a message queue will be used.
Read the complete guide about PDF processing here.
Passing background messages between server background workers
FarmBot is an open-source robotic hardware kit that enables interaction with agricultural projects in an efficient, fun way. FarmBot employs physical sensors and actuators that require a method of communication between the physical garden and the software.
Remote procedure calls or HTTP request/response patterns don’t always work for FarmBot, where a device could be forced to perform long polling activities and constantly making requests to the API for any new remote procedure calls. This would cause tremendous scalability issues for the web app and cause a sub-par real-time experience for users.
For instance, when FarmBot receives an emergency stop message, they should be received immediately upon creation rather than requiring the system to constantly check the API for them. Other examples include remote procedure calls and real-time data syncing.
A message queue is now an important component of the FarmBot web API and used to handle various tasks including:
- Passing push notifications between users and devices
- Passing background messages between server background workers
- Preventing unauthorized use through a set of custom authorization plug-ins
The message queue used is RabbitMQ, a real-time message broker, so there is no need to check for new messages. Messages such as a user clicking the "move" button on the interface are sent back and forth between client, device, and server and require no requests.
Think of the message broker acting as a machine-to-machine chat application. Any software package with the correct authorization, including REST API, FarmBot OS, or third-party firmware, can send a message to any other entity that is currently connected to the message broker.
Read the about how message queueing transformed the agri-tech app FarmBot.
Database backup handling at ElephantSQL (PostgreSQL database hosting service)
ElephantSQL (a PostgreSQL database as a service) is using queues in the architecture for daily backup handling. Once a day there are thousands of “perform-database-backup” messages added to the queue, one for each database. Another service is then subscribing to the messages and creating the backups.
Backups can also be created on-demand via a customer API or through a web interface. This backup should be available immediately after the request was sent - it is a priority, in other words. To solve this, your message queue needs to be able to prioritize messages in some way In this case RabbitMQ is used. The “perform-database-backup” message is added to the same queue, but this message has a higher message priority than all other requests. The message sent will be placed at the head of the queue immediately.
Read about message priority here.
Scaling a microservice architecture with message queues
CloudAMQP (RabbitMQ as a Service) is powered by a number of message queues that provide various functionality. This video goes into how AMQP and RabbitMQ can be used to power a microservice architecture from the bottom up with flexibility and reliability.
Why message queues?
What do all these success stories have in common? Message queues - separating the individual processes of applications and keeping them independent of each other. One process will never need to invoke another or interact with another or follow anything but its own process flow. The message is simply placed in the queue and the system continues to process independently. This decoupled architecture creates an easy-to-maintain system that is also easy to scale.
BETA: RabbitMQ Training tool
Are you ready to add a message queue into your architecture? Learn everything you need to know to master RabbitMQ (an open source message broker). Go all the way from learning the basics, via tips and tricks from a realistic case study, to master best practices at https://training.cloudamqp.com/
CloudAMQP
CloudAMQP is a hosting provider of RabbitMQ. The benefits of working with either message queueing service are obvious. Cloud-based messaging via CloudAMQP increases the speed of making the application available to the market. Developers can focus on the core of the application instead of managing and maintaining servers and infrastructure and updates.
