Microservices and Message Queues, Part 1: Understanding Message Queues

Congratulations, you got your first job as a junior developer. But you keep hearing your senior colleagues talk about distributed architectures and Message Queues, and wonder what these things are. Or maybe you’ve been a developer for some time, and you’ve heard of technologies like RabbitMQ, and Apache Kafka, but don’t really know what they are and how they work?

This 8-part series will progressively explore the concept of Message Queues and their practical use cases. But beyond just knowing the concept, you will also build a demo project that will get your hands dirty and by extension, learn how to work with a Message Queue first-hand.

As seen in the table of contents above, the focus of this article will be on what Message Queues are. But first, why do we even need Message Queues?

How a Message Queue saved a Pizza store

Imagine you just launched a pizza outlet and built a great web application for your store. The web application records each transaction processed and generates a receipt for that transaction.

Essentially, when a customer walks into your store and pays for a pizza, a chain of events occurs in this particular order:

• You record the transaction on your web application through its interface (e.g., enter the amount paid, the customer name, etc.)
• The web application records that transaction in the database
• The web application produces a receipt for the transaction
• You present the receipt and the pizza to the customer
• You proceed to accept payment from the next customer

You then noticed that processing each order takes some time, especially the part where you record the transaction and wait for the system to generate the receipt. However, your pizza store is still new and you only get 5-10 orders per day. So, even though each order takes some time, you can process an order before the next customer arrives.

Figure 1 - Orders are processed normally when the traffic is low

Because of the current low traffic, you only attend to one customer at a time. In fact, sometimes, you even have plenty of time to wait for the next customer to arrive. Your web application works well, and you have a small but mighty group of very happy customers. But then something changed.

The agony of having more customers

As time goes by, your pizza store makes a name for itself. Everybody loves your pizza. Happy customers spread the word, and now you are handling 5,000 - 10,000 orders daily. Your business is booming, and that’s a good thing. But then you notice a problem.

Lots of customers walk into your store at the same time, and you spend a reasonable amount of time recording and generating the receipt for each transaction. Because of these two factors, you end up having customers who walk into your store and wait for minutes and sometimes hours before it is their turn. But customers are not patient.

Figure 2 - When there is a surge in traffic, orders take longer to process

Additionally, because your web application now has to do a lot more work beyond its capacity, it starts to get overwhelmed. Indeed, there is a problem, but what do you do?

Message Queues to the rescue

A moment of brilliance hits you, and you realize customers can drop a note with their name, the amount paid, and their email in a box once they’ve paid for a pizza. That way, the customers get their pizza immediately and don’t have to wait. The transaction is then recorded and the receipt is emailed to the customer at a later time.

Figure 3 - Asynchronous processing

This is a drastic shift from the previous approach. Customers get their pizzas instantly, and the transaction receipt is emailed to them later. This is what is described as asynchronous processing. And this is fundamentally what Message Queues do. They buffer the message/request that needs to be handled, and the processor picks up the request/message and executes it at its own pace.

Now, let’s tie all this into software systems.

Software systems do not exist or work in isolation. Systems usually send data to and receive data from other systems. The interactive systems could be processes on the same computer, modules of the same application, services that might be running on different computers or technology stacks.

When designing these systems, sometimes we adopt a communication mechanism where the system:

• Sends data to another system/service
• Waits for the service it contacted to finish processing
• Gets a response from the service
• Then proceeds to the next task

This pattern of communication is formally called synchronous processing. We saw this in the pizza store example, where customers had to wait for receipts to be generated for an order to be considered complete. As mentioned earlier, this way of handling requests could result in an inordinately high response time in the event of a spike in traffic. Additionally, there is also high coupling between the client and the web application.

Again, as demonstrated in the pizza store example, Message Queues eliminate the limitations of synchronous processing by allowing two or more systems to communicate asynchronously. The Message Queue sits between two or more systems and acts as the buffer for messages. One system, called the producer, puts a message on the Message Queue and a second system, called the consumer, picks up the message from the queue and processes it.

Let's break down the roles of producers, consumers and messages, shall we?

Message Queues: A formal definition

The key to understanding the concept of Message Queues lies in understanding two terms: queues and messages.

A queue is a data structure that lines things up in the order they arrive- thus, the first item to arrive in the queue will be the first item out of the queue.

A message is the piece of data the sender application adds to the queue. For example, a message added to the queue could be a request for the receiving system to perform some task (e.g send an email or resize an image). A message could also just be plain text or information about a finished task.

Putting the two together, we can define a Message Queue as the technology that accepts messages from sender application(s), lines the messages up in a buffer, and allows the receiver application(s) to process the messages in the order they arrive.

Figure 4 - Messages in a Message Queue

One key thing to point out about Message Queues is that they foster asynchronous communication between the systems or processes involved. In other words, when the first process puts a message on the queue it does not wait for an immediate response. It proceeds with its execution and the receiving system processes the message at its own pace.

The architecture of a Message Queue is simple; client applications called Producer create messages and deliver them to the Message Queue. Another application, called a Consumer, connects to the queue and gets the messages to be processed. Messages placed in the queue are stored until the Consumer retrieves them. As mentioned earlier, this way of handling messages decouples the sender application from the receiver application. This is true because the Message Queue sits between the systems involved and these systems don’t have to talk to the Message Queue simultaneously.

The ability to decouple systems and have them communicate asynchronously is one of the key advantages of Message Queues. Let’s pursue this benefit of Message Queues further.

Decoupled systems with Message Queues

We mentioned that one of the key things Message Queues help us achieve is decoupling. But what does decoupling really mean?

Decoupling, as far as software goes, means developing software components that are self-reliant. In other words, software systems or components that don’t rely so much on other systems to function– so much so that if the systems they depend on fail then they also fail.

One way to achieve a decoupled system is to establish communication between two or more systems without connecting them directly. Even though the systems involved talk to each other, they can remain autonomous and unaware of each other. This is exactly what Message Queues help us achieve.

If one process in a decoupled system fails to handle messages from the queue, other messages can still be added to the queue and processed when the system has recovered.

Figure 5 - Producers and Consumers

Instead of building one large tightly-coupled application, it is recommended to break the monolith into smaller components. You can then have these smaller components communicate asynchronously using Message Queues.

This way, each component can evolve independently, be written in different languages, and/or be maintained by separate development teams.

A Message Queue will keep the processes in your application separate and independent of each other. The interacting systems don’t need to communicate directly. The client application can just put the message in the queue and then continue processing.

The consuming application can also handle its work independently, taking the messages from the queue when they are able to process them. This way of handling messages creates a system that is easy to maintain and scale.

Conclusion

This article gave an overview of what Message Queues are. Essentially, Message Queues foster asynchronous communication between systems. They accept messages from client applications called the Producers, buffer these messages in a line, and allow Consumers to pick them up and execute them at their own pace.

This asynchronous communication pattern decouples the Producer from the Consumer and by extension, encourages the implementation of systems that are easy to scale and fault-tolerant.

Now that we understand the basics of Message Queues, we will look at some of the practical use cases of Message Queues in the next article in the series.

Do you want to learn more about Message Queues before we continue? We wrote an article that explains the basics of Message Queues as well as Message Queues from a non-developer perspective.

For any suggestions, questions, or feedback, get in touch with us at contact@cloudamqp.com