The previous article reinforced how RabbitMQ can allow services to communicate asynchronously. This post will stretch our knowledge of the application of RabbitMQ in a microservice architecture beyond just “knowing”. To do that, it will walk us through building a demo project that uses RabbitMQ in a microservice architecture. But first, let’s introduce the demo project.
Note: this the part 4 of a 7-parts series. If you'd rather start from the beginning and learn ablut message queues from scratch, start from part 1, understanding message queues.
That being said, it is common for job hunters to regularly visit different job portals to find relevant jobs to apply for. As you’d imagine, doing this every day could be very time-consuming, but what if this could be automated?
Yes, we will be building a solution that solves this problem. We will create a web application that accepts a target keyword (e.g., Python Engineer), a location (e.g., Nigeria or Remote), and an email address from a user. The software will then grab relevant job postings from the web matching the target keyword and email them to the user.
Let’s call our piece of software GetMeHired. Essentially, GetMeHired is supposed to eliminate the need to visit different job portals and search for relevant jobs manually. This will make it painless for job hunters to discover new and relevant job postings.
Before we proceed, keep in mind that “supposed” has been deliberately emphasized in the previous paragraph to underscore an important point – our implementation of GetMeHired will be so simplified that it is not going to be anywhere near a good solution.
Let’s talk about the implementation details a little bit.
GetMeHired: High-level architecture
GetMeHired will adopt the microservice architecture, where it will be broken down into two smaller services. We will call the first service GetMeHired-producer and the second service GetMeHired-consumer.
We will then have RabbitMQ sit between these two services to allow them to communicate with each other. Let’s explore the function of each service.
GetMeHired-producer
As the name suggests, this service will serve as the producer of messages in our architecture. Essentially, it will expose just one REST API endpoint that accepts a POST request from an end user. Each request will carry the search keyword, location, and email address as its payload. For example, see the snippet below:
{
“search term” : “Python Engineer”,
“location” : “Nigeria”,
“email”: “random@gmail.com”
}
This service will then publish each request it receives from an end user to RabbitMQ. But before publishing a request to RabbitMQ, the service will perform some validations – for example, verify that the submitted email address is valid.
Remember that this service will publish each POST request it receives as a message to RabbitMQ. Think of each message published in this case as an instruction for the consumer to grab job postings matching the keyword in the payload from the web and send the job links to the email in the payload.
The low-level details of how the GetMeHired-consumer would fetch the relevant job postings and email them to the user is totally abstracted from the first service. We will implement the GetMeHired-producer service with FastAPI, a Python web framework.
GetMeHired-consumer
As its name implies, this service will function as the consumer in our architecture. This service will be responsible for all the heavy lifting – it will receive messages from RabbitMQ. For each message, it will search Linkedin for relevant jobs matching the parameters in the payload.
This service will use the Linkedin Jobs Search API for its searches. The service will then email the result of its search to the user whose email address was submitted in the payload.
Furthermore, this service will leverage Twilio Sendgrid to send emails to users. We will also implement this service with FastAPI.
Should you decide to extend this project beyond this tutorial, scraping, let's say, the top five job portals directly instead of connecting to an API is a good idea. That way, you can be in control of a lot of things. For example, you get to decide what portals to scrap.
You can also build a standalone user interface with which users can interact . Right now, the only way to interact with our software is through the first service's API endpoint .
Overall, this is what our architecture will look like in picture.
Figure 1 - GetMeHired high-level architecture
The genius of this architecture lies in the following:
- Even if GetMeHired-consumer accidentally crashes, GetMeHired-producer will keep producing messages and adding them to RabbitMQ and the end users wouldn’t even notice
- If we want to process more messages quickly we can create more copies of GetMeHired-consumer
Project requirements
To successfully follow the tutorials in the subsequent articles that would help you build this demo project, you should meet the following requirements.
- Knowledge of Python
- Basics of FasAPI, a Python web framework
- CloudAMQP account – to run RabbitMQ on the cloud
- RapidAPI account – to access the jobs search APIs
- Twilio Sendgrid account – to email users
Conclusion
This article gave us an overview of the demo project we will be building – GetMeHired. This project will adopt the microservices architecture, breaking it down into two smaller services. To begin, in the next article we will build the first service – GetMeHired-producer and have it connect to RabbitMQ.
For any suggestions, questions, or feedback, get in touch with us at contact@cloudamqp.com
