The property listing platform, Hemnet, moved from a full on-premise solution to a cloud-based solution in less than a year. This is the story about how they changed from in-house to the cloud, and how RabbitMQ became an important player during the migration.
Hemnet is Sweden’s leading online property site for home buyers, sellers and dreamers. It was founded in 1998, as an initiative from the Swedish real estate broker associations, and has since then been “growing with the internet”. Hemnet today ranks among the top websites in the country, making it a key destination for real estate listings and transactions.
Hemnet has about 2.8M visitors and 18M search hits a week. Around 40 million hours were spent on Hemmet during 2017 according to Anna Lagerborg, CTO at Hemnet.
CloudAMQP made a visit to Hemnet’s HQ in Stockholm, Sweden, where we were given the whole story about Hemnet’s tech journey; The migration from in-house to the cloud. We had a chat with the CTO Anna Lagerborg and the developer Jonas Liljestrand.
Background Hemnet
Hemnet has around 30 in-house developers and together they are building the website, mobile applications and products for real estate agents.
The website shows everything regarding real estate listings, transactions and statistics. Some examples are data from what homes that are being listed in different areas in the country, including a map of properties for sale in an area. As well as energy cost, living cost, insurances and history of final prices for listings in various areas etc. Making it easier for brokers, buyers and sellers to forecast a reasonable selling price for a property.
Before the migration, Hemnet talked a lot internally about the DevOps role. They concluded that the best solution for them would be to empower all developers to be able to respond to needs of the business in near real-time and make all developers familiar with the infrastructure in the cloud. This instead of having a single role handling all change requests. Therefore, the goal was to educate all developers about the infrastructure in the cloud, so that all of them could do configuration changes if needed. All developers should also be able to try out new services etc. easily by just entering the product catalogue at a cloud provider such as AWS.
The Migration Project
Hemnet had built their platform as an in-house project but decided to move to a full cloud hosted solution. The reason for this was that Hemnet wanted to be able to scale in an easy way, and that was the strongest argument to enter the cloud. They also wanted to outsource their message queues and their databases, using Software as a Service (SaaS) solutions instead of self-hosting. Great SaaS companies usually include easy ways to upgrade the software as part of the solution and other features that make life easier for in-house developers.
We wanted to be able to scale in an easy way, and that was the strongest argument to enter the cloud. - Anna Lagerborg, CTO
The migration project started in March 2017. Hemnet decided to switch from their on-premise solution to AWS, where they could put their data in different datacenters. One single datacenter located in only Stockholm was not an option since they required HA (high availability).
On-Premises to the Cloud with RabbitMQ
The project team started by investigating options for the migration. Questions such as “How should the migration be performed?”, “What tools should be used?”, and “How long time will the migration take?” needed to be answered. Once these questions had been dealt with, a protocol of things to do was established and then followed until the end. Lift and shift migration was a big no-no. The goal was to migrate all applications to a docker runtime hosted in a cloud environment. All services like databases and message queues were going to be outsourced. However, for some cases, there was no substitute third-party services, so then they built them themselves.
Hemnet had a well-designed architecture written in Ruby, with RabbitMQ as message broker in between many of the services, already before the migration. Thus, they did not spend too much time trying to change any part of the systems before the migration. The fact that the architecture was well-designed from the beginning facilitated the migration to the cloud.
The architecture made it possible to move one part at the time, off the system. - Jonas Liljestrand, System Developer
Hemnet’s use of Message queues / RabbitMQ
Hemnet uses RabbitMQ, among other things, as a pipeline for image scaling. Once a real estate broker adds a new image of the property, an image scaling task is given to RabbitMQ. The image scaling task stays in the message queue until Hemnet’s image scaling service grabs the task from the queue, scale the selected image, and in the end, the image is ready to be shown on the website or in the app with the new size.
This scaling service was easily moved to the cloud while scaling requests were simple idling in the queue until the scaling service was moved and ready to be started again.
Another simple use case is the visitor counter on objects. When a real estate broker adds an object to Hemnet on behalf of a seller, the seller is usually eager to see how many visits their specific object has gotten. Every visit on an object is sent to RabbitMQ and stored in Rabbit until the counter-service can handle the task and add the visit count into the database.
The counting service adding up the visitor-count could easily be moved while every visit-count made on the website was still collected. The visitor count task is idling in the queue until the consumer is ready to handle the database update requests.
One card at the time could be moved from the house of cards
Previously, Hemnet hosted RabbitMQ in-house. During the strive to outsource services to SaaS companies, and while looking for a managed solution for RabbitMQ to assist the journey to the cloud, Hemnet found CloudAMQP. One requirement Hemnet had, was that RabbitMQ had to be hosted in a private VPC (Virtual Private Cloud). CloudAMQP fulfilled Hemnet’s needs in regards to simple AWS VPC setup options, which did let them simply define a private network in the cloud, via the CloudAMQP control panel. Hemnet was also looking for a hosted RabbitMQ service with default availability of the RabbitMQ API.
During the migration, Hemnet simply moved one service at the time and, therefore, RabbitMQ become a central part of the migration. “One card at the time could be moved from the house of cards” said Jonas. RabbitMQ was still able to receive messages/tasks, even though the consumer of the messages were down.
Hemnet stopped the consumers for a service and redirected the data to the new consumer in the cloud. The lovely thing with RabbitMQ in between all the services was the fact that no data was lost, even though the service was down for a while, the queue just filled up.
Jonas said that even though the migration went good - the stress level could be a bit high sometimes. Hemnet have a message flow with around 400 msg/sec in the system, and when a consumer service is down, it quickly adds up lots of messages to the queue, and a too long queue is never recommended.
RabbitMQ and CloudAMQP became a key part of the migration since it gave Hemnet the freedom to move one service at a time. RabbitMQ was always able to receive messages and traffic, even though the consumer of the messages were not there. The whole migration was finished in January 2018, in less than 9 months.
Thank you Hemnet for your time!
We received a lot of great feedback about CloudAMQP during the visit. Feedback is of great value to us and always appreciated. Let us know if you have an interesting RabbitMQ story that you would like us to tell the world, just sending me an email. Happy coding!
