How to connect your SpringBoot application to multiple RabbitMQ brokers? But why do you need multiple RabbitMQ brokers in the first place? We wrote this simple-to-use open source library to solve different use cases from brokers in different PAAS providers to dedicated brokers.

Short biography
Wander Costa ( Twitter, GitHub ) is a Software Engineer based in Hamburg, Germany. He has over 13 years of experience with Java, many of them focused on microservices with Spring stack. He is interested in scalability and creating things.

When not programming or running side projects, he pets his dog Chewbacca and strives to make good cappuccinos.

How to use multiple RabbitMQ brokers with SpringBoot

My name is Wander Costa, and I work with a company called Free Now in Hamburg, Germany. I'd like to also thank the guys that are running this conference, and for inviting me. It’s a pleasure to be here. 

Today, I'm going to talk about RabbitMQ and SpringBoot. SpringBoot doesn't have a good way to connect to multiple RabbitMQs. I mean it has, but I'm going to talk about it a little and how to simplify it further.

Talking about simplicity, according to the dictionary, simplicity is the quality or condition of being easy to understand or to do. It turns out that life is not simple. We’ve got several complex things around us. A good example is programming. 

From the perspective of Java, if you take a look at this code, it's hard to understand. This is the implementation of a hashmap in Java 8. It’s pretty complicated, it would take me half an hour to study and understand it. 

When I got started, it turns out that there are several complexities around it. Hash maps, for example, are meant to be performant, and they have a lot of logic happening in the backend to make things happen, to prevent collisions, etc. 

I prefer to simplify things, and this is a kind of code that I don’t like to work with. But, we are lucky that we have Spring for the Java environment job and coding environment.

Springs is a web framework that has been around for years now. It was launched around 2014. Five years ago, we had the first SpringBoot launched immediately after the very first version of SpringBoot was just wrapping up. 

All the things that Spring does make it easy for Java developers to launch their services. With the help of microservices we got an excellent input from Spring with this.

For example, if you look at this code, it uses Spring Boot underneath, providing a hello world, endpoint, which is quite simple. It took some lines for me to make it happen. 

The Spring Boot running in the background is doing a lot of things. Holding a lot of complexities and making it easy for the developer to start with. 

In this case, we have Tomcat running, let's say with 2 megabits of limitation on the requests. If you entered a query, we have this simple endpoint working with the hello world which is straightforward.

And then, I came to the RabbitMQ, which is our main focus here at this conference. This is a simple implementation of publisher consumer logic using Spring Boots. Just to give you some thoughts about it. I'm consuming from Q1, which is automatically down here with the exchange 1, running key one. 

And I'm publishing messages here and consuming from here and it turns out that we got this in the console like receiving this message. It’s very simple, easy to start with Spring Boot and RabbitMQ. 

However, where is the configuration? Which service or which server am I connecting to? 

It turns out that we have this easy way to set up configuration with Springboot, like setting Spring, RabbitMQ for username and password. You can set up anything as it is very easy to couple, especially the configuration and the code.

This is viewed from the perspective of the developer. This is because Spring favors convention over configuration. So whatever you have in Spring Boot is already configured for you to start running the application. It would be production-ready, or you can just deploy. 

Everything is going to be configured. If you have to configure something differently, it can be done with ease. This approach makes things easier with Spring Boot.

Let’s go back a little bit. As I mentioned, I work with a company called FreeNow. This company is from Hamburg. I moved from Brazil to Germany at the beginning of last year.

The name of the company used to be mytaxi, and later on, in the middle of this year, we rebranded to FreeNow. 

Here are some numbers. We have got over 150 developers. We run more than 300 JVM bays, micro-services. In less than a month, in the live cluster alone, we handled over 21 million messages in RabbitMQ. Even though this number is not massive. It’s quite fair.

At the end of the last year, we had a different requirement, which was to introduce another RabbitMQ and another broker. We wanted this new broker to be shared amongst some of the variants that we had.

Eventually, we couldn't use our main broker or main RabbitMQ cluster and it was in the requirement. Let's simply configure our services to use the second brokers to connect to the second broker.

We found out that Spring does not offer this support. You cannot say, 

“hey Spring, Spring.RabbitMQ1, something RabbitMQ2.”

It cannot be configured this way because the Spring Boot expects one single instance or one single broker in the configuration. 

However, you can do this manually, but from the configuration, from the properties, you can only set one broker 

We found this problem, which showed that we cannot move with the second broker. We discovered that we could use Spring Cloud Streams to solve this problem. Spring cloud stream is fantastic. It was introduced in abstraction, so we can talk to either RabbitMQ or Kafka and eventually have multiple connections or multiple brokers.

Talking to multiple brokers in the same application. The point is that we had over 300 microservices, and we didn't want to add an abstraction layer, which would require us to change the 300 microservices. 

The cost of changing these bunch of services would be very high, and we didn't want to move in this direction. Instead, we thought if we could implement a library that does this for us. 

A library that instantiates bins, the connections with Spring, instantiate the connection factory, the container factory, the rabbit admins, and all the things that would make the connection work. The annotations in Spring still work just fine. 

Let's do this, but some of the requirements that we had were that we must have some library that was an extension to Spring AMQP. It would have minimal code change in the selves that would use this. We could not have any internal library. It should be decoupled from whatever we have from the internal systems.

It would require a little configuration to start with, and the more important thing is that it should not be intrusive. When this library service is introduced of which it was running before with the configuration with one broker, ideally, it shouldn’t break the system. That's what we were striving for. 

So we came up with this Spring multirabbit, which is a library we created. Spring multirabbit is going to be used in a great way as we put a lot of thought into it to meet several requirements. All you need to do is to import the library, spring multirabbit. Currently, it's still available under the group ID mytaxi. 

Just as mentioned earlier, we branded this year, so the library was launched at the start of the year. We wanted to move from one configuration to another and have everything set by default. 

Everything should be done and configured with Spring which is easy to be implemented. Whenever we have a default RabbitMQ server that you’re connecting to in your application. You can still have this connection as before, and you can have additional connections 

If you take a look at this configuration, we have broker 2 and broker 3. These are just names that I've created for that. I can call it whatever I want. I can also have thousands if I want it in the system.

What I’m saying is that it has no limitation on how many brokers you can configure. Eventually, we came from this kind of code which was connecting and listening and publishing from and to only one single broker or one cluster and we introduced this code which seems a little bit at first but I'm going to break it down a little. 

For example, before I had this consume one which was listening to q1, I have exactly the same method here. I introduced two more consumers, the only consistent difference between them is that these two other ones have this container factory broker. 

In this case, it is container factory broker 3 and that's where the dots are connected. Whenever you say container factory broker two, it's going to use the bins. It will connect to the bins, they're produced from the additional configuration that we added in the previous slide.

The same thing happens here when we publish messages, so before we were using rabbitTemplate.convertAndSend to the default broker and it was sent to additional brokers, I need to use a connection factory context wrapper.

This will bind the connection factory and unbind it afterward. Here, we can provide a lambda and like a runnable for example and i can ensure it’s binding and unbinding automatically done via this class. So even though it seems a little daunting, it's not. 

It comes by introducing more brokers. It is some kind of requirement that no application, no system has. Eventually, if you have, you can still figure out how to do it with this library. 

Then, we can take a look at these different servers that I was running just before the conference. I have the Q’s receiving each one in a different server running on docker, and I have a receiving message from all of them as an amplification of this code before. 

So we made an extension of spring AMQP, we have minimal code change, so whatever we have before running, Spring and RabbitMQ wouldn't crash. It wouldn't require too much change. 

Nothing related to our internal system and it requires minimal configuration for you to start with. Just a couple of additional properties, setting up the container factory and the contacts, and it's non-intrusive 

Why not share it?

We thought about sharing this library and that's what we did in January this year. After publishing in this library, I wrote this blog post in our internal blog and I was just teaching how to use the library, showing how to configure it and how you can easily get it done 

I got a comment from a nice guy called Gary Russell. Gary Russel is the project lead from Spring Integration AMQP. Gary asked: “Would you be interested if I should contribute my work to the framework?” 

I said “wow, this is awesome let's do this so this.”

Since then, we got very busy, but we created a pull request. This feature is going to be contributed to the framework soon. There's a pull request already open. I actually need to do two Pull requests to have it done. It is a work in progress but you can still use the library as it is now. In the future, it's going to be by default inside the Spring framework 

I want to just share some statistics with you, one month ago, I posted this on my LinkedIn account. We don't have this contribution yet, so people are still using the library and we have had more than 6,000 downloads so far. One month afterward, we've got 10,000 downloads which is still not large enough.

If you consider the use case, it is very strict to use. The application must be talking to multiple brokers at the same time, and at the same moment. 

If you want to take a look at our website or our blog, we have some tech information there which you can follow. This library is published on Github, Free Now tech, Spring, and multirabbit. In addition, the code that I shared here and the presentation are also available on my GitHub account. 

This are my social networks, my blog, if you're interested in following or anything, so I'm open for any questions that you might have if you have any

Q & A Section

I haven’t encountered any issues actually why one of the brokers is being threatened or like the is application is struggling while publishing to one and that impacts no other connection to the other broker or and they're completely separated 

Sorry, I'm not sure if I understood your question, but you're talking about the problems with the connection when there are different brokers

Yes, so like in your application, you suffer from a specific issue with one broker, not both but just one of them. Does this impact the application in its entirety or does the application end?

The answer is no, because when we use Spring, we have the bins and connection factories and in this case, for multirabbit, we create different connection factors for each of the brokers 

It's a caching connection factory, so they are pretty different, if there is a problem happening with one of the brokers, it cannot cause any trouble with the other connections or it should not. If you're having your application, some flow that depends on one to reach the other it might be a different answer, but the answer by default is no 

Any questions? 

Our use case was that we wanted to move on the configuration to start using the Spring cloud config it provides. It can enable the application to grab information from Git repositories and it has a feature of hot reloading. This hot reloading depends on RabbitMQ and the points that we wanted this RabbitMQ for hot reloading to be shared among some environments 

There are other use cases, if there are rules strict rules around payment data in companies. The payment data or the payment infrastructure must be in a separate account in native and this is another use case.

I mean it's not a problem RabbitMQ cannot handle, The problem is that whenever we have a strict use case that you must use a different RabbitMQ, this comes a little bit tough without the library and that's the thing that we wanted to achieve here with library

That's the point we wanted to make it as an extension of what Spring does today. So the way Spring constructs the factories and all the bins, we just plug into it. Should Spring change in the way it creates the connection factories, multirabbit will not use the same method as spring, so it's not going to be a problem 

If the broker is updated and it's still compatible with Spring, it's going to be compatible with multirabbit, and it's not the problem. That was what I was trying to say when I mentioned that we wanted to have an extension to what Spring does. We do not implement this ourselves, we simply plug into Spring, and it recreates other flows 

Are you configuring them on each broker level, or is it per default, and all others have the base configuration?

Whatever you have is possible to configure via properties in Spring, it's can also be done in SpringBoot. If you want to provide message converters and then you create a new bin with your own message converter that is going to be used for all of them. Unfortunately, as it is, there is no way to make a difference between one or the other whenever you produce your bins 

There's a way to plug into multi rabbit, which is identical here in bins and then we can talk about it afterward. You will provide the connection factories the way you want and multiple connection factories of how many you want, and then we can plug something like this.

[Applause]