Connection limit per user - a community requested feature made available by Erlang Solutions and CloudAMQP

One single faulty client application may in the worst case scenario exhaust the maximum number of connections that RabbitMQ can handle. Such a "connection leak" may eventually cause the RabbitMQ instance to crash when pushed to the limit. Thankfully it's a rare case, but it sometimes happens. The real question is: how can this be prevented?

According to CloudAMQP customers, Erlang Solutions clients as well as discussion boards online (such as GitHub), one suggested feature is to limit the number of connections a single RabbitMQ user can make.

The flawed implementation of the client application will still need some love and care, but this new feature will at least prevent it from jeopardising the RabbitMQ broker for the rest of the architecture.

Meeting our customers’ demands is always one of our top priorities and since this feature request has been on our minds for quite some time, we decided to reach out to Ayanda Dube at Erlang Solutions to help us finish the task. Before we’ve done manual work-arounds as stop-gap measures, but this helps us and the community long-term.
- Johan Rhodin, CloudAMQP

So, with the release of RabbitMQ 3.8.10, this feature became available, providing the ability to set connection limits per user. Made possible through great collaboration between Erlang Solution and CloudAMQP.

The new feature allows RabbitMQ administrators to set a connection and/or channel limit per user, whereas previous versions only offered a means to set connection limits per virtual host, leaving an inherent risk of client applications with connection leaks gradually building up unused connections which would ultimately exhaust and crash RabbitMQ when it ran out of resources.

In the end, we built a very very useful feature that is both straightforward and easy to use from a configuration perspective
- Ayanda Dube, Head of RabbitMQ Engineering - Erlang Solutions

Feature Specification

Whenever a connection is opened in RabbitMQ, there are multiple operations taking place which are of critical importance, therefore processing and validating limits needs to be fast. Hence, when building this new feature it was necessary to create a highly optimized processing layer, to avoid performance problems for the system.

Limiting the number of connections per user is, in terms of the design, very closely related to the already existing, virtual host connection limits. The internal lookup tables used for tracking the vhost-connections were extended with the addition of more internal lookup tables for tracking the active number of connections and channels for a specific user.

Usage

User limits may be set and controlled using HTTP API/Management UI, or from CLI tools as follows:

Setting user limits via RabbitMQ Management UI and HTTP API

Navigating to Admin -> Limits tab, under the User limits section, the following interface will be presented, on which connection and channel limits for specific users, may be set or cleared:

The illustration shows the Management user interface for controlling the limits

Alternatively, these may be set by directly executing the user limits HTTP API as follows:

To set 1000 maximum number of connections for user guest:

$ curl -4u 'guest:guest' -H 'content-type:application/json' -X PUT \
localhost:15672/api/user-limits/guest/max-connections -d '{"value": 1000}'

To set 500 maximum number of channels for user guest:

$ curl -4u 'guest:guest' -H 'content-type:application/json' -X PUT \
localhost:15672/api/user-limits/guest/max-channels -d '{"value": 500}'

To view all active connection/channel user limits:

$ curl -i -u guest:guest http://localhost:15672/api/user-limits
HTTP/1.1 200 OK
cache-control: no-cache
content-length: 70
content-security-policy: script-src 'self' 'unsafe-eval' 'unsafe-inline'; object-src 'self'
content-type: application/json
date: Wed, 09 Jun 2021 17:58:54 GMT
server: Cowboy
vary: accept-encoding, origin

[{"user":"guest","value":{"max-channels":500,"max-connections":1000}}]

To clear connection limits for user guest:

$ curl -4u 'guest:guest' -H 'content-type:application/json' -X DELETE \
localhost:15672/api/user-limits/guest/max-connections

To clear channel limits for user guest::

$ curl -4u 'guest:guest' -H 'content-type:application/json' -X DELETE \
localhost:15672/api/user-limits/guest/max-channels

Setting user limits via Command Line Interface (CLI)

User limits can also be controlled using the CLI tools as follows:

# limits user "guest" to up to 1000 connections and 500 channels
rabbitmqctl set_user_limits "guest" '{"max-connections": 1000, "max-channels": 500}'

# clears the maximum number of connections limit for the user
rabbitmqctl clear_user_limits "guest" "max-connections"

# clears all limits for the user
rabbitmqctl clear_user_limits "guest" "all"

Source:

https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.8.10

Design

Setting and Clearing Limits

User limits can be set via the CLI or the Management UI/HTTP API. When set, internally, RabbitMQ validates the specified limits and updates its user database with the limits set for a particular user. Prior to doing so, a prerequisite check on whether or not the user_limits feature flag is enabled is carried out. This means that RabbitMQ will only expose this feature if the user_limits feature flag is enabled.

The illustration shows the administrator controlling the limits through CLI options or the RabbitMQ Management interface.

Once a specific user's connection or channel limits have been set in the internal user database, RabbitMQ will update and emit internal statistics and events, marking completion of this operation. (Clearing limits follows this same operational path, however instead of setting connection and channel limits for the specific user in the internal user database, the user’s limits are cleared).

Connection/channel establishment

Once connection and channel limits have been set for a specific user, connecting client applications are then validated if their current active connections and channel counts exceed the configured user’s limits or not. Internal tracking tables are responsible for maintaining the current connection and channel count(s) per configured user. On creating a new connection or channel, if the connection or channel limits are exceeded respectively, the request fails. If the connection or channel limits are not exceeded, then the client request succeeds.

Enhancements and optimizations

Working with this project also led to the optimization of previous work when a few bottlenecks were discovered during the process. Previously, tracking active connection and channel activity (e.g. creation and deletion events) was handled by a single processing thread. In the new design, multiple activity events handling has been improved to a multi-threaded approach, implying faster processing and no bottlenecks during high connection/channel churn. Additionally, connections which were terminating were not always cleared out of their vhost tracking tables, something which the user never noticed. As part of this feature, this issue was detected and resolved, thereby improving the existing connection tracking implementation for virtual hosts.

Also, support was added for both network connections and direct connections. A lot of plugins use direct connections and this opens the possibility for the feature to be used with MQTT, Shovel, or federation plugins, and the like.

For more detail regarding the design and implementation, refer to the pull request:
Per user connection and channel limits

---

A special mention to Ms. Anupama Sighn as well, from Erlang Solutions, who collaborated with me in the Erlang and Elixir implementations of the significant and critical parts of the user limit validations on RabbitMQ’s internal auth-backends, the feature flag mediation modules, as well as the CLI tools
- Ayanda Dube, Head of RabbitMQ Engineering - Erlang Solutions

This has been an exciting project and we are pleased with how it turned out. We hope you will find the new feature useful, and if you have suggestions on how to further improve RabbitMQ and make it even more user-friendly, don’t hesitate to reach out to us!