Kubernetes Nginx Ingress: Consistent hash subset load balancer

Last month we got a Pull Request with a new feature merged into the Kubernetes Nginx Ingress Controller codebase. This feature request came from a client that needs a specific behavior of the Load Balancer not available on any Ingress Controller.

Link to the PR: https://github.com/kubernetes/ingress-nginx/pull/3396/

What’s the use case?

Our client’s software is a multi-tenant machine learning application that caches and store some information in local memory. Each tenant has multiple users, so this information is utilized multiple times by different users. Initially, all the users of the same tenant should be routed to the same instance to take benefit from caching, but if we have an increasing number of users on some specific tenant we could have a load issue. The proposed solution was to distribute the load to a subset of the instances. The hashing algorithm maps the key to a subset of nodes, and the distribution in the subset is random. The key can be a URL, a header, or any other information in the HTTP request header.

What’s the Nginx Ingress controller?

All the Kubernetes applications live in a private network inside the cluster. By default, applications are not exposed to the world. If you want to make them reachable, you have two options:

• Services: It’s a special object type that allows applications to be reachable internally and externally in the cluster. You can define a service to expose your applications and there are two options to expose them to the world: NodePort and LoadBalancer. Both have limitations when you have multiple applications, or if you have to manage virtual hosting.
• Nginx Ingress controller: It’s a reverse proxy running in your cluster, like any other application, but it’s the only one exposed to the world. It acts as proxy to route the external requests to the right pod in the internal Kubernetes network. Nginx is the most popular ingress controller, but there are others using HAProxy or Envoy in the backend.

What’s the Nginx load balancer?

Usually, there is not a single pod behind an external endpoint or URL in your cluster. Instead, you deploy a “Deployment” object with multiple replicas or instances of the same application. Internally, when you build this scenario in Kubernetes, Nginx is configured dynamically to route the traffic to multiple upstream servers. The load balancer algorithm works here, deciding how to distribute the traffic. You have multiple options: round robin, hashing, etc.

What’s the consistent hash subset load balancer?

The Nginx controller was shipped with the consistent hash algorithm which maps a client to a specific upstream server based on some configurable criteria, for example, an argument in the URL. So all the HTTP requests with the argument value are going to be routed to the same pod. The subset option is slightly different, because instead of mapping to a single instance, it maps to a subset of the instances. The traffic is distributed inside the subset at random.

How to use it?

The behavior of the Ingress Controller is defined in the Ingress object using annotations. The following example shows how to enable the consistent hash subset with a subset size of 3. The key is the URL argument predictorid.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-hash-by: "$arg_predictorid"
    nginx.ingress.kubernetes.io/upstream-hash-by-subset: "true"
    nginx.ingress.kubernetes.io/upstream-hash-by-subset-size: "3"
  name: nginxhello-ingress
  namespace: default
spec:
  backend:
    serviceName: nginxhello
    servicePort: 80

Conclusion

There were some options to satisfy this requirement, such as hacks or creating a custom version of this controller, but getting it into upstream was definitely a better option. The flexibility of open source and a friendly community like Kubernetes allow getting new features without being blocked by a vendor.