TL;DR

The problem

We want to define multiple ingresses for the same domain name in nginx ingress controller. A user-defined header should be used to decide which ingress to use.

The solution

The solution is to overwrite the generated upstream based on an incoming header value:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      if ($http_namespace){
                set $proxy_upstream_name "${http_namespace}-service-80";
                }
…

Background

To understand why something like the problem described above might even arise, it’s helpful to understand our basic setup and some of the restrictions that come with it.

We’re currently running our development stage in a Google Cloud Kubernetes environment (GKE). Our deployment follows a strict GitOps approach, which means we describe both infrastructure and applications in a git repository, which is then synchronised to the actual cluster with Google’s config sync. 

To reach applications from the outside, we decided to use the nginx ingress controller. This decision came with one major advantage, as well as one major disadvantage. On the pro side, nginx ingresses follow the ordinary declaration of ingresses in Kubernetes. So, no additional knowledge is needed if one is familiar with kubernetes as such. On the other hand, a full-fledged service mesh like istio for example, allows for a more detailed and sophisticated ingress configuration, including A-B deployments, header-based routing, etc.

Finally, we leverage the gitops approach in conjunction with GKE to allow developers to deploy their application to what we call a dynamic environment. Basically, it allows us to run any applications and their dependencies decoupled from the rest of a stage. This feature comes in handy when running automatic tests for new builds. It’s also the reason why we need a way to decide which pod to actually target, even though it is accessible under the same domain name.

For the sake of simplicity, we use the http echo service in the following examples, since it’s easy to understand and still conveys our needs. Furthermore, we assume that an nginx ingress controller is already running and supplied with a wildcard certificate. In our case, we use our own domain: *.unstable.viesure.io. 

Deployment & service

Let’s create an easy deployment and expose it with a Service and a ClusterIP in namespace1:

apiVersion: v1
kind: Service
metadata:
  name: echo
  namespace: namespace1
spec:
  type: ClusterIP
  ports:
    - port: 80
      protocol: TCP
      name: http
  selector:
    app: echo

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-v1
  namespace: namespace1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - name: echo
          image: "docker.io/hashicorp/http-echo"
          args:
            - -listen=:80
            - --text="echo-v1"
          ports:
            - name: http
              protocol: TCP
              containerPort: 80

Now let’s do the same for namespace2:

apiVersion: v1
kind: Service
metadata:
  name: echo
  namespace: namespace2
spec:
  type: ClusterIP
  ports:
    - port: 80
      protocol: TCP
      name: http
  selector:
    app: echo

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-v2
  namespace: namespace2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - name: echo
          image: "docker.io/hashicorp/http-echo"
          args:
            - -listen=:80
            - --text="echo-v2"
          ports:
            - name: http
              protocol: TCP
              containerPort: 80

Normal ingress

So far so good. Normally, we would now create an ingress that looks like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echo
  namespace: namespace1
spec:
  ingressClassName: nginx
  rules:
  - host: echo.unstable.viesure.io
    http:
      paths:
      - backend:
          service:
            name: echo
            port:
              number: 80
        path: /
        pathType: Prefix

While this is perfectly fine, this targets only the specified service in namespace1. To also use echo-v2, there are two different options:

• Expose it under a different domain name (i.e. echo2.unstable.viesure.io)
• Expose it under the same name, but a different path (i.e. path: /echo-v2)

For different reasons, both variants are non-viable solutions for us. Instead, we want to achieve having both variants exposed under the same hostname and path and decide with a request header, which one to use. So the desired outcome is this:

root@machine:~/$ curl https://echo.unstable.viesure.io

"echo-v1"



root@machine:~/$ curl https://echo.unstable.viesure.io \ 

-H “namespace: namespace2”

"echo-v2"

nginx ingress controller configuration

The last part to understand the final solution is the way that nginx ingress controller routes traffic. In a nutshell, each ingress generates a configuration section for nginx where the name of the upstream is set to <namespace>-<service>-<port> before it’s actually proxied to. Furthermore, multiple paths are distinguished by a location section in the configuration. Consequently, the above ingress example would result in the following line:

set $proxy_upstream_name "namespace1-echo-80";

nginx ingress controller configuration

Now finally, we can include a configuration snippet into the main ingress that yields to the desired result:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
            if ($http_namespace){
                      set $proxy_upstream_name "${http_namespace}-echo-80";
                      }
  name: echo
  namespace: namespace1
spec:
  ingressClassName: nginx
  rules:
  - host: echo.unstable.viesure.io
    http:
      paths:
      - backend:
          service:
            name: echo
            port:
              number: 80
        path: /
        pathType: Prefix

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echo
  namespace: namespace2
spec:
  ingressClassName: nginx
  rules:
  - host: echo.unstable.viesure.io
    http:
      paths:
      - backend:
          service:
            name: echo
            port:
              number: 80
        path: /dummylocationwhichisnevercalleddirectly
        pathType: Prefix

As you can see, we create two ingresses for the same host. The second ingress is generated for a dummy path that should never be called directly.

Instead, the code snippet from the first ingress is injected into the generated config and checks if an http header called “namespace” is present ($http_namespace). If this is the case, instead of the original upstream, the namespace is supplied by the header content. 

Of course it’s also possible to overwrite other elements than the namespace. In the end it’s only necessary to create the correct upstream with a dummy location first. 

And finally, should a wrong namespace be supplied, nginx will simply report back a 503.

Conclusion

In conclusion, it is possible to have header-based traffic routing in kubernetes systems running nginx ingress-controller, even though that’s not supported out of the box. However, it requires some decent amount of intervening to realise such a solution.

And last but not least, it might be a monetary consideration as well, because in the end, nginx is still a community solution, while most service meshes are paid solutions.