Add Helm-Chart compatibility with any ingress controller

[mladBlum]

Current Situation:

The ingress resources in the Helm chart contain numerous configurations (annotations) specific to the Kubernetes ingress-nginx controller (ingress-nginx GitHub repository) and are not universally compatible with other ingress controllers.
For individuals using a different ingress controller, the current options are:

1. Disable all ingress and create them manually
2. Override all the defaults in values.yaml

Both methods require considerable effort,  initially and also before each update of Ditto. I think the aim should be that someone who uses the helm chart should have as little effort as possible with installation and updates.

Proposal:

I believe we can achieve a big improvement if we manage to remove the ingress-nginx specific annotations to create a "neutral" ingress configuration. This ingress will align with typical ingress configurations in other charts, allowing operators to add their specific settings for whatever they want to achieve.

While I was validating this approach, I discovered that this should be quiety easy. Ditto-nginx already serves as entry point to ditto and the specific annotations in the current ingress resources are already present in the ditto-nginx config.

See ditto-nginx:

• etc/nginx/nginx.conf
• etc/nginx/nginx-cors.conf

Current ingress annotations:

• nginx.ingress.kubernetes.io/server-snippet
• nginx.ingress.kubernetes.io/configuration-snippet

If everything aligns well, the necessary changes might be minimal and simplifies the current ingress resources.
I welcome feedback if I've missed any points. I know, of course, that many things are not as easy as they seem at first.

Btw. I think there is something odd with the current setup anyway. 
The Ingress ressources apply nginx configurations that are already applied by ditto-nginx. So there are two almost identical configurations that are applied twice.

flowchart LR
subgraph chart [Helm-Chart]
ingress_controller --> ditto-nginx

headers_chart((nginx.config from ingress)) --> ingress_controller
headers_ditto((nginx.config included in ditto-nginx)) --> ditto-nginx

ditto-nginx --> ui(ui)
ditto-nginx --> swagger(swagger)
ditto-nginx --> any(...)
ingress_controller --> gateway
end

client --> ingress_controller(Ingress Controller)

Loading

[mladBlum]

I'm currently testing all of this with HAProxy. I will report back with the results in the next few days.

[mladBlum]

I've now run some tests with a minimal ingress. Below are the results.

What's working:

• /index.html
• /apidoc/
• /ui/
• /health
• /status
• /stats
• /devops
• /api
• /api/2/connections
• Authentication/Authorization with OAuth

What's not working:

• BasicAuth does not work because it was implemented with ingress-nginx specific annotations. However, I think this is fine. A Helm Chart should not add features that are not part of the bundled app itself. It should be quite easy for anyone to add this if they need it.

• OAuth redirect of Ditto UI: When logging in via OpenID, the port on which Ditto UI is running in the cluster is added to the redirect URL. This shouldn't be the case. I have to fix that, or at least find the issue.

The ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
  name: ditto
  namespace: ditto
spec:
  ## Define haproxy as my ingress controller
  ingressClassName: haproxy

  ## Ditch the default backend. The Ingress Controller itself should handle all not configured routes 
  # defaultBackend:
  #  service:
  #    name: ditto-nginx
  #    port:
  #      name: http

  ## Route every request to ditto-nginx
  rules:
  - host: ditto.example.com
    http:
      paths:
      - backend:
          service:
            name: ditto-nginx
            port:
              name: http
        path: /
        pathType: ImplementationSpecific