helm: stale TLS blocks in ingresses cause ssl-redirect loops and break websockets

[mosoriob]

Problem

When upgrading the mint Helm release, ingresses can retain stale spec.tls entries from previous deploys even when the current chart values set tls: []. Helm's 3-way merge does not remove fields that were added by a prior release but are absent from the current manifest.

Because nginx-ingress enables ssl-redirect by default on any ingress that contains a TLS block — regardless of whether the request host is listed in that block — all hosts on the affected ingress get a 308 redirect to HTTPS, and the backing self-signed cert causes browsers to fail.

Symptoms

• curl -I http://mint.local returns 308 Permanent Redirect to https://mint.local
• Chrome shows ERR_CERT_AUTHORITY_INVALID and HSTS makes it sticky
• UI websocket fails: WebSocket connection to 'ws://graphql.mint.local/v1/graphql' failed
• kubectl get ingress -n mint shows stale TLS hosts (e.g. mint.tacc.utexas.edu, graphql.mint.tacc.utexas.edu) on local-only installs

Reproduction

1. Install mint chart once with TLS values for a production host
2. Redeploy with a values file that does not set tls (e.g. a local dynamo-values.yaml)
3. Observe: live ingress still has spec.tls and redirects HTTP to HTTPS

Proposed fixes

Any of these would prevent the issue; ideally combine (1) and (2):

1. Set ssl-redirect: false by default in the chart for local/dev scenarios, or make it configurable per ingress component:

   components:
     ui:
       ingress:
         annotations:
           nginx.ingress.kubernetes.io/ssl-redirect: "false"

2. Always render the tls: key in ingress templates, even when empty, so Helm's merge correctly removes stale entries. Current template uses {{- if .Values.components.ui.ingress.tls }}, which omits the field entirely and leaves stale data intact.

3. Document the upgrade gotcha in the chart README with a one-liner to clean up:

   for ing in $(kubectl get ingress -n mint -o name); do
     kubectl patch $ing -n mint --type=json -p='[{"op":"remove","path":"/spec/tls"}]' 2>/dev/null
   done

4. Add a global.tls.enabled switch so local installs can disable TLS wiring across all ingresses at once.

Affected templates

• charts/mint/templates/ingress-ui.yaml
• charts/mint/templates/ingress-hasura.yaml
• charts/mint/templates/ingress-ensemble-manager.yaml
• charts/mint/templates/ingress-data-catalog.yaml
• charts/mint/templates/ingress-model-catalog*.yaml
• (all ingress templates use the same {{- if ... .tls }} pattern)

Workaround (current)

Manually patch stale ingresses after upgrade:

kubectl patch ingress mint-ui -n mint --type=json -p='[{"op":"remove","path":"/spec/tls"}]'
kubectl patch ingress mint-hasura -n mint --type=json -p='[{"op":"remove","path":"/spec/tls"}]'