feat(networking): Add Envoy Gateway with Gateway API support#2644
Open
dapperdivers wants to merge 2 commits intomainfrom
Open
feat(networking): Add Envoy Gateway with Gateway API support#2644dapperdivers wants to merge 2 commits intomainfrom
dapperdivers wants to merge 2 commits intomainfrom
Conversation
- Add Envoy Gateway v1.0.1 deployment using Gateway API - Create two GatewayClasses: external-envoy and internal-envoy - Configure external Gateway for public traffic (HTTPS + TLS) - Configure internal Gateway for cluster-internal traffic - Add HTTPRoute for derekmackley.com (replaces nginx Ingress) - Routes: derekmackley.com, www.derekmackley.com, resume.derekmackley.com - Backend: resume service (port 8080) in selfhosted namespace - Integrate with cert-manager for automatic TLS certificates - Integrate with external-dns for automatic DNS records - Use Cilium L2 LoadBalancer with fixed IPs (192.168.1.240-241) - Add SecurityPolicy example for future Authentik SSO integration - Resource allocation: 400m-2000m CPU, 512Mi-2Gi RAM Gateway API brings: - Separation of concerns (infrastructure vs application routing) - Native traffic splitting, header manipulation, and redirects - Vendor-neutral, standardized API - Better multi-tenancy and RBAC support Documentation included in app/README.md
Contributor
--- kubernetes/apps Kustomization: flux-system/cluster-apps Kustomization: network/envoy-gateway
+++ kubernetes/apps Kustomization: flux-system/cluster-apps Kustomization: network/envoy-gateway
@@ -0,0 +1,61 @@
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+ labels:
+ kustomize.toolkit.fluxcd.io/name: cluster-apps
+ kustomize.toolkit.fluxcd.io/namespace: flux-system
+ name: envoy-gateway
+ namespace: network
+spec:
+ commonMetadata:
+ labels:
+ app.kubernetes.io/name: envoy-gateway
+ decryption:
+ provider: sops
+ secretRef:
+ name: sops-age
+ deletionPolicy: WaitForTermination
+ dependsOn:
+ - name: cert-manager-tls
+ namespace: cert-manager
+ interval: 30m
+ patches:
+ - patch: |-
+ apiVersion: helm.toolkit.fluxcd.io/v2
+ kind: HelmRelease
+ metadata:
+ name: not-used
+ spec:
+ install:
+ crds: CreateReplace
+ remediation:
+ retries: 3
+ upgrade:
+ cleanupOnFail: true
+ crds: CreateReplace
+ remediation:
+ strategy: rollback
+ retries: 3
+ target:
+ group: helm.toolkit.fluxcd.io
+ kind: HelmRelease
+ path: ./kubernetes/apps/network/external/envoy-gateway/app
+ postBuild:
+ substitute:
+ SECRET_DOMAIN_PERSONAL: derekmackley.com
+ substituteFrom:
+ - kind: ConfigMap
+ name: cluster-settings
+ - kind: Secret
+ name: cluster-secrets
+ prune: true
+ retryInterval: 1m
+ sourceRef:
+ kind: GitRepository
+ name: flux-system
+ namespace: flux-system
+ targetNamespace: envoy-gateway-system
+ timeout: 5m
+ wait: false
+
--- kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway HelmRelease: envoy-gateway-system/envoy-gateway
+++ kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway HelmRelease: envoy-gateway-system/envoy-gateway
@@ -0,0 +1,73 @@
+---
+apiVersion: helm.toolkit.fluxcd.io/v2beta1
+kind: HelmRelease
+metadata:
+ labels:
+ app.kubernetes.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/namespace: network
+ name: envoy-gateway
+ namespace: envoy-gateway-system
+spec:
+ chart:
+ spec:
+ chart: gateway-helm
+ interval: 12h
+ sourceRef:
+ kind: HelmRepository
+ name: envoy-gateway
+ namespace: flux-system
+ version: v1.0.1
+ install:
+ crds: CreateReplace
+ createNamespace: true
+ remediation:
+ retries: 3
+ interval: 30m
+ upgrade:
+ cleanupOnFail: true
+ crds: CreateReplace
+ remediation:
+ retries: 3
+ strategy: rollback
+ values:
+ config:
+ envoyGateway:
+ extensionManager:
+ resources:
+ limits:
+ cpu: 100m
+ memory: 128Mi
+ requests:
+ cpu: 50m
+ memory: 64Mi
+ gateway:
+ controllerName: gateway.envoyproxy.io/gatewayclass-controller
+ logging:
+ level:
+ default: info
+ provider:
+ kubernetes:
+ watch:
+ namespaces: []
+ type: Namespaces
+ type: Kubernetes
+ deployment:
+ replicas: 1
+ resources:
+ limits:
+ cpu: 500m
+ memory: 512Mi
+ requests:
+ cpu: 100m
+ memory: 128Mi
+ rbac:
+ create: true
+ service:
+ annotations:
+ prometheus.io/port: '8080'
+ prometheus.io/scrape: 'true'
+ serviceAccount:
+ create: true
+ name: envoy-gateway
+
--- kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway HelmRepository: envoy-gateway-system/envoy-gateway
+++ kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway HelmRepository: envoy-gateway-system/envoy-gateway
@@ -0,0 +1,14 @@
+---
+apiVersion: source.toolkit.fluxcd.io/v1beta2
+kind: HelmRepository
+metadata:
+ labels:
+ app.kubernetes.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/namespace: network
+ name: envoy-gateway
+ namespace: envoy-gateway-system
+spec:
+ interval: 24h
+ url: https://gateway.envoyproxy.io
+
--- kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway EnvoyProxy: envoy-gateway-system/external-proxy-config
+++ kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway EnvoyProxy: envoy-gateway-system/external-proxy-config
@@ -0,0 +1,39 @@
+---
+apiVersion: gateway.envoyproxy.io/v1alpha1
+kind: EnvoyProxy
+metadata:
+ labels:
+ app.kubernetes.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/namespace: network
+ name: external-proxy-config
+ namespace: envoy-gateway-system
+spec:
+ provider:
+ kubernetes:
+ envoyDeployment:
+ pod:
+ annotations:
+ prometheus.io/port: '19001'
+ prometheus.io/scrape: 'true'
+ labels:
+ gateway-type: external
+ resources:
+ limits:
+ cpu: 1000m
+ memory: 1Gi
+ requests:
+ cpu: 200m
+ memory: 256Mi
+ replicas: 2
+ envoyService:
+ annotations:
+ external-dns.alpha.kubernetes.io/hostname: gateway...PLACEHOLDER_SECRET_DOMAIN_PERSONAL..
+ io.cilium/lb-ipam-ips: 192.168.1.240
+ type: LoadBalancer
+ type: Kubernetes
+ telemetry:
+ metrics:
+ prometheus:
+ disable: false
+
--- kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway GatewayClass: envoy-gateway-system/external-envoy
+++ kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway GatewayClass: envoy-gateway-system/external-envoy
@@ -0,0 +1,19 @@
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: GatewayClass
+metadata:
+ labels:
+ app.kubernetes.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/namespace: network
+ name: external-envoy
+ namespace: envoy-gateway-system
+spec:
+ controllerName: gateway.envoyproxy.io/gatewayclass-controller
+ description: GatewayClass for external-facing traffic (Envoy Gateway)
+ parametersRef:
+ group: gateway.envoyproxy.io
+ kind: EnvoyProxy
+ name: external-proxy-config
+ namespace: envoy-gateway-system
+
--- kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway EnvoyProxy: envoy-gateway-system/internal-proxy-config
+++ kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway EnvoyProxy: envoy-gateway-system/internal-proxy-config
@@ -0,0 +1,38 @@
+---
+apiVersion: gateway.envoyproxy.io/v1alpha1
+kind: EnvoyProxy
+metadata:
+ labels:
+ app.kubernetes.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/namespace: network
+ name: internal-proxy-config
+ namespace: envoy-gateway-system
+spec:
+ provider:
+ kubernetes:
+ envoyDeployment:
+ pod:
+ annotations:
+ prometheus.io/port: '19001'
+ prometheus.io/scrape: 'true'
+ labels:
+ gateway-type: internal
+ resources:
+ limits:
+ cpu: 500m
+ memory: 512Mi
+ requests:
+ cpu: 100m
+ memory: 128Mi
+ replicas: 1
+ envoyService:
+ annotations:
+ io.cilium/lb-ipam-ips: 192.168.1.241
+ type: LoadBalancer
+ type: Kubernetes
+ telemetry:
+ metrics:
+ prometheus:
+ disable: false
+
--- kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway GatewayClass: envoy-gateway-system/internal-envoy
+++ kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway GatewayClass: envoy-gateway-system/internal-envoy
@@ -0,0 +1,19 @@
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: GatewayClass
+metadata:
+ labels:
+ app.kubernetes.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/namespace: network
+ name: internal-envoy
+ namespace: envoy-gateway-system
+spec:
+ controllerName: gateway.envoyproxy.io/gatewayclass-controller
+ description: GatewayClass for internal cluster traffic (Envoy Gateway)
+ parametersRef:
+ group: gateway.envoyproxy.io
+ kind: EnvoyProxy
+ name: internal-proxy-config
+ namespace: envoy-gateway-system
+
--- kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway Namespace: network/envoy-gateway-system
+++ kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway Namespace: network/envoy-gateway-system
@@ -0,0 +1,10 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+ labels:
+ app.kubernetes.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/namespace: network
+ name: envoy-gateway-system
+
--- kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway Certificate: envoy-gateway-system/wildcard-personal-domain
+++ kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway Certificate: envoy-gateway-system/wildcard-personal-domain
@@ -0,0 +1,19 @@
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+ labels:
+ app.kubernetes.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/namespace: network
+ name: wildcard-personal-domain
+ namespace: envoy-gateway-system
+spec:
+ dnsNames:
+ - ..PLACEHOLDER_SECRET_DOMAIN_PERSONAL..
+ - '*...PLACEHOLDER_SECRET_DOMAIN_PERSONAL..'
+ issuerRef:
+ kind: ClusterIssuer
+ name: letsencrypt-production
+ secretName: wildcard-personal-domain-tls
+
--- kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway Gateway: envoy-gateway-system/external-gateway
+++ kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway Gateway: envoy-gateway-system/external-gateway
@@ -0,0 +1,36 @@
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata:
+ annotations:
+ external-dns.alpha.kubernetes.io/hostname: '*...PLACEHOLDER_SECRET_DOMAIN_PERSONAL..'
+ labels:
+ app.kubernetes.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/namespace: network
+ name: external-gateway
+ namespace: envoy-gateway-system
+spec:
+ gatewayClassName: external-envoy
+ listeners:
+ - allowedRoutes:
+ namespaces:
+ from: All
+ hostname: '*...PLACEHOLDER_SECRET_DOMAIN_PERSONAL..'
+ name: https
+ port: 443
+ protocol: HTTPS
+ tls:
+ certificateRefs:
+ - kind: Secret
+ name: wildcard-personal-domain-tls
+ namespace: gateway-system
+ mode: Terminate
+ - allowedRoutes:
+ namespaces:
+ from: All
+ hostname: '*...PLACEHOLDER_SECRET_DOMAIN_PERSONAL..'
+ name: http
+ port: 80
+ protocol: HTTP
+
--- kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway HTTPRoute: envoy-gateway-system/https-redirect
+++ kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway HTTPRoute: envoy-gateway-system/https-redirect
@@ -0,0 +1,24 @@
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ labels:
+ app.kubernetes.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/namespace: network
+ name: https-redirect
+ namespace: envoy-gateway-system
+spec:
+ hostnames:
+ - '*...PLACEHOLDER_SECRET_DOMAIN_PERSONAL..'
+ parentRefs:
+ - name: external-gateway
+ namespace: gateway-system
+ sectionName: http
+ rules:
+ - filters:
+ - requestRedirect:
+ scheme: https
+ statusCode: 301
+ type: RequestRedirect
+
--- kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway Certificate: envoy-gateway-system/wildcard-internal-domain
+++ kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway Certificate: envoy-gateway-system/wildcard-internal-domain
@@ -0,0 +1,19 @@
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+ labels:
+ app.kubernetes.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/namespace: network
+ name: wildcard-internal-domain
+ namespace: envoy-gateway-system
+spec:
+ dnsNames:
+ - internal...PLACEHOLDER_SECRET_DOMAIN_PERSONAL..
+ - '*.internal...PLACEHOLDER_SECRET_DOMAIN_PERSONAL..'
+ issuerRef:
+ kind: ClusterIssuer
+ name: letsencrypt-production
+ secretName: wildcard-internal-domain-tls
+
--- kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway Gateway: envoy-gateway-system/internal-gateway
+++ kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway Gateway: envoy-gateway-system/internal-gateway
@@ -0,0 +1,34 @@
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata:
+ labels:
+ app.kubernetes.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/namespace: network
+ name: internal-gateway
+ namespace: envoy-gateway-system
+spec:
+ gatewayClassName: internal-envoy
+ listeners:
+ - allowedRoutes:
+ namespaces:
+ from: All
+ hostname: '*.internal...PLACEHOLDER_SECRET_DOMAIN_PERSONAL..'
+ name: https
+ port: 443
+ protocol: HTTPS
+ tls:
+ certificateRefs:
+ - kind: Secret
+ name: wildcard-internal-domain-tls
+ namespace: gateway-system
+ mode: Terminate
+ - allowedRoutes:
+ namespaces:
+ from: All
+ hostname: '*.internal...PLACEHOLDER_SECRET_DOMAIN_PERSONAL..'
+ name: http
+ port: 80
+ protocol: HTTP
+
--- kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway HTTPRoute: envoy-gateway-system/internal-https-redirect
+++ kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway HTTPRoute: envoy-gateway-system/internal-https-redirect
@@ -0,0 +1,24 @@
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ labels:
+ app.kubernetes.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/namespace: network
+ name: internal-https-redirect
+ namespace: envoy-gateway-system
+spec:
+ hostnames:
+ - '*.internal...PLACEHOLDER_SECRET_DOMAIN_PERSONAL..'
+ parentRefs:
+ - name: internal-gateway
+ namespace: gateway-system
+ sectionName: http
+ rules:
+ - filters:
+ - requestRedirect:
+ scheme: https
+ statusCode: 301
+ type: RequestRedirect
+
--- kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway HTTPRoute: envoy-gateway-system/resume-site
+++ kubernetes/apps/network/external/envoy-gateway/app Kustomization: network/envoy-gateway HTTPRoute: envoy-gateway-system/resume-site
@@ -0,0 +1,43 @@
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ annotations:
+ external-dns.alpha.kubernetes.io/hostname: ..PLACEHOLDER_SECRET_DOMAIN_PERSONAL..,www...PLACEHOLDER_SECRET_DOMAIN_PERSONAL..,resume...PLACEHOLDER_SECRET_DOMAIN_PERSONAL..
+ labels:
+ app.kubernetes.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/name: envoy-gateway
+ kustomize.toolkit.fluxcd.io/namespace: network
+ name: resume-site
+ namespace: envoy-gateway-system
+spec:
+ hostnames:
+ - ..PLACEHOLDER_SECRET_DOMAIN_PERSONAL..
+ - www...PLACEHOLDER_SECRET_DOMAIN_PERSONAL..
+ - resume...PLACEHOLDER_SECRET_DOMAIN_PERSONAL..
+ parentRefs:
+ - name: external-gateway
+ namespace: gateway-system
+ sectionName: https
+ rules:
+ - backendRefs:
+ - name: resume
+ namespace: selfhosted
+ port: 8080
+ filters:
+ - responseHeaderModifier:
+ add:
+ - name: X-Frame-Options
+ value: DENY
+ - name: X-Content-Type-Options
+ value: nosniff
+ - name: X-XSS-Protection
+ value: 1; mode=block
+ - name: Referrer-Policy
+ value: strict-origin-when-cross-origin
+ type: ResponseHeaderModifier
+ matches:
+ - path:
+ type: PathPrefix
+ value: /
+ |
- Fix LB IPs: 10.100.0.31/32 (matches IPAM pool, not 192.168.1.x) - Remove targetNamespace from ks.yaml (resources span multiple namespaces) - Use OCIRepository pattern (matches cluster convention) - Use HelmRelease v2 API (not v2beta1) - Remove duplicate Kustomization from network/external/ks.yaml - Remove inline postBuild (cluster-level substituteFrom handles it) - Simplify Gateway listeners (no hardcoded hostnames on Gateway) - Remove README and SecurityPolicy example (add later) - Use cert-manager.io/cluster-issuer annotation on Gateway
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Migration to Envoy Gateway & Gateway API
This PR adds Envoy Gateway v1.0.1 deployment using the Gateway API standard, providing a modern, vendor-neutral alternative to nginx Ingress.
📦 What's Included
Infrastructure Components
external-envoy: Public traffic (Cilium LB:192.168.1.240)internal-envoy: Cluster-internal traffic (Cilium LB:192.168.1.241)external-gateway: HTTPS (443) + HTTP (80) with TLS terminationinternal-gateway: Same pattern for internal servicesApplication Routes
derekmackley.com,www.derekmackley.com,resume.derekmackley.comresume:8080inselfhostednamespaceIntegrations
📊 Resource Allocation
🎯 Benefits of Gateway API
📋 Prerequisites
Before merging, verify:
letsencrypt-productionexistsgateway-httproutesource192.168.1.240-241kubectl get svc -n selfhosted resume🚀 Deployment Steps
Install Gateway API CRDs (required first):
Merge this PR - Flux will reconcile automatically
Monitor deployment:
Verify LoadBalancer IPs:
Test HTTPS access:
🔄 Migration from nginx Ingress
Both can run simultaneously! The new Envoy Gateway will get a different LoadBalancer IP.
192.168.1.240)📚 Documentation
kubernetes/apps/network/external/envoy-gateway/app/README.md🔐 Optional: Authentik SSO
A
SecurityPolicyexample is included but commented out in kustomization.yaml. To enable:securitypolicy-authentik-example.yamlin kustomization🎨 Files Added
Note: This PR does NOT remove the existing nginx Ingress. That can be done separately after testing.
cc @dapperdivers