Migrating off Ingress-NGINX: Our Production Gateway API Journey

When our platform team announced Ingress-NGINX was approaching end-of-life for our Kubernetes versions, a cold sweat ran down my spine. We had 18 production clusters, some with over 200 ingress rules, heavily relying on its annotation-driven features. The clock was ticking, and we needed a robust, production-ready Gateway API implementation that could handle our scale and complex routing without a single minute of downtime. The biggest headache, as always, turned out to be the subtle annotation translations that broke applications built on years of Ingress-NGINX muscle memory.

The Deadline and the Decision Matrix

Our mandate was clear: migrate all clusters off Ingress-NGINX v1.9.4 within 16 weeks. The first 4 weeks were spent evaluating options for Gateway API implementations. We looked at three main contenders:

• Envoy Gateway (v0.4.0): Seemed like the natural successor, direct Envoy control, lightweight.
• Istio (v1.17.2): Already in use for service mesh in some clusters, powerful, but heavy for pure ingress.
• Cilium (v1.14.0): Exciting eBPF approach, but still maturing its Gateway API support at the time.

Our core requirements included native TLS termination, HTTP/2 support, advanced traffic splitting, and crucially, preserving existing Ingress-NGINX custom annotations or finding direct Gateway API equivalents.

The Annotation Translation Nightmare: An Early Gotcha

The gotcha that bit us hard in week three was the sheer volume and nuance of Ingress-NGINX annotations. We had a script to translate Ingress objects to Gateway API, but it missed many subtleties. For example, nginx.ingress.kubernetes.io/proxy-read-timeout translated straightforwardly to a Gateway API HTTPRoute Timeout, but things like nginx.ingress.kubernetes.io/server-snippet, often used for custom Lua scripts or header manipulations not exposed by standard Ingress, had no direct Gateway API equivalent in Envoy Gateway. This forced us to either refactor applications or implement custom EnvoyFilter resources.

Why Envoy Gateway Won (and Where it Fell Short)

Ultimately, we chose Envoy Gateway. Its direct integration with Envoy gave us control, and its lower operational footprint compared to a full-blown Istio service mesh was attractive for our ingress-only clusters. Istio, while powerful, felt like overkill for simple HTTP routing and introduced more CRDs and an increased learning curve for our L1 support team. Cilium's Gateway API support was too nascent for our comfort. However, Envoy Gateway's native feature set sometimes lagged behind Ingress-NGINX's comprehensive annotation library.

Our Migration Playbook: Blue/Green with DNS Cutover

Our migration strategy was a blue/green deployment with DNS cutover. For each cluster:

1. Deploy Envoy Gateway: We deployed the controllers and provisioned an EnvoyProxy pointing to a new LoadBalancer IP.
2. Translate and Deploy Gateway API Resources: Our script generated Gateway, HTTPRoute, and TLSRoute resources from existing Ingresses.
3. Test Internally: We used host file modifications and internal DNS to send traffic to the new Gateway IP for thorough testing by QA.
4. DNS Cutover (TTL 30s): Once confidence was high, we changed the DNS A records.
5. Monitor and Rollback Window: We intensely monitored metrics for 24 hours. The old Ingress-NGINX was kept running, ready for a quick DNS rollback.

This process was repeated across all 18 clusters over a 10-week period, averaging two clusters per week.

What I'd Do Differently

Looking back, I'd have invested more time in an exhaustive pre-migration audit of all Ingress-NGINX annotations, categorizing them by complexity and native Gateway API support. This would have given us a clearer picture of the unavoidable manual interventions upfront. The assumption that most annotations would have direct translations led to several late-stage scrambles.

Lingering Limitations

Even after the migration, some limitations persist. Advanced WAF capabilities, which Ingress-NGINX sometimes handled through external integrations or custom Lua modules, required us to deploy a separate layer of security proxies in front of our new Gateways. We're still evaluating how to consolidate these security features without adding too much architectural complexity.