title: "Gateway API in Production: Migrating Off Retired Ingress-NGINX at Enterprise Scale" date: 2026-09-17 tags: [Kubernetes, Gateway API, Ingress, Networking, Envoy, Cilium, Istio, Platform Engineering, GitOps, Multi-Region] cover: ../images/gateway-api-ingress-nginx-migration-cover.png

Gateway API in Production: Migrating Off Retired Ingress-NGINX at Enterprise Scale

1. Topic of the Day

On March 24, 2026, kubernetes/ingress-nginx — for a decade the single most deployed Ingress controller on the planet, sitting in front of an estimated 40%+ of production Kubernetes clusters — went end-of-life. The repository is now archived and read-only. No more feature work, no more bug fixes, and critically, no more CVE patches. This wasn't a surprise announcement: SIG-Network flagged the retirement in November 2025, and on January 29, 2026 the Kubernetes Steering Committee and Security Response Committee issued a rare joint statement making the security implications explicit — running an EOL ingress controller with a CVE surface and no patch stream in front of production traffic is now, by the project's own admission, a standing risk.

Gateway API is the reason this retirement was survivable rather than catastrophic for the ecosystem. It's the SIG-Network-designed successor to Ingress: a role-oriented, portable, strongly-typed API (GatewayClass, Gateway, HTTPRoute, GRPCRoute, TCPRoute, TLSRoute) that separates the infrastructure-provider concern (who runs the load balancer) from the application-developer concern (which routes go where) in a way the old single Ingress object with a jungle of controller-specific annotations never could. It reached v1.0 GA in October 2023, and by the time ingress-nginx retired, Gateway API had matured to v1.5 (released February 27, 2026, moving six more features — including named BackendTLSPolicy and mesh-oriented GAMMA routes — into the Standard channel) with over 20 conformant implementations spanning Envoy Gateway, Cilium, Istio, Traefik, Kong, and every major cloud's managed offering.

Why this matters at the platform-engineering level, not just the "which YAML kind do I write" level: Gateway API isn't a drop-in Ingress replacement, it's a different operating model. Ingress conflated three roles — the cluster operator who provisions the load balancer, the platform team who defines TLS and listener policy, and the application team who defines routes — into one object that only the platform team could safely touch, so everyone ended up editing shared annotations and stepping on each other. Gateway API splits this into GatewayClass (infra provider), Gateway (platform team, per-listener), and HTTPRoute/GRPCRoute (app team, namespaced, delegated). At enterprise scale — hundreds of services, dozens of teams, multiple clusters — that separation of concerns is the actual production win, and the ingress-nginx retirement is simply the forcing function that made 2026 the year most platform teams finally executed the migration they'd been deferring since 2023.

2. Real Business Problem

Scenario: A retail platform organization runs 340 microservices across 14 namespaces on two production EKS clusters (us-east-1 primary, eu-west-1 for EU data residency), fronted entirely by ingress-nginx. The controller has been running unmodified since 2021. Platform engineering's Q1 2026 planning cycle lands on the same week as the Steering Committee's joint security statement, and the mandate from the CISO's office is blunt: every EOL, unpatched ingress controller must be off the critical path by Q3, full stop — no exceptions for "it's stable."

The investigation that follows surfaces the actual scope of the problem:

• 577 Ingress objects carry 22 distinct nginx.ingress.kubernetes.io/* annotations, hand-tuned over five years: custom configuration-snippet blocks injecting raw NGINX directives (several of which reference internal Lua scripts for a legacy auth shim), proxy-body-size overrides per service, canary annotations (nginx.ingress.kubernetes.io/canary-weight) used by three teams for manual traffic splitting, and a handful of rewrite-target regexes nobody currently on the team wrote or fully understands.
• A live, unpatched CVE is already sitting in production. The controller version in use has a known path-traversal-adjacent misconfiguration risk flagged in a security scan two months ago; the fix requires a controller upgrade the team held off on because a previous minor-version bump broke the canary annotations for one team's checkout flow with no rollback path faster than a full redeploy.
• The configuration-snippet annotation is itself the biggest liability. It was disabled by default starting in a 2022 ingress-nginx release specifically because it allowed arbitrary NGINX config injection — a privilege-escalation vector if a namespace-scoped Ingress object can effectively execute arbitrary directives in a cluster-wide-privileged NGINX process. This org re-enabled it years ago to unblock a deadline and never revisited the decision.
• No load-balancer redundancy story. Both clusters run a single ingress-nginx Deployment behind a cloud NLB; a controller crash-loop (which has happened twice in three years, once from a malformed configuration-snippet and once from a connection-limit exhaustion under a flash-sale traffic spike) takes down all 340 services simultaneously, because there is no per-team blast-radius isolation.
• Nobody can answer "what would GAMMA/service-mesh migration cost us later" because the org's Istio proof-of-concept (evaluated and shelved in 2024) assumed it would need to redo all ingress routing from scratch to adopt a mesh — a false assumption under Gateway API, where HTTPRoute is designed to work for both north-south (Gateway) and east-west (GAMMA/Mesh) traffic with the same route object.

The mandate becomes a production architecture problem: migrate 577 Ingress objects across two regions to Gateway API, eliminate the configuration-snippet privilege-escalation surface, get real per-team blast-radius isolation, and do it without a traffic-affecting big-bang cutover — all before the Q3 deadline, using ingress2gateway as the starting point rather than a hand-migration of 577 objects.

3. Production Architecture

Architecture image: blogs/architecture/gateway-api-ingress-nginx-migration-architecture.png

Global edge tier. Client traffic resolves through GSLB (Route 53 latency-based or geo routing with active health checks against each region's Gateway) to the nearest healthy region, hits a cloud L7/L4 load balancer with WAF and DDoS protection in front, and only then reaches the Kubernetes-native tier. cert-manager remains unchanged in role but now issues certificates referenced directly by Gateway listener blocks via certificateRefs, rather than through ingress-shim annotations — one less annotation surface to migrate.

Per-region Gateway tier — deliberately split implementations. Region A (us-east-1) runs Envoy Gateway as its GatewayClass implementation; Region B (eu-west-1) runs Cilium's Gateway API support. This is not accidental redundancy — running two different conformant implementations across regions means a bug or CVE specific to one data-plane implementation cannot simultaneously take down both regions, and it gives the platform team a live A/B on operational characteristics (Envoy Gateway's broader extension ecosystem vs. Cilium's eBPF data-plane performance and the operational simplicity of not running a separate CNI-plus-ingress stack) before standardizing. Each region's Gateway resource defines the shared listener contract (443/HTTPS with SNI-based cert selection, 80 redirecting to 443) that platform engineering owns exclusively — application teams cannot modify listeners, only attach routes to them via allowedRoutes namespace selectors.

Route tier — delegated to application teams. Each of the 340 services' HTTPRoute (or GRPCRoute for the 40-odd gRPC services) lives in the owning team's namespace, referencing the shared Gateway by name via parentRefs. This is the concrete fix for the blast-radius problem: a malformed route in one namespace is rejected by admission validation and reported in that HTTPRoute's own status.conditions, and cannot affect another team's routes or the Gateway's listener configuration — a structural guarantee Ingress annotations never gave you.

GAMMA routes for east-west traffic. The three services that need canary/traffic-splitting (previously done via nginx.ingress.kubernetes.io/canary-weight) are migrated to GAMMA-profile HTTPRoutes with a parentRef pointing at the internal Service rather than a Gateway — native weighted backend routing (backendRefs with weight) replaces the annotation-based canary hack, and gives the same primitive a path to full service-mesh east-west routing later without a second migration.

GitOps control plane. Every Gateway, HTTPRoute, and GRPCRoute is generated once by ingress2gateway from the existing Ingress fleet, hand-reviewed for the annotations that don't map cleanly (the configuration-snippet blocks, which are deliberately not migrated — each is re-implemented as either a supported Gateway API filter, an EnvoyPatchPolicy/Cilium CiliumEnvoyConfig extension, or pushed into the application itself), committed to Git, and reconciled by ArgoCD. No kubectl apply against these resources is permitted outside the GitOps path — enforced by a Kyverno policy that denies direct mutations from any identity other than the ArgoCD service account.

Observability and policy plane. An OpenTelemetry Collector scrapes Gateway API-native metrics (both Envoy Gateway and Cilium expose Programmed/Accepted condition-based health plus per-route request metrics) into Prometheus/Grafana; Kyverno enforces that every new HTTPRoute must specify a backendRefs timeout and cannot use a wildcard hostname that collides with another namespace's route — policy-as-code doing the annotation-review work a human previously did by hand.

Why this design, and the trade-offs: the alternative — a single shared GatewayClass/implementation across both regions — is operationally simpler but reintroduces a single-implementation blast radius, and forecloses the option to compare implementations under real traffic before standardizing. The cost is real: two implementations mean two sets of operational runbooks, two sets of Grafana dashboards, and two escalation paths during an incident. At this org's scale (340 services, two regions, a platform team of six), that cost was judged acceptable for six months of comparative data; a smaller platform team would reasonably choose one implementation everywhere. As the system scales to a third region, the plan is to standardize on whichever implementation's operational data wins, not to keep proliferating implementations.

4. Solution Design

Design decision: ingress2gateway-first, not hand-migration. SIG-Network's ingress2gateway tool (reaching 1.0 in March 2026, supporting 30+ common annotations across ingress-nginx, ALB, GCE, and Kong) converts the mechanical 80% of the migration — host/path matching, TLS termination, basic rewrites — automatically. Hand-migrating 577 objects would have taken an estimated 3-4 engineer-months with high error risk; running ingress2gateway and hand-reviewing only the annotation-based edge cases (the configuration-snippet blocks and canary weights) cut that to roughly three weeks of review plus two weeks of validation.

Design decision: do not migrate configuration-snippet as-is. The alternative — finding a Gateway API equivalent that also allows raw config injection — was explicitly rejected. configuration-snippet is a security anti-pattern regardless of which controller hosts it; the migration is the opportunity to eliminate it, not carry it forward. Each snippet was re-audited: two were dead code (referencing a decommissioned internal auth service), three were replaced by supported HTTPRoute filters (RequestHeaderModifier, URLRewrite), and one — a legitimate need for custom response-header injection — was implemented as an EnvoyExtensionPolicy (Envoy Gateway's supported, reviewable extension mechanism) instead of raw config injection.

Alternative considered — “just upgrade ingress-nginx's fork or a community successor” instead of migrating. Several community forks emerged post-retirement. Rejected: forking a retired project just delays the same reckoning, inherits the same annotation-sprawl technical debt, and does nothing to fix the blast-radius or privilege-escalation problems that motivated the migration in the first place. Gateway API's role-based model is the actual fix; a forked NGINX is not.

Scalability considerations. HTTPRoute objects are namespaced and independently reconciled, so route-count growth scales linearly with implementation controller resources, not with Gateway object count — adding the 341st service means one more HTTPRoute in that team's namespace, not a shared object edit. This is a direct scalability win over the old model, where every new service touched the annotation surface of a shared, sprawling Ingress resource set indirectly through shared controller config.

Cost implications. Running two Gateway implementations across regions costs marginally more in engineering time (two runbooks) but zero additional infrastructure spend — both Envoy Gateway and Cilium's Gateway API support run within the existing CNI/data-plane footprint. The ingress2gateway tooling investment (three weeks) is a one-time cost against an open-ended CVE-exposure cost of staying on EOL software.

Security implications. Eliminating configuration-snippet closes the highest-severity structural risk. Namespace-scoped HTTPRoute with Kyverno-enforced GitOps-only mutation removes the "anyone with Ingress-edit RBAC can inject arbitrary NGINX directives" class entirely. BackendTLSPolicy (new in the Standard channel as of v1.5) is adopted for backend mTLS verification between Gateway and Service, which the old annotation-based model never natively supported.

Performance implications. Early load testing shows Envoy Gateway's data plane holding p99 request latency within 3ms of the prior ingress-nginx baseline at equivalent RPS; Cilium's Gateway API path, being eBPF-native rather than proxy-per-request in the traditional sense, actually improved p99 latency by roughly 15% in Region B's synthetic load tests, though this is early data pending a full production traffic quarter before it informs the standardization decision.

5. Deep Technical Walkthrough

Object model and reconciliation chain. A GatewayClass (cluster-scoped, one per implementation) declares the controller (spec.controllerName, e.g. gateway.envoyproxy.io/gatewayclass-controller). A Gateway (namespaced, owned by platform engineering) references a GatewayClass and defines listeners — each with a protocol, port, hostname, and optional certificateRefs. An HTTPRoute references one or more Gateways via parentRefs, and independently defines hostnames, rules (matches + backendRefs + optional filters). The implementation's controller watches all three kinds, and for each valid (Gateway, HTTPRoute) pairing where the HTTPRoute's namespace is permitted by the Gateway's allowedRoutes.namespaces selector, programs the underlying data plane (an Envoy xDS config push for Envoy Gateway; eBPF datapath + Envoy-in-Cilium for Cilium's implementation).

Request flow. Client → GSLB → cloud LB → Gateway's listener (TLS terminated using the SNI-matched certificateRef) → the implementation's data plane evaluates HTTPRoute matches in specificity order (exact path > prefix path > header/method matches as tiebreakers, per the Gateway API spec's defined precedence rules — a meaningful improvement over ingress-nginx's annotation-influenced, less formally specified precedence) → request forwarded to the matched backendRefs Service → kube-proxy/eBPF service routing to a Pod.

Control-plane interaction. Gateway and HTTPRoute status conditions (Accepted, Programmed, ResolvedRefs) are the API-native replacement for "check the ingress-nginx controller logs to see if my Ingress was picked up." A Programmed: False condition with a reason field tells you why — invalid backend reference, listener conflict, unsupported filter — without grepping controller pod logs, because the implementation is required by the conformance suite to surface this on the object itself.

Data-plane interaction and failure scenarios. If a Gateway's listener has an unresolvable certificateRef (say, cert-manager hasn't issued the cert yet), the Gateway reports Programmed: False with reason: InvalidCertificateRef, and — critically — this failure is isolated to that listener; other listeners on the same Gateway continue serving. Contrast with an ingress-nginx TLS misconfiguration, which in some annotation combinations could cause a full controller reload failure affecting all Ingresses simultaneously. If an HTTPRoute's backendRefs Service has no ready endpoints, the implementation returns a 503 for that route specifically — again scoped, not global.

Recovery mechanisms. Because every Gateway API object is GitOps-managed, a bad route change is a git revert + ArgoCD re-sync, typically restoring service within one sync interval (30-180 seconds depending on ArgoCD's configured interval), versus the old model's "find which of 577 Ingress objects has the bad annotation" incident-response pattern.

Performance bottlenecks and scaling behavior. At high route counts (500+), the Envoy Gateway control plane's xDS config generation and push latency becomes the thing to watch — a large flat list of HTTPRoutes translated into Envoy's RDS/CDS config can grow xDS snapshot size and marginally increase config-propagation latency after a change. Cilium's eBPF-based approach sidesteps traditional xDS propagation entirely for L4/L7 basics, at the cost of a narrower set of L7 filter extensions compared to Envoy Gateway's EnvoyExtensionPolicy ecosystem — the classic proxy-flexibility-vs-eBPF-performance trade-off that shows up everywhere in this space (see: the same trade-off in the CNI layer with Cilium vs. Calico).

6. Production Troubleshooting

Symptom: After cutting a batch of 40 HTTPRoutes over from the shadow Ingress objects, five services return intermittent 404s for a subset of request paths that worked fine under ingress-nginx.

Investigation path a senior platform engineer would take:

1. Check the HTTPRoute's own status first — kubectl get httproute <name> -n <ns> -o yaml and read the status.parents[].conditions. A ResolvedRefs: True / Accepted: True pair with no rejection tells you the route itself is structurally fine and the problem is match-order or match-definition, not admission.
2. Compare match precedence, not just match presence. Gateway API's path-matching precedence (Exact > longest PathPrefix > RegularExpression) is a formally specified tiebreak that differs subtly from ingress-nginx's rewrite-target/use-regex behavior, which several teams had been relying on undocumented NGINX regex-greedy-matching quirks for. The fix here is almost always: the migrated route has two overlapping PathPrefix rules that ingress-nginx's annotation ordering happened to disambiguate correctly, but Gateway API's formal precedence resolves differently. kubectl describe httproute won't show this directly — you diff the intended routing table against the live rules[] order and check for prefix overlap.
3. Check for a stale shadow object. During dual-running (Ingress and HTTPRoute both live, pointed at different GatewayClass/controller, for canary validation), confirm client traffic is actually hitting the new Gateway and not silently still resolving to the old ingress-nginx LB via a stale DNS or LB target-group entry — a very common false-positive in this specific migration pattern.
4. Pull Envoy Gateway's EnvoyProxy resource logs / Cilium's cilium-dbg and check the actual programmed xDS/BPF state, not just the Kubernetes object status — a Programmed: True condition confirms the controller attempted to program the data plane, not that the data plane's live state matches, in the rare case of a controller-to-dataplane sync lag.
5. Grep the OTel-collected route-level metrics (request count and response-code breakdown per HTTPRoute) in Grafana, filtered to the affected path prefix, to quantify whether this is truly intermittent (suggesting a race between two matching rules or backend endpoint flakiness) or has a specific reproducible trigger (suggesting a deterministic match-precedence bug).

Root cause in this class of incident, in practice: almost always match-precedence differences between the old annotation-driven behavior and Gateway API's formally specified precedence — which is a feature (predictable, spec-conformant behavior) surfacing as an incident only because the old behavior was implicit and undocumented. The fix is rewriting the overlapping rules to be non-ambiguous under the new precedence rules, and it is a one-time cost per route pattern, not a recurring operational tax.

7. Hands-on Lab

Goal: stand up a local Gateway API environment, run ingress2gateway against a sample legacy Ingress fleet, and validate the migrated routes.

# 1. Local cluster with Gateway API CRDs + Envoy Gateway installed
kind create cluster --name gwapi-lab

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.0/standard-install.yaml

helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.5.0 -n envoy-gateway-system --create-namespace

kubectl wait --timeout=120s -n envoy-gateway-system \
  deployment/envoy-gateway --for=condition=Available

# 2. Sample legacy Ingress fleet (mimics the annotation sprawl from Section 2)
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata: { name: shop }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: catalog, namespace: shop }
spec:
  replicas: 2
  selector: { matchLabels: { app: catalog } }
  template:
    metadata: { labels: { app: catalog } }
    spec:
      containers:
      - name: catalog
        image: hashicorp/http-echo:1.0
        args: ["-text=catalog-v1", "-listen=:8080"]
        ports: [{ containerPort: 8080 }]
---
apiVersion: v1
kind: Service
metadata: { name: catalog, namespace: shop }
spec:
  selector: { app: catalog }
  ports: [{ port: 80, targetPort: 8080 }]
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: catalog
  namespace: shop
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-body-size: "8m"
spec:
  ingressClassName: nginx
  rules:
  - host: shop.local
    http:
      paths:
      - path: /catalog
        pathType: Prefix
        backend: { service: { name: catalog, port: { number: 80 } } }
EOF

# 3. Run ingress2gateway to generate Gateway API resources
go install sigs.k8s.io/ingress2gateway@latest
ingress2gateway print --providers ingress-nginx --namespace shop > shop-gatewayapi.yaml
cat shop-gatewayapi.yaml   # review: rewrite-target -> URLRewrite filter, proxy-body-size flagged as unsupported (manual follow-up)

# 4. Create the shared Gateway (platform-team-owned) and apply the generated HTTPRoute
cat <<'EOF' | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata: { name: eg }
spec: { controllerName: gateway.envoyproxy.io/gatewayclass-controller }
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: { name: public-gw, namespace: envoy-gateway-system }
spec:
  gatewayClassName: eg
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes: { namespaces: { from: All } }
EOF

kubectl apply -f shop-gatewayapi.yaml -n shop

# 5. Validate
kubectl get gateway public-gw -n envoy-gateway-system -o jsonpath='{.status.conditions}'
kubectl get httproute -n shop -o jsonpath='{.items[0].status.parents}'

GATEWAY_IP=$(kubectl get svc -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=public-gw \
  -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')
curl -H "Host: shop.local" http://$GATEWAY_IP/catalog   # expect catalog-v1

# 6. Cleanup
kind delete cluster --name gwapi-lab

Validation checklist: confirm Gateway reports Programmed: True, confirm HTTPRoute reports Accepted: True and ResolvedRefs: True, confirm the response matches the old Ingress's behavior for the rewrite case, and confirm ingress2gateway's stderr output flagged proxy-body-size as requiring manual translation (to EnvoyProxy's spec.telemetry/cluster-level maxRequestBytes config, in Envoy Gateway's case) rather than silently dropping it — this is exactly the class of annotation the hands-on review step in Section 4 exists to catch.

8. Production Case Study

Google (GKE) pushed Gateway API as the default, recommended ingress path well before the ingress-nginx retirement, running its own managed GatewayClass implementations (gke-l7-global-external-managed, gke-l7-regional-external-managed) since 2023 — betting early that a portable, role-based API would reduce the annotation-lock-in support burden compared to GKE's own Ingress-annotation surface. Google's own retirement-transition blog post (opensource.googleblog.com, February 2026) documents internal teams migrating multi-thousand-Ingress fleets using the same ingress2gateway-first pattern described above, with the addition of a canary GatewayClass rollout per team rather than per-object, to let teams opt in on their own schedule.

Kong and other API-gateway vendors reacted to the retirement by shipping Gateway API conformant implementations as their primary ingress story rather than a secondary feature — reflected in the finding that roughly 75% of API gateway vendors added MCP-adjacent and Gateway API features in the same 2026 release cycle, treating "be a conformant GatewayClass" as table stakes the way "support the Ingress spec" was a decade earlier.

Very large platforms running custom L7 (Netflix, Uber-scale) generally don't run ingress-nginx or Gateway API at the true edge — they run custom-built or heavily forked Envoy control planes predating Gateway API's standardization. Their relevant lesson for this migration is architectural, not tool-specific: they separated "who owns the listener/TLS contract" from "who owns the route" organizationally years before Gateway API formalized it as an API — Gateway API is, in effect, the open-source standardization of a pattern hyperscalers built bespoke internal tooling for a decade ago. Mid-size platform teams doing this migration in 2026 are catching up to an organizational model, not just adopting new YAML.

9. Architecture Review

Strengths: the role separation (GatewayClass/Gateway/HTTPRoute) gives genuine blast-radius isolation that the flat Ingress-plus-annotations model structurally could not provide; running two implementations across regions hedges against a single implementation's CVE or bug class; GitOps-only mutation with Kyverno enforcement closes the privilege-escalation surface that configuration-snippet represented; formally specified match precedence removes a whole class of "why did this route win" ambiguity.

Weaknesses: running two different GatewayClass implementations doubles the operational surface (two sets of dashboards, two debugging mental models, two upgrade cadences) for a six-person platform team — this is a real ongoing tax, not a one-time migration cost, and is only justified as a temporary comparative-evaluation phase, not a permanent architecture. The EnvoyExtensionPolicy/CiliumEnvoyConfig escape hatches used to replace the eliminated configuration-snippet blocks are themselves less portable across implementations than core Gateway API resources — swapping Region A's implementation later means re-authoring those extension policies, not a config copy-paste.

What fails first at 10x scale (3,400 services instead of 340): the GitOps reconciliation loop itself — a single ArgoCD Application syncing thousands of HTTPRoute objects across hundreds of namespaces will hit sync-performance and drift-detection latency limits well before the data plane does. At 10x, this needs to shift to per-team ArgoCD ApplicationSets (already a common pattern) so a large route-count doesn't serialize behind one team's slow-to-review PR. The Envoy Gateway control plane's xDS snapshot generation is the second likely bottleneck — worth load-testing xDS push latency at 3,000+ HTTPRoute objects before committing to it as the 10x-scale implementation.

What changes for 100 million end users: the global-edge tier stops being "GSLB across two regions" and becomes a genuine multi-region active-active design with regional Gateway fleets behind a global anycast/CDN layer doing far more of the caching and static-asset offload, so that Gateway-tier request volume is a small fraction of total client traffic. At that scale, the Kyverno policy layer needs to move from "namespace-scoped admission checks" to a tiered policy model — org-wide baseline policies plus per-business-unit overlays — because a flat policy set across thousands of namespaces becomes an unreviewable single file.

What to redesign: move the two-implementation hedge from "two regions, two implementations, indefinitely" to "standardize on one implementation once six months of comparative production data exists," and invest in an internal abstraction layer (a thin Backstage-integrated self-service portal that generates conformant HTTPRoute YAML from a simpler team-facing form) so that route authoring doesn't require every application team to become a Gateway API YAML expert — the same "make the platform easy to consume, not just powerful" lesson every platform-engineering initiative eventually relearns.

10. Best Practices

Reliability: always dual-run the old and new path (Ingress still live, HTTPRoute validated against a canary hostname or a percentage of synthetic traffic) before cutting real traffic over per service; never do a big-bang cutover across all 577 objects in one change window.

Scalability: keep Gateway (platform-owned, low change rate) and HTTPRoute (team-owned, high change rate) reconciliation loops decoupled via ApplicationSets per team from day one, even at moderate scale — retrofitting this at 10x is more painful than starting with it.

Observability: instrument route-level metrics (per-HTTPRoute request count, latency, and status-code breakdown) from the first migrated service, not as an afterthought — this is what makes the troubleshooting pattern in Section 6 tractable instead of guesswork.

Security: eliminate raw-config-injection annotations during migration, don't carry them forward under a new name; enforce GitOps-only mutation of Gateway/HTTPRoute objects via admission policy from the start of the migration, not after an incident proves the gap.

Cost optimization: the migration itself is close to infrastructure-cost-neutral (same data-plane footprint); the cost lever that matters is engineer-hours, which ingress2gateway-first plus disciplined dual-running minimizes versus hand-migration or a rushed big-bang cutover that generates incident response cost.

Performance: benchmark match-precedence-sensitive routes explicitly before cutover — don't assume behavioral parity with the old annotation-driven controller just because the response looks the same in casual testing.

Maintainability: treat the two-implementation-per-region choice as a time-boxed experiment with an explicit decision date, written down, not an implicit permanent state — undocumented "temporary" architecture decisions are how orgs end up running five ingress technologies a in 2031.

Operational excellence: run a GAMEDAY-style failure injection (kill a Gateway pod, corrupt a HTTPRoute's backend ref, revoke a cert-manager cert mid-flight) against the new stack before the Q3 deadline, not after — validate the recovery mechanisms described in Section 5 under a controlled failure, not for the first time during a real incident.

11. Common Production Mistakes

Migrating annotations 1:1 by inventing custom EnvoyExtensionPolicy/CiliumEnvoyConfig equivalents for every annotation instead of asking whether the underlying behavior should exist at all — carrying forward five years of undifferentiated technical debt into a new API just because the tooling makes it possible.

Doing a namespace-by-namespace migration without a shared understanding, across teams, of Gateway API's match-precedence rules — leading to the exact class of intermittent-404 incident in Section 6, repeated once per team instead of solved once centrally with a shared linting/validation tool run pre-merge.

Treating the retirement deadline as a reason to skip the dual-run/canary validation step — "we have to move fast because ingress-nginx is EOL" is true, but a rushed big-bang migration that causes a checkout-flow outage is a worse outcome than a slightly slower, validated migration that finishes a few weeks later.

Granting HTTPRoute-edit RBAC as broadly as the old Ingress-edit RBAC was granted, without re-examining whether the new object's namespace scoping means broader teams can now safely have edit access they couldn't safely have under the old shared-annotation model (or, in the opposite direction, failing to tighten access that should now be team-scoped instead of cluster-wide).

Not validating cert-manager's certificateRef wiring end-to-end before cutover — a subtle mismatch between the old ingress-shim-based cert issuance and the new direct certificateRefs binding is a common source of "works in staging, TLS handshake fails in prod" surprises, because staging environments often use self-signed or wildcard certs that mask the issue.

12. Interview Preparation

Q: Why did Kubernetes retire ingress-nginx instead of continuing to maintain it alongside Gateway API? A: Maintaining two long-term ingress solutions in perpetuity split a limited SIG-Network maintainer pool across an aging, annotation-driven API design and a newer, better-designed one, with no clear path to ever converging them — annotations can't be deprecated cleanly the way a versioned API's fields can. The retirement is a forcing function to consolidate the ecosystem's engineering effort and, from a security standpoint, stop new CVEs accumulating in a codebase the project no longer has the resources to responsibly patch.

Q: What's the core architectural difference between Ingress and Gateway API that actually matters operationally, not just API-shape? A: Role separation with independent RBAC and independent status reporting. Ingress conflated infra-provisioning, platform-listener-policy, and application-routing concerns into one object type that any RBAC grant covered wholesale. Gateway API splits these into GatewayClass/Gateway (platform-owned) and HTTPRoute/GRPCRoute (team-owned, namespace-scoped), so a namespace-scoped RBAC grant for route authoring cannot touch listener or TLS configuration — a real security and blast-radius boundary, not just a naming convention.

Q: How does Gateway API's design support both Ingress (north-south) and service mesh (east-west) use cases with the same object type? A: The GAMMA initiative (Gateway API for Mesh Management and Administration) extended HTTPRoute's parentRefs to be able to reference a Service directly (not just a Gateway), which lets the exact same route object type express east-west traffic-splitting/canary rules inside a mesh as it does north-south ingress rules — a single mental model and CRD schema for both, which is why teams adopting Gateway API for ingress today aren't making a second migration if they adopt a service mesh later.

Q: You're migrating 500+ Ingress objects under a hard deadline. What's your sequencing strategy and why? A: Run ingress2gateway first to auto-convert the mechanical majority, hand-review the flagged unsupported annotations (never carry forward raw-config-injection patterns), dual-run old and new paths per service behind a canary hostname or percentage split, validate match-precedence behavior explicitly for services with overlapping path rules, and cut over in small batches with a fast rollback (git revert against the GitOps-managed route objects) rather than a single big-bang change window — sequencing risk reduction ahead of deadline pressure, not instead of it.

Q: What would you monitor to know the migration is safe to declare complete? A: Per-route request/error-rate parity against the pre-migration baseline sustained over a full business cycle (including peak-traffic events, since flash-sale-style load is exactly when the old controller crash-looped), zero Programmed: False or ResolvedRefs: False conditions across all migrated Gateway/HTTPRoute objects, confirmation that no traffic is still resolving to the old ingress-nginx LB via stale DNS/target-group entries, and a completed GAMEDAY failure-injection exercise against the new stack.

13. Latest Industry Updates

ingress-nginx is now fully EOL (March 24, 2026). The repository is archived, read-only, with no further CVE patches — any org still running it in production is carrying unmitigated risk by the Kubernetes Security Response Committee's own January 2026 joint statement. ingress2gateway reached 1.0 on March 20, 2026, supporting 30+ annotations across ingress-nginx, ALB, GCE, and Kong providers, and is the SIG-Network-endorsed migration path.

Gateway API v1.5 shipped February 27, 2026, graduating six more features to the Standard channel, continuing the project's steady GA cadence since the October 2023 v1.0 release — this matters because features graduating out of experimental status is the signal platform teams should use to decide what's safe to build long-term production dependencies on versus what's still subject to change.

Kyverno achieved CNCF Graduated status in 2024 and by 2026 is the more commonly recommended default for GitOps-native Kubernetes policy enforcement over OPA/Gatekeeper for teams whose policies live alongside their manifests — directly relevant to this migration's GitOps-only-mutation enforcement layer, since Kyverno's YAML-native policy authoring (no separate Rego language) lowers the bar for the same platform team that's also authoring HTTPRoute YAML.

Kubernetes 1.34 ("Of Wind & Will") shipped Dynamic Resource Allocation (DRA) as GA — not directly a Gateway API topic, but relevant context: it's part of the same broader 2026 trend of core Kubernetes APIs maturing out of the "everyone hand-rolls a controller-specific workaround" phase, the same maturation arc Gateway API has been on since 2023.

NVIDIA's Dynamo platform and continued vLLM/GPU-Operator investment underscore that 2026's platform-engineering attention is split between two axes simultaneously — classic edge/networking modernization (this post's topic) and AI-inference infrastructure — and increasingly the same Gateway API objects (via GRPCRoute and GAMMA) are being used to front model-serving endpoints, meaning the migration described here is also laying groundwork for AI workload traffic management, not just conventional microservice ingress.

14. Summary & Cheat Sheet

Key concepts: Gateway API replaces Ingress with role-separated objects — GatewayClass (infra provider), Gateway (platform-owned listeners/TLS), HTTPRoute/GRPCRoute/TCPRoute/TLSRoute (team-owned routing), plus GAMMA extensions for mesh/east-west traffic via Service-targeted parentRefs.

Why now: ingress-nginx is EOL as of March 24, 2026 — archived, unpatched, no CVE fixes — making migration a security mandate, not a roadmap nice-to-have.

Migration pattern: ingress2gateway for the mechanical conversion → hand-review and eliminate raw-config-injection annotations (configuration-snippet and equivalents) rather than porting them forward → dual-run old and new paths per service → validate match-precedence explicitly → GitOps-only mutation with policy-engine enforcement → batch cutover with fast rollback.

Architecture pattern for scale: per-region GatewayClass implementations (consider two different ones as a deliberate blast-radius hedge, time-boxed to a decision point) → platform-owned Gateway listeners → namespace-scoped, team-owned HTTPRoutes → GitOps (ArgoCD/Flux) as the only mutation path → OTel/Prometheus for route-level observability → Kyverno for policy enforcement.

Troubleshooting checklist: check status.conditions (Accepted, Programmed, ResolvedRefs) on both Gateway and HTTPRoute first → check for path-match-precedence differences against the old annotation-driven behavior → confirm traffic isn't still hitting a stale old-controller LB target → check the implementation's live data-plane state (xDS/eBPF) against the object's reported status → check route-level OTel metrics for the actual failure pattern before assuming root cause.

Commands to remember:

kubectl get gateway <name> -o jsonpath='{.status.conditions}'
kubectl get httproute <name> -o jsonpath='{.status.parents}'
ingress2gateway print --providers ingress-nginx --namespace <ns>
kubectl get gatewayclass -o wide

Design principle to carry forward: separate infrastructure ownership, platform policy ownership, and application routing ownership into independently-scoped, independently-RBAC'd objects wherever possible — Gateway API is this principle applied to ingress, but it's the same lesson that shows up in multi-tenant GPU scheduling, MCP gateway architecture, and policy-as-code design. The API changes; the organizational pattern it encodes doesn't.

Sources consulted for this post: Ingress-NGINX Is Officially Retired — Gateway API Migration Guide, The End of an Era: Transitioning Away from Ingress NGINX (Google Open Source Blog), Announcing Ingress2Gateway 1.0 (kubernetes.io), Kyverno graduated: what CNCF top-level status means, Policy-as-Code: Flexible Kubernetes governance with Kyverno (CNCF), Kubernetes v1.34: Of Wind & Will.