Production AI Gateway Architecture — Multi-Model LLM Routing, Token-Aware Rate Limiting & Failover at Scale

Production AI Gateway Architecture: Multi-Model LLM Routing, Token-Aware Rate Limiting & Failover at Scale

Daily DevOps Mentor — 2026-09-04

1. Topic of the Day

Every platform team that shipped more than one LLM-backed feature has independently reinvented the same piece of infrastructure: a proxy layer sitting between application code and model providers that handles auth, retries, fallback across providers, per-tenant rate limiting, cost attribution, and observability. Two years ago that layer was a thin wrapper script around the OpenAI SDK. Today, with agentic workflows fanning out dozens of concurrent tool-augmented calls per user request, a mix of self-hosted OSS models (Llama, Qwen, DeepSeek, Mistral) running on internal GPU fleets alongside hosted APIs (OpenAI, Anthropic, Bedrock, Vertex), and finance demanding per-team token budgets, that wrapper script has become a first-class piece of production infrastructure: the AI Gateway.

An AI Gateway is functionally an API gateway (Envoy, Kong, Apigee-lineage) specialized for inference traffic. It exists because inference traffic breaks the assumptions every existing L7 gateway was built on. HTTP gateways route on path and header and assume request cost is roughly uniform and completion is fast; LLM requests vary in cost by 100-1000x depending on prompt and output length, streaming responses can run for tens of seconds to minutes, "backend health" isn't just up/down but includes GPU KV-cache occupancy and queue depth, and the thing you're rate-limiting isn't requests-per-second but tokens-per-minute against a provider's actual billing unit. A gateway that doesn't understand these differences will load-balance round-robin across GPU replicas with wildly different queue depths, retry a failed streaming request from the beginning and double-bill the tokens already generated, or rate-limit on request count while a single request silently blows a $50k/month model budget in an afternoon.

At the platform-engineering layer, the AI Gateway is also where organizational policy actually gets enforced instead of hoped for: which teams can call which models, whether PII has to be redacted before it leaves the VPC, whether a request needs to be routed to a region-pinned self-hosted deployment for compliance reasons instead of a third-party API, and what happens when the primary provider returns a 529 overloaded error during a product launch. This is the same reason API gateways exist in front of microservices — centralize cross-cutting concerns once instead of re-implementing retry logic and auth in every service — except now the "service" is a non-deterministic, expensive, streaming, occasionally-down third-party model.

By September 2026 this space has consolidated around two architectural patterns that compose rather than compete: Kubernetes-native inference-aware gateways (Envoy AI Gateway, kgateway, GKE Inference Gateway — all building on the now-GA kubernetes-sigs/gateway-api-inference-extension) that do smart routing to self-hosted model servers using live KV-cache and queue-depth signals, and multi-provider LLM gateways (LiteLLM Proxy, Kong AI Gateway, Portkey, Cloudflare AI Gateway) that do provider abstraction, fallback chains, and cost governance across self-hosted and third-party endpoints. Production platforms increasingly run both layered: the multi-provider gateway as the outer edge making routing/fallback/budget decisions, delegating to the inference-aware gateway for anything landing on the internal GPU fleet.

Today's session designs that composed architecture end to end, walks the request-scheduling internals of the Gateway API Inference Extension (which is genuinely novel Kubernetes engineering, not just "another ingress controller"), and covers where token-based rate limiting and multi-provider failover break down under real production load.

2. Real Business Problem

Symptom: A fintech company's customer-support product ships an agentic assistant that fans out 3-8 LLM calls per user turn (intent classification, retrieval, tool calls, response drafting, safety check). It's built directly against the OpenAI SDK, hardcoded to gpt-4.1, with an internal Llama-3-70B deployment on a vLLM cluster used only for the cheap intent-classification step. Three things go wrong within the same quarter:

  • Provider-side incident. OpenAI has a 45-minute regional degradation with elevated 5xx and overloaded_error responses. There is no fallback path — every in-flight agent turn either times out or returns a raw provider error to the end user. Support ticket volume from users seeing "Something went wrong" spikes 12x during the incident window, on a support product, which is its own special kind of embarrassing.
  • Runaway cost. A prompt-engineering change ships that increases average output tokens per response by 4x (a verbose system prompt update that wasn't caught in review because token cost isn't visible in code review). Finance discovers the anomaly three weeks later reconciling the OpenAI invoice — the delayed detection alone costs roughly $180K before anyone notices, because there is no real-time token-spend dashboard, only a monthly bill.
  • GPU fleet under-utilization next to provider overspend. The internal vLLM cluster serving Llama-3-70B for intent classification sits at 20% GPU utilization most of the day, while the exact same intent-classification workload class (short prompt, short output, latency-tolerant) is also being sent to GPT-4.1 from two other teams' features that never discovered the internal cluster exists, because there's no central catalog or routing layer — each team just calls whatever SDK example they copied from a wiki page.

The ask, verbatim from the CTO after the incident review: "I want one place that owns every call to every model — internal or external — so that a provider outage degrades gracefully instead of user-facing, a runaway prompt trips a budget alarm in minutes not weeks, and a team building a new feature routes to whatever's cheapest and available without having to know our GPU fleet's utilization by heart." That's the AI Gateway mandate: centralize provider abstraction, cost governance, and inference-aware routing behind one control plane, with per-team policy enforced declaratively instead of by convention.

3. Production Architecture

Production AI Gateway — Multi-Model Routing, Rate Limiting & Failover Architecture

Edge layer — multi-provider LLM gateway. All application traffic (agent backends, RAG services, batch pipelines) calls a single internal endpoint, ai-gateway.internal, fronted by an LLM-aware proxy (LiteLLM Proxy or Kong AI Gateway deployed as a Kubernetes Deployment behind a standard Service, HPA'd on CPU/connections since the proxy itself is stateless). This layer owns: API-key-per-provider secret injection (via External Secrets synced from Vault/AWS Secrets Manager, so application code never holds a provider key), per-team/per-application virtual keys with attached budgets and rate limits, request/response logging with PII redaction (Presidio or a regex/NER pass) before anything hits long-term storage, and the routing decision: is this request going to a self-hosted model (forward to the internal inference gateway) or a third-party provider (forward with fallback chain)?

Fallback and circuit-breaking. Every model alias configured in the gateway (support-agent-primary) maps to an ordered list of concrete deployments — e.g., [gpt-4.1 (openai), claude-sonnet-4.5 (anthropic), llama-3.3-70b (internal-vllm)] — with per-deployment circuit breakers tracking error rate and latency over a sliding window. On a provider 5xx/429/timeout, the gateway retries against the next deployment in the chain without the caller knowing a fallback happened (a fallback-provider header is added for observability, not surfaced to the app). Circuit breakers trip per-deployment on sustained error rate, removing that deployment from rotation for a cooldown window and alerting — this is what turns a 45-minute OpenAI regional incident into a brief latency blip instead of a user-facing outage.

Inference layer — Kubernetes-native inference gateway. For requests routed to self-hosted models, the multi-provider gateway forwards to an internal Gateway API Gateway running Envoy Gateway (or kgateway) with the gateway-api-inference-extension installed. Each self-hosted model (Llama-3.3-70B, Qwen2.5-72B, a fine-tuned classifier) is fronted by an InferencePool — a CRD that groups a set of vLLM/SGLang/Triton replica pods and, critically, delegates routing within the pool to an Endpoint Picker (EPP) extension via Envoy's ext_proc protocol. Instead of round-robin or least-connections, the EPP scores candidate pods on live metrics scraped from the model server: KV-cache utilization, queue depth, and — for LoRA-adapter-serving deployments — which adapters are already loaded on which replica, so a request for a rarely-used fine-tune doesn't get routed to a pod that would need a cold adapter swap.

Data plane details. GPU node pools are provisioned via Karpenter with GPU-specific NodePools (topology-aware, MIG-partitioned where the model fits in a fraction of an H100/H200), and the NVIDIA GPU Operator manages driver/device-plugin lifecycle. InferencePool health checks feed both the EPP's scoring function and cluster-autoscaling signals — sustained high queue depth across a pool is a KEDA trigger for horizontal scale-out (new vLLM replicas), while sustained low utilization drives scale-in, with the caveat that model-load time (tens of seconds to load 70B-parameter weights) means scale-out latency has to be masked by keeping a warm-pool buffer, not scaling purely reactively.

Cross-cutting: observability and security. Every hop — edge gateway, EPP routing decision, model server — emits OpenTelemetry spans correlated by a single trace ID, so a slow agent turn can be attributed to "which of the 6 sub-calls, to which model, waited on what" rather than a single opaque latency number. Prometheus scrapes token-level metrics (prompt tokens, completion tokens, time-to-first-token, inter-token latency) per virtual key, feeding both Grafana dashboards and the budget-enforcement Redis counters the edge gateway reads synchronously on every request. SPIFFE/SPIRE-issued workload identities authenticate service-to-gateway calls (no shared API keys between internal services), and network policy restricts which namespaces can reach the edge gateway at all — a compromised low-trust workload can't directly exfiltrate data via an unrestricted LLM call to an external provider.

Why this two-layer shape, and the trade-off. Collapsing both layers into one (either running LiteLLM's own Kubernetes routing to raw pods, or trying to make the inference-aware gateway also understand third-party provider billing/fallback) is possible for smaller platforms and reduces one hop of latency. It was rejected here because the two layers optimize genuinely different things — the edge layer is about business policy (budgets, provider fallback, PII) and changes with product/finance requirements, while the inference layer is about GPU scheduling efficiency (KV-cache-aware routing) and changes with model-serving infrastructure. Coupling them means a finance-driven budget-policy change requires redeploying the same component that does latency-critical GPU routing, which is an unnecessary blast-radius coupling once you're running enough self-hosted models for the EPP's smarter routing to matter.

4. Solution Design

Provider abstraction: build vs. adopt. The default instinct on a platform team is "we'll just write a thin FastAPI proxy that normalizes provider SDKs" — this is exactly what the fintech example above had, and it scales to about two providers and one team before it collapses under retry/fallback/budget logic nobody wants to maintain. LiteLLM Proxy (open-source, extensive provider coverage, virtual-key budget management built in) and Kong AI Gateway (if already standardized on Kong for regular API traffic — one control plane for both) are the two most production-proven adopt options as of late 2026; Portkey and Cloudflare AI Gateway are strong SaaS alternatives when a team doesn't want to operate the proxy layer itself, at the cost of routing sensitive prompt/response data through a third party, which is a hard no for regulated fintech/healthcare data without a signed DPA and, often, a compliance review that kills the timeline anyway.

Alternatives considered and rejected:

  • Client-side SDK abstraction (a shared internal Python/TypeScript library wrapping multiple provider SDKs, imported by every service) was the fintech team's original design. Rejected as the long-term answer because policy changes (new budget limit, new fallback order) require a library version bump and redeploy across every consuming service — there's no way to change routing policy without a code change, which is precisely the "convention not enforcement" problem the CTO called out.
  • Service-mesh-level routing (Istio/Envoy VirtualService weighted routing treating each LLM provider as a mesh destination) handles basic weighted splitting but has no concept of token-based budgets, provider-specific error semantics (insufficient_quota vs rate_limit_exceeded vs overloaded_error need different retry behavior), or streaming-response-aware retry — you'd end up bolting an AI-gateway-shaped layer onto the mesh anyway.
  • Cloud-native managed gateways (Azure API Management's GenAI gateway capabilities, AWS Bedrock's cross-region inference profiles) are excellent when the org is single-cloud and models are single-provider, but this fintech platform runs multi-cloud (OpenAI + Anthropic + internal GPU on-prem/EKS) so a single cloud's managed gateway can't be the sole control plane — it can still front that cloud's own model traffic as one deployment in the fallback chain.

Scalability considerations. The edge gateway is stateless and horizontally scales trivially; the real scale constraint is the budget-enforcement path — every request needs a synchronous read-and-decrement against a shared counter (Redis, typically) before it's allowed through, which becomes a shared-state bottleneck at very high QPS. This is addressed with local token-bucket approximation (each gateway replica holds a local budget slice, reconciled against Redis periodically) trading strict budget accuracy for throughput — acceptable because budget enforcement needs to catch "team burned 10x their daily allocation," not enforce the Nth token to the exact unit.

Cost implications. The gateway itself is cheap to run relative to the inference spend it governs — the ROI case is entirely about the incidents and overspend it prevents (see Section 2's $180K delayed-detection example) plus the GPU utilization gains from actually routing eligible traffic to the underutilized internal fleet instead of paying provider-API prices for workloads that could run in-house.

Security implications. The gateway becomes the single point where a compromised or over-privileged internal service could exfiltrate data via prompt content to an external provider, so response/request logging with PII redaction and per-virtual-key egress policy (this key can only reach internal models, that key can reach Anthropic but not OpenAI) matter as much as the routing logic. It's also now a credential vault by proxy — losing control of the gateway's own secrets access is equivalent to losing every provider API key at once, so its own RBAC and Workload Identity binding need the same scrutiny as a secrets-management system, not a generic proxy.

Performance implications. Every hop adds latency — edge gateway parsing/policy check, EPP scoring for internal routing, network hop to the model server — typically 5-20ms added per hop, which is negligible against a multi-second LLM completion but matters for latency-sensitive, low-token workloads (a 50ms classification call where the gateway overhead is a double-digit percentage of total latency). Time-to-first-token is the metric to protect for streaming responses; naive proxy implementations that buffer the full response before forwarding destroy the point of streaming and need explicit testing to catch.

5. Deep Technical Walkthrough

Request flow, end to end. An application calls POST https://ai-gateway.internal/v1/chat/completions with an OpenAI-compatible payload and a virtual API key identifying the calling team/app. The edge gateway: (1) authenticates the virtual key, (2) checks the team's remaining token budget against the local+Redis counter, (3) resolves the requested model alias to its ordered deployment list, (4) applies PII redaction/policy filters to the request body if configured, (5) for the first deployment in the chain, either calls the external provider directly (OpenAI/Anthropic SDK call with the org's real provider key injected) or forwards to the internal Gateway API Gateway if the deployment is self-hosted.

Inside the inference gateway — the ext_proc handshake. Envoy Gateway, on receiving the request, is configured with an EnvoyExtensionPolicy pointing at the EPP as an external processor. Per the Gateway API Inference Extension's ext_proc protocol, Envoy pauses request processing and calls out to the EPP over gRPC with the request headers/body (enough to extract the model name and, for LoRA-serving pools, the adapter identifier). The EPP maintains a near-real-time view of every pod in the target InferencePool's metrics — each vLLM pod exposes its running queue length, KV-cache block utilization percentage, and loaded-adapter set via a metrics endpoint the EPP polls on a short interval (sub-second in tuned deployments). The EPP's scheduling algorithm scores candidates factoring in queue depth (avoid piling onto an already-backed-up pod), KV-cache headroom (a pod with a mostly-full cache will evict and re-compute prefixes, hurting latency), and adapter affinity (route to a pod that already has the needed LoRA adapter resident over one that would need to load it), then returns the chosen pod's address to Envoy, which completes the route.

Why this beats simple load balancing. Round-robin or least-connections load balancing treats every request as equal cost, which is false for LLM inference — a request with a 8K-token prompt and a request with a 50-token prompt consume wildly different KV-cache and compute resources, and a pod that "has the fewest active connections" can still be the most backed-up pod if its in-flight requests are long-context. Field data from the Inference Extension project consistently shows meaningfully better p99 tail latency and throughput under this cache-aware scheduling versus naive load balancing, precisely because it's optimizing for the actual bottleneck resource (GPU memory/KV-cache and compute queue) instead of a proxy metric (connection count) that doesn't correlate well with it.

Streaming and retry semantics. Chat completion requests are typically Server-Sent-Events streams; the edge gateway must proxy the stream token-by-token rather than buffering, and — this is the detail that bites teams new to this space — a mid-stream provider failure cannot simply be "retried" by resending the same request to a fallback provider, because the client may have already rendered partial output. Production implementations either buffer a small look-ahead window and only commit to a provider after confirming a healthy first-token response (adding a little latency but avoiding user-visible truncation-then-restart), or expose a client-side contract where a stream error triggers an application-level retry that explicitly clears and re-renders, never silently splicing two providers' outputs into one response.

Failure scenarios and recovery. Provider outright down (connection refused / 5xx storm): circuit breaker trips after N consecutive failures or an error-rate threshold over a sliding window, deployment pulled from rotation, alert fires, traffic shifts to next fallback. Provider rate-limited (429 with Retry-After): gateway respects the backoff hint, and if the team's own configured rate limit for that provider is already the bottleneck (not the provider's global limit), the request queues briefly rather than immediately failing over, since failing over to a more expensive provider for a transient self-inflicted rate limit is often the wrong trade. Internal InferencePool pod failure: standard Kubernetes pod eviction/replacement, but the EPP's fast-refreshing metrics view means traffic stops routing to a failing pod within one polling interval rather than waiting for a slow-to-trip readiness probe.

6. Production Troubleshooting

Symptom: p99 latency for support-agent-primary doubled starting 14:20 UTC, no deploy correlated.

Investigation path a senior platform engineer would follow:

  1. Split by hop, not by symptom. Pull the OTel trace waterfall for a handful of slow requests from that window. Is the added latency in the edge gateway's own processing (budget check, policy filter), the EPP scoring round-trip, queueing at the model server, or actual token-generation time (inter-token latency)? This single step usually eliminates 80% of the wrong-turn debugging.
  2. Check circuit-breaker and fallback metrics first. ai_gateway_fallback_total{deployment="gpt-4.1"} spiking means the primary is failing and every request is eating a full timeout-then-retry cycle before landing on a slower fallback — that's often the actual root cause disguised as "the model got slower." Cross-check against the provider's public status page.
  3. If internal routing: query InferencePool queue-depth and KV-cache-utilization metrics (exposed by the vLLM /metrics endpoint, scraped into Prometheus) — vllm:num_requests_waiting climbing steadily indicates the pool is under-provisioned for current load, not a routing bug. Compare against EPP scoring latency (epp_scoring_duration_seconds) — if the EPP itself is slow (e.g., its own metrics-polling loop backed up, or gRPC connection pool exhaustion to Envoy), every request pays that tax regardless of backend health.
  4. Check for a "hot pod" pattern. Grafana panel of per-pod queue depth across the InferencePool — if one or two pods show sustained high queue depth while siblings idle, the EPP's adapter-affinity logic may be over-concentrating traffic (common when a single LoRA adapter is disproportionately popular and only loaded on a subset of replicas); the fix is either pre-loading that adapter on more replicas or tuning the affinity-vs-load-balance weighting in the EPP config.
  5. Token-length drift. Query average prompt/completion tokens per request over the window (ai_gateway_completion_tokens_sum / ai_gateway_requests_total) — a silent prompt-template change (someone added few-shot examples, RAG context grew) inflating average tokens explains both the latency increase and an unexplained cost bump, and is the single most common "mystery latency regression" root cause in gateways fronting agentic workloads.
  6. Confirm via a synthetic canary. A scheduled synthetic request with a fixed, known prompt run every minute against each deployment in the fallback chain isolates "the model provider is genuinely slower right now" from "our routing/queueing logic degraded" — if the canary's latency is flat while real traffic's isn't, the problem is upstream of the model (request composition, RAG retrieval step, tool-call round-trips), not the gateway.

Sample debugging commands:

# Check InferencePool pod-level queue depth and cache utilization
kubectl get inferencepool llama-3-70b-pool -n inference -o yaml

# Tail EPP logs for scoring decisions on a specific trace
kubectl logs -n inference deploy/epp-llama-3-70b --since=10m | grep <trace-id>

# Prometheus: fallback rate by deployment over last hour
sum by (deployment) (rate(ai_gateway_fallback_total[5m]))

# Prometheus: p99 time-to-first-token by pool
histogram_quantile(0.99, sum by (le, pool) (rate(vllm_time_to_first_token_seconds_bucket[5m])))

7. Hands-on Lab

A local reproduction using kind, Envoy Gateway, and the Gateway API Inference Extension against a lightweight vLLM simulator (no GPU required for the routing-logic exercise).

# 1. Create a kind cluster and install Gateway API CRDs
kind create cluster --name ai-gateway-lab
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/latest/download/standard-install.yaml

# 2. Install Envoy Gateway
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.2.0 \
  -n envoy-gateway-system --create-namespace

# 3. Install the Gateway API Inference Extension CRDs (InferencePool, InferenceModel)
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/latest/download/manifests.yaml

# 4. Deploy the vLLM simulator (mock model server exposing OpenAI-compatible API + metrics)
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api-inference-extension/main/tools/simulator/deploy.yaml

# 5. Define an InferencePool + Endpoint Picker for the simulated pool
cat <<EOF | kubectl apply -f -
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: sim-model-pool
spec:
  targetPortNumber: 8000
  selector:
    app: vllm-sim
  extensionRef:
    name: epp-sim
EOF

# 6. Create a Gateway + HTTPRoute pointing at the InferencePool
kubectl apply -f gateway-and-route.yaml

# 7. Send test traffic and observe routing distribution across replicas
for i in $(seq 1 50); do
  curl -s -X POST http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"sim-model","messages":[{"role":"user","content":"test"}]}' \
    -o /dev/null -w "%{http_code}\n"
done

# 8. Simulate one replica going unhealthy and confirm traffic shifts away
kubectl exec deploy/vllm-sim-0 -- curl -X POST localhost:8000/debug/set-queue-depth?value=999

# Validation: check per-pod request counts skew away from the "overloaded" replica
kubectl logs -n envoy-gateway-system deploy/epp-sim --tail=100 | grep "selected_pod"

# Cleanup
kind delete cluster --name ai-gateway-lab

For the edge-layer half of the lab, deploy LiteLLM Proxy with two mock "providers" (a fast one and a deliberately-flaky one returning 500s on a random 30% of requests) and confirm the fallback chain and circuit breaker behave as expected — the LiteLLM config file's router_settings.fallbacks and retry_policy blocks are the pieces to exercise, validating that a client sees a successful response even while the primary is failing.

8. Production Case Study

Large-scale platforms converged on this same layered pattern independently, which is a strong signal it's the right shape rather than a coincidence. Cloud providers now ship it as a managed capability — Azure API Management added GenAI/LLM gateway features (token-limiting policies, semantic caching, load balancing across OpenAI deployments) directly into their existing APIM product rather than building a separate service, betting that enterprises want AI traffic governed by the same control plane as their REST APIs. Google's GKE Inference Gateway is the managed packaging of exactly the gateway-api-inference-extension pattern described above, reflecting that Google sees KV-cache-aware routing as important enough to productize as a first-class GKE feature rather than leaving it to a community extension alone.

On the multi-provider-abstraction side, the pattern that companies running agentic products at scale describe is consistent: never call a model provider SDK directly from application code, always through an internal gateway, with fallback chains defined declaratively and reviewed like any other production configuration (a fallback-chain change goes through the same review/rollout process as a deployment config, because getting it wrong has the same blast radius as a bad deploy). The KV-cache-aware scheduling work in the Gateway API Inference Extension itself grew out of Google's internal experience serving models at scale and was contributed as a CNCF-adjacent community project specifically so the ecosystem wouldn't fragment into a dozen incompatible vendor-specific inference-routing implementations — the same "avoid re-inventing it per company" logic that has driven prior infrastructure standardization (CNI, CSI, service mesh interfaces).

The consistent lesson across these production stories: the gateway's routing intelligence (cache-aware scheduling, fallback, budgets) delivers most of its value not in the steady state but during the tail-risk events — a provider incident, a runaway prompt, a popularity spike on one fine-tuned adapter — which is exactly why it's worth building before the incident that makes the case for it retroactively.

9. Architecture Review

Strengths. Clean separation of policy (edge gateway) from GPU-scheduling mechanism (inference gateway) means each evolves independently and can be owned by different teams (platform/FinOps owns the edge policy, ML infra owns the InferencePool layer). The ext_proc-based extension model means the inference-aware routing logic isn't forked into a custom Envoy build — it's a pluggable sidecar-style extension, so upgrading Envoy Gateway itself doesn't require re-implementing the EPP.

Weaknesses. Two extra network hops (edge gateway, EPP round-trip) on every request add latency that's negligible for multi-second completions but proportionally painful for very low-latency, low-token workloads — teams sometimes end up needing a documented "bypass path" for hyper-latency-sensitive internal calls, which reintroduces exactly the ungoverned-traffic problem the gateway was built to prevent, so that exception needs its own guardrail (still logged, still budget-tracked, just skipping the EPP scoring round-trip). The Redis-backed budget counter is a shared-state dependency that, while approximated with local token buckets for throughput, is still a single logical point of failure for budget enforcement (though not for routing) — a Redis outage should fail open on routing but fail toward conservative (deny or heavily throttle) on budget checks, and that fail-mode decision needs to be explicit, not discovered during an incident.

What breaks first at 10x scale. The EPP's metrics-polling loop against every pod in a pool — at 10x the model-server replica count, sub-second polling intervals against hundreds of pods per pool starts to strain the EPP's own resource budget and the gRPC connection count to Envoy; this needs either sharding the EPP per pool-subset or moving to a push-based metrics model (pods push state changes rather than being polled) before it becomes the bottleneck. The synchronous Redis budget check also becomes the first shared-state chokepoint — at 10x request volume it needs to move fully to the local-token-bucket-with-async-reconciliation model rather than any synchronous path, accepting looser budget precision as the trade.

What changes at 100M-user scale. The architecture would need to go multi-region with region-local edge gateways and InferencePools (not a single global gateway), with budget/cost aggregation becoming an eventually-consistent global rollup rather than a synchronous check, and provider fallback chains becoming region-aware (falling back to a same-region alternative before a cross-region one, to protect both latency and data-residency requirements). Semantic caching (returning a cached response for a near-duplicate prompt, using embedding-similarity matching) becomes essential at that scale purely for cost reasons — it's a nice-to-have at today's scale and a load-bearing cost-control mechanism at 100M-user scale.

What to redesign. The EPP's polling-based metrics collection should move to an event-driven push model to remove the scaling ceiling described above. The Redis-based synchronous budget path should be redesigned as fully async with local enforcement and eventual reconciliation from day one rather than retrofitted under pressure — this is the one piece of the current design that was a deliberate "good enough for now" trade-off, explicitly flagged for revisit as traffic grows.

10. Best Practices

Route every model call, self-hosted or third-party, through the gateway with no exceptions carved out by convention — an ungoverned direct-SDK call is invisible to budget enforcement, fallback, and audit, and it will exist in every codebase where "just this once" was allowed. Treat fallback-chain and rate-limit configuration as reviewed, version-controlled config (GitOps-managed, same as any other production manifest) rather than a value set once in a dashboard and forgotten. Instrument token counts, not just request counts and latency, as a first-class metric from day one — cost anomalies are visible in token-rate dashboards hours before they show up on an invoice. Build circuit breakers per-deployment with sane defaults out of the box rather than per-team opt-in, since the team least likely to configure a circuit breaker correctly is the one that needs it most during their first provider incident. Keep the fallback chain provider-diverse (different underlying infrastructure, not just different model names from the same provider) so a single provider's regional incident doesn't take out your entire fallback path. Cap max-tokens and enforce prompt-length ceilings at the gateway as a backstop even when application code is supposed to enforce them — defense in depth against exactly the kind of silent prompt-template regression from Section 2.

11. Common Production Mistakes

Hardcoding provider SDKs directly into application code "temporarily" during a prototype and never migrating to the gateway before the feature ships — prototypes become production faster than the migration gets prioritized. Treating request-count rate limiting as equivalent to cost control — a team can stay well under a requests-per-minute limit while still burning an enormous token budget if each request is expensive, and vice versa; the two need independent limits. Retrying a failed streaming request by resending to a fallback provider without accounting for tokens already streamed to the client, producing visibly spliced or duplicated output. Configuring the EPP or InferencePool health checks with the same readiness-probe assumptions as a stateless web service — a model server can be "ready" (process up, port open) while its KV-cache is fully saturated and every new request will queue for tens of seconds; readiness needs to reflect actual serving capacity, not process liveness. Skipping PII redaction on the assumption that "we only send internal data to our own models" — internal-only today doesn't mean internal-only after the next fallback-chain change adds a third-party provider to the same alias.

12. Interview Preparation

Q: Why can't you just use a standard API gateway (Kong, Apigee) for LLM traffic without any inference-specific extensions? A: Standard gateways route on stable, cheap-to-evaluate signals (path, header, weighted round-robin) and assume uniform, fast request cost. LLM traffic has wildly variable per-request cost (token count), backend health defined by resource state (KV-cache occupancy, queue depth) rather than simple up/down, and long-lived streaming responses where naive retry semantics corrupt output. You can bolt provider abstraction and budget logic onto a standard gateway (that's what LiteLLM/Kong AI Gateway do), but routing to self-hosted model replicas efficiently needs cache-aware scheduling that a generic L7 gateway has no visibility into without an inference-specific extension.

Q: Walk through what happens end to end when a self-hosted model's primary replica pool is saturated. A: EPP's live metrics polling detects rising queue depth and shrinking KV-cache headroom across the pool; new requests get scored lower for those replicas and routed to any replica with more headroom; if the whole pool is saturated, queue depth keeps rising and, assuming KEDA/HPA is wired to pool-level queue-depth metrics, triggers scale-out of new replicas — masked by a warm-pool buffer if model load time is significant. If scale-out capacity (GPU nodes) isn't available fast enough, sustained saturation should trip a circuit breaker at the edge gateway, failing over to the next deployment in the chain (another provider) rather than queuing indefinitely and blowing latency SLOs.

Q: How do you rate-limit fairly across teams sharing a model without a synchronous global counter becoming a bottleneck? A: Token-bucket approximation with local buckets per gateway replica, refilled from a shared budget periodically (e.g., every few seconds) rather than checked synchronously per request — trades strict per-token accuracy for throughput, which is the right trade because budget enforcement needs to catch large deviations quickly, not enforce the exact Nth token. Pair with async reconciliation against the source of truth (Redis or a dedicated budget service) and alert on drift beyond a threshold.

Q: What's the failure mode you're most worried about in this architecture, and how do you mitigate it? A: Silent correctness/cost regressions that don't trip any binary alarm — a prompt template gradually growing, a fallback chain quietly routing everything to the most expensive provider because a cheaper one's health check is subtly misconfigured, an adapter-affinity routing skew concentrating load on one replica. Binary up/down alerting misses all of these; the mitigation is trend-based alerting on token-rate, fallback-rate, and per-pod load-distribution metrics, not just error-rate and latency thresholds.

Q: When would you not build a two-layer gateway architecture like this? A: A platform with a single model provider, no self-hosted models, and one or two consuming teams doesn't need the inference-aware layer at all — a managed multi-provider gateway or even direct SDK calls with basic retry logic is proportionate. The two-layer design earns its complexity once there's meaningful self-hosted GPU capacity to route intelligently against and enough teams/traffic that centralized budget/fallback policy has real payoff; building it earlier is premature infrastructure investment.

13. Latest Industry Updates

The kubernetes-sigs/gateway-api-inference-extension project has reached general availability, with the InferencePool and Endpoint Picker (EPP) pattern now supported across Envoy Gateway, kgateway, and GKE's managed Inference Gateway — meaning inference-aware routing is no longer a single-vendor bet but an interoperable, Gateway-API-native standard, the same trajectory CSI and CNI took from vendor-specific to standardized (Kubernetes blog: Introducing Gateway API Inference Extension, project docs). This matters for production teams because it de-risks the "which inference gateway vendor do we bet on" decision the same way Gateway API itself de-risked ingress-controller lock-in — the routing logic and CRDs are portable across implementations.

Envoy AI Gateway (envoyproxy/ai-gateway) has matured as the reference implementation for unifying access to generative AI services on top of Envoy Gateway, giving teams already standardized on Envoy/Gateway API a first-party path to both the provider-abstraction and inference-routing layers without adopting a separate proxy technology stack (envoyproxy/ai-gateway). Alibaba Cloud's ACK and other managed Kubernetes offerings have shipped their own packaging of the inference extension pattern, reinforcing that KV-cache-aware, queue-aware LLM routing has become an expected baseline capability of enterprise Kubernetes platforms rather than a niche optimization (Alibaba Cloud ACK Gateway with Inference Extension).

On the routing-intelligence front, semantic-router style projects (vLLM Semantic Router and similar) are integrating directly with the Gateway API Inference Extension to add embedding-similarity-based semantic caching and intent-based model selection at the gateway layer — routing a request to a cheaper/smaller model when the semantic classifier determines the query doesn't need the largest model, purely at the infrastructure layer with no application code change (vLLM Semantic Router k8s docs). This is the direction to watch: routing decisions that used to require application-level prompt engineering (which model should handle this?) are moving down into infrastructure, the same layering shift that happened with service-mesh traffic policy moving out of application code a decade ago.

Sources:

14. Summary & Cheat Sheet

Core architecture: two composed layers — an edge multi-provider LLM gateway (LiteLLM Proxy / Kong AI Gateway) owning policy: virtual keys, per-team token budgets, PII redaction, provider fallback chains with circuit breakers; and a Kubernetes-native inference gateway (Envoy Gateway/kgateway + gateway-api-inference-extension) owning GPU-scheduling-aware routing to self-hosted model replicas via InferencePool + Endpoint Picker.

Key CRDs/components: InferencePool (groups model-server replicas), InferenceModel (maps a logical model name to pool + priority/criticality), Endpoint Picker / EPP (scores pods via ext_proc on queue depth, KV-cache headroom, adapter affinity).

Rate limiting: token-based, not request-count-based; local token-bucket approximation reconciled async against a shared budget store for throughput at scale; fail-mode for the budget path (open vs. closed) must be an explicit decision.

Failover: per-deployment circuit breakers on error-rate/latency sliding windows; provider-diverse fallback chains; streaming responses need look-ahead buffering or explicit client-retry contracts, never silent mid-stream provider splicing.

Troubleshooting checklist: split latency by hop (edge/EPP/queueing/token-generation) → check fallback-rate metrics before assuming model slowness → check InferencePool queue-depth/cache-utilization → look for hot-pod adapter-affinity skew → check token-length drift → confirm with a synthetic canary against a fixed prompt.

Best practice one-liners: no direct-SDK bypass of the gateway, ever. Token metrics as first-class as latency metrics. Circuit breakers on by default, not opt-in. Fallback chains provider-diverse, GitOps-managed, and reviewed like production config. Cap prompt/output length at the gateway as a backstop even when application code should already enforce it.

Failure points to watch as you scale: EPP metrics-polling load (move to push-based before it saturates), synchronous budget-check shared state (move fully async before it becomes a global chokepoint), and semantic caching moving from nice-to-have to load-bearing cost control well before 100M-user scale.