DailyDevOps — 2026-08-25

Topic: Taming AI Inference Latency Spikes with vLLM, KServe & Distributed Serving on Kubernetes

1. Topic of the Day

Every hyperscaler that ships LLM products (OpenAI, Anthropic, Google, Microsoft, Amazon, NVIDIA-powered neoclouds) faces the same core operational problem: GPU inference is expensive, latency-sensitive, and bursty, while GPUs themselves are supply-constrained and slow to provision. Unlike stateless CPU microservices, LLM inference workloads carry large in-memory state (KV cache), have highly variable per-request cost (prompt length × output length), and are extremely sensitive to co-location and memory fragmentation.

vLLM exists because naive HuggingFace transformers.generate() serving wastes 60-80% of GPU memory on KV-cache fragmentation and cannot batch requests efficiently. vLLM's PagedAttention algorithm treats the KV cache like OS virtual memory pages, enabling near-zero-waste memory management and continuous batching — this is why it became the de facto inference engine at scale (Stripe, Character.AI–style workloads, and multiple frontier labs' internal serving stacks use vLLM or vLLM-derived engines).

On top of vLLM, Kubernetes provides the orchestration layer: KServe (or a hand-rolled operator) wraps vLLM pods with K8s-native autoscaling, canary rollout, and multi-model routing. Ray handles distributed training and multi-node tensor/pipeline parallel inference for models too large for a single GPU or single node. This stack — vLLM + KServe + Kueue + Ray — is now the 2026 industry-consensus pattern for production LLM serving on Kubernetes, replacing the fragmented 2023-era landscape of bespoke Flask + Triton deployments.

Where this is used in production: any org serving LLM inference at scale behind an API — chat products, RAG pipelines, agentic backends, internal copilots. If you operate a /v1/chat/completions-shaped endpoint backed by open-weight models (Llama, DeepSeek, Qwen, Mistral, Gemma) on your own GPU fleet instead of a hosted API, you are running this exact stack.

2. Real Business Problem

Incident scenario: Your company serves a RAG-based internal copilot on a fleet of 40 × A100-80GB GPUs, using vLLM behind KServe on EKS. At 9:15am on a Tuesday (standup time — everyone opens the copilot at once), p99 latency for time-to-first-token (TTFT) spikes from 400ms to 11 seconds. Some requests time out entirely. On-call gets paged. Grafana shows GPU utilization pinned at 100%, but throughput (tokens/sec) is flat or declining — a classic sign of KV-cache thrashing, not raw compute exhaustion.

This is the kind of problem senior platform engineers are expected to diagnose in minutes, not hours, because every minute of degraded latency during a burst is visible to the entire org.

3. Production Architecture

                                   ┌─────────────────────────────┐
                                   │        Global Load           │
                                   │   Balancer / API Gateway     │
                                   │   (Envoy / AWS ALB + WAF)    │
                                   └──────────────┬───────────────┘
                                                  │
                          ┌───────────────────────┼───────────────────────┐
                          │                        │                       │
                 ┌────────▼────────┐     ┌─────────▼────────┐   ┌─────────▼────────┐
                 │  AI Gateway /    │     │  AI Gateway /     │   │  AI Gateway /     │
                 │  Router (region  │     │  Router (region   │   │  Router (region   │
                 │  us-east-1)      │     │  us-west-2)       │   │  eu-west-1)       │
                 │  - auth/rate     │     │                    │   │                    │
                 │  - semantic      │     │                    │   │                    │
                 │    caching       │     │                    │   │                    │
                 │  - request       │     │                    │   │                    │
                 │    routing by    │     │                    │   │                    │
                 │    model/tenant  │     │                    │   │                    │
                 └────────┬─────────┘     └────────┬───────────┘   └─────────┬──────────┘
                          │                          │                        │
        ┌─────────────────┴─────────────────┐        │                        │
        │        Kubernetes Cluster (EKS)     │        │                        │
        │  ┌────────────────────────────────┐│        │                        │
        │  │   KServe InferenceService       ││        │      (same pattern,    │
        │  │   ┌──────────────────────────┐ ││        │       replicated per    │
        │  │   │  vLLM Prefill Pool        │ ││        │       region for DR)   │
        │  │   │  (prefill-optimized,      │ ││        │                        │
        │  │   │   H100 nodes, tensor      │ ││        │                        │
        │  │   │   parallel=4)             │ ││        │                        │
        │  │   └──────────────┬───────────┘ ││        │                        │
        │  │                  │ KV cache      ││        │                        │
        │  │                  │ transfer      ││        │                        │
        │  │                  │ (NIXL/RDMA)   ││        │                        │
        │  │   ┌──────────────▼───────────┐ ││        │                        │
        │  │   │  vLLM Decode Pool         │ ││        │                        │
        │  │   │  (decode-optimized,       │ ││        │                        │
        │  │   │   continuous batching,    │ ││        │                        │
        │  │   │   PagedAttention)         │ ││        │                        │
        │  │   └───────────────────────────┘ ││        │                        │
        │  └────────────────────────────────┘│        │                        │
        │  ┌────────────────────────────────┐│        │                        │
        │  │  Kueue (GPU queue/quota mgmt)   ││        │                        │
        │  │  Karpenter (GPU node autoscale) ││        │                        │
        │  │  NVIDIA GPU Operator + Device   ││        │                        │
        │  │  Plugin (MIG / time-slicing)    ││        │                        │
        │  └────────────────────────────────┘│        │                        │
        │  ┌────────────────────────────────┐│        │                        │
        │  │  Observability: Prometheus +    ││        │                        │
        │  │  DCGM-exporter + Grafana +      ││        │                        │
        │  │  OpenTelemetry (GenAI semconv)  ││        │                        │
        │  └────────────────────────────────┘│        │                        │
        └───────────────────────────────────┘        │                        │
                          │                            │                        │
                 ┌────────▼────────┐          ┌────────▼────────┐    ┌─────────▼────────┐
                 │ Model Registry / │          │  Vector DB       │    │  Redis Semantic   │
                 │ Object Store     │          │  (Qdrant/PGVector│    │  Cache             │
                 │ (S3 + MLflow)    │          │  for RAG)        │    │                    │
                 └──────────────────┘          └──────────────────┘    └────────────────────┘

Key design decisions:

• Prefill/decode disaggregation. Prefill (processing the prompt, compute-bound, parallelizable) and decode (generating tokens one at a time, memory-bandwidth-bound) have opposite resource profiles. Colocating them on the same GPU pool causes decode requests to stall behind long prefill batches — this is the single biggest lever for tail latency in 2026-era vLLM deployments. Disaggregated serving (vLLM + NIXL/LMCache for KV transfer) is now standard at scale.
• Security boundaries: the AI Gateway terminates auth/rate-limiting/PII redaction before traffic ever reaches the cluster; GPU node pools run in a dedicated node group with no direct external ingress; KV cache and prompts never persist outside the pod's ephemeral memory unless explicitly logged (compliance boundary for PII-bearing prompts).
• HA/multi-region: stateless router layer fronts region-local KServe deployments; cross-region failover is DNS/gateway-based (Route53 latency routing) rather than live GPU failover, because GPU capacity is regionally scarce and pre-warming standby capacity in every region is cost-prohibitive — this is a deliberate trade-off, not an oversight.
• Multi-cloud consideration: GPU capacity is the scarcest resource in 2026; many orgs run a primary cluster on the cloud with best GPU availability/pricing (often Azure or a GPU-neocloud like CoreWeave/Lambda) and burst to AWS/GCP during capacity crunches, using a common OCI-based model artifact format so model registries are cloud-portable.

Trade-offs & evolution path: at small scale, colocated prefill+decode on a single vLLM deployment is simpler to operate and sufficient. Disaggregation adds operational complexity (KV transfer network, two autoscaling policies, more failure modes) and is only worth it once tail-latency SLOs are being violated by prefill/decode contention — typically past ~20-30 QPS sustained per model.

4. Solution Design

Design decisions and alternatives:

Cost implications: GPU-hours dominate cost. Prefill/decode disaggregation improves GPU utilization 20-40% by letting each pool use the optimal batch size for its workload, directly reducing $/1K tokens. Spot/preemptible GPU capacity for decode pools (with fast rescheduling via Karpenter) is common for cost-sensitive tiers; prefill pools are usually on-demand due to tighter latency requirements.

Security implications: prompts often carry PII/secrets — enforce workload identity (IRSA/Azure Workload Identity) so vLLM pods can only pull models from an approved registry, and route all prompt/response logging through a redaction layer before it hits observability backends.

Performance implications: continuous batching + PagedAttention typically yields 2-24x throughput over naive HF serving; speculative decoding (draft model + verification) can cut decode latency further for latency-critical tiers at the cost of extra GPU memory for the draft model.

5. Deep Technical Walkthrough

Request flow:

1. Client request hits the AI Gateway → auth, rate limit, semantic cache check (Redis embedding lookup — if a near-duplicate query was answered recently, short-circuit here).
2. Gateway routes to the KServe InferenceService for the target model, using a router that's aware of current queue depth per replica (not just round-robin).
3. Request enters the vLLM scheduler: on the prefill pool, it's added to a running batch based on available KV-cache blocks (PagedAttention allocates fixed-size pages on demand rather than reserving max-context-length up front).
4. Prefill computes the full attention over the prompt in one forward pass, producing the initial KV cache, which is then transferred to a decode-pool replica over RDMA/NIXL (this transfer is the new bottleneck class introduced by disaggregation — sub-10ms on NVLink/InfiniBand fabrics, much worse on plain TCP).
5. Decode pool performs iterative, continuous-batched generation: at every scheduler tick, it can add newly-arrived requests to the running batch (not wait for the whole batch to finish) and evict/preempt lowest-priority requests if memory pressure hits.
6. Tokens stream back through the gateway to the client via SSE/gRPC streaming.

Control plane vs data plane: KServe's control plane (K8s controller) only touches scaling decisions, rollout, and pod lifecycle — it never sits in the token-generation hot path. The data plane is entirely inside the vLLM engine process; this separation is why K8s-level slowness (API server latency, scheduler delays) doesn't directly cause token-generation latency, but it does cause slow reaction to load spikes (new pods take too long to become Ready).

Failure scenarios & recovery:

• OOM on decode pool: vLLM's scheduler preempts (evicts) lowest-priority sequences and recomputes their KV cache later — this shows up as a latency spike for evicted requests, not a crash. Mitigate with --gpu-memory-utilization headroom (typically 0.90, not 0.95+) and cache-aware admission control.
• KV transfer failure between prefill/decode: falls back to recomputing prefill on the decode node (slow-path) — must be monitored explicitly, since it silently degrades latency without erroring.
• Node loss (spot reclaim): in-flight requests on that pod are lost; client-side retry with idempotency + gateway-level request replay is required, since GPU inference is not naturally idempotent-safe without careful design.

Scaling behavior: vLLM throughput scales sub-linearly with batch size until it hits a "memory wall" (all KV cache pages consumed) — beyond that point, adding more concurrent requests only increases queueing, not throughput. This is exactly the ceiling your autoscaler must trigger on, not GPU% utilization.

6. Production Troubleshooting

Symptom: p99 TTFT spikes from 400ms → 11s at 9:15am; GPU util pinned at 100%; token throughput flat/declining.

Step-by-step senior-SRE investigation:

1. Check the vLLM engine metrics first (vllm:num_requests_waiting, vllm:num_requests_running, vllm:gpu_cache_usage_perc):

   kubectl exec -it vllm-decode-0 -n inference -- curl -s localhost:8000/metrics | grep -E "num_requests_waiting|gpu_cache_usage"
   

   If gpu_cache_usage_perc is pinned near 100% while num_requests_waiting climbs, this confirms KV-cache exhaustion, not compute exhaustion — the fix is memory-related, not "add more GPUs of the same shape."

2. Check for preemption/eviction events in vLLM logs:

   kubectl logs -n inference vllm-decode-0 --since=15m | grep -i "preempt\|evict"
   

   A high preemption rate means requests are being kicked out of the running batch and recomputed — visible as sawtooth latency in Grafana.

3. Correlate with request shape. Pull a sample of requests from the gateway's access logs — is this an unusually long-context burst (e.g., a batch job dumping large documents into the RAG pipeline at the same time as interactive standup traffic)? Long-context requests consume disproportionate KV-cache pages and starve short interactive requests.

4. Check Karpenter/Kueue reaction time:

   kubectl get events -n inference --sort-by=.lastTimestamp | grep -i "FailedScheduling\|Provisioned"
   

   If new GPU nodes are still being provisioned 5+ minutes after the spike started, your autoscaling trigger metric fired too late — the fix belongs in KEDA's scaling rule (should trigger on queue depth trend, not steady-state GPU%).

5. Dashboards: Grafana panel showing gpu_cache_usage_perc vs num_requests_waiting vs p99 TTFT overlaid — this correlation is the single most diagnostic view for LLM-serving incidents and should be the first panel any on-call engineer opens.

6. Root cause in this scenario: standup-time traffic burst + several long-context RAG requests in the same window exhausted KV-cache pages on the decode pool; HPA was scaled on GPU utilization (already saturated, so no signal to scale further) instead of queue depth; Karpenter took 3 minutes to provision new GPU nodes because no pre-warmed capacity buffer existed.

7. Immediate mitigation: enable --max-num-seqs cap with priority-aware admission (interactive traffic gets priority over batch RAG ingestion), and set a KEDA ScaledObject on vllm:num_requests_waiting with a low threshold and fast scale-up.

8. Config change (KEDA GPU-aware scaling):

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     name: vllm-decode-scaler
     namespace: inference
   spec:
     scaleTargetRef:
       name: vllm-decode
     minReplicaCount: 4
     maxReplicaCount: 32
     triggers:
     - type: prometheus
       metadata:
         serverAddress: http://prometheus.monitoring:9090
         query: avg(vllm:num_requests_waiting{pod=~"vllm-decode.*"})
         threshold: "5"
   

7. Hands-on Lab

Goal: deploy vLLM on a local kind cluster (CPU-only mode for lab purposes, or GPU node if available), observe PagedAttention metrics, and simulate load to see queueing behavior.

# 1. Create a lightweight kind cluster
kind create cluster --name vllm-lab

# 2. Install KServe quickstart (includes Knative + Istio lightweight)
curl -s "https://raw.githubusercontent.com/kserve/kserve/release-0.14/hack/quick_install.sh" | bash

# 3. Deploy a small open model with vLLM runtime (use a small model for lab feasibility)
cat <<EOF | kubectl apply -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: qwen-lab
  namespace: default
spec:
  predictor:
    model:
      modelFormat:
        name: vLLM
      runtime: kserve-vllmserver
      storageUri: "hf://Qwen/Qwen2.5-0.5B-Instruct"
      resources:
        limits:
          cpu: "4"
          memory: 8Gi
        requests:
          cpu: "2"
          memory: 4Gi
EOF

# 4. Wait for readiness
kubectl get inferenceservice qwen-lab -w

# 5. Port-forward and send a test request
kubectl port-forward svc/qwen-lab-predictor 8080:80 &
curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen-lab","messages":[{"role":"user","content":"Explain PagedAttention in one sentence."}]}'

# 6. Load test to observe queueing (requires `hey` or `k6`)
hey -z 60s -c 20 -m POST -H "Content-Type: application/json" \
  -d '{"model":"qwen-lab","messages":[{"role":"user","content":"Summarize Kubernetes networking."}]}' \
  http://localhost:8080/v1/chat/completions

# 7. Observe metrics during load
kubectl exec -it deploy/qwen-lab-predictor -- curl -s localhost:8000/metrics | grep -E "num_requests|gpu_cache"

# --- Cleanup ---
kubectl delete inferenceservice qwen-lab
kind delete cluster --name vllm-lab

Validation: confirm num_requests_waiting rises during the hey load test and drains afterward; confirm p99 latency in the hey output correlates with the queue-depth spike.

8. Production Case Study

Stripe (per late-2025 reporting) migrated internal LLM-backed fraud-detection and support-copilot workloads from a legacy HF Transformers serving stack to vLLM, reporting roughly 73% inference cost reduction while serving ~50M daily API calls on roughly one-third of their prior GPU fleet — driven primarily by continuous batching and better memory utilization from PagedAttention rather than raw hardware upgrades. [Source: Introl blog, industry reporting]

Broader industry pattern (2026 consensus): frontier labs and large platform teams converge on a layered stack — vLLM (or an internally-forked variant) for the engine, Kueue for multi-tenant GPU fair-share scheduling, KServe or a custom operator for K8s-native lifecycle management, and Ray for distributed training plus multi-node inference for models exceeding single-node GPU memory. NVIDIA's own recommended reference architecture for enterprise inference platforms now explicitly includes GPU Operator + KServe + vLLM/Triton as interchangeable engine backends behind a common serving API, reflecting the industry's move toward engine-agnostic serving abstractions so platform teams aren't locked into one inference runtime.

This validates the disaggregated, autoscaled, queue-aware design in Section 3 — it's not a one-off pattern, it's converging industry practice.

9. Architecture Review

Strengths:

• Prefill/decode disaggregation directly targets the actual latency driver (cache contention), not a proxy metric.
• Queue-depth-based autoscaling (via KEDA) reacts to the real leading indicator of SLO breach, not the lagging GPU% signal.
• Clear security boundary at the gateway keeps PII redaction and auth out of the latency-critical inference path.

Weaknesses:

• Cross-region failover is DNS-based, not live — a regional GPU capacity outage means real degraded service for the failover window, not seamless continuity.
• KV-cache transfer over RDMA/NIXL is a new single point of subtle failure (silent fallback to recompute) that's easy to under-monitor.
• Spot-instance decode pools introduce request-loss risk that pushes complexity onto client-side retry logic — this is an incomplete solution without idempotency keys end-to-end.

What breaks first at 10x scale: the semantic cache (Redis) becomes a bottleneck/hot-key problem, and the AI Gateway's per-request routing logic (queue-depth-aware routing across replicas) starts adding meaningful latency itself once replica counts exceed what a single router can track in real time — this typically pushes toward a sharded/hierarchical gateway layer.

At 100M-user scale: you'd need multi-region active GPU capacity (not just DNS failover), a global model registry with regional artifact caching to avoid cross-region cold-start pulls, and almost certainly a move toward speculative decoding and smaller distilled models for the majority of traffic, reserving frontier-size models for a minority of high-value requests (cost is the forcing function, not just latency).

What I'd redesign: replace DNS-based regional failover with a capacity-reservation system that keeps a small warm standby pool per region (accepting the cost) for tier-1 SLA traffic, and move semantic caching to a sharded/consistent-hashed cache layer before it becomes a bottleneck.

10. Best Practices

Reliability: always run prefill and decode pools with independent PodDisruptionBudgets; never let Karpenter consolidate/terminate a node mid-generation without graceful drain (vLLM supports SIGTERM-triggered request draining — wire it into your terminationGracePeriodSeconds).

Scalability: scale on request queue depth and KV-cache occupancy, not GPU utilization; separate autoscaling policies for prefill vs decode pools since their load profiles differ.

Observability: adopt the OpenTelemetry GenAI semantic conventions (span attributes for prompt tokens, completion tokens, model name) so tracing is comparable across engines/vendors; track TTFT and inter-token-latency (ITL) as separate SLOs, not just end-to-end latency.

Security: enforce workload identity for model registry pulls; redact prompts/completions before they hit long-term log storage; treat GPU nodes as a distinct trust boundary from general compute nodes with tighter network policies.

Cost optimization: track $/1M tokens as your primary cost KPI, not $/GPU-hour; use spot for decode pools where request-loss tolerance allows, on-demand for prefill; right-size context windows (don't let default max-context waste KV-cache pages nobody needs).

Performance: benchmark with realistic request-size distributions from production, not synthetic fixed-length prompts — batch efficiency is highly sensitive to prompt/output length variance.

Maintainability: pin vLLM versions per model family (the engine evolves fast; a version bump can change default batching behavior) and canary every engine upgrade against a latency/throughput regression suite before full rollout.

Operational excellence: run regular GPU-node chaos drills (simulate spot reclaim, node failure) so the on-call team has muscle memory before a real 9:15am incident.

11. Common Production Mistakes

• Autoscaling on GPU utilization alone — GPU% stays near 100% even while thrashing, giving false confidence that "capacity is fine" during an actual incident.
• Colocating prefill and decode on identical pods at scale, causing long prompts to stall short interactive generations behind them in the same batch.
• No admission control / priority tiers — batch jobs (bulk RAG ingestion, evaluation runs) compete unthrottled with interactive user traffic for the same KV-cache budget.
• Ignoring KV-cache transfer failures in disaggregated setups — silent fallback to recompute quietly doubles latency without tripping alerts.
• Treating inference pods as stateless for retry purposes without idempotency keys, leading to duplicate side effects (e.g., duplicate downstream API calls triggered by agentic workflows) on client retries after a dropped connection.
• Under-provisioning headroom on --gpu-memory-utilization (setting it to 0.98+) leaving no slack for activation memory spikes, causing OOM crashes instead of graceful preemption.

12. Interview Preparation

Q1 (Kubernetes expert): "Why does GPU utilization mislead you when diagnosing LLM inference latency, and what would you monitor instead?" Answer: GPU util measures whether SM cores are executing instructions, not whether the workload is making forward progress efficiently. A vLLM engine can show 100% GPU util while thrashing on KV-cache evictions/recomputation — the GPU is "busy" doing wasted work. Better signals: num_requests_waiting (queue depth), gpu_cache_usage_perc (KV-cache pressure), and the ratio of running vs waiting requests. Autoscaling policies should trigger on queue depth trend, since it's a leading indicator; GPU util is at best a lagging, ambiguous signal for this workload class.

Q2 (AI Infrastructure): "When would you choose prefill/decode disaggregation over a simple colocated vLLM deployment?" Answer: Once request volume is high enough that prefill batches (compute-bound, benefit from large batch sizes) start delaying decode-phase token generation (memory-bandwidth-bound, latency-sensitive) for concurrently running requests. Below roughly 20-30 sustained QPS per model, the operational complexity of disaggregation (separate pools, KV transfer network, two autoscaling policies) usually isn't worth it. Above that, disaggregation is close to mandatory for consistent tail latency.

Q3 (Platform Engineering): "How do you handle GPU node preemption (spot reclaim) mid-inference without breaking client requests?" Answer: Combine graceful SIGTERM handling in the inference engine (vLLM drains in-flight requests within terminationGracePeriodSeconds), client-side retry with idempotency keys so a dropped/failed request can be safely replayed, and reserve on-demand capacity for latency-critical tiers while relegating spot to fault-tolerant batch/decode-only pools where request loss is acceptable.

Q4 (Cloud Architecture): "How would you design multi-region failover for a GPU inference service, given GPU capacity is regionally scarce?" Answer: Full active-active with pre-warmed capacity in every region is usually cost-prohibitive given GPU scarcity. A common pattern is DNS/gateway-based latency routing to region-local clusters, with a small warm standby buffer sized for tier-1 SLA traffic only, accepting degraded (not zero) capacity during a regional outage for lower-priority traffic. The trade-off should be made explicit and cost-justified, not left implicit.

Q5 (Kubernetes internals): "Explain what happens inside the vLLM scheduler when GPU memory pressure hits during a running batch." Answer: The scheduler preempts (evicts) the lowest-priority sequences from the running batch, freeing their KV-cache pages. Evicted sequences are either swapped to CPU memory (if configured) or their KV cache is dropped and recomputed from scratch when they're rescheduled. This shows up as a latency spike (not a crash) for evicted requests — it's vLLM's graceful degradation mechanism, and monitoring preemption rate is essential to catch this before it becomes visible as a broad SLO violation.

13. Latest Industry Updates

• Kubernetes 1.34 / 1.35: Dynamic Resource Allocation (DRA) reached GA in 1.34 — directly relevant to GPU scheduling, since DRA replaces the older device-plugin model with a more expressive resource-claim API that better represents GPU topology (NVLink domains, MIG partitions) to the scheduler. 1.35 stabilizes structured auth config for the API server (JWT-based auth without restarts) — useful for AI gateways doing dynamic per-tenant token issuance. Cloudsmith, Kubernetes.io
• AI/ML on Kubernetes production stack consensus (2026): vLLM + Kueue + KServe + Ray has solidified as the reference architecture across multiple vendor and practitioner guides this year, reducing the "which serving stack" decision fatigue that characterized 2023-2024. KubernetesGuru
• vLLM production cost outcomes: reported real-world migrations (e.g., Stripe-scale workloads) show 70%+ inference cost reductions from continuous batching and PagedAttention adoption, reinforcing that engine choice remains one of the highest-leverage cost levers available to platform teams — more impactful than most infrastructure-level cost optimizations. Introl
• Why this matters operationally: DRA's GA status means GPU-aware scheduling is finally a first-class Kubernetes primitive rather than a device-plugin workaround — expect Karpenter, Kueue, and the NVIDIA GPU Operator to converge on DRA as the standard interface over the next few release cycles, which will change how you write GPU resource requests in pod specs.

14. Summary & Cheat Sheet

Key concepts:

• PagedAttention: OS-style paged memory management for KV cache — the core innovation behind vLLM's throughput advantage.
• Continuous batching: add/remove requests from a running batch per scheduler tick, instead of static batch windows.
• Prefill/decode disaggregation: split compute-bound prompt processing from memory-bandwidth-bound token generation across separate pools.
• TTFT (time-to-first-token) and ITL (inter-token-latency) are the two SLOs that matter — track them separately, never just end-to-end latency.

Architecture pattern: AI Gateway (auth, semantic cache, routing) → KServe InferenceService → vLLM prefill pool → KV transfer (NIXL/RDMA) → vLLM decode pool → streamed response. Kueue + Karpenter + GPU Operator handle scheduling/provisioning underneath.

Diagnostic commands:

# vLLM engine metrics
kubectl exec -it <pod> -n inference -- curl -s localhost:8000/metrics | grep -E "num_requests|gpu_cache"

# Preemption/eviction check
kubectl logs <pod> -n inference --since=15m | grep -i "preempt\|evict"

# Scheduling delay check
kubectl get events -n inference --sort-by=.lastTimestamp | grep -i "FailedScheduling\|Provisioned"

Best-practice checklist:

• Autoscale on queue depth / KV-cache occupancy, not GPU utilization
• Separate prefill and decode pools once sustained QPS exceeds ~20-30 per model
• Priority/admission control between interactive and batch traffic
• Idempotency keys for client-side retries on spot-reclaimed pods
• --gpu-memory-utilization headroom (~0.90) to allow graceful preemption over OOM
• OpenTelemetry GenAI semconv tracing wired end-to-end
• Redaction layer between inference pods and long-term log storage

Troubleshooting checklist for latency spikes:

1. Check gpu_cache_usage_perc and num_requests_waiting first.
2. Check preemption/eviction rate in logs.
3. Correlate with request-shape anomalies (long-context bursts).
4. Check autoscaler reaction time (Karpenter provisioning events).
5. Fix the triggering metric for autoscaling, not just add capacity.

Next session candidates (not yet covered): Cilium/eBPF service mesh migration, ArgoCD multi-cluster GitOps at scale, Azure Workload Identity deep dive, SPIFFE/SPIRE zero-trust workload identity, GPU Operator + MIG partitioning internals.