AI Model Serving at Scale: KServe, Multi-Model Serving & GPU-Aware Autoscaling on Kubernetes

Daily DevOps Mentor — 2026-09-01

1. Topic of the Day

Every AI infrastructure team eventually hits the same wall: training gets a model to "works in a notebook," but serving it to production traffic — with SLOs, multi-tenant isolation, cost accountability, and zero-downtime rollouts — is a distinct engineering discipline. KServe (the CNCF project that absorbed KFServing) exists to give Kubernetes a standard, cloud-native abstraction for that discipline: the InferenceService CRD, built on Knative Serving's request-based autoscaling and networking, with pluggable ServingRuntimes for Triton, vLLM, TorchServe, SKLearn, XGBoost, and custom containers.

The reason this matters at enterprise scale isn't "how do I run one model" — that's a Deployment and a Service. It's "how do I run 200 models, owned by 30 teams, on a shared GPU fleet, where model A gets 10,000 req/s and model B gets 3 req/s once an hour, without either statically over-provisioning GPUs for the long tail or starving the hot path." That's a bin-packing, autoscaling, and multi-tenancy problem simultaneously, and it's why the AI model serving platform has become its own layer in the stack — distinct from the inference engine (vLLM, Triton) and distinct from raw GPU scheduling (device plugins, MIG, DRA) — sitting between the two and orchestrating both.

Organizations serving LLMs and classical ML models at fleet scale (Bloomberg, Salesforce, Zillow, and most large AI-native platforms publicly documented as KServe adopters) converged on this pattern because the alternative — every team hand-rolling their own Deployment + HPA + Ingress per model — produces exactly the mess this session's business problem describes: dozens of bespoke serving stacks, no consistent canary mechanism, no shared GPU bin-packing, and a FinOps team that cannot answer "which model costs us money."

Today's session covers why naive per-model deployment collapses past a few dozen models, how to design a multi-model serving platform with GPU-aware autoscaling that doesn't waste GPU-hours on idle capacity, how KServe's control plane actually schedules and scales a request end-to-end, and how this compares to Seldon Core, BentoML, and rolling your own on top of Ray Serve.

2. Real Business Problem

Symptom: A platform team supports 180 model endpoints across 12 product teams — a mix of fine-tuned LLMs (Llama/Qwen variants), classical XGBoost fraud-scoring models, and CV models for image moderation. Three things surface in the same quarter:

• GPU utilization dashboards show a fleet-wide average of 22% GPU utilization, yet the team is blocked on GPU quota requests to finance — because every model was deployed as a static Deployment with a dedicated GPU request, sized for each team's peak traffic, and peaks don't overlap in time. The fraud model peaks at market open; the moderation model peaks during evening user activity; the internal chatbot peaks during business hours. Nobody's peak is anybody else's peak, but nobody shares capacity.
• A new model version rollout for the fraud-scoring model causes a nine-minute total outage during cutover, because the team's deploy script does kubectl delete on the old Deployment before the new one reports ready — there is no canary mechanism, no traffic-shifting primitive, and no automatic rollback when the new model's error rate spikes.
• The on-call platform engineer gets paged for cold-start latency SLO violations — one team's low-traffic model scales to zero between requests to save GPU cost, but the 45-second cold start (pulling a 14GB model image, loading weights onto the GPU) blows through the caller's 5-second timeout on the first request after any idle period.

The business ask: "let teams self-serve model deployment without hand-rolling infrastructure, share the GPU fleet efficiently across uncorrelated traffic patterns, support safe canary rollouts by default, and make cold starts either fast enough to not matter or avoidable for latency-sensitive models." That is precisely the problem KServe plus GPU-aware autoscaling is designed to solve — and precisely where a from-scratch Deployment-per-model approach runs out of road.

3. Production Architecture

Request path. Clients call an AI gateway (Envoy or Istio ingress with model-aware routing, API-key auth, and per-tenant rate limiting) rather than a model's Service directly — this is the layer that does token metering for cost allocation and enforces the "no team calls another team's GPU budget" boundary. The gateway routes to the KServe controller's networking layer, which for single-model serving is a Knative Route/Revision pair, and for high-density multi-model serving is ModelMesh, a separate high-throughput multi-model server that packs many small models into shared runtime pods with LRU-based load/unload.

Control plane. The InferenceService CRD is the unit of deployment: it declares a predictor (model URI, framework, resource requests, min/max replicas, scale-to-zero policy) and optional transformer/explainer components for pre/post-processing. The KServe controller reconciles this into a Knative Service (for single-model, scale-per-revision serving) or a ModelMesh ServingRuntime deployment (for multi-model density). Knative Serving owns request-based autoscaling (concurrency or RPS target) and traffic splitting between revisions — this is what gives KServe its canary/blue-green primitives for free, inherited from Knative rather than reimplemented.

Data plane. Model runtime pods run Triton Inference Server (multi-framework: ONNX, TensorRT, PyTorch, dynamic batching, model ensembles), vLLM (LLM-specific: PagedAttention, continuous batching, tensor parallelism), or TorchServe/custom runtimes via the ServingRuntime CRD for anything else. These pods run on a GPU node pool managed by the NVIDIA GPU Operator (device plugin, DCGM exporter for GPU telemetry, MIG or time-slicing configuration) and provisioned/consolidated by Karpenter, which right-sizes GPU instance types and bin-packs nodes as workloads scale.

Autoscaling signal chain. This is the architectural crux: KEDA scales pods based on external metrics — vLLM's exposed queue depth and time-per-token, Triton's queue-to-compute ratio, or a Prometheus query blending both — rather than raw CPU/memory, which are meaningless signals for GPU-bound inference. Knative provides request-concurrency-based scaling for the HTTP-facing revision; KEDA layers GPU-aware, workload-specific signals on top for the runtime pods themselves. Karpenter then reacts to pending pods from either scaler by provisioning right-sized GPU nodes (or consolidating them away once utilization drops), closing the loop from "request queue growing" to "new GPU capacity online" without a human in between.

Security boundaries. Each tenant's InferenceService objects live in a dedicated namespace with ResourceQuota capping GPU request count, NetworkPolicy restricting east-west traffic to the gateway and shared observability endpoints only, and RBAC scoping who can create/modify InferenceService and ServingRuntime objects — a tenant can deploy models but not touch cluster-scoped GPU Operator or Karpenter configuration. Model artifacts are pulled from a registry (MLflow backed by S3/GCS) with signed URLs scoped per namespace, so a compromised model-serving pod cannot enumerate other tenants' model weights.

HA/DR and multi-region. The Model Registry replicates cross-region (S3 CRR or equivalent), and InferenceService manifests live in Git, deployed via ArgoCD — the actual DR mechanism is "redeploy the manifests in the failover region and let KServe pull weights from the replicated registry," not stateful failover of running pods. The AI gateway does latency- and capacity-aware routing across regional GPU pools, so failover is a gateway-level routing decision, not something the client needs to know about. This mirrors the design principle from the GPU-scheduling and ArgoCD sessions: push complexity into a control plane the platform team owns, keep the failure-recovery mechanism declarative and Git-driven rather than imperative and pod-state-dependent.

Why this shape, and how it evolves. Below roughly 10-20 models on a small team, a hand-rolled Deployment + HPA per model is genuinely simpler and KServe's control-plane overhead (Knative's extra networking hop, the CRD learning curve) isn't worth it. The inflection point is the same one seen in every session this rotation: once the coordination cost of N teams each reinventing canary rollout, GPU bin-packing, and cold-start handling exceeds the cost of operating a shared control plane, standardize. At 10x scale (thousands of models, hundreds of GPUs), the next axis that has to change is moving from Knative's per-revision scaling toward ModelMesh's shared-runtime multi-model density for the long tail of low-traffic models, while keeping dedicated Knative-scaled InferenceServices only for the handful of genuinely hot, latency-sensitive models — a tiered architecture rather than one-size-fits-all.

4. Solution Design

Design decision: KServe vs. Seldon Core vs. BentoML vs. Ray Serve vs. hand-rolled. KServe is the right default when the platform is already Knative/Kubernetes-native and needs a CNCF-governed, framework-agnostic standard with strong multi-framework runtime support (Triton) and first-class LLM support (vLLM ServingRuntime, now upstream). Seldon Core is a reasonable alternative with a richer built-in explainability/drift-detection pipeline (useful for regulated industries needing model governance out of the box) but a smaller OSS community post-Seldon's 2024 licensing change toward Seldon Core v2's business-source license for some components — a real consideration for teams that need permissive OSS licensing guarantees. BentoML optimizes for developer experience packaging a model as a deployable unit from Python with less Kubernetes-native ceremony, which is attractive for smaller ML platform teams but historically weaker on the shared-fleet GPU bin-packing story KServe+KEDA+Karpenter provides. Ray Serve is the strongest choice when the workload is fundamentally a Ray application already (distributed training-to-serving pipelines, complex multi-model DAGs with Python-native composition) — its serving layer is excellent, but adopting it purely for serving when the rest of the platform is plain Kubernetes adds an entire second orchestration paradigm to operate.

Alternative approaches considered and rejected.

• Deployment + HPA per model, no KServe. Rejected past the scale in Section 2 — HPA's CPU/memory-based scaling is the wrong signal for GPU inference, there's no standard canary primitive, and every team reinvents cold-start and model-loading logic independently, which is the exact duplicated-effort problem this session's business case describes.
• Cluster Autoscaler instead of Karpenter for GPU nodes. Works, but Cluster Autoscaler's ASG/node-group-based provisioning is materially slower to react and worse at bin-packing heterogeneous GPU instance types (A100 vs. H100 vs. L40S) than Karpenter's just-in-time, instance-type-flexible provisioning — for GPU nodes specifically, where a single node costs $30-100+/hour, provisioning speed and packing efficiency have outsized cost impact.
• One giant shared GPU pod per model family, no autoscaling at all. Simpler operationally, but reintroduces the static-peak-sizing waste from Section 2 and removes the platform's ability to reclaim capacity from an idle model for a busy one — defeats the entire purpose of a shared fleet.

Scalability, cost, security, performance implications. GPU-aware autoscaling directly targets the 22%-utilization problem: KEDA scaling on actual inference queue depth rather than static replica counts, combined with Karpenter consolidation, routinely pushes fleet-wide GPU utilization from the 20-30% range into 55-70% for platforms with sufficiently uncorrelated tenant traffic — the FinOps win is proportional to how uncorrelated tenant peaks are, which is worth measuring before promising savings to leadership. Security-wise, multi-tenancy on a shared GPU fleet needs explicit ResourceQuota and node-pool segmentation for anything regulated (PCI/HIPAA workloads should get dedicated node pools, not just namespace isolation, since GPU-level tenant isolation guarantees are weaker than CPU cgroup isolation). Performance-wise, scale-to-zero saves cost for the long tail but must be paired with either a warm-pool minimum for latency-sensitive models or aggressive model-weight caching (Section 5) — the naive version of this architecture trades cost for tail latency, and getting that trade-off wrong in either direction is the single most common mistake teams make adopting this pattern.

5. Deep Technical Walkthrough

Internal working — from InferenceService to running pod. Applying an InferenceService triggers the KServe controller to reconcile a Knative Service, which creates a Configuration (desired state) and Route (traffic split). Knative's Configuration controller creates a new Revision for each unique spec — this is the mechanism canary rollouts use: two Revisions coexist, and the Route splits traffic between them by percentage. For ModelMesh-managed multi-model serving, instead of a per-model Knative Revision, the controller registers the model with a shared ServingRuntime pod pool, which loads the model into an already-running runtime process on demand (avoiding a full pod cold start for models that fit the density profile).

Control plane interactions — the autoscaling decision chain. Knative's autoscaler component watches request concurrency (or RPS) per Revision via a sidecar-reported metric (queue-proxy), and scales the Revision's pod count within minReplicas/maxReplicas bounds — this handles the HTTP-facing scaling decision. Independently, KEDA's ScaledObject polls an external metrics source (a Prometheus query against vLLM's /metrics endpoint for queue depth, or Triton's queue-to-inference-time ratio) and scales the underlying Deployment KServe manages — this is the GPU-workload-aware layer that Knative's generic concurrency model doesn't capture well for variable-cost-per-request workloads like LLM generation, where "concurrent requests" is a much weaker cost proxy than "tokens currently queued for generation."

Data plane interactions — from HTTP request to GPU compute. A request lands at the Knative queue-proxy sidecar, which enforces the configured concurrency limit and forwards to the user container (Triton or vLLM). Triton's scheduler batches compatible requests (same model, compatible shapes) into a single GPU forward pass per its configured dynamic-batching window; vLLM's continuous batching does this at the token level, admitting new requests into an in-flight batch between generation steps rather than waiting for a batch window to close — the reason vLLM specifically is the default runtime choice for LLM InferenceServices over generic Triton-hosted PyTorch models.

Failure scenarios and recovery. If a model-loading pod OOMs during weight loading (common with underestimated memory-per-parameter budgeting for large LLMs), Knative's readiness probe never passes, the Revision never receives traffic, and the old Revision keeps serving 100% of traffic — a fail-safe default, since Knative won't cut over until the new Revision is ready. If a GPU node fails mid-serving, pods on it are rescheduled, but any in-flight generation requests on that node are lost (KServe/Knative doesn't checkpoint mid-inference state) — clients need request-level retry logic, and idempotency matters more for generation endpoints than most REST APIs because a retried request may re-generate tokens a client already partially received via streaming.

Performance bottlenecks and scaling behavior. The most common bottleneck at scale isn't the model runtime itself — it's cold start dominated by model artifact pull and weight-loading time, not container start. A 70B-parameter model in FP16 is ~140GB; even from a fast internal registry, pulling and loading that onto GPU memory can take minutes, which is incompatible with scale-to-zero for latency-sensitive endpoints. The standard mitigation is a model cache (a warm PVC or node-local NVMe cache pre-populated with frequently used model weights) so scale-from-zero reloads from local disk rather than a remote registry, cutting cold start from minutes to tens of seconds — still not free, which is why truly latency-critical InferenceServices should set minReplicas: 1 and eat the idle-GPU cost rather than scale to zero at all.

6. Production Troubleshooting

Walking a representative incident the way a senior platform engineer would:

Symptom. The fraud-scoring InferenceService starts returning p99 latency of 8+ seconds (SLO: 500ms) during a traffic spike, despite maxReplicas being set high enough on paper to absorb the load.

Step 1 — confirm whether the bottleneck is scaling lag or GPU saturation.

kubectl get inferenceservice fraud-scorer -n fraud -o yaml | yq '.status'
kubectl get scaledobject -n fraud fraud-scorer-keda -o yaml
kubectl get hpa -n fraud  # KEDA creates a backing HPA object
kubectl top pods -n fraud -l serving.kserve.io/inferenceservice=fraud-scorer

If replica count is still climbing toward maxReplicas several minutes into the spike, the bottleneck is scaling lag, not a hard ceiling — check the KEDA ScaledObject's polling interval (pollingInterval, default 30s) and cooldown settings, which for GPU workloads should usually be tightened (5-10s polling) given how expensive a slow reaction is in GPU-minutes and SLA risk.

Step 2 — check whether new pods are actually getting GPU capacity, or stuck pending on node provisioning.

kubectl get pods -n fraud -l serving.kserve.io/inferenceservice=fraud-scorer -o wide
kubectl get events -n fraud --field-selector reason=FailedScheduling
kubectl logs -n karpenter deploy/karpenter | grep fraud-scorer

A cluster of Pending pods with FailedScheduling events citing insufficient nvidia.com/gpu is the classic case: Karpenter hasn't provisioned new GPU nodes fast enough, either because the configured NodePool doesn't include an available instance type in the current AZ, or because GPU instance provisioning itself (which can take 3-8 minutes depending on cloud and instance type) is simply slower than the traffic spike's growth rate — this is a capacity-planning problem, not a bug, and the fix is a small warm buffer of pre-provisioned GPU capacity for SLO-critical models rather than pure reactive autoscaling.

Step 3 — isolate GPU-level saturation from application-level queuing.

kubectl exec -n fraud <triton-pod> -- curl -s localhost:8002/metrics | grep -E "nv_inference_queue_duration|nv_gpu_utilization"

High nv_inference_queue_duration with GPU utilization already near 100% means the model is compute-bound and needs either more replicas (if scaling isn't the bottleneck per Step 1) or a cheaper/quantized model variant; high queue duration with low GPU utilization points back to a scheduling or batching misconfiguration — check Triton's dynamic-batching window (max_queue_delay_microseconds) isn't set so conservatively that requests sit idle waiting for a batch that never fills at off-peak concurrency.

Step 4 — root cause and remediation. In the composite incident: Karpenter's NodePool was scoped to a single GPU instance type with limited AZ capacity, so during the spike, Karpenter's provisioning requests queued behind AZ-level capacity exhaustion at the cloud provider. Remediation: broaden the NodePool's instance type flexibility (multiple GPU SKUs, multiple AZs) so Karpenter can fall back automatically, add a small minReplicas warm buffer sized to absorb typical spike-onset latency, and tighten the KEDA polling interval — each change independently reduces cold-provisioning exposure, and together they moved p99 back under SLO even during comparable spikes afterward.

7. Hands-on Lab

Goal: deploy a multi-model InferenceService setup on a local kind cluster with GPU-metric-based KEDA autoscaling simulated via a synthetic metrics endpoint (no real GPU required to validate the control-plane wiring).

# 1. Create cluster and install Knative Serving + KServe (Raw Deployment mode works without Istio for a lab)
kind create cluster --name kserve-lab
kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.16.0/serving-crds.yaml
kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.16.0/serving-core.yaml
kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.14.0/kserve.yaml

# 2. Deploy a sample sklearn InferenceService
kubectl create ns models
cat <<EOF | kubectl apply -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris
  namespace: models
spec:
  predictor:
    minReplicas: 0
    maxReplicas: 5
    sklearn:
      storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"
EOF

# 3. Confirm it scales to zero, then wake on request
kubectl get inferenceservice -n models -w   # watch READY and URL columns
SERVICE_URL=$(kubectl get inferenceservice sklearn-iris -n models -o jsonpath='{.status.url}')
curl -v -H "Content-Type: application/json" "$SERVICE_URL/v1/models/sklearn-iris:predict" \
  -d '{"instances": [[6.8, 2.8, 4.8, 1.4]]}'

# 4. Install KEDA and wire a queue-depth-based ScaledObject for a Triton/vLLM runtime
helm repo add kedacore https://kedacore.github.io/charts && helm install keda kedacore/keda -n keda --create-namespace
cat <<EOF | kubectl apply -f -
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-queue-scaler
  namespace: models
spec:
  scaleTargetRef:
    name: vllm-llama-predictor
  minReplicaCount: 1
  maxReplicaCount: 8
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.observability:9090
      query: vllm_num_requests_waiting
      threshold: "5"
EOF

# 5. Validate canary traffic split between two Revisions
kubectl get revisions -n models
# Patch InferenceService's predictor with a new storageUri, then split traffic 90/10 via the Route/Service spec

# Cleanup
kubectl delete ns models keda
kind delete cluster --name kserve-lab

8. Production Case Study

Bloomberg has published on running KServe at scale for internal ML platforms, using it as the standard abstraction across hundreds of models spanning classical ML and deep learning, explicitly citing the multi-framework ServingRuntime model as the reason teams could avoid per-framework bespoke serving stacks. Salesforce's Einstein platform similarly standardized on KServe for its multi-tenant model-serving layer, leaning on Knative's traffic-splitting for safe rollout across a large number of internally-owned models. At the hyperscaler end, Google's Vertex AI and AWS's approach to SageMaker multi-model endpoints solve the same "shared GPU fleet, many low-traffic models" problem KServe+ModelMesh targets, but as a fully managed service rather than a self-operated control plane — the trade-off enterprises weigh here is exactly the build-vs-buy question that recurs across this entire curriculum: a managed platform removes the Knative/KEDA/Karpenter operational burden but caps how deeply the platform team can customize scheduling, batching, and cost-allocation logic for workload patterns specific to their business. Uber's Michelangelo platform, predating KServe's maturity, took the build-your-own route early and has since publicly discussed converging pieces of its serving layer toward more standardized OSS primitives as the ecosystem matured — a pattern worth noting: platforms built before a category's OSS standard existed often carry years of migration debt once one does.

9. Architecture Review

Strengths. The separation of Knative's request-based scaling from KEDA's GPU-workload-aware scaling from Karpenter's node provisioning is a clean layering — each component owns a distinct decision (pod count for HTTP concurrency, pod count for GPU-specific load, node count for capacity) and can be reasoned about and debugged independently, which is precisely what Section 6's troubleshooting walkthrough depends on. Standardizing on InferenceService also gives every model a uniform canary/rollback mechanism for free, closing the nine-minute-outage gap from Section 2 without every team reimplementing it.

Weaknesses. The control-plane surface area is genuinely large — Knative, KServe, KEDA, Karpenter, GPU Operator, and ModelMesh (if used) is six separately-versioned systems that must be compatibility-tested together, and a version skew bug between any two (a recurring real-world pain point with Knative/Istio compatibility windows specifically) can take down serving fleet-wide. Cold-start behavior remains a fundamentally hard problem this architecture mitigates but doesn't solve — any model too large to cache aggressively will always have a real cold-start floor set by weight size and network/disk bandwidth, no amount of orchestration cleverness removes that physics.

What breaks first at 10x scale. The Knative Route/networking layer's per-Revision overhead (an extra Envoy/Istio hop per request) becomes a measurable tax at very high fleet-wide QPS, and the etcd/API-server load from thousands of InferenceService/Revision/ScaledObject objects being reconciled becomes a control-plane scaling concern in its own right — this is the same API-server-latency failure mode that shows up in any sufficiently large Kubernetes cluster, just triggered here by AI-platform-specific object proliferation rather than raw pod count.

Redesign for 100 million users / massive scale. At that scale, the architecture shifts from "one shared cluster with namespace multi-tenancy" to multiple regional/purpose-segmented clusters (a hot-path cluster for the highest-QPS, lowest-latency models running dedicated minReplicas capacity with no scale-to-zero at all, and a long-tail cluster leaning heavily on ModelMesh density for the thousands of rarely-called models), with the AI gateway doing cross-cluster routing based on model registration rather than a single cluster's Knative Route table. Model caching moves from a nice-to-have to mandatory infrastructure — a dedicated, tiered model-weight cache (node-local NVMe, then regional object storage) becomes as central to the architecture as the CDN is to a web platform.

10. Best Practices

Reliability. Set minReplicas: 1 for any InferenceService with a real latency SLO — scale-to-zero is a cost feature, not a reliability feature, and mixing them for SLO-bound endpoints is a common self-inflicted incident. Scalability. Tier models by traffic pattern explicitly (hot/dedicated vs. long-tail/ModelMesh-packed) rather than treating every model identically; a one-size-fits-all autoscaling config wastes either capacity or engineering effort. Observability. Instrument GPU utilization, queue depth, and time-per-token (for LLMs) as first-class SLIs, not just HTTP latency — HTTP p99 alone hides GPU-level saturation until it's already an incident. Security. Namespace-scope InferenceService RBAC per tenant, and use dedicated node pools (not just namespace isolation) for regulated workloads sharing a GPU fleet. Cost Optimization. Track GPU-hours per model, not per namespace — namespace-level attribution hides the actual cost driver when one model in a shared namespace dominates GPU spend. Operational Excellence. Version-pin the entire stack (Knative + KServe + KEDA + Karpenter + GPU Operator) together and test upgrades in a non-production cluster running representative model traffic before touching the fleet — this stack's compatibility matrix is a real operational cost, not a one-time setup task.

11. Common Production Mistakes

Treating GPU inference autoscaling like CPU-workload HPA — scaling on CPU/memory utilization for a GPU-bound workload measures the wrong resource entirely and produces scaling decisions disconnected from actual load. Enabling scale-to-zero fleet-wide without measuring real cold-start time per model first — a blanket policy applied without per-model cold-start data guarantees some team's SLO gets silently violated the first time their model goes idle. Sizing GPU node pools for a single, static instance type — this removes Karpenter's ability to fall back to alternate SKUs during capacity crunches, turning a transient cloud-provider capacity shortage into a self-inflicted outage. Skipping canary rollout for "low-risk" model updates — model behavior regressions (accuracy drift, not just crashes) are often invisible to standard health checks and only show up in business metrics, which is exactly the class of failure canary analysis with real traffic is designed to catch before 100% rollout.

12. Interview Preparation

Q: Why is HPA's default CPU-based scaling inappropriate for GPU inference workloads, and what should replace it? CPU utilization on a GPU inference pod largely reflects request marshaling/preprocessing overhead, not the actual bottleneck resource (GPU compute/memory). The correct signal is workload-specific: request queue depth, GPU utilization via DCGM, or engine-specific metrics like vLLM's num_requests_waiting — surfaced to Kubernetes via KEDA's external-metrics adapter rather than the standard resource-metrics-based HPA.

Q: Walk through what happens end-to-end when an InferenceService scales from zero. A request arrives at the Knative activator (which buffers requests when no Revision pods are ready), triggering the autoscaler to scale the target Deployment from 0 to 1. The new pod pulls the runtime image (cached if previously run on that node) and loads model weights (from local cache if warm, otherwise from the model registry) before passing readiness checks; the activator then forwards the buffered request(s) once a pod is ready. Total latency is container start + image pull (if cold) + weight load — the dominant term for large models is almost always weight load, not container start.

Q: How would you design GPU fleet sharing across teams with wildly different latency SLOs? Tier by SLO: latency-critical models get dedicated minReplicas-backed capacity on a node pool sized for their peak, never scaling to zero; latency-tolerant, low-traffic models get packed via ModelMesh or scale-to-zero Knative Revisions on a shared, elastic node pool that Karpenter consolidates aggressively. Enforce the split with ResourceQuota and node affinity/taints so the elastic tier's autoscaling churn can never starve the dedicated tier's capacity.

Q: What's the failure mode of scale-to-zero for a 70B-parameter LLM endpoint, and how do you mitigate it without disabling scale-to-zero fleet-wide? The failure mode is a multi-minute cold start dominated by weight loading, incompatible with most caller timeouts. Mitigate with a node-local model-weight cache so scale-from-zero reloads from local NVMe rather than the registry, and/or set a small non-zero minReplicas specifically for that model while leaving genuinely low-traffic models on true scale-to-zero — the mitigation should be per-model, not a global policy change.

13. Latest Industry Updates

KServe continues consolidating as the CNCF-governed standard for model serving, with the ModelMesh multi-model serving path and native vLLM ServingRuntime support now both stable upstream — a direct response to the LLM-serving-specific batching and memory-management needs classical Triton runtimes weren't originally designed for. KEDA has expanded its GPU/AI-relevant scaler ecosystem, with Prometheus-based scalers remaining the most common integration path for vLLM/Triton queue-depth metrics, reflecting the broader trend of AI workloads needing custom, workload-aware autoscaling signals rather than generic resource metrics. Karpenter's GPU-instance-type flexibility and consolidation logic remain an active area of improvement as cloud providers continue fragmenting the GPU SKU landscape (H100, H200, GB200, L40S, and successive generations) — instance-type flexibility in NodePool configuration is increasingly treated as a capacity-risk-mitigation practice, not just a cost optimization, given how frequently single-SKU capacity crunches cause exactly the incident walked through in Section 6. Across the CNCF landscape broadly, "AI model serving" has solidified as its own recognized category distinct from general workload serving, reflected in the growing number of projects (KServe, Ray Serve, BentoML, Seldon) explicitly scoped to this problem rather than treating it as a generic Kubernetes deployment pattern.

14. Summary & Cheat Sheet

Key concepts: InferenceService as the standard model deployment unit; Knative Serving for request-based autoscaling and canary traffic-splitting; ModelMesh for high-density multi-model packing; KEDA for GPU/queue-depth-aware scaling beyond what CPU-based HPA can express; Karpenter for GPU-aware node provisioning and consolidation; NVIDIA GPU Operator for device plugin/telemetry/MIG configuration underneath it all.

Architecture pattern: Client → AI Gateway (auth, routing, metering) → KServe/Knative control plane (canary, autoscaling decision) → Triton/vLLM/custom runtime pods → GPU node pool (Karpenter-provisioned, GPU-Operator-managed), with Prometheus/DCGM/OpenTelemetry observability and Kyverno/OPA policy enforcement wrapping the whole namespace-isolated tenant boundary.

Commands to remember:

kubectl get inferenceservice -A                       # fleet-wide model status
kubectl get scaledobject -n <ns>                       # KEDA scaling config/status
kubectl logs -n karpenter deploy/karpenter              # node provisioning decisions
kubectl exec <triton-pod> -- curl localhost:8002/metrics  # per-model GPU/queue metrics

Design patterns: tier models by SLO (dedicated warm capacity vs. elastic shared pool); scale on workload-specific signals (queue depth, tokens-in-flight), never raw CPU/memory, for GPU inference; treat model-weight caching as core infrastructure, not an optimization, once any served model is large enough that cold start exceeds caller timeouts.

Troubleshooting checklist: (1) is the bottleneck scaling lag or a hard capacity ceiling — check replica count trend against maxReplicas; (2) are new pods actually scheduling — check for FailedScheduling events and Karpenter logs; (3) is the GPU actually saturated or is this a batching/queuing misconfiguration — check DCGM utilization against queue-duration metrics; (4) confirm root cause against node-pool instance-type flexibility and KEDA polling/cooldown settings before declaring the incident resolved.