Daily DevOps Mentor — 2026-08-25

Topic: Karpenter — Next-Generation Kubernetes Cluster Autoscaling (AWS)

1. Topic of the Day

Karpenter is a Kubernetes-native, high-performance node lifecycle manager originally built by AWS, now a CNCF project with providers for AWS, Azure, and (community) GCP/bare-metal. It replaces the Cluster Autoscaler (CAS) model of "scale a fixed-shape ASG/node pool" with direct, workload-aware node provisioning: it reads unschedulable pod specs and calls the cloud API directly to launch the exact instance type, size, and AZ needed — no pre-baked node groups required.

Why it exists: Cluster Autoscaler scales node groups defined ahead of time (fixed instance types per ASG). This creates binpacking waste, slow scale-up (CAS evaluates node groups serially, ASG launch templates, warm pool complexity), and poor fit for heterogeneous workloads — especially GPU fleets where you have 6+ instance families (g5, g6, p4d, p5, trn1) and workloads with wildly different resource shapes. In enterprise production, Karpenter is used for: general-purpose compute consolidation (cost), spot-heavy batch/CI fleets, and increasingly as the default GPU capacity manager for AI inference/training clusters, because it can express diversified instance-type fallback (if p5 is unavailable, fall back to p4d, then on-demand) in a single NodePool.

Production usage: Amazon (EKS default recommendation since 2023), Airbnb, Grafana Labs, Prime Video (AWS), and many AI infra shops running vLLM/Triton on EKS use Karpenter for GPU node scaling because CAS's node-group model can't do capacity-aware fallback fast enough during GPU shortages.

2. Real Business Problem

Scenario: Your platform team runs an EKS cluster serving both a web API tier (general purpose, c6i/m6i) and an LLM inference tier (vLLM on g5.12xlarge). At 2 AM UTC, a marketing campaign triggers a 4x traffic spike. Cluster Autoscaler is configured across three ASGs (web, batch, gpu-inference). Symptoms:

Web tier pods stay Pending for 6–9 minutes because CAS scale-up needs to identify the right ASG, respect max-size, and wait for the ASG's own scaling cooldown plus EC2 launch + kubelet register + node-ready (~90–150s), but ASG launch templates use a single instance type that is capacity-constrained in the AZ.
GPU inference pods stay Pending for 20+ minutes because the g5.12xlarge ASG hits InsufficientInstanceCapacity in us-east-1a, and CAS has no fallback — it just keeps retrying the same ASG.
On-call is paged for SLA breach (P99 inference latency >30s due to queueing on too few replicas).

This is the canonical "cluster autoscaler capacity fallback" failure mode that Karpenter is specifically designed to solve via NodePool requirements with multiple instance types/AZs and native spot-to-on-demand fallback.

3. Production Architecture

                         ┌─────────────────────────────────────────┐
                         │              EKS Control Plane            │
                         │  API Server │ etcd │ Scheduler │ CCM       │
                         └───────────────────┬───────────────────────┘
                                              │ watch/list (Pending pods,
                                              │ NodePool/NodeClass CRDs)
                                              ▼
                         ┌───────────────────────────────────────────┐
                         │      karpenter controller (Deployment)     │
                         │  - Provisioning controller                 │
                         │  - Disruption controller (consolidation)   │
                         │  - Interruption handler (spot 2-min notice)│
                         └───────────────────┬───────────────────────┘
                                              │ EC2 Fleet / CreateFleet API
                                              │ (instance-type diversification)
                                              ▼
              ┌─────────────────────────────────────────────────────────┐
              │                      AWS EC2 Capacity                     │
              │  On-Demand pools │ Spot pools │ Capacity Blocks (GPU)     │
              │  AZ-a  AZ-b  AZ-c   x  m6i/c6i/g5/g6/p5 families          │
              └───────────────────┬─────────────────────────────────────┘
                                   │ node bootstraps (bottlerocket/AL2023)
                                   ▼
              ┌─────────────────────────────────────────────────────────┐
              │        Worker Nodes (join cluster via kubelet)            │
              │  Labels: karpenter.sh/nodepool, node.kubernetes.io/      │
              │  instance-type, topology.kubernetes.io/zone               │
              │  Taints: dedicated=gpu:NoSchedule (per NodePool)          │
              └─────────────────────────────────────────────────────────┘

Security boundaries:
- karpenter controller IRSA role: scoped to ec2:RunInstances/CreateFleet/
  TerminateInstances/DescribeInstances, iam:PassRole (only the node role),
  restricted via aws:RequestTag conditions to prevent privilege escalation.
- NodeClass defines instanceProfile — nodes get minimal IAM (SSM, CNI, ECR pull).
- No SSH; access via SSM Session Manager only.
- Node network: private subnets only, egress via NAT/VPC endpoints (S3, ECR,
  STS) to avoid NAT cost blowup at GPU-node scale.

HA / DR:
- karpenter controller runs 2 replicas w/ leader election (client-go lease).
- Multi-AZ NodePool requirements (topology.kubernetes.io/zone In [a,b,c]).
- PodDisruptionBudgets protect inference replicas during consolidation.
- Cross-region: Karpenter is single-region/single-cluster by design — DR is
  handled at a higher layer (multi-cluster + Argo Rollouts/Route53 failover),
  not by Karpenter itself.

Why this shape: Karpenter deliberately has no concept of "node groups" — it decouples capacity type selection from capacity reservation, letting the scheduler's own bin-packing simulation run inside the controller (it re-implements a fast subset of kube-scheduler predicates) so it can decide node shape per batch of pending pods in one shot, rather than reasoning about pre-defined ASGs.

Trade-offs: You give up the "known fleet" simplicity of ASGs (harder to reason about exact capacity at a glance; mitigated with karpenter.sh/nodepool labels + Kubecost/CCM dashboards). You take on a new failure domain: the karpenter controller itself becomes a single point of provisioning (mitigated by 2 replicas + leader election + PDB on itself).

At scale (1000+ nodes, GPU-heavy): Move to NodePool per workload class (web, batch-spot, gpu-inference, gpu-training) with disruption budgets tuned per class (disruption.budgets to cap consolidation churn during business hours), and enable Karpenter's native spot interruption handling (SQS + EventBridge rule for GPU spot, since GPU capacity is spot-fragile).

4. Solution Design

Design decisions:

One NodePool per workload shape/priority tier (not one giant NodePool) — enables independent disruption budgets, weight-based selection priority, and per-tier consolidation policy (WhenEmptyOrUnderutilized for web, WhenEmpty for GPU to avoid killing warm model-loaded nodes).
NodeClass (EC2NodeClass) pins AMI family (Bottlerocket for security/immutability on web tier; AL2023 + NVIDIA driver on GPU tier via user-data or GPU AMI), subnet/SG selectors by tag, not hardcoded IDs — survives infra-as-code re-applies.
Instance type diversification: node.kubernetes.io/instance-type In [g5.12xlarge, g5.24xlarge, g6.12xlarge] — lets Karpenter pick whichever has capacity, critical during GPU shortages.
karpenter.sh/capacity-type In [spot, on-demand] with weighted NodePools: spot-first NodePool with higher weight, on-demand fallback NodePool with lower weight and only used if spot fully exhausted (CAS could never express this).

Alternatives considered:

Cluster Autoscaler + Managed Node Groups: simpler mental model, mature, but slow scale-up, no capacity fallback, poor GPU shortage resilience. Choose CAS when the fleet is small/homogeneous and team unfamiliar with Karpenter's CRDs.
Static capacity (fixed node pools, no autoscaling): predictable cost/perf, but wastes money at low utilization and can't absorb spikes — used sometimes for latency-critical GPU inference where cold-start (~90s AMI boot + model load) is unacceptable; combine with Karpenter for the overflow tier only.
Fargate (serverless pods): no node management at all, but doesn't support GPU workloads or DaemonSets (no CNI plugins like Cilium, no NVIDIA device plugin) — unsuitable for GPU inference.

Cost implications: Karpenter's consolidation feature actively repacks pods onto fewer/cheaper nodes (bin-packing improvement typically 20–40% cost reduction over static node groups per AWS case studies) but adds churn risk (pods rescheduled mid-flight) — must gate with PDBs and do-not-disrupt annotations on stateful pods.

Security implications: Broader RunInstances permission blast radius than CAS's ASG-scoped UpdateAutoScalingGroup. Mitigate with IAM condition keys restricting instance types/AMIs Karpenter can launch, and enforce NodeClass immutability via GitOps + OPA/Kyverno admission policy blocking manual edits.

Performance: Scale-up latency drops from CAS's ~3–5 min typical to ~30–60s (EC2 Fleet API call + boot), because Karpenter skips the ASG launch-template/cooldown layer entirely.

5. Deep Technical Walkthrough

Control flow:

Pod created → scheduler tries to place it → fails all nodes (insufficient resources / anti-affinity / taint mismatch) → pod becomes Pending, scheduler emits FailedScheduling event.
Karpenter's provisioning controller watches for pending pods (via informer, not polling), batches them (debounced ~1s window) to avoid thrashing on bursts.
For each batch, Karpenter runs an internal scheduling simulation: it evaluates all NodePool CRDs the pods' node-affinity/taints/tolerations allow, computes the minimal set of nodes (with real instance-type sizing) that would let the scheduler place all pods, applying real kube-scheduler predicate logic (topology spread, pod affinity/anti-affinity, resource requests) — this is why Karpenter can right-size instead of guessing.
Karpenter calls EC2 CreateFleet with an instance-type list ranked by price (or lowest-latency/capacity-optimized for spot), diversified across the NodePool's allowed AZs/types.
EC2 returns launched instances; Karpenter creates a Node object (via NodeClaim CRD — the internal representation before an actual Node registers) and taints it so it doesn't get raced by other pending pods until kubelet joins.
kubelet on the new instance registers with API server; node becomes Ready; scheduler places the pending pods.

Disruption/consolidation control flow:

Disruption controller periodically (default every ~10s reconcile loop, event-driven on pod changes) evaluates nodes for: Empty (no non-daemonset pods), Underutilized (consolidation candidate — replace 2 half-empty nodes with 1 full node), Drifted (NodeClass/NodePool changed, e.g., new AMI), Expired (exceeds expireAfter).
Before terminating, Karpenter cordons, drains respecting PDBs, and waits terminationGracePeriod. For GPU nodes running long inference sessions, this drain can be slow (model unload) — tune terminationGracePeriod and use karpenter.sh/do-not-disrupt: "true" pod annotation on active-inference pods.

Failure scenarios:

Spot interruption: AWS issues a 2-minute interruption notice via EC2 metadata + EventBridge. Karpenter's interruption controller (subscribes to an SQS queue fed by an EventBridge rule you must provision) immediately cordons and drains proactively instead of waiting for the hard kill — critical for GPU spot where losing a $30k/hr p5 node ungracefully mid-batch is costly.
Controller crash mid-provisioning: NodeClaim CRD is the source of truth (not in-memory state), so on restart the new leader reconciles from NodeClaim status — no double-provisioning, but there can be a brief gap where pending pods wait for the new leader to acquire the lease (typically <15s).
ICE (InsufficientCapacityError) on all diversified types: Karpenter marks that instance-type+AZ combination in a short-lived internal "unavailable offerings" cache (default ~3 min TTL) and retries with the next-ranked type — this is the exact gap CAS lacks.

6. Production Troubleshooting

Symptom: GPU inference pods Pending for >5 minutes during a traffic spike.

Step-by-step investigation (senior SRE flow):

kubectl get pods -n inference --field-selector=status.phase=Pending → confirm which pods, check kubectl describe pod events for FailedScheduling reason (insufficient GPU vs. taint mismatch vs. no matching NodePool).
kubectl get nodeclaims -A → check if Karpenter already launched a NodeClaim that's stuck (kubectl describe nodeclaim <name>) — look for Launched=False condition with a reason like InsufficientCapacity.
Karpenter controller logs (structured JSON):
```
kubectl logs -n karpenter deploy/karpenter -c controller | \
  jq 'select(.logger=="controller.provisioner")'
```
Look for "could not get instance types that satisfy requirements" or InsufficientInstanceCapacity errors per AZ/instance-type.
Check Prometheus metrics (Karpenter exposes native metrics):
- karpenter_nodeclaims_disrupted_total
- karpenter_provisioner_scheduling_duration_seconds (simulation taking too long = too many pending pods/nodes, consider batching limits)
- karpenter_cloudprovider_instance_type_offering_available — 0 means AWS reports no capacity for that (type, AZ, capacity-type) tuple right now.
Grafana dashboard: overlay pending_pods vs nodeclaims_created vs time-to-ready histogram — if NodeClaims are created but never reach Ready, it's an EC2/AMI/user-data boot problem, not a scheduling problem — check kubectl get events --field-selector involvedObject.kind=Node.
Root cause in this scenario: NodePool only listed g5.12xlarge; AWS had zero on-demand capacity for that type in the account's AZs during the spike (regional GPU shortage).
Fix: Expand NodePool requirements to include g5.24xlarge, g6.12xlarge, and enable Capacity Blocks/On-Demand Capacity Reservations for baseline GPU floor, with Karpenter only handling burst above the reserved floor (reservation-aware NodePool via karpenter.k8s.aws/capacity-reservation-id).

Config change (YAML):

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-inference
spec:
  template:
    spec:
      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["g5.12xlarge","g5.24xlarge","g6.12xlarge"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-nodeclass
  disruption:
    consolidationPolicy: WhenEmpty
    budgets:
      - nodes: "10%"

7. Hands-on Lab

Goal: Stand up Karpenter on a local/dev EKS cluster and observe capacity-fallback behavior.

# 1. Create a minimal EKS cluster (eksctl) with Karpenter's IAM/OIDC prereqs
eksctl create cluster -f - <<EOF
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: karpenter-lab
  region: us-east-1
iam:
  withOIDC: true
managedNodeGroups:
  - name: system
    instanceType: m6i.large
    desiredCapacity: 2
EOF

# 2. Install Karpenter via Helm (assumes IRSA role + instance profile already created,
#    typically via the Karpenter Terraform/CFN quickstart)
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version 1.1.0 --namespace karpenter --create-namespace \
  --set settings.clusterName=karpenter-lab \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=$KARPENTER_IAM_ROLE_ARN

# 3. Apply a NodePool + EC2NodeClass (general purpose, spot-first)
kubectl apply -f - <<EOF
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: Bottlerocket
  role: KarpenterNodeRole-karpenter-lab
  subnetSelectorTerms:
    - tags: {karpenter.sh/discovery: karpenter-lab}
  securityGroupSelectorTerms:
    - tags: {karpenter.sh/discovery: karpenter-lab}
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot","on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      nodeClassRef: {group: karpenter.k8s.aws, kind: EC2NodeClass, name: default}
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
EOF

# 4. Trigger scale-up: deploy a workload requesting more CPU than current capacity
kubectl create deployment inflate --image=public.ecr.aws/eks-distro/kubernetes/pause:3.9
kubectl scale deployment inflate --replicas=20
kubectl set resources deployment inflate --requests=cpu=1,memory=1Gi

# 5. Validate
kubectl get nodeclaims -w          # watch new nodes being provisioned
kubectl get nodes -L karpenter.sh/capacity-type,node.kubernetes.io/instance-type
kubectl logs -n karpenter deploy/karpenter -c controller --tail=50

# 6. Cleanup
kubectl delete deployment inflate
kubectl delete nodepool default
kubectl delete ec2nodeclass default
eksctl delete cluster --name karpenter-lab

Validation checklist: confirm NodeClaims transition Launched→Registered→Initialized→Ready; confirm consolidation removes nodes ~30–60s after scaling deployment back to 0 (watch kubectl get nodes -w).

8. Production Case Study

AWS/Amazon internally migrated large EKS fleets from Cluster Autoscaler + Managed Node Groups to Karpenter specifically to solve GPU capacity fallback for internal ML platforms — public AWS blog case studies report 30–40%+ compute cost reduction from consolidation alone, plus scale-up latency dropping from minutes to under a minute.

Grafana Labs uses Karpenter across multi-tenant EKS clusters to handle bursty tenant workloads without maintaining dozens of pre-sized node groups, relying on NodePool-per-tenant-class isolation with taints to prevent noisy-neighbor scheduling.

AI inference platforms (pattern seen across OpenAI-adjacent and NVIDIA-partner infra shops) commonly pair Karpenter with Karpenter's native GPU + Capacity Block support: baseline GPU capacity is reserved via Capacity Blocks (guaranteed for training runs), and Karpenter handles only the elastic inference overflow tier on spot/on-demand — this "reserved floor + elastic ceiling" pattern is the dominant production architecture for cost-sensitive GPU fleets in 2025–2026.

9. Architecture Review

Strengths: fast, capacity-aware scale-up; strong cost consolidation; single controller instead of N node-group configs; native spot interruption handling; GitOps-friendly CRDs.

Weaknesses: single-cluster/single-region scope (no built-in multi-cluster capacity brokering); consolidation churn can disrupt long-lived stateful workloads if PDBs/annotations aren't disciplined; broader IAM blast radius than ASG-scoped CAS; relatively young project (GA 2023, v1 API stabilized 2024) — fewer battle-tested large-scale (10k+ node) public postmortems than CAS.

What fails first at 10x scale: the provisioning controller's scheduling simulation becomes CPU/latency-bound with very large pending-pod batches (thousands of pods pending simultaneously during a mass rescheduling event, e.g., an AZ failure) — mitigate with pod priority classes so Karpenter provisions for critical workloads first, and split NodePools to bound simulation scope per workload class.

At 100M users: you'd move to a cell-based multi-cluster architecture (each cell = one EKS cluster with its own Karpenter instance) with a higher-level capacity broker/service-mesh-based traffic router (e.g., Istio multi-cluster or a custom control plane) doing cross-cell load shedding — Karpenter itself stays cluster-scoped; the redesign happens above it, not within it.

What to redesign: introduce a capacity-reservation-aware wrapper (Capacity Blocks/ODCR integration) as a first-class NodePool tier rather than an afterthought, and invest in pre-warmed node pools (NodePool with a floor via karpenter.sh/nodepool + a small always-on limits) for latency-critical GPU tiers where 30–60s cold start still breaches SLA.

10. Best Practices

Set disruption.budgets per NodePool to cap concurrent node churn during business hours (e.g., nodes: "0" during a defined blackout window via schedule). Use karpenter.sh/do-not-disrupt: "true" on pods running irreplaceable long jobs (model training checkpoints, batch jobs near completion). Always diversify instance types (minimum 3–5 per NodePool) — single-type NodePools reintroduce the exact ICE problem Karpenter is meant to solve. Pin AMI via amiSelectorTerms with digest/hash pinning in regulated environments rather than amiFamily alone, to avoid untested AMI drift. Emit Karpenter metrics to Prometheus/Grafana and alert on karpenter_nodeclaims_launch_failure_total and sustained Pending pod count. Use resource limits on NodePools to hard-cap runaway cost from a misconfigured HPA/KEDA scaler.

11. Common Production Mistakes

Running a single catch-all NodePool for all workload types — causes GPU pods to be starved by web-tier consolidation churn and makes disruption budgets impossible to tune sensibly. Forgetting PodDisruptionBudgets — consolidation will happily terminate nodes and break availability SLOs for services with only 1–2 replicas. Not diversifying instance types on GPU NodePools — teams hit the exact ICE failure Karpenter was adopted to prevent. Leaving consolidationPolicy: WhenEmptyOrUnderutilized on stateful/GPU workloads without do-not-disrupt — leads to mid-inference node churn and dropped requests. Granting the Karpenter IRSA role unscoped ec2:*/iam:PassRole — a real privilege-escalation risk if the controller is ever compromised via a supply-chain issue.

12. Interview Preparation

Q: How does Karpenter decide which instance type to launch, and how is this fundamentally different from Cluster Autoscaler? A: Karpenter runs an internal scheduling simulation over pending pods against all NodePool-allowed instance types/AZs/capacity-types, computing the minimal-cost node set that satisfies real scheduler predicates (affinity, topology spread, resource requests), then calls EC2 CreateFleet with a ranked, diversified instance list. CAS instead scales pre-defined, fixed-shape ASGs/node groups and has no per-pod-shape awareness or cross-type capacity fallback — it just requests +1 instance of whatever type the target ASG has.

Q: How would you prevent Karpenter's consolidation from disrupting a stateful GPU inference workload mid-request? A: Combine a PodDisruptionBudget (minAvailable) with the karpenter.sh/do-not-disrupt: "true" pod annotation during active inference windows, set consolidationPolicy: WhenEmpty (not WhenEmptyOrUnderutilized) on the GPU NodePool, and tune terminationGracePeriodSeconds to allow in-flight requests/model unload to complete before hard kill.

Q: Your GPU NodePool only lists one instance type and you keep hitting InsufficientInstanceCapacity. How do you fix it without redesigning the whole platform? A: Diversify the NodePool's instance-type requirements across a family of comparable GPU shapes (e.g., g5.12xlarge, g5.24xlarge, g6.12xlarge), optionally add capacity-type fallback (spot+on-demand), and consider backing a baseline floor with On-Demand Capacity Reservations or Capacity Blocks so burst-only traffic hits Karpenter's elastic path.

Q: What's the security concern unique to Karpenter's IAM model compared to CAS, and how do you mitigate it? A: Karpenter's controller role needs broad ec2:RunInstances/CreateFleet/iam:PassRole permissions (not scoped to a single ASG like CAS), which is a larger blast radius if compromised. Mitigate with IAM condition keys restricting allowed instance types/AMIs/tags, restrict PassRole to only the node instance role, and enforce NodeClass/NodePool changes via GitOps + admission policy (Kyverno/OPA) so the controller's own permissions can't be silently widened.

13. Latest Industry Updates

Karpenter's v1 API (stable, GA'd in 2024) is now the AWS-recommended default for new EKS clusters over Managed Node Group + Cluster Autoscaler; AWS continues investing in tighter Capacity Blocks/ODCR integration for GPU-heavy AI platforms, which matters directly for teams running vLLM/Triton/KServe fleets that need both a guaranteed training floor and elastic inference burst. The broader CNCF ecosystem trend (Karpenter, KEDA, Cluster API convergence) is toward workload-shape-aware, declarative capacity management replacing static node-group sizing everywhere — relevant because it changes how platform teams budget and forecast cloud spend (moving from "N nodes reserved" to "SLA + cost ceiling declared, capacity emergent").

14. Summary & Cheat Sheet

Key concepts: NodePool = capacity policy (instance types, AZs, capacity-type, disruption rules). EC2NodeClass = infra template (AMI, subnets, SGs, IAM). NodeClaim = internal reconciliation object between "pending pods" and "registered Node." Consolidation = active bin-packing/cost optimization, distinct from scale-down.

Core commands:

kubectl get nodepools
kubectl get nodeclaims -A
kubectl describe nodeclaim <name>
kubectl logs -n karpenter deploy/karpenter -c controller
kubectl get nodes -L karpenter.sh/capacity-type,node.kubernetes.io/instance-type

Design pattern: one NodePool per workload class + weighted spot/on-demand fallback + diversified instance types + PDBs + do-not-disrupt annotations for stateful/GPU workloads.

Troubleshooting checklist: (1) kubectl describe pod for scheduling reason → (2) kubectl get nodeclaims for launch failures → (3) controller logs for ICE/capacity errors → (4) Prometheus karpenter_cloudprovider_instance_type_offering_available → (5) widen instance-type diversity or add capacity reservation floor.

Best practice one-liner: never run Karpenter with a single instance type per NodePool — diversification is the entire point.

Karpenter — Next-Generation Kubernetes Cluster Autoscaling (AWS)