title: "Policy as Code at Scale: Kyverno, OPA/Gatekeeper & CEL-Native Admission Control in Production Kubernetes" date: 2026-09-20 tags: [Kubernetes, Policy as Code, Kyverno, OPA, Gatekeeper, ValidatingAdmissionPolicy, CEL, Admission Control, DevSecOps, Platform Engineering, CNCF] cover: ../images/policy-as-code-kyverno-cel-cover.png

Cover

Policy as Code at Scale: Kyverno, OPA/Gatekeeper & CEL-Native Admission Control in Production Kubernetes

1. Topic of the Day

Every mature Kubernetes platform eventually hits the same wall: RBAC and NetworkPolicy answer "who can do what" and "what can talk to what," but neither answers "is this object allowed to exist in this shape." Nothing in core Kubernetes stops a developer from deploying a container running as root, pulling :latest from an untrusted registry, requesting no resource limits, or omitting the cost-center label your FinOps chargeback pipeline depends on. That gap is what admission control — and specifically policy as code — exists to close: intercept the object between "the user submitted it" and "etcd persisted it," and either reject it, silently fix it, or spin off a companion resource, all as versioned, reviewable, testable code instead of a wiki page nobody reads.

The landscape here has genuinely shifted through 2025 and 2026, and it's worth being precise about where things stand today rather than repeating advice that was accurate two years ago. For most of Kubernetes' history there were two real choices: OPA Gatekeeper, which wraps Open Policy Agent's general-purpose Rego language in Kubernetes CRDs (ConstraintTemplate + Constraint), and Kyverno, a Kubernetes-native engine where policies are plain YAML ClusterPolicy objects with no new language to learn. Kyverno graduated to CNCF Graduated status in November 2024, and by 2026 multiple ecosystem surveys report it has overtaken Gatekeeper in new production adoption — not because Rego is bad, but because "write a Kubernetes manifest to govern Kubernetes manifests" turned out to have a much shorter path to platform-team productivity than "learn a general-purpose logic language first."

The bigger shift, though, is the arrival of native, in-process admission policy. ValidatingAdmissionPolicy (VAP), built on CEL (Common Expression Language) and running directly inside kube-apiserver, went GA in Kubernetes 1.30. Its sibling, MutatingAdmissionPolicy (MAP), went stable in Kubernetes 1.37 ("Garhwal"), which shipped GA on August 26, 2026 — so as of this post, the two halves of native CEL admission control (validate and mutate) are both stable, in every cluster running a current Kubernetes version, with zero external webhook infrastructure required. Kyverno itself responded by shipping CEL-based policies to v1 in its 1.17 release (reported this month), letting teams write one policy language that spans admission, background audit, and cleanup — instead of the JMESPath-flavored expression language Kyverno used historically, which is now on a deprecation path (legacy JMESPath ClusterPolicy/CleanupPolicy types are slated for removal in Kyverno v1.20, October 2026).

None of this makes Kyverno or Gatekeeper obsolete — VAP/MAP are deliberately scoped to what CEL can express against a single object in isolation, with no cross-resource context, no external data lookups, no mutation-then-generate chaining, and no built-in reporting UI. What it does is collapse the "why do we even need an external admission controller for a label-format check" argument: in 2026, the correct production stack is a layered one — native CEL policy for the high-volume, low-latency, single-object rules (label conventions, securityContext requirements, registry allow-lists), and Kyverno (increasingly with its own CEL policy type) or Gatekeeper reserved for what needs an external engine — context-aware validation, generate/mutate workflows, image signature verification against an external Sigstore/Rekor endpoint, and org-wide reporting. Getting that layering wrong — or getting webhook failure-policy and timeout tuning wrong — is precisely how a "we added a security policy" change becomes a cluster-wide production outage, which is the incident this post is built around.

2. Real Business Problem

Scenario: A B2B SaaS platform runs a 340-node, multi-tenant EKS cluster hosting ~180 customer-facing microservices across 65 namespaces, shared by four internal product teams and isolated per-customer workloads via namespace-per-tenant. The security team, closing out a SOC 2 finding, rolls out a Kyverno ClusterPolicy requiring every new Pod's images to carry a verified Cosign signature from the internal supply-chain pipeline (Section covers the supply-chain angle only at the edges — the focus here is what the admission path itself did under load).

The policy is validated in staging, passes a canary rollout to two low-traffic namespaces, and ships cluster-wide on a Tuesday morning. Nothing breaks — for six hours.

At 2:14 PM, a marketing-driven traffic spike (a customer's product launch) triggers the platform's Horizontal Pod Autoscalers across 40+ Deployments simultaneously, and Karpenter begins provisioning nodes to match. Within about ninety seconds:

  • The Kyverno webhook's signature-verification step, which calls out to the internal Rekor transparency-log mirror to verify each image's signature, was never load-tested at this request rate. Under normal traffic, admission requests trickle in; under a 40-Deployment simultaneous scale-out, the webhook receives several hundred concurrent AdmissionReview calls, and the Rekor mirror — a single-replica internal service nobody flagged as being on the pod-creation critical path — starts queuing, then timing out.
  • The ClusterPolicy's webhook was configured with failurePolicy: Fail and a 10-second timeout, cluster-wide, with no namespaceSelector exclusions. This was a deliberate security choice (fail closed, don't let unsigned images slip through on a webhook hiccup) — but it means that when the signature-verification backend degrades, every Pod creation in every namespace, not just the ones actually needing strict enforcement, starts failing at the API server, because the webhook itself is now the single slowest, least reliable component in the entire pod-admission critical path.
  • Admission webhooks execute synchronously, in the request path, for every matching request cluster-wide. As the Rekor mirror's p99 latency climbs past the 10-second webhook timeout, kube-apiserver starts returning Internal error occurred: failed calling webhook "verify-image-signatures.kyverno.svc": context deadline exceeded on Pod creates across the fleet — not just from the four teams whose deploys triggered the surge, but from an unrelated team's CronJob-spawned Pods, from the cluster autoscaler's own node-readiness probes indirectly (via DaemonSet Pod creation on new nodes), and from ArgoCD's own reconciliation loop trying to sync unrelated Applications.
  • HPA-driven scale-out silently stalls. The HPA controller keeps recording the target replica count correctly, but the ReplicaSet controller's repeated Pod-create attempts against the API server keep failing and retrying with backoff — from the outside, this looks like "autoscaling isn't working" with no obvious webhook-shaped root cause in the first dashboards anyone opens, because the person on call is looking at HPA and node-capacity graphs, not apiserver_admission_webhook_admission_duration_seconds.
  • The customer's product launch traffic keeps growing while the platform's ability to add capacity is effectively frozen, because the thing standing between "HPA decided to scale" and "a schedulable Pod exists" is a webhook call to an overloaded internal service that has nothing to do with capacity at all.

By 2:31 PM — seventeen minutes after the spike started — the platform is in a full-blown incident: five customer-facing services are returning elevated 5xx rates from being under-provisioned, the on-call engineer has restarted the Kyverno pods twice (no effect, since the bottleneck is downstream of Kyverno itself), and nobody yet suspects a security policy shipped six hours earlier is the cause, because "it's been running fine all day."

The business problem the postmortem has to solve isn't "was the signature-verification requirement wrong" — the auditors correctly required it. It's: how do you design admission control so that a degraded external dependency behind one policy cannot become a cluster-wide capacity-provisioning outage, while still meeting a fail-closed security requirement for the specific policy that actually needs it.

3. Production Architecture

Architecture image: blogs/architecture/policy-as-code-kyverno-cel-architecture.png

Layered admission, ordered by cost. The redesigned architecture treats admission control as a pipeline with an explicit cost order, evaluating the cheapest, most reliable checks first and only reaching the riskiest external dependency last, for the narrowest possible slice of traffic. kube-apiserver evaluates built-in admission plugins and Pod Security Admission (PSA, at the restricted level for tenant namespaces) first — these are in-process, sub-millisecond, and cannot fail from an external dependency. Next in the chain: ValidatingAdmissionPolicy and MutatingAdmissionPolicy (both stable as of Kubernetes 1.37) handle every rule that can be expressed as CEL against a single object in isolation — required labels, securityContext enforcement (no root, no privilege escalation, read-only root filesystem), resource-request/limit presence, registry-prefix allow-lists, replica-count sanity bounds. These run inside the API server process itself — no network hop, no separate Deployment to scale, no webhook timeout to tune, and critically, no way for an external dependency's degradation to touch them.

External webhooks, scoped to exactly what needs them. Only what genuinely requires an external engine reaches a webhook at all: Kyverno owns mutate/generate workflows (auto-injecting NetworkPolicies per namespace, generating default ResourceQuotas, sidecar injection coordination) and the one rule that actually needs an external service call — Cosign signature verification against the Rekor mirror. That signature-verification ClusterPolicy is deliberately isolated onto its own ValidatingWebhookConfiguration, separate from Kyverno's other (in-cluster-only, no-external-call) policies, so that a degraded Rekor mirror can only ever fail the one webhook that depends on it — not the entire Kyverno admission surface. Its namespaceSelector scopes enforcement to namespaces actually in the compliance boundary (customer-tenant namespaces and anything shipping to production), explicitly excluding platform-internal namespaces (kube-system, the autoscaler's own namespace, CI/CD runner namespaces) that were never in scope for the SOC 2 control but were getting caught by the blanket original policy.

The webhook itself is now horizontally scaled and fronted by a local cache. Kyverno's signature-verification policy runs behind an HPA keyed on admission request rate (via a custom Prometheus adapter metric, not just CPU), and — the change that mattered most in the postmortem — a short-TTL, in-cluster signature-verification cache (Redis, 5-minute TTL keyed on image digest) sits between Kyverno and the Rekor mirror. Because the same handful of image digests get deployed hundreds of times across a scale-out event, the cache converts what was "one Rekor call per Pod create" into "one Rekor call per unique image digest per five minutes," cutting real external-dependency load by two orders of magnitude during a burst. The Rekor mirror itself moved from a single replica to a 3-replica Deployment behind its own readiness-gated Service, since the postmortem also surfaced that a security-critical dependency being a single point of failure was never something anyone had explicitly signed off on.

Failure-policy is tiered, not blanket. failurePolicy: Fail is retained — deliberately — but now only on the signature-verification webhook, only for the namespaces actually in the compliance boundary, with a tuned 3-second timeoutSeconds (down from 10) paired with the cache so the common case never gets close to timing out, and an explicit, monitored PolicyException escape hatch (Kyverno 1.11+) that lets platform engineers grant a time-boxed, audited exception during an active incident without having to edit or delete the policy itself under pressure. Every other Kyverno policy in the cluster — the ones with no external dependency — runs failurePolicy: Fail too, safely, precisely because "fail closed" is only dangerous when the thing that can fail is an unreliable external call; for a pure in-cluster mutation or a CEL check, failing closed has no real downside and is the correct default.

matchConditions as a pre-filter. Both the native VAP/MAP objects and Kyverno's webhook configurations use CEL matchConditions to skip evaluation entirely for requests that can never match — system namespaces, non-Pod resources for Pod-only policies, updates that don't touch the fields a mutate policy cares about — cutting webhook invocation volume before a single line of policy logic runs, which is the single highest-leverage lever for reducing webhook load under burst traffic, because the fastest admission call is the one that never happens.

Observability wired into the admission path itself. apiserver_admission_webhook_admission_duration_seconds (per webhook, per operation) and apiserver_admission_webhook_rejection_count are first-class SLO signals in Grafana, alerting through the same pipeline as node-capacity and HPA metrics — deliberately co-located on the same on-call dashboard the incident commander opens first, closing the exact "nobody thought to look at admission latency" gap from the original incident. Kyverno's PolicyReport CRDs (the CNCF Policy Working Group's adopted reporting standard) feed a PolicyReporter UI for audit and drift visibility, and every policy ships through GitOps (ArgoCD) from a dedicated platform-policies repo with required PR review, staged rollout via Audit mode before Enforce, and a canary namespace tier before cluster-wide application — the same progressive-delivery discipline the platform already applies to application code, now applied to the policies governing that code.

Why this design, and the trade-offs. The alternative — routing everything through native CEL policy and eliminating Kyverno/Gatekeeper entirely — was evaluated and rejected: CEL has no mechanism for calling out to an external service (by design, for exactly the isolation reasons this incident illustrates), so signature verification against Rekor structurally cannot move to VAP/MAP without either embedding trust material directly as in-cluster CEL-checkable data (rejected — that reintroduces the supply-chain trust problem the policy exists to solve) or waiting for a future native mechanism that doesn't yet exist. The chosen split — native CEL for everything context-free and single-object, an external engine narrowly scoped and hardened for the one rule that genuinely needs external verification — is the point on the reliability/capability curve that isolates blast radius without abandoning the security requirement. As the platform scales past its current 340 nodes, the next evolution is moving the signature cache from a single Redis deployment to a properly HA-replicated cache tier, and revisiting whether an in-toto/SLSA attestation check that can be resolved from data already admitted into the cluster (rather than a live external call on every request) can replace the live Rekor lookup for previously-verified digests.

4. Solution Design

Design decision: split Kyverno's signature-verification webhook onto its own ValidatingWebhookConfiguration, isolated from other policies. Kyverno, by default, can register multiple ClusterPolicy objects under a shared webhook configuration for efficiency. That efficiency is exactly the coupling that turned "the Rekor mirror is slow" into "every Kyverno-governed admission in the cluster is slow." Isolating the one policy with an external dependency costs a small amount of additional webhook-registration overhead and is unambiguously worth it — blast-radius isolation was judged the higher-priority property once the incident made the coupling's cost concrete.

Alternative considered and rejected: move signature verification to a background/audit-only check instead of blocking admission. This was seriously discussed — Kyverno supports validationFailureAction: Audit for exactly this pattern, flagging non-compliant Pods in a PolicyReport rather than blocking them. Rejected for the production tenant namespaces specifically, because the SOC 2 control the security team is closing requires preventive, not detective, enforcement for unsigned images reaching production — audit-only was judged acceptable for lower-risk namespaces (internal tooling, staging) but not for the compliance boundary the finding actually covers. This is a real trade-off or nothing here: audit mode trivially solves the outage-risk problem by removing the blocking dependency, at the direct cost of the security guarantee the policy exists to provide — the resolution wasn't "audit is better" or "blocking is better" in the abstract, it was scoping which enforcement mode applies to which namespaces based on actual compliance boundary, not applying one mode cluster-wide out of convenience.

Alternative considered and rejected: standardize entirely on OPA Gatekeeper instead of Kyverno. Evaluated during the postmortem, since Gatekeeper's smaller footprint (roughly 270MB versus Kyverno's ~600MB across four controllers) was raised as a simplification option. Rejected: the incident's root cause was never Kyverno's resource footprint or its policy language — it was webhook coupling and an unbounded external call — and Gatekeeper's Rego-based ConstraintTemplate model would reproduce the identical webhook-coupling risk if the signature-verification logic were ported as-is. The team's existing YAML-fluent policy library, Kyverno's GA mutate/generate support (which Gatekeeper's own mutation support still sees little production use, per current ecosystem surveys), and PolicyReport being the CNCF-standardized reporting format were all judged to outweigh the footprint difference, which is "noise" at this cluster's scale per the same surveys.

Scalability considerations. Webhook admission latency scales with concurrent admission volume, not cluster size directly — the binding constraint the incident surfaced was concurrent Pod-create rate during a correlated multi-team scale-out event, which is precisely the traffic pattern load tests need to target (steady-state admission-rate testing would never have caught this; the test now explicitly simulates a correlated 40-Deployment scale-out). matchConditions-based pre-filtering and the signature cache both directly target this axis rather than throwing more webhook replicas at the problem, since more replicas don't help when the bottleneck is a downstream dependency's own capacity.

Cost implications. The added Redis cache tier and the Rekor mirror's move from 1 to 3 replicas are the primary new cost — modest in absolute terms (a handful of small instances) against the cost of the incident itself (customer-facing 5xx during a customer's own product launch, an SLA-relevant event). The webhook HPA's added replicas during bursts are a small, bursty cost, not a steady-state one.

Security implications. Narrowing namespaceSelector scope to the actual compliance boundary is itself a security-posture improvement, not just a reliability one — the original blanket policy was, in practice, providing no additional assurance for platform-internal namespaces that were never in the audit scope, while adding exactly the risk this incident realized. The PolicyException escape hatch is deliberately audited and time-boxed (a Kyverno PolicyException object with an expiry and a required context.reason annotation, alerted on creation) specifically so it can't quietly become a permanent bypass.

Performance implications. Moving the high-volume, context-free checks to in-process CEL (VAP/MAP) removes an entire network hop and a separate controller's scheduling/resource contention from the hot path for the majority of admission traffic — measured post-migration at roughly a 60-70% reduction in mean admission latency for the policies that moved, since "in-process CEL evaluation" has no comparable failure mode to "call a webhook Service, which the kube-proxy/CNI data plane then has to route."

5. Deep Technical Walkthrough

The admission request lifecycle, precisely. When kube-apiserver receives a write request, it runs, in order: authentication, authorization (RBAC), mutating admission (built-in plugins, then MutatingAdmissionPolicy objects, then mutating webhooks, each potentially modifying the object and each seeing the previous stage's output), object schema validation, and finally validating admission (built-in plugins, then ValidatingAdmissionPolicy objects, then validating webhooks — these run against a stable, already-mutated object and, unlike the mutating chain, can be evaluated in parallel since none of them can change what the others see). Only after every stage succeeds does the object get persisted to etcd. This ordering is why a slow validating webhook doesn't just delay that rule's own check — it delays the entire request past everything that already successfully ran, and why the incident's Pod creates weren't failing at authorization or at Kyverno's other (fast) policies; they were failing at the very last gate, after all cheaper work had already been done and thrown away on rejection.

Why CEL in VAP/MAP is structurally different from a webhook, not just "a faster webhook." A ValidatingAdmissionPolicy compiles its CEL expressions once, at policy-load time, into a form the API server evaluates as part of its own request-handling goroutine — there is no serialization to JSON, no HTTP round-trip, no separate process to schedule, restart, or lose network connectivity to. This is also its hard boundary: CEL expressions in VAP/MAP can see the incoming object, the old object (on updates), and a small set of injected variables (namespace object, authorizer for authz.* checks), but they cannot make an outbound network call, cannot query another cluster resource not explicitly provided via paramRef, and cannot maintain state across requests. paramRef (pointing a VAP at a CRD instance holding configurable parameters, like an allow-listed registry set) is the mechanism for making CEL policies configurable without a redeploy — the policy's logic ships once; the parameters it evaluates against ship as data.

Kyverno's request flow and where the external call actually sits. Kyverno's admission controller receives the AdmissionReview, matches it against loaded ClusterPolicy/Policy objects by match/exclude block and (as of Kyverno 1.17's CEL policy type) optional matchConditions, and for the signature-verification rule specifically, its verifyImages handler extracts the image reference, checks the local signature cache first (the change from this postmortem), and on a cache miss calls out via cosign's verification library to the configured Rekor endpoint — this is the one point in Kyverno's entire request flow that leaves the cluster's network boundary, and correspondingly the one point that needed isolation. Every other Kyverno capability used here — mutate, generate, the CEL-based label/securityContext rules — resolves entirely from data already in the AdmissionReview object or from in-cluster API server lookups, with no equivalent external-latency exposure.

Control-plane and data-plane interaction during a webhook timeout. When Kyverno's webhook exceeds its configured timeoutSeconds, kube-apiserver does not retry the webhook call itself — it applies the failurePolicy immediately: Fail returns the admission error to the original client (kubectl apply, the ReplicaSet controller, ArgoCD's sync); Ignore proceeds as if the webhook had approved the request. This is a control-plane-only interaction — no data-plane component (kube-proxy, CNI, kubelet) is involved at all, which is precisely why the incident's symptoms looked like a scheduling or capacity problem from the data-plane side (Pods never got created, so there was nothing for the data plane to do) when the actual fault was entirely in the admission stage of the control plane, several steps before scheduling is ever reached.

Failure scenarios and recovery mechanisms.

  • External dependency (Rekor mirror) degrades: with the cache in place, only cache-miss traffic (new, previously-unseen image digests) is exposed to the degraded dependency at all — the blast radius shrinks from "every Pod create cluster-wide" to "Pod creates using a brand-new image digest during the degradation window," a materially smaller and more defensible failure mode.
  • Webhook Service itself is unreachable (pod crash, network partition): failurePolicy: Fail on the narrowly-scoped webhook fails only compliance-boundary Pod creates, not the whole cluster — and the PolicyException path gives on-call a deliberate, audited way to restore service without editing the policy under incident pressure.
  • A CEL expression in a VAP has a runtime error (e.g., a paramRef object is missing): VAP/MAP support a failurePolicy too, and — separately — a CEL Cost budget that the API server enforces per-expression to prevent a pathological policy from degrading kube-apiserver itself; a CEL policy that exceeds its cost budget or errors is rejected (or ignored, per failurePolicy) per-request, never able to cascade into an apiserver-wide slowdown the way an external webhook's queuing can.
  • A GitOps-shipped policy change is bad (too strict, matches unintended resources): staged rollout (canary namespace → Audit → Enforce) and ArgoCD's own sync-wave ordering mean a bad policy is caught in the canary tier or in audit-only reporting before it can block production traffic cluster-wide, which is the process control that should have caught this specific incident's root policy change before it ever reached "cluster-wide, Fail, no namespace scoping" in one step.

Performance bottlenecks and scaling behavior. Post-redesign, the dominant remaining cost in the admission path is the mutating-webhook chain for generate-type Kyverno policies (default NetworkPolicy/ResourceQuota generation on namespace creation) — infrequent (namespace creation is rare relative to Pod creation) and therefore not a burst-sensitivity concern the way Pod-level signature verification was. The next anticipated scaling constraint, flagged but not yet hit, is CEL Cost budget headroom as more VAP/MAP rules accumulate per object type — mitigated by consolidating related checks into fewer, more cost-efficient expressions rather than one VAP per individual rule.

6. Production Troubleshooting

Incident: Pod creation cluster-wide fails intermittently during a traffic-driven scale-out, symptoms initially read as an autoscaling failure.

Symptoms. HPA desiredReplicas climbs correctly on the dashboards; actual Ready replica counts lag far behind and eventually plateau; kubectl describe replicaset on affected workloads shows repeated FailedCreate events. No obvious node-capacity or scheduler-pressure signal in the first graphs the on-call engineer checks.

Logs, metrics, traces to pull first.

  • kubectl get events -A --field-selector reason=FailedCreate -o wide — the fastest way to surface the actual admission error text (failed calling webhook "..."), which immediately redirects investigation from "capacity problem" to "admission problem," and is the single command that should be muscle memory for this failure class.
  • apiserver_admission_webhook_admission_duration_seconds_bucket and apiserver_admission_webhook_rejection_count, broken down by name label (the specific webhook) — identifies which webhook is slow/rejecting, not just that "something" is.
  • kubectl logs -n kyverno deploy/kyverno-admission-controller filtered for the affected time window, checking specifically for context deadline exceeded or Rekor-endpoint connection errors, distinguishing "Kyverno itself is unhealthy" from "Kyverno is healthy but its downstream dependency isn't."
  • The Rekor mirror's own latency/error-rate dashboard — this is the step that was missing from the original incident's early investigation entirely, because nobody on the on-call rotation had it wired into the same dashboard as admission and capacity metrics, a gap this postmortem's remediation explicitly closed.
  • kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o yaml cross-referenced against the affected namespace, to confirm exactly which webhooks were actually in scope for the failing requests (this is how the missing namespaceSelector gap was confirmed — platform-internal namespaces were provably in scope for a policy that was never meant to cover them).

Root cause analysis, step by step. Correlating apiserver_admission_webhook_admission_duration_seconds against the Rekor mirror's own request-rate graph showed the two curves rising in lockstep starting at 2:14 PM, with webhook duration crossing the 10-second timeout threshold around 2:15:30 — a clean causal chain, once the right two graphs were on the same screen. The FailedCreate events confirmed the failing webhook by name. kubectl get validatingwebhookconfigurations kyverno-resource-validating-webhook-cfg -o yaml confirmed no namespaceSelector exclusion existed, explaining why platform-internal namespaces were affected despite never being in the compliance boundary the policy was written for.

Configuration changes made. Split the signature-verification rule onto its own ValidatingWebhookConfiguration with a namespaceSelector scoped to a policy-tier: compliance-boundary namespace label (applied only to actual production tenant namespaces). Deployed the Redis signature-cache layer in front of the Rekor mirror call. Scaled the Rekor mirror from 1 to 3 replicas behind a readiness-gated Service. Reduced the webhook timeoutSeconds from 10 to 3, paired with the cache so normal-path latency stays well under it. Added apiserver_admission_webhook_admission_duration_seconds and the Rekor mirror's own latency to the same Grafana dashboard as HPA and node-capacity metrics.

Verification. Re-ran a synthetic 40-Deployment correlated scale-out against a staging replica of the cluster (the load pattern that was never tested pre-incident), first without the fix (reproduced the timeout cascade cleanly, confirming root cause) and then with each change applied incrementally — namespace scoping alone bounded the blast radius to the compliance-boundary namespaces; adding the cache kept even those namespaces' webhook latency under 200ms p99 during the synthetic burst. Both were kept as permanent additions to the platform's load-test suite, run pre-merge on any policy change touching an external dependency.

7. Hands-on Lab

A local, provider-agnostic lab covering all three layers discussed: native CEL (ValidatingAdmissionPolicy), Kyverno with an external-dependency-style check, and the failure-policy/namespace-scoping pattern that fixed the incident.

Setup.

kind create cluster --name policy-lab --image kindest/node:v1.35.0
kubectl create namespace tenant-a
kubectl create namespace platform-internal
kubectl label namespace tenant-a policy-tier=compliance-boundary

Part A — native CEL ValidatingAdmissionPolicy (in-process, no webhook).

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-non-root
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["CREATE"]
      resources: ["pods"]
  matchConditions:
  - name: skip-system-namespaces
    expression: "!(object.metadata.namespace in ['kube-system','kube-public'])"
  validations:
  - expression: "object.spec.securityContext.runAsNonRoot == true"
    message: "Pods must set securityContext.runAsNonRoot: true"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-non-root-binding
spec:
  policyName: require-non-root
  validationActions: ["Deny"]

kubectl apply -f require-non-root.yaml
kubectl run bad-pod -n tenant-a --image=nginx --restart=Never   # expect: denied
kubectl run good-pod -n tenant-a --image=nginx --restart=Never --overrides='{"spec":{"securityContext":{"runAsNonRoot":true,"runAsUser":1000}}}'

Part B — Kyverno policy simulating an external dependency, with a deliberately narrow scope.

helm repo add kyverno https://kyverno.github.io/kyverno/ && helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-team-label
spec:
  validationFailureAction: Enforce
  rules:
  - name: check-team-label
    match:
      any:
      - resources:
          kinds: ["Pod"]
          namespaceSelector:
            matchLabels:
              policy-tier: compliance-boundary
    validate:
      message: "label 'team' is required in the compliance boundary"
      pattern:
        metadata:
          labels:
            team: "?*"

kubectl apply -f require-team-label.yaml
kubectl run test1 -n tenant-a --image=nginx --restart=Never          # expect: denied, no team label
kubectl run test2 -n platform-internal --image=nginx --restart=Never # expect: allowed, out of scope

Part C — simulate the incident and the fix: failurePolicy under a downstream timeout.

# Point Kyverno's webhook timeout down and watch the difference between Fail and Ignore
kubectl patch clusterpolicy require-team-label --type=merge -p '{"spec":{"webhookConfiguration":{"timeoutSeconds":1}}}'
# Simulate load with parallel pod creation
for i in $(seq 1 50); do kubectl run burst-$i -n tenant-a --image=nginx --restart=Never & done; wait
kubectl get events -n tenant-a --field-selector reason=FailedCreate

Observe FailedCreate events referencing the Kyverno webhook under load, then re-apply with the cache/scoping pattern (in this lab, simply widening timeoutSeconds back and confirming namespaceSelector keeps platform-internal unaffected) to validate the isolation property.

Cleanup.

kubectl delete clusterpolicy require-team-label
kubectl delete validatingadmissionpolicybinding require-non-root-binding
kubectl delete validatingadmissionpolicy require-non-root
kind delete cluster --name policy-lab

8. Production Case Study

Google's internal admission-control equivalent (predating and informing much of the upstream Kubernetes admission design) has long treated policy evaluation cost as a first-class SLO input to the control plane itself, not an afterthought bolted on per-team — the CEL Cost budget enforcement in VAP/MAP is a direct, public-facing descendant of that discipline, making pathological policy expressions a rejected-at-load-time problem rather than a live-incident one. Netflix's platform engineering has spoken publicly about treating policy-as-code changes with the same canary-and-automated-rollback machinery (built on their Spinnaker lineage) as application deployments — a policy that starts rejecting an anomalous volume of requests post-rollout triggers an automatic revert, the same reflex this incident's remediation adopted via staged Audit-then-Enforce rollout. Uber's infrastructure teams have documented similar lessons from their own service-mesh and admission-layer work: synchronous, in-request-path checks against services not explicitly designed and capacity-planned for admission-tier traffic patterns are a recurring, cross-company failure class, not a one-off mistake specific to any single org's Kyverno rollout — which is exactly why the CNCF's own Policy Working Group has pushed PolicyReport standardization and why the ecosystem's overall direction (native CEL for the cheap/common case, external engines narrowly scoped) mirrors what large-scale platform teams converge on independently once they've been burned by the coupled version once.

9. Architecture Review

Strengths. Blast-radius isolation (one webhook, one external dependency, one narrow namespace scope) means a future degradation of any single downstream service can no longer become a cluster-wide capacity incident. Layering native CEL ahead of external engines removes an entire class of failure (network/process failures) from the majority of admission traffic. GitOps-staged rollout with canary and audit-before-enforce closes the process gap that let an under-tested policy reach cluster-wide Fail in one step.

Weaknesses. The signature-verification cache introduces a new correctness question — a 5-minute TTL means a signature revocation (an image later found to be compromised and un-signed/re-flagged) has up to a 5-minute window where a cached "verified" result could admit a Pod using it; this is an accepted trade-off today but is a real weakening of the "always check live" guarantee the original design assumed, and needs an explicit cache-invalidation hook on revocation events to fully close. The PolicyException escape hatch, while audited, is still a human-operated bypass under incident pressure — its blast radius is bounded but not zero, and its usage needs periodic review to ensure exceptions expire rather than becoming permanent shadow policy.

What fails first at 10x scale. At 10x current Pod-creation volume, the Redis cache tier itself becomes the new bottleneck candidate — a single-region Redis deployment serving cache reads for every admission request cluster-wide is a new single point of contention that didn't exist in the original (simpler, more fragile) design; it would need to move to a properly sharded/replicated cache before 10x, not just a bigger single instance. CEL Cost budget headroom, currently comfortable, becomes a real constraint as more VAP/MAP rules accumulate per object type across more teams shipping their own policies independently.

What changes for 100 million users. At that scale, per-request signature verification (even cached) stops being the right model entirely — the natural evolution is moving to an admission-time attestation check against data already resolved into the cluster (an in-toto/SLSA provenance record synced in-cluster as a CRD, checkable via native CEL with no external call at all) rather than a live external lookup on any code path, cache or not. Multi-cluster policy distribution (the platform currently runs one cluster; 100M users implies a fleet) would also need Kyverno's or an equivalent fleet-policy-distribution mechanism to keep policy versions consistent across clusters, with drift detection as a first-class signal, not just per-cluster GitOps sync.

What I'd redesign. Given the chance to redo this from scratch rather than retrofit it post-incident, the signature-verification model would be attestation-based (data resolved once, checked natively) from day one rather than live-external-call-with-a-cache-bolted-on-later — the cache is the right immediate fix for the incident that happened, but it's treating a symptom (external call is slow under burst) rather than removing the structural cause (an external call exists on the hot path at all).

10. Best Practices

Reliability: isolate any admission webhook with an external dependency onto its own WebhookConfiguration, never sharing registration with dependency-free policies, so one degraded backend cannot fail unrelated checks. Scalability: prefer native VAP/MAP for anything expressible as single-object CEL before reaching for an external engine, and load-test admission specifically against correlated-burst traffic patterns (many Deployments scaling simultaneously), not just steady-state rate. Observability: put admission-webhook latency and rejection-rate metrics on the same dashboard as capacity/autoscaling metrics — they must be checked together, because a capacity-shaped symptom can have an admission-shaped root cause. Security: scope every webhook's namespaceSelector/objectSelector to its actual compliance boundary explicitly; a blanket cluster-wide policy is not "more secure," it's "equally strict everywhere including where it was never required, with correspondingly larger blast radius." Cost optimization: cache external-dependency lookups aggressively for admission-tier checks where staleness within a bounded window is an acceptable trade against per-request live calls. Performance: use matchConditions on every policy (native or Kyverno) to skip evaluation for non-matching requests as early as possible. Maintainability: one policy language where possible — Kyverno's CEL policy type converging with native VAP/MAP syntax is explicitly reducing the "our team knows Rego, our other team knows JMESPath, nobody knows both" fragmentation that made policies hard to review across teams. Operational excellence: ship every policy change through the same staged-rollout discipline (canary namespace → audit → enforce) as application code, with an explicit, audited, time-boxed exception mechanism for incident response.

11. Common Production Mistakes

Shipping a new failurePolicy: Fail policy cluster-wide in one step instead of staged canary-then-audit-then-enforce is the single most common way a well-intentioned security policy becomes an outage — the fix isn't "never use Fail," it's "never skip the staging path that would have caught the blast radius before it mattered." Registering an external-dependency-backed rule under the same webhook as dependency-free rules is the second most common mistake, and it's often invisible until the exact failure mode this post describes actually happens, because in normal operation the coupling has zero observable cost. Leaving default webhook timeouts (often 10s) unexamined against what the actual downstream call's realistic p99 looks like under burst, rather than under steady state, is a close third — a timeout tuned against calm-traffic latency numbers is not tuned at all. Treating namespaceSelector/objectSelector as optional boilerplate rather than the primary blast-radius control they are is a recurring gap, especially in policies inherited from a vendor example or copied from another cluster without re-scoping. And reaching for an external policy engine by default, out of habit, for checks that are pure single-object CEL and belong in VAP/MAP, adds unnecessary webhook surface area, latency, and failure modes to rules that never needed any of it.

12. Interview Preparation

"Walk me through what happens, step by step, when a Pod create request hits an API server with both a ValidatingAdmissionPolicy and a Kyverno mutating webhook configured." A strong answer traces the full pipeline in order — authn, authz, mutating admission (built-in plugins, then MutatingAdmissionPolicy objects evaluated in-process, then mutating webhooks called over HTTP in a defined order, each seeing the prior stage's output) — followed by schema validation, then validating admission (built-in plugins, ValidatingAdmissionPolicy objects, then validating webhooks, evaluated against the now-stable mutated object) — and explains why validating webhooks cannot mutate and why that ordering exists (so validation always sees final, post-mutation state).

"Why would you choose ValidatingAdmissionPolicy over Kyverno for a given rule, and vice versa?" The strong answer isn't "VAP is newer/better" — it's a capability boundary: VAP/MAP for single-object CEL-expressible rules with no external dependency and no cross-resource context beyond paramRef data; Kyverno (or Gatekeeper) for anything needing mutate/generate chaining, external service calls (image signature verification, external data lookups), or built-in reporting via PolicyReport. A candidate should also flag that CEL's Cost budget and lack of network access are deliberate safety properties, not gaps to work around.

"A webhook-based policy is causing intermittent Pod-create failures under load. How do you diagnose it without assuming the fix upfront?" Look for: check kubectl get events --field-selector reason=FailedCreate first for the actual error text; correlate apiserver_admission_webhook_admission_duration_seconds against the specific webhook name; check whether the webhook has any external dependency and whether that dependency's own latency correlates with the timing; confirm failurePolicy and namespaceSelector scope match what was actually intended versus what's actually configured — a candidate who jumps straight to "just set failurePolicy to Ignore" without diagnosing first should be pushed on the security trade-off they just proposed blind.

"How do you prevent a runaway or expensive admission policy from degrading the API server itself?" For native CEL: the built-in per-expression Cost budget, enforced by the API server, rejecting policies that exceed it. For webhook-based engines: timeoutSeconds tuned against realistic burst-case downstream latency (not steady-state), horizontal scaling of the webhook's own Deployment keyed on admission request rate, and — the lesson from this incident — never coupling an external-dependency check to the same webhook registration as high-volume, dependency-free checks.

"Design a policy-as-code rollout process for a 60-namespace multi-tenant cluster that avoids a cluster-wide outage from a bad policy." Expect: GitOps-sourced policies with mandatory PR review; canary namespace tier first; Audit mode (PolicyReport-only, non-blocking) before Enforce; explicit, scoped namespaceSelector/objectSelector per policy tied to actual compliance boundaries rather than blanket application; load-testing admission changes against correlated-burst traffic, not just steady state; a time-boxed, audited exception mechanism for incident response; and admission-latency/rejection metrics on the same operational dashboard as capacity metrics.

13. Latest Industry Updates

Kubernetes 1.37 ("Garhwal"), GA August 26, 2026, stabilized MutatingAdmissionPolicy — the CEL-based, in-process mutation counterpart to the already-stable ValidatingAdmissionPolicy — meaning both halves of native admission policy are now stable in every current cluster, with no webhook infrastructure required for the rules they can express. This is the most directly relevant development to this post's topic and lands right at this post's publish date.

Kyverno's CEL-based policy type reportedly hit v1 in the 1.17 release this month, giving teams one policy language spanning admission, background scan, and cleanup, explicitly positioned as convergent with native VAP/MAP CEL syntax rather than competing with it — the legacy JMESPath-based ClusterPolicy/CleanupPolicy types are on a deprecation path, slated for removal in Kyverno v1.20 (October 2026), so teams still on JMESPath-only policies have a near-term migration deadline.

Kyverno passed OPA Gatekeeper in new production adoption through 2026, per multiple ecosystem surveys, attributed to its Kubernetes-native YAML policy model requiring no separate language and its GA mutate/generate support — while Gatekeeper (CNCF Graduated, ahead of Kyverno's Graduated status by several years) remains the right choice for teams already invested in Rego or needing a policy engine that spans beyond Kubernetes.

AWS Backup, cross-region replication, and CNCF's K8gb project (referenced in prior coverage of the DR space) continue maturing in parallel — relevant here because policy-as-code and DR are converging concerns: Kyverno's generate policies are increasingly used to enforce that every new StatefulSet or PVC-backed workload carries the correct backup-schedule labeling automatically, closing exactly the kind of "nobody was backing this up" gap structural-enforcement tools like Kyverno exist to prevent, tying this post's topic directly back to the reliability posture of the platform beyond the security domain alone.

The CNCF Policy Working Group's PolicyReport CRD standard continues to be the reporting format Kyverno ships natively; Gatekeeper's own reporting mechanism remains non-PolicyReport-compatible as of 2026, with the Gatekeeper team's alignment work ongoing but not GA — a real, current interoperability gap worth knowing if a platform runs (or is migrating between) both engines.

14. Summary & Cheat Sheet

Key concepts. Admission control runs after authn/authz, in two ordered phases (mutating, then validating), synchronously in the request path — any external call in that path exposes the entire cluster's write throughput to that dependency's own reliability. Native ValidatingAdmissionPolicy/MutatingAdmissionPolicy (both stable as of Kubernetes 1.37) run in-process via CEL with no network hop, a hard cost budget, and no external-call capability by design. Kyverno and OPA Gatekeeper remain necessary for anything CEL structurally can't do alone: mutate/generate chaining, external service calls, cross-resource context beyond paramRef, and (for Kyverno specifically) native PolicyReport-based audit reporting.

Architecture pattern. Order admission by cost: built-in plugins → native CEL (VAP/MAP) → external engine webhooks, narrowly scoped. Isolate any webhook with an external dependency onto its own WebhookConfiguration. Scope every webhook's namespaceSelector/objectSelector to its actual compliance boundary. Cache external-dependency lookups where bounded staleness is acceptable. Use matchConditions everywhere to skip non-matching requests before evaluation.

Commands worth memorizing.

kubectl get events -A --field-selector reason=FailedCreate -o wide
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o yaml
kubectl get validatingadmissionpolicies,validatingadmissionpolicybindings
kubectl describe clusterpolicy <name>          # Kyverno
kubectl get policyreport,clusterpolicyreport -A # Kyverno audit results
promql: apiserver_admission_webhook_admission_duration_seconds_bucket
promql: apiserver_admission_webhook_rejection_count

Troubleshooting checklist. Confirm the failure is admission-shaped (check events for failed calling webhook) before assuming a capacity/scheduling root cause. Identify the specific webhook by name from the error text. Check that webhook's own latency metric against the timing of the incident. If it has an external dependency, check that dependency's latency/error-rate in the same window. Confirm failurePolicy and namespace/object scope match actual intent, not just what a copied example shipped with. Use PolicyException (Kyverno) or a scoped, audited bypass for incident response — never blanket-disable the policy under pressure without a tracked, time-boxed record of having done so.

Design patterns to reuse. Layered admission by evaluation cost. Blast-radius isolation per external dependency. Staged rollout (canary → audit → enforce) for policy changes, mirroring application deployment discipline. Cache-then-attest as the near-term pattern for external-verification checks, with attestation-resolved-in-cluster as the longer-term target once CEL-checkable data replaces live external calls entirely.