Service Mesh Migration: Sidecar to Istio Ambient Mesh at Scale

Daily DevOps Mentor — 2026-08-31

1. Topic of the Day

Service mesh exists because mutual TLS, retries, circuit breaking, fine-grained authorization, and golden-signal telemetry are cross-cutting concerns that no team wants re-implemented per-language, per-service. The sidecar model (Istio pre-1.24, Linkerd, Consul Connect) solved this by injecting an L4/L7 proxy into every pod — correct, but expensive: every pod pays a second container's CPU/memory reservation, every hop pays two extra proxy traversals of latency, and every application restart is entangled with a sidecar's own lifecycle (the "sidecar hangs on kubectl rollout restart" class of incident that anyone running Istio sidecars at scale has hit at least once).

Ambient mesh is Istio's answer to that tax. It reached General Availability in Istio 1.24 (November 2024), and by 2026 it is the default recommendation for new Istio installations — ztunnel, waypoints, and the ambient APIs are all marked Stable by the Istio Technical Oversight Committee. Ambient splits the mesh into two independently-scaled layers instead of one monolithic per-pod proxy: ztunnel, a lightweight per-node L4 proxy handling mTLS and identity-based authorization for every pod on that node, and waypoint proxies, full Envoy instances deployed per-namespace or per-service-account, opted into only where L7 policy (HTTP routing, retries, fault injection, AuthorizationPolicy on paths/headers) is actually needed. The core architectural bet: most services never need L7 mesh features, they need mTLS and mesh telemetry — so make the expensive thing (a full Envoy proxy) optional and the cheap thing (an L4 tunnel) mandatory.

In enterprise production this shows up as the difference between a mesh rollout that stalls at 20% cluster adoption because platform teams can't justify doubling every pod's resource footprint, and one that reaches 100% adoption because the L4-only baseline costs a shared per-node DaemonSet instead of N sidecars. Organizations running Istio sidecars at fleet scale (thousands of pods) commonly report 50-70% memory reduction and meaningfully lower P99 latency after migrating to ambient, because most services drop from two extra proxy hops (client sidecar → server sidecar) to two ztunnel hops that do no L7 parsing at all.

Today's session covers why service mesh migrations fail in practice, how to design and execute a sidecar-to-ambient migration that doesn't page anyone, how HBONE and waypoints actually move a packet, and how ambient mesh stacks up against Cilium's eBPF-native mesh and Linkerd's lightweight-proxy model — including for the newest workload class asking for mesh identity: AI agent-to-agent traffic.

2. Real Business Problem

Symptom: A platform team runs Istio 1.19 sidecar mode across 40 clusters, ~6,000 services, ~35,000 pods. Three things are true simultaneously, and none of them individually looks like an emergency:

• Finance flags that the mesh's sidecar containers now account for roughly 18% of total cluster CPU/memory spend — pure proxy overhead, not application work — after a cost-allocation exercise tags istio-proxy containers separately for the first time.
• SRE's P99 latency dashboards show a mesh tax of 4-8ms per hop on the median service, which is invisible for a single call but compounds badly for the checkout service's 12-hop fan-out, where the mesh alone now accounts for a measurable slice of the SLA budget.
• Platform engineers dread istio upgrades: every control plane bump requires restarting every meshed pod to pick up the new sidecar image, which for a 35,000-pod fleet is a multi-day, staggered rollout that risks capacity dips during the restart wave, and any sidecar bug (a stuck preStop hook, an Envoy hot-restart bug) blast-radiuses to every single meshed pod at once.

The business ask arrives as: "cut mesh overhead without losing mTLS, authZ, or observability, and without an all-at-once flag-day cutover across 6,000 services owned by 40 different teams." That last clause is the real constraint — this is a migration problem as much as an architecture problem. A wrong migration strategy (mass relabeling a namespace at once, no coexistence window, no per-service rollback) turns a cost optimization into a self-inflicted multi-team outage. The correct fix has to let sidecar and ambient traffic interoperate mid-migration, because 6,000 services do not migrate in one maintenance window.

3. Production Architecture

Layered data plane. Ambient's core design decision is decomposing "the sidecar" into two independently deployed, independently scaled components. ztunnel runs as a DaemonSet, one pod per node, and handles every pod on that node: it terminates and originates mTLS using SPIFFE-format identities issued by istiod's CA (or an external CA/SPIRE via the Istio CA's pluggable signer), enforces L4 AuthorizationPolicy (identity, namespace, port), and load-balances at L4 — all without ever parsing HTTP. Waypoint proxies are full Envoy deployments, one per namespace or per service account depending on granularity needs, deployed only where a workload's AuthorizationPolicy, VirtualService, DestinationRule, retry policy, or WASM plugin actually requires L7 visibility. A namespace with no waypoint still gets mTLS, mesh telemetry, and L4 policy for free from ztunnel alone — this is the mechanism that lets 80% of a fleet run mesh-secured with zero per-pod proxy cost.

Data flow and the HBONE tunnel. All ambient mesh traffic between nodes is encapsulated in HBONE (HTTP-Based Overlay Network Environment) — an HTTP/2 CONNECT tunnel carrying the original TCP stream, authenticated with mTLS between ztunnels. For L4-only traffic: source pod → source-node ztunnel --HBONE(mTLS)--> dest-node ztunnel → dest pod, with the source ztunnel picking the destination based on Kubernetes Service/Endpoints the same way kube-proxy would, but with mesh identity attached to the connection instead of only IP:port. For L7-managed traffic, the flow gains a hop: source pod → source ztunnel --HBONE--> waypoint (Envoy, L7 policy + routing) --HBONE--> dest ztunnel → dest pod. Crucially, the waypoint hop is inserted only for traffic to a workload that has a waypoint attached (via istio.io/use-waypoint label) — traffic between two waypoint-less services never touches an Envoy L7 proxy at all.

Security boundaries. Identity is SPIFFE-format (spiffe://cluster.local/ns/<namespace>/sa/<service-account>), issued by istiod's built-in CA by default, or delegatable to an external CA/SPIRE deployment for organizations that already run SPIFFE/SPIRE as a cross-platform identity substrate (common where the same workload identity needs to span Kubernetes and non-Kubernetes VMs). ztunnel enforces mTLS and L4 AuthorizationPolicy in-kernel-adjacent (userspace but zero-copy where the kernel supports it) on every single packet, meaning zero-trust identity enforcement is mandatory and free, not an opt-in sidecar you might forget to inject. Waypoints add defense-in-depth at L7: header-based authz, JWT validation, and request-level audit logging for services carrying regulated data — HIPAA/PCI workloads should specifically get a per-service-account waypoint rather than sharing one at the namespace level, so an authz policy misconfiguration for one service account cannot leak into another's L7 boundary.

HA and DR. ztunnel's DaemonSet nature means a single node failure only affects that node's pods — there's no shared-fate blast radius across nodes the way a buggy control-plane push can affect all sidecars simultaneously. Waypoints are ordinary Deployments and get PodDisruptionBudgets and HPA like any other workload; because they're shared across a namespace's pods, they need to be treated as a semi-critical shared dependency (unlike a sidecar, which fails independently per-pod) — a waypoint crash-looping degrades L7 policy enforcement for every workload routed through it until it recovers or kube-proxy/ztunnel falls back to L4-only forwarding rules. For multi-region: Istio's Ambient Multicluster (Beta as of KubeCon EU 2026, Amsterdam) extends cross-cluster mesh identity and east-west gateway routing to ambient the same way multi-primary/primary-remote topologies worked for sidecar Istio, letting a waypoint in one cluster apply policy to traffic destined for another region without requiring every remote pod to carry a sidecar.

Why this shape, and how it evolves. At small scale (a few hundred pods, one team), plain sidecar Istio or even no mesh (mTLS via cert-manager + application-level retries) is simpler and the ambient control plane's extra moving parts (ztunnel DaemonSet, waypoint lifecycle, HBONE debugging unfamiliarity) aren't worth it yet. The inflection point is the same shape as the ArgoCD- and GPU-scheduling sessions': once per-pod sidecar cost (CPU/memory reservation × pod count, plus upgrade blast radius) exceeds the operational cost of learning ztunnel/waypoint debugging, migrate. At 10x scale, the granularity question shifts from "namespace vs cluster-wide waypoint" to "per-service-account waypoint for every workload with distinct authz requirements," and multicluster ambient stops being optional the moment mesh identity needs to span more than one region's failure domain.

4. Solution Design

Design decision: ambient vs. staying on sidecar vs. switching to Cilium's mesh vs. Linkerd. If the organization already runs Istio sidecar and the primary complaint is resource overhead and upgrade blast radius (not the mesh's feature set), ambient is the correct default in 2026 — it keeps the same CRDs (VirtualService, DestinationRule, AuthorizationPolicy), the same control plane (istiod), and the same operational knowledge, while removing the sidecar tax. If the cluster is greenfield and Cilium is already the chosen CNI, Cilium's own service mesh (eBPF-native, no per-pod or per-node userspace proxy for L4, lowest possible L4 latency) avoids running two overlapping networking stacks — CNI and mesh — with separate control planes, at the cost of a less mature L7 policy story than Istio/Envoy's. If the team is small, wants the simplest possible mental model, and doesn't need Istio's breadth of traffic-management CRDs, Linkerd remains the right "genuine simplicity" choice, with its own lightweight Rust micro-proxy per pod (still a sidecar, but a much lighter one than Envoy) — the trade-off there is Linkerd's commercial buildpack for the multi-cluster/HA feature depth Istio provides for free in the OSS project.

Alternative approaches considered and rejected.

• Big-bang cutover (relabel every namespace to ambient in one change window). Rejected outright for any fleet above a few dozen services — ambient changes the L7 policy attachment model (VirtualService/AuthorizationPolicy binding to a waypoint instead of a sidecar), and a big-bang cutover means discovering every service's hidden dependency on sidecar-specific behavior (e.g., EnvoyFilter customizations that assume a sidecar's listener architecture) all at once, in production.
• mTLS via cert-manager + app-level libraries, no mesh at all. Valid for small, single-language shops. Rejected at the scale described in Section 2 because it pushes retries, circuit breaking, and consistent authz enforcement into N different service codebases in N languages — exactly the duplicated-effort problem meshes exist to solve — and loses the uniform mesh telemetry that made the original sidecar rollout worthwhile in the first place.
• Keep sidecars, just tune resource requests down. A common first instinct from finance pressure. Rejected as a durable fix: under-provisioned sidecars manifest as worse tail latency under load (Envoy queuing/GC pressure) precisely when the mesh's retry/circuit-breaking behavior matters most, trading a steady-state cost problem for an intermittent reliability problem.

Scalability, cost, security, performance implications. Ambient's per-node ztunnel cost is roughly constant regardless of pod density per node, so cost scales with node count, not pod count — the more pods-per-node a fleet runs (bin-packing-optimized clusters, which is most cost-optimized fleets), the bigger the ambient win over per-pod sidecars. Security-wise, moving mTLS enforcement to a node-level DaemonSet means a ztunnel compromise has a larger per-node blast radius than a single sidecar compromise would — this needs to be explicitly named in a threat model, and argues for node-level hardening (dedicated node pools for anything security-sensitive, seccomp/AppArmor profiles on ztunnel itself) as a compensating control. Performance-wise, L4-only traffic sees close to bare-TCP latency; L7-managed traffic through a waypoint sees Envoy-equivalent latency to the old sidecar model for that hop, but only for the fraction of traffic that actually needs it — the net fleet-wide P99 improvement comes from the majority of hops that no longer touch Envoy at all.

5. Deep Technical Walkthrough

Internal working — how a pod gets "into" ambient without a sidecar. Labeling a namespace istio.io/dataplane-mode: ambient triggers the ambient CNI plugin (a chained CNI plugin, not a webhook-based injector like sidecar mode) to program pod-level networking so traffic is redirected to the node's local ztunnel — no init container, no sidecar injection, no pod restart required to join the mesh for L4 purposes (this is one of ambient's biggest migration wins: kubectl label namespace foo istio.io/dataplane-mode=ambient enrolls existing running pods without evicting them). Redirection happens at the node network layer: ztunnel programs iptables/eBPF rules so that pod-to-pod traffic transparently traverses the local ztunnel without the application or its Kubernetes manifest ever being aware a proxy exists.

Control plane interactions. istiod remains the single control plane for both sidecar and ambient workloads simultaneously — this is what makes coexistence during migration possible. istiod pushes ztunnel its per-node configuration (which pods are in-mesh, their SPIFFE identities, L4 AuthorizationPolicy) over the same xDS protocol used for sidecars, and separately pushes waypoints their Envoy configuration (VirtualService, DestinationRule, L7 AuthorizationPolicy) exactly as it would push a sidecar — a waypoint is, from istiod's perspective, just another Envoy data-plane target for xDS, deployed as a Deployment instead of injected as a sidecar.

Data plane interactions — the HBONE handshake. When ztunnel needs to send traffic to a remote pod, it opens (or reuses, via connection pooling) an HTTP/2 CONNECT tunnel to the destination node's ztunnel, authenticated by mTLS with both sides presenting SPIFFE certs from istiod's CA. The original TCP stream is tunneled inside that HTTP/2 CONNECT stream — HBONE deliberately reuses HTTP/2 multiplexing so many logical pod-to-pod connections share a small number of actual mTLS connections between any two nodes, which is a meaningful efficiency win at high pod density (avoiding a full TLS handshake per pod-pair, per node-pair). If a waypoint is in the path, it terminates the inbound HBONE tunnel from the source ztunnel, applies L7 policy as a normal Envoy listener/route/cluster chain, and originates a new HBONE tunnel toward the destination ztunnel — the waypoint is HBONE-aware on both sides, unlike a plain sidecar which only ever spoke mTLS directly pod-to-pod.

Failure scenarios and recovery. If ztunnel crashes on a node, the ambient CNI's redirection rules fail closed for new connections on that node by default (no silent fallback to unencrypted, unauthenticated traffic) — existing established connections may continue briefly depending on kernel state, but new connections queue or fail until ztunnel's DaemonSet controller restarts the pod, which is fast (ztunnel is a small, purpose-built Rust binary with a cold-start measured in low seconds, versus a full Envoy sidecar's heavier startup). If a waypoint crash-loops, workloads routed through it lose L7 policy enforcement and L7-dependent routing (retries, header-based splits) for the duration — L4 connectivity via ztunnel is unaffected because ztunnel doesn't depend on the waypoint being healthy to forward traffic to workloads that don't require L7 processing on that specific path, which is the isolation ambient is designed to provide (a broken waypoint degrades a scoped feature set, not raw connectivity for the entire namespace).

Performance bottlenecks and scaling behavior. ztunnel's own resource curve scales with the aggregate connection count and bandwidth on a node, not linearly with the DaemonSet's own workload — a "hot" node running many high-throughput pods needs a correspondingly larger ztunnel resource allocation, which is a capacity-planning axis platform teams historically didn't need to think about with per-pod sidecars (where resourcing was naturally per-workload). Waypoints scale like any Envoy Deployment: HPA on CPU/RPS, and the main bottleneck at very high fan-in is connection pool exhaustion to backend pods if DestinationRule connection pool settings are left at sidecar-era defaults sized for a 1:1 sidecar-to-backend relationship rather than a shared, namespace-wide waypoint fronting many more backends.

6. Production Troubleshooting

Walking a representative ambient migration incident the way a senior SRE would:

Symptom. After labeling the checkout namespace ambient and attaching a waypoint for its AuthorizationPolicy, checkout-api starts returning intermittent 503s under load, and a subset of calls from cart-service are unauthenticated-rejected (RBAC: access denied) that were working fine under sidecar mode.

Step 1 — confirm ztunnel is healthy and pods are actually redirected.

kubectl get pods -n istio-system -l app=ztunnel -o wide
kubectl exec -n istio-system ds/ztunnel -- ztunnel-tool logs --level debug | tail -100

# confirm the workload is actually captured (ambient-mode pods show a "ztunnel" annotation, not a sidecar container)
kubectl get pod -n checkout <pod> -o jsonpath='{.metadata.annotations.ambient\.istio\.io/redirection}'

If redirection shows disabled, the pod predates the namespace label and needs a restart to pick up ambient CNI redirection — unlike sidecar injection, ambient redirection for existing pods is applied at the CNI level on the next pod network setup, so a namespace-wide relabel does not retroactively redirect already-running pods without at least one restart cycle (rolling restart, not necessarily immediate).

Step 2 — check whether traffic is actually routing through the waypoint as expected.

kubectl get gtw -n checkout   # waypoint is provisioned as a Gateway API resource
kubectl logs -n checkout deploy/checkout-waypoint -c istio-proxy --tail=200 | grep -i "503\|upstream_reset"
istioctl proxy-config listener deploy/checkout-waypoint.checkout

A 503 with upstream_reset_before_response_started{connection_termination} from the waypoint, cross-referenced with the connection-pool sizing from Section 5, usually points to DestinationRule connection pool limits sized for the old per-pod sidecar traffic pattern, now bottlenecking a namespace-wide shared waypoint fronting far more aggregate concurrent connections than any single sidecar saw.

Step 3 — isolate the authz rejection.

istioctl x authz check deploy/checkout-api.checkout
kubectl get authorizationpolicy -n checkout -o yaml

The common migration bug here: an AuthorizationPolicy written for sidecar mode scopes source.principal using an identity format that assumed sidecar-terminated mTLS metadata propagation; under ambient, the same policy needs to be re-validated against ztunnel/waypoint's identity propagation, since the mTLS termination point moved. Cross-check the actual presented identity:

kubectl exec -n istio-system ds/ztunnel -- ztunnel-tool workload dump | grep -A3 cart-service

to confirm ztunnel sees the expected SPIFFE identity for cart-service — if it's missing or malformed, the root cause is usually a stale CA root propagation issue from a recent istiod CA rotation that hadn't finished distributing to every node's ztunnel yet.

Step 4 — root cause and remediation. In the composite incident: (1) the connection pool DestinationRule inherited from the sidecar-era config under-provisioned the shared waypoint for its new namespace-wide fan-in, and (2) two cart-service pods had been created after the namespace was labeled ambient but before a CNI daemon rollout completed on their node, leaving them briefly un-redirected and therefore invisible to ztunnel's identity propagation, which the AuthorizationPolicy interpreted as an unauthenticated caller. Remediation: resize the waypoint's DestinationRule connection pool for namespace-wide traffic (not per-pod sidecar-era sizing), add a readiness gate that blocks pod scheduling until the node's ambient CNI plugin reports ready, and add a pre-migration checklist item verifying CNI DaemonSet rollout status before labeling any namespace.

7. Hands-on Lab

Goal: stand up Istio ambient mode on a local kind cluster, run a workload with mTLS-only (no waypoint) traffic, then attach a waypoint and observe the added L7 hop — validating the coexistence and incremental-adoption model before touching production.

# 1. Create a kind cluster and install Istio with the ambient profile
kind create cluster --name ambient-lab
istioctl install --set profile=ambient --skip-confirmation

# confirm ztunnel and istiod are running
kubectl get pods -n istio-system

# 2. Deploy a sample two-service app (httpbin as the "backend", sleep as the "caller")
kubectl create ns demo
kubectl label namespace demo istio.io/dataplane-mode=ambient
kubectl apply -n demo -f https://raw.githubusercontent.com/istio/istio/release-1.24/samples/httpbin/httpbin.yaml
kubectl apply -n demo -f https://raw.githubusercontent.com/istio/istio/release-1.24/samples/sleep/sleep.yaml

# 3. Confirm mTLS is enforced with zero sidecars present (no istio-proxy container in `kubectl get pod`)
kubectl exec -n demo deploy/sleep -- curl -s http://httpbin:8000/headers
kubectl get pod -n demo -o jsonpath='{.items[*].spec.containers[*].name}'   # only app containers, no istio-proxy

# 4. Verify L4 mTLS is actually happening via ztunnel logs
kubectl logs -n istio-system ds/ztunnel | grep httpbin | tail -5

# 5. Now attach a waypoint to unlock L7 policy for httpbin specifically
istioctl waypoint apply -n demo --service-account httpbin --enroll-namespace=false

# 6. Add an L7 AuthorizationPolicy that requires a specific header — this only works once a waypoint exists
kubectl apply -n demo -f - <<'EOF'
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: httpbin-require-header
  namespace: demo
spec:
  targetRefs:
    - kind: Service
      group: ""
      name: httpbin
  action: ALLOW
  rules:
    - to:
        - operation:
            methods: ["GET"]
      when:
        - key: request.headers[x-lab-token]
          values: ["allow-me"]
EOF

# 7. Validate: request without the header is denied, with the header succeeds
kubectl exec -n demo deploy/sleep -- curl -s -o /dev/null -w "%{http_code}\n" http://httpbin:8000/get
kubectl exec -n demo deploy/sleep -- curl -s -o /dev/null -w "%{http_code}\n" -H "x-lab-token: allow-me" http://httpbin:8000/get

# Cleanup
kubectl delete ns demo
kind delete cluster --name ambient-lab

This lab deliberately sequences L4-only (no waypoint) before L7 (waypoint attached), because that sequence is the production migration pattern: land every workload on ztunnel-only ambient first (cheap, low-risk, no L7 behavior change), then attach waypoints incrementally, only to the specific services whose AuthorizationPolicy/VirtualService actually needs L7 semantics.

8. Production Case Study

Google — Istio's originator and still its largest production operator — has publicly discussed operating service mesh at a scale where per-pod sidecar overhead becomes a first-order cost line item, which is precisely the pressure that motivated ambient's L4/L7 split in the first place; Google Cloud's own GKE managed Istio (Anthos Service Mesh, and its ambient successor) exists specifically to absorb this operational complexity for customers who don't want to run istiod/ztunnel/waypoint lifecycle themselves. More broadly, the pattern of "separate a cheap mandatory L4 tier from an expensive optional L7 tier" mirrors how hyperscale networking teams (the lineage running from Google's own internal Traffic Director/GFE architecture through to modern eBPF-native networking at Meta and Cloudflare) have long approached the same trade-off: put identity and encryption as close to the kernel/network layer as possible and reserve full request parsing for the minority of traffic that actually needs it.

The emerging 2026 case that's specific to this curriculum's AI infrastructure track is agentic AI traffic: as internal platforms stand up fleets of AI agents calling each other and calling tool/MCP servers, that traffic needs the exact same primitives service mesh already solved for human-facing microservices — mutual authentication between agents, authorization scoped to which agent can call which tool, and audit-grade telemetry of who-called-whom-with-what — but the traffic pattern (many short-lived, bursty agent-to-agent and agent-to-tool calls) is far more sensitive to per-hop proxy latency than typical east-west microservice traffic. Solo.io and others have specifically pointed to ambient's L4-cheap/L7-optional split as a better fit for this workload class than sidecar mesh, since most agent-to-tool calls need identity and mTLS but not full L7 traffic management, and the volume of short bursty calls makes sidecar's per-pod overhead proportionally worse.

9. Architecture Review

Strengths. Ambient genuinely decouples mesh adoption cost from mesh feature depth — a fleet gets mTLS and identity for the DaemonSet's flat cost, and pays the Envoy tax only where L7 policy is actually exercised. Coexistence with sidecar mode during migration is a real, load-bearing feature, not a marketing footnote — it's what makes an incremental, per-namespace migration across 40 clusters and 6,000 services actually tractable instead of a flag-day risk. Reusing istiod and the existing CRD surface means teams don't relearn a new control plane or a new policy language, which is a meaningfully lower migration cost than switching mesh vendors entirely.

Weaknesses. The two-tier model adds a genuinely new failure mode class (partial mesh degradation: L4 fine, L7 broken, or vice versa) that sidecar mode's simpler "the sidecar is up or it isn't" model didn't have, and it requires operators to build new debugging muscle around HBONE, ztunnel logs, and waypoint-vs-ztunnel routing — Section 6's incident is exactly this kind of subtlety. Shared waypoints introduce a namespace-wide blast radius and a capacity-planning axis (aggregate fan-in sizing) that per-pod sidecars never had, shifting risk from "many small, independently-failing units" to "fewer, larger, shared units" — a trade every platform team should make consciously, not by default.

What fails first at 10x scale. ztunnel's per-node resource sizing, if left at generic defaults, is the first thing to buckle — a node running an unusually high density of high-throughput pods needs ztunnel resourced for that node's aggregate traffic, and generic fleet-wide ztunnel sizing will produce exactly the kind of hot-node latency spike that's hard to distinguish from a network problem without already knowing to look at ztunnel specifically. Second: waypoint sprawl — if per-service-account waypoints proliferate without a clear ownership and lifecycle model, the fleet ends up with as many Envoy Deployments to manage as it had sidecars, just organized differently, eroding much of ambient's operational-simplicity win.

How it changes at 100 million users. Multicluster ambient (Beta as of 2026) becomes non-optional, and the waypoint tier likely needs its own dedicated node pools separate from application workloads, so a waypoint capacity incident can't starve application pods for the same node resources. The identity layer (SPIFFE issuance and rotation) becomes a hard dependency deserving the same SRE rigor as DNS or the CA itself — at this scale, a CA rotation bug (as hinted at in Section 6) is a mesh-wide outage, not a namespace-scoped one, and needs staged rollout and canary verification with the same seriousness as a control-plane version upgrade.

What I'd redesign. Waypoint capacity planning should be a first-class, automated function of downstream fan-in (derived from Service/EndpointSlice cardinality and observed RPS) rather than manually-set HPA targets inherited from sidecar-era assumptions — this is the single most common source of the Section 6 failure mode and it's mechanically fixable rather than a fundamental architecture limitation.

10. Best Practices

Reliability calls for running ztunnel with resource requests sized to per-node aggregate traffic rather than generic fleet defaults, and treating waypoints as tier-1 shared infrastructure with their own PDBs, HPA, and on-call ownership — not an afterthought Deployment. Scalability favors starting every migration with L4-only ambient adoption fleet-wide before attaching a single waypoint, so the highest-value, lowest-risk win (sidecar removal) lands first and L7 migration proceeds service-by-service against real production traffic. Observability should preserve the sidecar-era mesh dashboards (per-service golden signals, mTLS success rate) through the migration by validating that ztunnel and waypoint telemetry map to the same Prometheus metric names and Grafana panels the org already trusts — a migration that silently degrades observability is a migration nobody will trust for the next one. Security demands validating every AuthorizationPolicy's identity assumptions against ambient's actual identity propagation path (Section 6) before relying on it in production, and treating waypoint-less namespaces as L4-only, not "unmeshed" — they still get mTLS and telemetry, so it should be an explicit, informed decision when a namespace stays L4-only rather than a status nobody re-evaluates. Cost optimization is largely automatic once migration completes, but should be measured explicitly (per-namespace CPU/memory before/after) to make the finance case for continuing the rollout across the remaining fleet. Operational excellence means keeping a documented, tested rollback path (relabel back to sidecar injection) for every migration wave until that wave has run in production long enough to trust it.

11. Common Production Mistakes

Relabeling an entire large namespace to ambient in one step without validating a canary subset first, discovering AuthorizationPolicy identity assumptions were sidecar-specific only after the whole namespace is affected. Assuming a namespace-wide waypoint deployment is ambient's default behavior — it isn't; waypoints are opt-in per-workload via istio.io/use-waypoint, and skipping that step means teams sometimes believe they've migrated to full ambient with L7 policy intact when only L4 features are actually active, silently dropping any VirtualService/L7 AuthorizationPolicy enforcement they had under sidecar mode. Inheriting sidecar-era DestinationRule connection pool settings unchanged onto a shared waypoint, producing the fan-in bottleneck from Section 6. Forgetting that ambient CNI redirection needs a pod restart cycle to take effect on already-running pods, then being confused when a namespace relabel appears to do nothing. Treating ztunnel like "just another DaemonSet" for resourcing purposes instead of capacity-planning it against actual per-node traffic aggregate, which is the architectural review's first-to-fail item at scale.

12. Interview Preparation

Q: Explain HBONE and why Istio ambient needed a new tunneling protocol instead of reusing plain mTLS like sidecar mode did. A strong answer covers: HBONE (HTTP-Based Overlay Network Environment) tunnels the original TCP stream inside an HTTP/2 CONNECT request between ztunnels, reusing HTTP/2 multiplexing so many logical pod-to-pod flows share few actual mTLS connections between node pairs — a meaningful efficiency win once identity moved from per-pod (sidecar terminates mTLS locally) to per-node (ztunnel terminates for many pods), where a naive one-mTLS-connection-per-pod-pair model would explode connection counts at high pod density.

Q: Design a zero-downtime migration plan from Istio sidecar to ambient for a 6,000-service fleet across 40 clusters. What's your sequencing and how do you validate each wave? Expect: coexistence-first (sidecar and ambient can run simultaneously fleet-wide, per Section 3), L4-before-L7 sequencing (label namespaces ambient without waypoints first, validate mTLS/telemetry parity against pre-migration dashboards, then attach waypoints service-by-service only where L7 policy is actually used), canary a small, low-risk namespace first, explicit rollback path per wave, and a hard pre-check on CNI DaemonSet rollout status before any namespace label change (per Section 6's root cause).

Q: When would you choose Cilium's service mesh over Istio ambient, and when would you choose Linkerd over both? Cilium when the CNI is already Cilium and the priority is lowest possible L4 latency via eBPF with a single unified networking control plane instead of running CNI and mesh as separate stacks; Istio ambient when rich L7 traffic management (Envoy's ecosystem, VirtualService/AuthorizationPolicy breadth) matters and the org either already runs Istio sidecar or is building fresh with L7 needs beyond basic mTLS; Linkerd when the team is small, wants the simplest possible mental model, and doesn't need Istio's CRD surface — while budgeting for its commercial layer if multicluster/HA depth is needed.

Q: A waypoint is crash-looping in production. What's the actual customer-facing impact, and why is it different from a crashed sidecar under the old model? Expect: L7 policy and L7-dependent routing (retries, header-based splits, path-based authz) for workloads routed through that specific waypoint degrade or fail, but L4 connectivity via ztunnel is unaffected for traffic that doesn't require L7 processing — a fundamentally smaller and more scoped blast radius than a crashed sidecar (which broke everything for that one pod, including basic connectivity), but the flip side is a shared waypoint's blast radius spans every workload that opted into it, versus a sidecar's single-pod scope.

Q: How does ambient's identity model interact with SPIFFE/SPIRE for organizations running non-Kubernetes workloads? Expect: Istio issues SPIFFE-format identities via istiod's built-in CA by default, but the CA is pluggable — organizations already running SPIRE as a cross-platform (VM + Kubernetes) identity substrate can delegate certificate issuance to SPIRE so ambient mesh workloads share a single trust domain and identity model with non-Kubernetes workloads authenticated via the same SPIFFE identities, avoiding a split-brain identity system across the platform.

13. Latest Industry Updates

Istio's ambient mode has been Stable/GA since 1.24 (November 2024), and by 2026 is the recommended default for new Istio deployments, with ztunnel performance reported to have improved significantly release-over-release since GA. At KubeCon + CloudNativeCon Europe 2026 in Amsterdam, the Istio project announced Ambient Multicluster reaching Beta, extending cross-cluster mesh identity and waypoint policy to multi-cluster topologies, plus movement on Gateway API Inference Extension support — relevant to this curriculum's AI infrastructure track, since it standardizes how Gateway API-based routing understands LLM inference-specific semantics (token-based load signals, model-aware routing) rather than treating every backend as a generic HTTP service. On the competitive landscape, Cilium's service mesh continues to mature as the default choice for greenfield Cilium-CNI clusters wanting a single eBPF-native networking and mesh stack, and Gateway API itself continues displacing Ingress as the standard L7 entry point across all three mesh ecosystems (Istio, Cilium, Linkerd), which matters because waypoints themselves are provisioned as Gateway API resources rather than a bespoke Istio-only CRD — a deliberate alignment with the broader Kubernetes networking API standardization effort.

14. Summary & Cheat Sheet

Key concepts: ztunnel (per-node L4 DaemonSet, mandatory, mTLS + L4 authz for every pod), waypoint (per-namespace/per-service-account Envoy, optional, L7 policy and routing), HBONE (HTTP/2 CONNECT-based mTLS tunnel between ztunnels, multiplexed for connection efficiency), SPIFFE identity (spiffe://cluster.local/ns/<ns>/sa/<sa>, pluggable to SPIRE), coexistence (sidecar and ambient run simultaneously under one istiod, the mechanism that makes incremental migration possible).

Architecture pattern: mandatory cheap L4 tier (ztunnel) + optional expensive L7 tier (waypoint), attached per-workload via istio.io/use-waypoint, both driven by the same control plane and CRD surface as sidecar mode.

Migration checklist: validate CNI DaemonSet rollout health before any namespace label change → label a canary namespace ambient (L4-only, no waypoint) → validate mTLS/telemetry parity against existing dashboards → roll out L4-only ambient fleet-wide → attach waypoints service-by-service only where L7 policy is actually used → re-validate every AuthorizationPolicy's identity assumptions post-migration → resize DestinationRule connection pools for namespace-wide waypoint fan-in, not sidecar-era per-pod sizing → keep a tested rollback path per wave.

Troubleshooting checklist: confirm ztunnel health and pod redirection status first (ambient.istio.io/redirection annotation) → check whether traffic is routing through the expected waypoint (istioctl proxy-config listener) → isolate authz rejections with istioctl x authz check and cross-check ztunnel's observed identity (ztunnel-tool workload dump) → check for CA rotation propagation lag across ztunnels → check waypoint connection pool exhaustion (upstream_reset_before_response_started) before assuming an application bug.

Decision matrix: Istio ambient — rich L7 needs, existing Istio investment, sidecar overhead is the pain point. Cilium mesh — greenfield, Cilium CNI already chosen, lowest L4 latency priority. Linkerd — small team, simplest mental model, L7 needs are minimal, commercial tier budgeted for HA/multicluster depth.