Daily DevOps Mentor — 2026-08-25

Topic: Cilium & eBPF — Replacing kube-proxy for Kubernetes Networking at Scale

Cilium eBPF Kubernetes datapath architecture

1. Topic of the Day

Cilium is a CNCF-graduated Container Network Interface (CNI) plugin built on eBPF (extended Berkeley Packet Filter) that replaces the traditional iptables/IPVS-based Kubernetes service datapath with programs that run inside the Linux kernel. It provides three things in one control plane: pod networking (CNI), service load-balancing (kube-proxy replacement), and identity-aware network policy enforcement — plus, via Hubble, flow-level observability without sidecars.

Why it exists: kube-proxy's default iptables mode (and even IPVS mode) implements Service VIP routing as a chain of Netfilter rules that is evaluated sequentially per packet. As Service and Endpoint counts grow past a few thousand, rule-chain traversal becomes the dominant cost on the data path — sync latency after an Endpoint change grows superlinearly, and per-packet CPU cost climbs. eBPF programs attached to tc, XDP, and sockops hooks replace that chain traversal with O(1) hash-map lookups executed directly in kernel context, and they can intercept traffic at the socket layer before a packet is even constructed — bypassing Netfilter/conntrack entirely for pod-to-pod traffic on the same node.

This matters in production because the failure mode isn't cosmetic — it shows up as DNS timeouts, SNAT port exhaustion, and multi-second Service update propagation delay in any cluster running more than roughly 2,000–3,000 Services, which is a completely ordinary size for a mid-size platform team, let alone a multi-tenant one.

Cilium is the default CNI/dataplane for GKE Dataplane V2, is offered as a first-class CNI option on EKS and AKS, and is in production at Adidas, Datadog, GitLab, Form3, Bell Canada, and most large-scale AI infrastructure shops that also need eBPF-based security (Tetragon) for GPU node fleets. As of Kubernetes 1.35, IPVS mode in kube-proxy itself has been marked deprecated in favor of nftables — meaning the entire iptables/IPVS conversation is heading toward EOL, and eBPF dataplanes are increasingly the "senior engineer default," not an exotic choice.

2. Real Business Problem

Scenario: A platform team runs a multi-tenant EKS cluster that has grown from 200 Services at launch to 4,800 Services and 38,000 Endpoints as more product teams onboarded. kube-proxy runs in iptables mode. Symptoms start appearing over a two-week period:

Intermittent DNS resolution failures (i/o timeout from CoreDNS) under load — roughly 1 in 2,000 lookups, worse during deploy storms.
New Service/Endpoint changes take 8–12 seconds to propagate to all 400 nodes instead of the sub-second latency seen at launch, because iptables-restore on each node has to re-render and atomically swap a ruleset with hundreds of thousands of rules.
Nodes running high-churn batch workloads intermittently show nf_conntrack: table full, dropping packet in dmesg, and connections silently fail — this is SNAT port exhaustion combined with conntrack table pressure, a classic symptom of NAT-based Service routing under high connection churn.
On-call is paged for a spike in 5xx errors from the checkout service that correlates with nothing in application logs — the failures are happening in the kernel networking stack, invisible to APM tools that only see request time inside the container.

This is the canonical "iptables doesn't scale" failure class that eBPF-based kube-proxy replacement is purpose-built to eliminate: O(1) map lookups instead of O(n) rule traversal, incremental map updates instead of full ruleset re-render, and (in strict mode) elimination of conntrack/SNAT for a large fraction of traffic via socket-level load balancing.

3. Production Architecture

                    ┌───────────────────────────────────────────────────┐
                    │                 Kubernetes Control Plane            │
                    │   kube-apiserver │ etcd │ scheduler │ CCM            │
                    └───────────────────────┬───────────────────────────┘
                                             │ watches Services, Endpoints,
                                             │ NetworkPolicies, CiliumIdentity
                                             ▼
        ┌───────────────────────────────────────────────────────────────┐
        │                        cilium-operator (Deployment)             │
        │  - IPAM (cluster-pool / ENI / Azure IPAM)                       │
        │  - CiliumIdentity allocation & garbage collection               │
        │  - CiliumEndpoint / CiliumNode CRD reconciliation                │
        │  - Derives kvstore-less state sync via CRDs (no etcd dependency) │
        └───────────────────────────┬───────────────────────────────────┘
                                     │ CRD watch (per node)
                                     ▼
        ┌───────────────────────────────────────────────────────────────┐
        │                cilium-agent (DaemonSet, 1 per node)             │
        │  - Compiles eBPF programs (BPF bytecode) per endpoint/policy     │
        │  - Loads programs onto tc hooks (ingress/egress on veth), XDP    │
        │    (on NIC driver for early drop / DSR), sockops (socket LB)     │
        │  - Maintains BPF maps: lb4_services, lb4_backends, ct (conntrack │
        │    replacement), policy map (identity → identity verdict)        │
        │  - Hubble Observer: exports flow events per node                 │
        └───────┬───────────────────────────────────────────┬────────────┘
                │ tc/XDP/sockops attach                       │ gRPC flow stream
                ▼                                              ▼
   ┌─────────────────────────────────┐            ┌─────────────────────────┐
   │   Linux Kernel eBPF Datapath      │            │   Hubble Relay (Deploy)  │
   │  Pod A veth ──tc──▶ policy lookup │            │  aggregates per-node     │
   │  (BPF map, O(1)) ──▶ NAT/LB       │            │  flow streams cluster-   │
   │  rewrite ──▶ forward (native      │            │  wide → Hubble UI/CLI    │
   │  routing or VXLAN/Geneve overlay) │            └─────────────────────────┘
   └─────────────────────────────────┘
                │
                ▼
   ┌─────────────────────────────────────────────────────────────────────┐
   │            Node NIC → underlay network → destination node             │
   │  Native routing mode: BGP-advertised pod CIDRs, no encapsulation       │
   │  Overlay mode: VXLAN/Geneve encapsulation (needed across L2 domains)   │
   └─────────────────────────────────────────────────────────────────────┘

Security boundaries:
- Identity-based policy: every pod gets a CiliumIdentity derived from its
  labels (not its IP). Policy enforcement is a map lookup on
  (source identity, dest identity, port) — IP churn from pod restarts does
  NOT require policy re-render, unlike IP-based NetworkPolicy backends.
- Encryption: WireGuard (kernel-native, lower overhead) or IPsec, transparent
  to the eBPF datapath, node-to-node.
- cilium-agent runs privileged (needs CAP_BPF/CAP_NET_ADMIN) — treat the
  DaemonSet as part of the trusted computing base; restrict who can edit it.
- ClusterMesh extends identity-aware policy across clusters over an
  authenticated, encrypted control-plane and data-plane connection.

HA / DR:
- cilium-agent is a DaemonSet — inherently per-node, no single point of
  failure for the datapath itself (a crashed agent leaves existing eBPF
  programs attached and forwarding; only *new* policy/service updates stall).
- cilium-operator runs 2+ replicas with leader election — only IPAM/identity
  GC is centralized, and it is not on the packet-forwarding critical path.
- Hubble Relay is horizontally scalable and independent of the datapath —
  losing it loses observability, not connectivity.
- Multi-region: Cilium ClusterMesh links clusters for cross-cluster service
  discovery and failover; region failover itself still lives at a higher
  layer (global load balancer / Route53 / Traffic Manager).

Why this shape: eBPF programs live in the kernel and are attached per-hook (tc, XDP, sockops), so the "control plane" (cilium-agent) only needs to push map updates, not re-render an entire ruleset. This is the structural reason updates are incremental and fast: adding one Endpoint is one map insert, not a full iptables-restore. Identity-based policy (labels → numeric identity → map key) means policy doesn't have to be recomputed when pod IPs change, which is the other structural win over IP-based NetworkPolicy implementations.

Trade-offs: You take on a kernel-version dependency (meaningful eBPF feature parity needs 5.10+, and some features like BPF host-routing / DSR want 5.13+ or a distro backport-heavy kernel like Bottlerocket/AL2023), and you introduce a new debugging surface — when something breaks, tcpdump alone won't show you why a packet was dropped; you need cilium monitor / hubble observe / cilium-dbg bpf tooling. Teams without kernel/eBPF familiarity underestimate this ramp.

At scale (5,000+ nodes, multi-tenant): Move to native routing (BGP, no overlay encapsulation overhead) where the underlay allows it, shard CiliumIdentity GC intervals to avoid operator churn, and enable bpf.masquerade + DSR to remove the SNAT hop entirely for LoadBalancer/NodePort traffic, which is the exact failure mode from Section 2.

4. Solution Design

Design decisions:

Full kube-proxy replacement (kubeProxyReplacement: true), not CNI-chaining alongside an existing kube-proxy — running both is a supported migration step, not an end state; it doubles the datapath surface and gives you two things to debug.
Native routing mode over VXLAN where the underlying VPC/subnet topology allows direct pod-CIDR routing (AWS ENI mode, or BGP peering with the fabric) — removes ~50 bytes/packet encapsulation overhead and an entire decap/hash step per packet.
Socket-level load balancing (sockops/sk_msg) enabled for pod-to-ClusterIP traffic on the same node — this is the mechanism that lets same-node east-west calls skip the network stack (and conntrack) almost entirely, addressing the SNAT exhaustion symptom directly.
Direct Server Return (DSR) for NodePort/LoadBalancer ingress paths — response traffic returns directly to the client instead of hairpinning back through the ingress node, cutting a network hop for high-fan-out services.
Default-deny CiliumNetworkPolicy per namespace, rolled out in audit mode first (policyEnforcementMode: default, verify via Hubble flow logs for unexpected drops) before flipping to always — the single most common Cilium production incident is enabling default-deny before verifying DNS/kube-apiserver egress is explicitly allowed.

Alternatives considered:

Calico (iptables or eBPF mode): mature, huge install base, has its own eBPF dataplane option with similar performance characteristics; choose it when the team already has deep Calico operational knowledge or needs Calico Enterprise's specific compliance tooling. Its eBPF mode is less feature-rich than Cilium's for L7-aware policy and service mesh.
AWS VPC CNI + kube-proxy: simplest on EKS, native ENI-per-pod IPs, but you keep iptables' scaling ceiling and get no identity-based policy — fine for small/medium clusters, wrong choice once Service count crosses the low thousands.
Cilium as CNI only, kube-proxy left running: lower blast radius during migration, but you get none of the performance win and still carry two datapaths — only justified as a temporary bridge state during a phased rollout.
Istio ambient mesh / Cilium Service Mesh (sidecar-less): if L7 mesh features (mTLS, traffic shifting, retries) are also needed, Cilium's eBPF can handle L3/L4 for the whole mesh while an ambient L7 proxy (ztunnel/Envoy) handles only the L7-needing subset — avoids the sidecar-per-pod tax entirely. Evaluate only after the L3/L4 replacement is stable; don't adopt both at once.

Cost implications: CPU savings from eliminating iptables chain traversal and SNAT/conntrack are typically the largest line item — teams commonly report 20–40% lower per-node CPU spent on network processing at high Service counts, which translates directly to smaller node counts for the same workload. Hubble adds a modest but real cost (flow export CPU + Relay/UI compute) that should be sized like any other observability pipeline, not treated as free.

Security implications: identity-based policy is a strict improvement over IP-based NetworkPolicy for dynamic environments (autoscaling, spot interruption, frequent redeploys) because policy correctness doesn't degrade during pod churn. WireGuard encryption is nearly free on modern kernels (in-kernel, hardware-accelerated on many NICs) versus IPsec's higher CPU cost — default to WireGuard unless FIPS compliance specifically requires IPsec.

Performance implications: the biggest win is tail latency stability under Service/Endpoint churn, not just average throughput — this is what actually fixes the DNS timeout and deploy-storm symptoms from Section 2, because map updates are atomic and incremental instead of requiring a full ruleset swap that briefly stalls all nodes syncing near-simultaneously.

5. Deep Technical Walkthrough

Request flow, same-node pod-to-ClusterIP call (the common case in a dense bin-packed cluster):

Application in Pod A calls http://checkout-svc.prod.svc.cluster.local.
CoreDNS resolves to the Service ClusterIP (not a backend Pod IP) — this part is unchanged from vanilla Kubernetes.
With sockops/sk_msg enabled, an eBPF program attached at the socket layer intercepts the connect() syscall before a packet is ever built. It looks up the ClusterIP in the lb4_services BPF map (hash map, O(1)), picks a backend via the configured algorithm (round-robin, maglev consistent-hashing for session affinity at scale), and rewrites the destination to the backend Pod IP directly at the socket level.
If both pods are on the same node, the kernel can splice the two sockets together without ever building an IP packet that traverses the veth/bridge/Netfilter stack — this is the "socket-level load balancing" performance mode, and it is why same-node Cilium traffic is dramatically cheaper than the equivalent iptables DNAT path.
For cross-node traffic, the tc (traffic control) hook on the egress veth performs the same map lookup, DNAT-rewrites the packet, and either sends it natively routed (BGP-advertised pod CIDR) or encapsulates it (VXLAN/Geneve) for the receiving node's tc ingress hook to decapsulate and deliver to the destination pod's veth.
Policy enforcement happens as a separate map lookup keyed on (source identity, destination identity, port/protocol) at the tc hook — identities are small integers derived from label sets by the operator, so this lookup is O(1) regardless of how many CiliumNetworkPolicy objects exist, unlike iptables policy chains which grow linearly with rule count.

Control plane sync flow:

kube-apiserver Service/Endpoint/NetworkPolicy changes are watched by every cilium-agent directly (informer pattern) — there is no separate kvstore dependency in the default CRD-backed mode (identity-allocation-mode: crd), which removes an entire external dependency (etcd/Consul) that older Cilium deployments required.
cilium-operator allocates/garbage-collects CiliumIdentity objects centrally (labels → identity mapping must be cluster-consistent) and reconciles IPAM (which IP ranges are assigned to which node).
Each cilium-agent translates the current desired state into BPF map updates and pushes them via the bpf() syscall — these are incremental, not full-table swaps, which is the core mechanism behind sub-100ms propagation versus iptables' multi-second full-ruleset-render behavior at scale.

Failure scenarios and recovery:

cilium-agent crash/restart: existing eBPF programs remain attached to their hooks (they live in the kernel independent of the userspace agent process), so already-established connections and policy continue to work. New pods scheduled during the outage window will be stuck in ContainerCreating (no CNI ADD can complete) until the agent recovers — this is the main blast radius, not existing traffic.
BPF map exhaustion: maps have fixed sizes set at agent startup (e.g., --bpf-ct-global-tcp-max); hitting the limit silently drops new connections rather than erroring visibly — this is a top production gotcha at high connection-churn workloads (e.g., serverless-style fan-out, or GPU inference gateways under load-testing) and requires proactive sizing, not reactive discovery.
eBPF verifier rejection on upgrade: a kernel upgrade or Cilium version bump can hit the in-kernel BPF verifier's complexity limits on older kernels, causing agent CrashLoopBackOff at startup — this is why Cilium publishes a supported kernel/version compatibility matrix and why you pin both together in upgrade planning.
MTU mismatch in overlay mode: VXLAN/Geneve adds ~50 bytes of overhead; if the underlay MTU isn't reduced accordingly on the pod-facing interface, you get silent fragmentation or blackholed large packets — a classic "works for small requests, breaks on large payloads" bug.

6. Production Troubleshooting

Symptom: intermittent connection failures reported by an application team, no clear pattern in app logs.

Step 1 — confirm datapath health cluster-wide:

cilium status --wait
# Look for: KubeProxyReplacement, Cluster health, Controller status (any "failing")
kubectl -n kube-system get pods -l k8s-app=cilium -o wide

Step 2 — check for BPF map pressure (the conntrack-exhaustion equivalent):

kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf ct list global | wc -l
kubectl -n kube-system exec ds/cilium -- cilium-dbg statedb bpf-maps
# Compare current entries against configured max (bpf-ct-global-tcp-max);
# >85% utilization is the alerting threshold to set proactively

Step 3 — trace the actual dropped flow with Hubble instead of guessing from app-side symptoms:

hubble observe --pod checkout-svc --verdict DROPPED --last 200
# Output includes the exact policy or datapath reason: "Policy denied",
# "Reassembly", "Invalid state", etc — this replaces hours of tcpdump
# correlation with a single, identity-aware, cluster-wide flow log

Step 4 — validate policy intent vs. reality (the default-deny footgun from Section 4):

cilium policy trace --src-k8s-pod default:checkout-svc-7d9 \
  --dst-k8s-pod prod:payments-api-2f1 --dport 443

Step 5 — end-to-end connectivity regression test (run before/after any Cilium version or kernel upgrade):

cilium connectivity test

Step 6 — for DNS-specific timeouts (the exact symptom from Section 2): check whether toFQDNs/DNS-aware policy is causing extra DNS proxy hops, and inspect cilium-dbg fqdn cache list; also confirm CoreDNS itself isn't CPU-throttled (kubectl top pod -n kube-system -l k8s-app=kube-dns) — eBPF fixes the kube-proxy layer, not application-layer DNS server capacity, and conflating the two wastes debugging time.

Senior-level habit: always reach for hubble observe --verdict DROPPED before tcpdump. Hubble gives you the why (which policy, which identity) in one command; raw packet capture only gives you the that.

7. Hands-on Lab

Goal: stand up a local cluster with Cilium as full kube-proxy replacement, verify the datapath, and observe policy enforcement with Hubble.

# 1. Create a kind cluster WITHOUT kube-proxy (Cilium will replace it entirely)
cat <<EOF | kind create cluster --name cilium-lab --config -
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
  kubeProxyMode: none
EOF

# 2. Install Cilium via Helm with kube-proxy replacement enabled
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --version 1.19.0 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=cilium-lab-control-plane \
  --set k8sServicePort=6443 \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

# 3. Verify the datapath is fully up and kube-proxy-free
cilium status --wait

# 4. Deploy a sample app and a default-deny policy in AUDIT mode first
kubectl create deployment checkout --image=hashicorp/http-echo -- -text="ok"
kubectl expose deployment checkout --port=80 --target-port=5678
cat <<EOF | kubectl apply -f -
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: checkout-default-deny
spec:
  endpointSelector:
    matchLabels:
      app: checkout
  ingress:
  - fromEndpoints:
    - matchLabels:
        role: frontend
EOF

# 5. Watch live flows to confirm what the policy actually allows/denies
cilium hubble port-forward &
hubble observe --pod checkout --follow

# 6. Run the built-in connectivity test suite
cilium connectivity test

# 7. Cleanup
kind delete cluster --name cilium-lab

Validation checkpoints: cilium status reports KubeProxyReplacement: True with no kube-proxy pods running anywhere in the cluster; hubble observe shows FORWARDED for traffic from correctly-labeled pods and DROPPED (policy denied) for everything else before you add explicit allow rules.

8. Production Case Study

Google (GKE Dataplane V2): GKE's default dataplane since 2020 is built directly on Cilium's eBPF stack, replacing kube-proxy cluster-wide for all new GKE clusters — Google's own stated motivation was identical to Section 2's failure mode: iptables scaling limits at their multi-tenant cluster sizes, plus the need for identity-based policy that doesn't degrade under the pod churn rate of a large autoscaled fleet.

Datadog: runs Cilium across its Kubernetes fleet specifically for the combination of kube-proxy replacement performance and Hubble-based network observability, treating flow-level visibility as a first-class SRE tool rather than a nice-to-have — their scale (many thousands of nodes) makes iptables rule-chain latency a hard operational blocker, not a theoretical concern.

Adidas: migrated large e-commerce Kubernetes platforms to Cilium explicitly to solve Service-count scaling limits during peak sales events (Black Friday-class traffic spikes), where iptables sync latency during rapid autoscale-driven Endpoint churn was directly implicated in checkout-path incidents — structurally the same root cause as Section 2's business problem, at bigger scale.

AI infrastructure shops (GPU fleets): teams running vLLM/Triton inference gateways with high east-west fan-out between routers and model replicas increasingly pair Cilium's eBPF dataplane with Tetragon (eBPF-based runtime security) to get both networking performance and syscall-level observability on GPU nodes without adding per-pod sidecar tax — relevant because GPU nodes are expensive enough that any CPU spent on sidecar proxies is a direct tax on inference throughput.

9. Architecture Review

Strengths: O(1) service lookup and policy enforcement regardless of scale; incremental map updates instead of full-ruleset re-render, which directly fixes propagation-latency incidents; identity-based policy that stays correct under pod churn; built-in flow-level observability (Hubble) without a service mesh sidecar tax; a credible path to sidecar-less L7 mesh if needed later.

Weaknesses: real operational learning curve — debugging requires eBPF-specific tooling (hubble observe, cilium-dbg bpf), and teams that only know iptables-save/tcpdump need retraining; kernel-version coupling means upgrade planning must jointly consider kernel LTS status and Cilium's supported matrix; BPF map sizing is a capacity-planning exercise most teams skip until they hit it in production.

What fails first at 10x scale: cilium-operator's CiliumIdentity GC and IPAM reconciliation becomes the new bottleneck once node count crosses several thousand — it's centralized (with leader election, not sharded), so identity churn from very high pod turnover rates (spot interruption storms, aggressive HPA) can queue up faster than the operator reconciles, delaying new pod scheduling even though the datapath itself is fine. BPF map size limits, set per-node at agent startup, also need re-tuning as connection-churn workloads grow — this doesn't autoscale by default.

What changes at 100M-user scale: ClusterMesh becomes mandatory (single-cluster stops being viable well before that), which introduces its own operational surface (cross-cluster identity sync, encrypted mesh links) that needs the same rigor as the single-cluster datapath; native routing (BGP) becomes essential since VXLAN encapsulation overhead compounds across a much larger east-west traffic volume; a dedicated network-platform team owning eBPF/kernel compatibility becomes a real org-chart line item, not a shared responsibility.

What to redesign: shard identity GC/IPAM reconciliation per node pool or per namespace tier rather than one global operator loop; move BPF map sizing into the same capacity-planning process as CPU/memory requests, with per-node-pool overrides for high-churn workload classes (batch, serverless-style, GPU inference gateways); treat kernel version as a first-class dependency pinned alongside the CNI version in upgrade runbooks, not an afterthought owned by a separate OS team.

10. Best Practices

Pin Cilium version to a kernel version explicitly validated in Cilium's own compatibility matrix, and upgrade both together in staging before production — never let kernel patching and CNI upgrades drift independently. Monitor BPF map utilization (cilium-dbg bpf counters exported via the Cilium Prometheus metrics, e.g. cilium_bpf_map_pressure) with alerting at 80%, not just at exhaustion. Prefer native routing over overlay encapsulation wherever the underlay topology allows it, and re-verify MTU end-to-end whenever overlay mode is unavoidable. Roll out CiliumNetworkPolicy default-deny in audit mode first, verify with hubble observe, and only then flip to enforcing — never enforce-first. Run cilium connectivity test as a required gate in CI/CD before promoting any Cilium version or Helm-values change to production. Enable WireGuard transparent encryption by default for node-to-node traffic unless a specific compliance requirement mandates IPsec. Size cilium-agent resource requests/limits based on node pod density, not a flat default — high-density nodes compile and hold more eBPF programs.

11. Common Production Mistakes

Enabling default-deny CiliumNetworkPolicy before explicitly allowlisting DNS (kube-dns/CoreDNS) and kube-apiserver egress — this is the single most common Cilium-adjacent outage, and it looks exactly like a DNS problem because it is one, just self-inflicted. Running Cilium and kube-proxy simultaneously as a permanent state instead of a short migration window, doubling the datapath surface with no corresponding benefit. Ignoring BPF map size defaults on high-connection-churn workloads (serverless-style fan-out, GPU inference gateways, aggressive load testing) and discovering the ceiling via silent connection drops in production rather than proactive sizing. Enabling IPsec or WireGuard encryption without first measuring the throughput/CPU hit on representative traffic, then being surprised by a capacity regression. Upgrading the Linux kernel without checking it against Cilium's supported-kernel matrix, causing eBPF verifier rejections and agent CrashLoopBackOff at the worst possible time — during a routine OS patch cycle, not a planned migration. Treating Hubble as purely a nice-to-have dashboard rather than the primary troubleshooting tool, and defaulting back to tcpdump-first debugging that's slower and doesn't surface policy verdicts.

12. Interview Preparation

Q: Why does iptables-based kube-proxy stop scaling, precisely — what's the mechanism? A: iptables implements Service routing as a linear chain of Netfilter rules (one set per Service, referencing per-Endpoint DNAT rules); every packet traverses the chain sequentially until it matches, so per-packet cost grows with Service/Endpoint count. Beyond that, every Endpoint change requires iptables-restore to atomically replace the entire ruleset on every node (there's no incremental update primitive), so sync latency also grows with total rule count, not just the size of the delta.

Q: How does eBPF avoid that, mechanically? A: eBPF programs attached at tc/sockops/XDP hooks look up Service→Backend mappings in BPF hash maps, which are O(1) regardless of map size, and updates to those maps are incremental (single map insert/delete), not full-table swaps. Socket-level load balancing (sockops) goes further by rewriting the destination at connect() time, before a packet exists, which for same-node traffic can avoid the kernel network stack (and conntrack) almost entirely.

Q: Explain identity-based network policy and why it matters for autoscaled clusters. A: Cilium derives a small integer "identity" from a pod's label set via the operator; policy is enforced as a map lookup on (source identity, dest identity, port). Because identity is derived from labels, not IP, policy stays correct across pod restarts/rescheduling/IP churn without any policy recomputation — critical in clusters with high autoscaling or spot-interruption churn, where IP-based policy either lags reality or requires constant re-render.

Q: What's the practical difference between tc and XDP hooks? A: XDP runs earliest, at the NIC driver level, before the kernel allocates an sk_buff — it's used for very cheap early-drop decisions (DDoS mitigation, DSR return path) but has a more restrictive execution environment. tc hooks run after sk_buff allocation, with access to more kernel networking context, and are where most of Cilium's policy/NAT/LB logic lives because it needs that richer context.

Q: A cluster is seeing SNAT port exhaustion under Cilium. What are your first three diagnostic steps? A: (1) Confirm whether socket-level LB (sockops) is actually enabled and eligible for the affected traffic pattern — same-node hairpin traffic that should skip SNAT but isn't configured to. (2) Check whether DSR is enabled for the LoadBalancer/NodePort path in question, since DSR removes the SNAT hop on the return path. (3) Check BPF conntrack map utilization and the configured bpf-ct-global-tcp-max — if it's undersized for the workload's connection churn rate, that's a direct capacity fix, not an architecture problem.

Q: When would you not choose Cilium? A: When the team has deep existing Calico operational expertise and no acute scaling pain, when the cluster is small enough that iptables' O(n) cost is genuinely irrelevant, or when the underlying kernel fleet can't be brought to a version in Cilium's supported matrix in a reasonable timeframe (e.g., regulated environments frozen on old LTS kernels) — forcing an eBPF dataplane onto an unsupported kernel is a worse outcome than staying on a well-understood iptables setup.

13. Latest Industry Updates

Kubernetes 1.35 deprecates IPVS mode in kube-proxy, with nftables positioned as the supported replacement for teams staying on kube-proxy rather than moving to an eBPF dataplane — SIG-Network cited the difficulty of maintaining three separate backends (iptables, IPVS, nftables) as the driver. This matters directly to this topic: it confirms the industry-wide move away from Netfilter-chain-based service routing, whether teams land on nftables (better than iptables, still fundamentally chain-based) or skip straight to an eBPF dataplane like Cilium.

nftables mode has been stable since Kubernetes 1.33 and requires a 5.13+ kernel — relevant as a fallback/comparison point for teams not ready for a full eBPF migration but wanting to escape iptables' worst scaling characteristics.

Cilium 1.19 (current stable line as of this cluster's Kubernetes version) is validated against Kubernetes 1.31–1.34 and requires Linux kernel 5.10+ (or an equivalent distro-backported kernel such as RHEL 8.10's 4.18). Teams planning a Kubernetes 1.35 upgrade should confirm Cilium's compatibility matrix before moving, per the kernel/version coupling discussed in Section 10.

Sidecar-less service mesh continues converging on eBPF-plus-ambient-proxy architectures — Cilium's own Service Mesh and Istio's ambient mode both increasingly delegate L3/L4 to an eBPF dataplane and reserve per-pod sidecar-equivalent proxying only for traffic that actually needs L7 features (mTLS, retries, traffic shifting), which is the direction most large platform teams are now defaulting to instead of blanket sidecar injection.

Tetragon (Cilium's sister eBPF runtime-security project) is seeing growing adoption specifically on GPU inference node fleets, where teams want syscall-level security observability without adding per-pod sidecar CPU tax on already-expensive GPU nodes — a direct extension of the same eBPF investment made for networking.

Sources:

14. Summary & Cheat Sheet

Core concept: eBPF replaces Netfilter chain traversal (O(n), full-ruleset re-render on change) with kernel hash-map lookups (O(1), incremental updates) for Service routing, and replaces IP-based policy with label-derived identity-based policy that survives pod churn.

Architecture in one line: cilium-agent (DaemonSet) compiles and loads eBPF programs onto tc/XDP/sockops hooks per node; cilium-operator centralizes IPAM and identity GC; Hubble exports flow-level observability without sidecars.

Key commands:

cilium status --wait                         # datapath + kube-proxy-replacement health
hubble observe --verdict DROPPED --last 200  # why a flow was dropped (use before tcpdump)
cilium-dbg bpf ct list global                # conntrack-equivalent BPF map contents
cilium policy trace --src-k8s-pod ... --dst-k8s-pod ...  # policy decision trace
cilium connectivity test                     # end-to-end regression gate for upgrades

Troubleshooting checklist: (1) cilium status for datapath health → (2) BPF map pressure for silent drops → (3) hubble observe for policy verdicts → (4) cilium policy trace for expected-vs-actual → (5) MTU check if overlay mode and large-payload failures → (6) kernel/Cilium version compatibility if failures started right after an OS or Cilium upgrade.

Design patterns: native routing over overlay where possible; socket-level LB + DSR to eliminate SNAT hops; identity-based default-deny rolled out audit-first; WireGuard over IPsec by default; kernel version pinned alongside CNI version in upgrade runbooks.

Failure mode to remember: BPF map size limits and cilium-operator identity GC/IPAM reconciliation are the new bottlenecks at extreme scale — the datapath itself doesn't degrade the way iptables does, but the control-plane-adjacent pieces still need capacity planning.

Cilium & eBPF — Replacing kube-proxy for Kubernetes Networking at Scale