title: "CI/CD at Enterprise Scale: Actions Runner Controller, Ephemeral Runners, and Distributed BuildKit Caching on Kubernetes" date: 2026-09-14 tags: [Kubernetes, CI/CD, GitHub Actions, ARC, Platform Engineering, BuildKit, Karpenter, DevOps] cover: ../images/cicd-actions-runner-controller-buildkit-caching-cover.png

CI/CD at Enterprise Scale: Actions Runner Controller, Ephemeral Runners, and Distributed BuildKit Caching on Kubernetes

1. Topic of the Day

Every platform team eventually hits the same wall: CI/CD that worked fine at 50 repositories falls over at 500. GitHub-hosted runners are simple and zero-ops, but they cap out on concurrency, can't touch VPC-internal resources without exposed self-hosted fallbacks, and get expensive fast at scale (per-minute billing on shared infrastructure with no control over instance shape, no persistent cache, and no path to GPU runners for ML build/test workloads). The answer every large-scale org converges on is the same one we've used elsewhere in this series for stateful, bursty workloads: run it on Kubernetes, and let the platform own the scaling.

Actions Runner Controller (ARC) is GitHub's own Kubernetes operator for self-hosted runners — it watches a AutoscalingRunnerSet (the modern gha-runner-scale-set Helm chart, the successor to the older, now-deprecated RunnerDeployment/HorizontalRunnerAutoscaler CRDs) and creates ephemeral runner pods on demand, one per queued job, scaling to zero when idle. This matters because it converts CI capacity from "a fixed fleet of VMs you pay for 24/7" into "a Kubernetes workload that behaves like every other autoscaled workload on the cluster" — the same node pools, the same Karpenter or Cluster Autoscaler provisioning logic, the same bin-packing and spot-instance economics you already run for application traffic.

The second half of the problem — and the one teams solve badly far more often — is that ephemeral runners have no persistent local disk. A runner pod that lives for the duration of exactly one job cannot rely on Docker's local layer cache, npm's node_modules cache, or a compiler's incremental build cache surviving between runs, because there is no "between runs" on the same filesystem. Solving that requires a distributed build cache — BuildKit's registry-backed cache being the dominant pattern — that lives outside any individual runner and gets pulled fresh by whichever ephemeral pod picks up the next job. Getting both pieces right — ARC for elastic compute, BuildKit remote caching for build speed — is what separates a CI platform that scales linearly with organizational growth from one that becomes the top complaint in every engineering all-hands.

2. Real Business Problem

Scenario: A 400-engineer product org runs 900 repositories against GitHub-hosted Actions runners. Growth from 150 to 900 repos over eighteen months surfaces the following, in order:

• Queue times balloon during peak hours. Standard GitHub-hosted runners are capped per-org on concurrency (the default concurrency limits scale with plan tier, but even generous limits get saturated when every team's CI fires on every push during a 10am-to-noon commit rush). PRs start waiting 8-12 minutes just to get a runner assigned, before a single test executes.
• Build times are dominated by cold cache, not compute. Every GitHub-hosted job starts from a pristine VM. A monorepo's Docker image build re-downloads base layers and re-compiles unchanged dependency layers on every single run, because GitHub-hosted runners' opt-in actions/cache action has size limits and eviction policies that don't hold up for multi-gigabyte image layer caches at this scale — cache misses become the norm, not the exception.
• GPU-dependent test suites have nowhere to run. The ML platform team's model-serving repo needs a GPU to run its integration test suite (inference correctness checks against a real model). GitHub-hosted runners offer no GPU tier at the org's current plan, so this team hand-rolled a single, un-autoscaled, always-on self-hosted VM with a GPU attached — a shared, un-monitored bottleneck and a single point of failure for that entire team's CI.
• Security review flags org-wide, statically-provisioned self-hosted runners. A few teams, frustrated with queue times, stood up long-lived self-hosted runner VMs registered directly to the org. Security review finds these runners are not ephemeral — a compromised job on one repo's workflow can leave persistent state (cached credentials, planted binaries) for the next job that happens to land on the same runner, including jobs from a different repository entirely. This is the single most common self-hosted-runner security finding in real audits: shared, long-lived, non-ephemeral runners are a lateral-movement vector across repos and teams that share the fleet.

The fix has to solve compute elasticity (queue times, GPU access) and cache locality (build speed) simultaneously, without recreating the ephemerality/security problem the ad-hoc VMs introduced.

3. Production Architecture

Architecture image: blogs/architecture/cicd-actions-runner-controller-buildkit-caching-architecture.png

Layer 1 — GitHub control plane: GitHub.com (or GHES) is the job source of truth. Workflow runs queue jobs against a named runs-on: [self-hosted, gha-runner-scale-set-label] runner group. GitHub's Actions service exposes a scale-set API that ARC's listener polls (webhook-driven in newer ARC versions, reducing polling latency) to learn "N jobs are queued for this scale set right now."

Layer 2 — ARC control plane (in-cluster): the gha-runner-scale-set-controller (cluster-scoped, one per cluster) manages one or more AutoscalingRunnerSet custom resources, each corresponding to one runner scale set / label combination. A listener pod per scale set holds the persistent connection to GitHub's scale-set API and is the sole component that decides "create N more ephemeral runner pods" — this centralizes the scaling decision so you don't get thundering-herd double-provisioning from multiple uncoordinated watchers.

Layer 3 — Ephemeral runner pods (the data plane): each queued job gets exactly one runner pod, created fresh, that registers with GitHub, executes the job, and is deleted immediately after — no reuse, no shared filesystem state across jobs, by design. Runner pods use the dind (Docker-in-Docker) or, in the current recommended pattern, rootless containerd-based "Kubernetes mode" container runtime, avoiding the privileged-container requirement that classic DinD imposed. Pods are scheduled onto a dedicated CI node pool, tainted so application workloads can't land there and CI jobs can't starve production pods for node resources.

Layer 4 — Node autoscaling: Karpenter (covered in an earlier session) provisions nodes for this CI node pool on demand — spot instances for the bulk of stateless test/lint jobs (tolerating interruption, since a killed job just gets re-queued and re-run on a fresh pod), on-demand for jobs tagged as interruption-sensitive (release builds, signing steps). A separate, GPU-flavored NodePool with a taint like workload=gpu-ci handles the ML team's GPU test suite, scaling from zero — this is the direct fix for the "hand-rolled always-on GPU VM" problem in Section 2, and it reuses the exact GPU-node provisioning pattern from the GPU scheduling and Ray sessions earlier in this series.

Layer 5 — Distributed build cache: runner pods push and pull BuildKit cache manifests to/from a registry-backed cache store (a dedicated ECR/ACR/GCR repository or an internal Harbor instance, type=registry cache backend) using --cache-to=type=registry,ref=<cache-repo>:<branch-or-layer-key>,mode=max and --cache-from on the next build. This is the component that makes ephemeral, stateless runners not pay a cold-cache tax on every single job — the cache lives in the registry, not on any runner's disk, so it survives every pod's death.

Layer 6 — Security boundary and governance: each AutoscalingRunnerSet maps to exactly one GitHub organization/repo-group scope with its own Kubernetes namespace, its own service account, and — critically — its own registry-cache repository, so a compromised job in one team's runner pool cannot poison another team's build cache or read another team's cached secrets. Ephemeral-by-default means a compromised job's blast radius ends at pod deletion; there is no persistent runner for an attacker to plant anything on.

Why this shape, and how it evolves: the split between "ARC decides how many runner pods" and "Karpenter decides how many nodes" mirrors the same separation of concerns used everywhere else in Kubernetes autoscaling — pod-level and node-level scaling are different problems with different signal sources, and coupling them tightly (e.g., a single custom controller trying to do both) loses the ability to swap either half independently. As the fleet grows past a few thousand jobs/day, the bottleneck shifts from "can we get enough runner pods scheduled" to "is the registry cache backend's read/write throughput and the container registry's pull-through rate limits keeping up" — which is a capacity-planning problem for the cache and registry tier, not the Kubernetes scheduler.

4. Solution Design

Design decisions and alternatives:

Scalability considerations: the listener-pod-per-scale-set model scales horizontally by adding more scale sets (one per label/team/repo-group), not by making one listener handle more load — plan the number of scale sets around organizational boundaries, not just total job volume. Node-level scaling is Karpenter's problem and scales near-linearly with spot capacity availability in the target region/AZ.

Cost implications: spot instances for the bulk of CI traffic is the single biggest cost lever — most CI jobs are short-lived, idempotent, and safely re-runnable, which is exactly the workload profile spot interruption tolerates well. The registry cache itself has a real, easily-underestimated storage and egress cost at scale (multi-GB cache manifests per branch, retained for a rolling window) — cache retention/GC policy is a cost control, not just a hygiene task.

Security implications: ephemeral runners remove the persistent-state attack surface but shift the security question to "what can a job's runner pod reach" — network policy scoping (no route to the Kubernetes API server beyond what the runner needs, no route to other namespaces' secrets) matters as much as the ephemerality itself.

Performance implications: cold-cache job duration (no matching registry cache entry — first build on a new branch, or after a base-image bump) can be several times slower than warm-cache; this variance needs to be an accepted, understood part of CI latency SLOs, not treated as a regression every time it happens.

5. Deep Technical Walkthrough

Internal working — the scale-up path for one queued job:

1. A workflow run queues a job with runs-on: [self-hosted, my-scale-set-label]. GitHub's Actions backend records this in the scale set's pending-job queue and notifies (webhook, in current ARC versions — a major latency improvement over the older polling-only model) the corresponding listener.
2. The listener pod, which holds the live connection to GitHub's scale-set API for exactly this AutoscalingRunnerSet, receives the notification and reconciles: it computes desired runner-pod count as min(maxRunners, currentDesired + pendingJobs) and patches an EphemeralRunnerSet resource (an internal ARC CRD) to that count.
3. The EphemeralRunnerSet controller creates the corresponding number of EphemeralRunner custom resources, each of which owns exactly one runner Pod spec.
4. Kubernetes scheduling picks up the new pods. If the CI node pool has no available capacity, they go Pending, which Karpenter (or Cluster Autoscaler) observes via unschedulable-pod events and provisions new nodes to satisfy — this is the node-level half of the scale-up, decoupled from ARC's pod-level decision.
5. Once scheduled and running, the runner pod's entrypoint registers itself with GitHub as a just-in-time (JIT) runner using a short-lived registration token scoped to exactly one job — this is what makes the runner genuinely one-job-only rather than "registered and reusable until deleted," closing a subtle gap that existed in older, less-ephemeral runner registration flows.
6. The job executes: checkout, then typically a container build step where BuildKit is invoked with --cache-from=type=registry,ref=<cache-repo>:cache-<branch> — BuildKit queries the registry for a matching cache manifest, pulls only the layers needed (content-addressed, so unchanged layers are genuinely skipped, not re-downloaded), and builds only what changed.
7. On completion, --cache-to=type=registry,ref=<cache-repo>:cache-<branch>,mode=max pushes the updated cache manifest (including intermediate layers, not just the final image, when mode=max is set) back to the registry for the next job to consume.
8. The runner pod reports job completion to GitHub, and the EphemeralRunner controller tears the pod down immediately — no idle period, no reuse.

Control plane vs. data plane: the ARC listener + EphemeralRunnerSet/EphemeralRunner controllers are the control plane — they decide how many runners should exist and when. The runner pods themselves, plus the BuildKit cache read/write against the registry, are the data plane — they do the actual job execution and I/O. This separation is why a listener pod restart (control plane blip) doesn't kill in-flight jobs (data plane keeps running), and why data-plane throughput (registry pull bandwidth) is a completely separate scaling axis from control-plane decision latency.

Failure scenarios and recovery:

• Listener pod crash: in-flight runner pods are unaffected (they're already registered and running their job independently); new job intake pauses until the listener restarts (Kubernetes restarts it per its Deployment spec) and re-establishes its connection to GitHub — a brief scale-up delay, not a running-job failure.
• Runner pod OOM-killed mid-job: the job fails, GitHub marks it failed, and (if retry or re-run is configured) a fresh EphemeralRunner is created for the retry — no state carries over from the killed pod, which is again the ephemerality property working as intended, not a special-cased recovery path.
• Registry cache backend unavailable: builds fall back to a full cold build automatically (BuildKit treats a cache-from miss/error as "no cache available," not a hard failure) — slower, but not broken; this graceful-degradation property is why registry cache should be treated as a performance optimization with a monitored SLO, not a hard dependency the pipeline can't run without.
• Karpenter can't provision capacity fast enough (spot exhaustion in the AZ): runner pods stay Pending and jobs queue longer; a fallback NodePool with on-demand instances and a higher priority weight is the standard mitigation, accepting higher cost during spot-scarce windows in exchange for bounded queue time.

Performance bottlenecks at scale: registry pull throughput under high concurrent job counts (many runner pods pulling large cache manifests simultaneously) is the most common ceiling — mitigated by registry pull-through caching close to the cluster (same region, ideally same AZ) and by scoping cache keys tightly enough that not every job needs the full multi-gigabyte cache, only the layers relevant to what it's building.

6. Production Troubleshooting

Symptom: PR CI queue times spike from the usual 1-2 minutes to 15+ minutes fleet-wide, with no corresponding spike in merged PR volume.

Investigation path a senior platform engineer follows:

1. Check whether it's a control-plane (ARC) problem or a node-capacity (Karpenter) problem first — these have completely different fixes:

   kubectl get ephemeralrunners -n arc-runners -o wide | grep -c Pending
   kubectl get pods -n arc-runners --field-selector=status.phase=Pending
   kubectl describe pod <pending-runner-pod> -n arc-runners | grep -A5 Events
   

   A Pending pod's Events will say FailedScheduling: Insufficient cpu (node-capacity problem, hand off to Karpenter investigation) versus no pods being created at all (ARC listener/controller problem).

2. If pods aren't being created, check the listener's connection health:

   kubectl logs -n arc-systems -l app.kubernetes.io/component=runner-scale-set-listener --tail=100
   

   Look for repeated reconnect/auth errors — a common root cause is a GitHub App installation token nearing expiry or rate-limit throttling from GitHub's Actions API on a particularly bursty morning.

3. If it's node capacity, check Karpenter's provisioning decisions:

   kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter --tail=200 | grep -i "insufficient\|error\|launch"
   

   Spot capacity exhaustion in the target AZ/instance-family combination is the most common finding — Karpenter will log launch failures or fall back to alternative instance types if the NodePool allows multiple types, and a too-narrow instance-type constraint is a frequent self-inflicted cause.

4. Check whether it's actually a cache-miss slowdown, not a queueing problem:

   # Compare job duration, not just queue time, across recent runs on the same workflow
   gh run list --workflow=build.yml --json databaseId,status,conclusion,createdAt,updatedAt --limit 20
   

   If queue time is normal but job duration jumped, suspect the registry cache backend — check the registry's own health/throttling metrics and confirm --cache-from is actually finding hits (BuildKit's build log shows CACHED per step when it does; a sudden absence of CACHED lines across many jobs points at a cache backend outage or a cache-key scheme that just broke, e.g., after a base-image bump invalidated the whole cache at once).

5. Correlate with a recent change: a maxRunners value on the AutoscalingRunnerSet that's too low for current job volume, a Karpenter NodePool limit hit, or a registry quota/rate-limit newly enforced are the three most common "this used to work" root causes — check recent GitOps commits to the ARC and Karpenter configs before assuming an upstream GitHub incident.

Common root causes ranked by frequency: (1) node-capacity/spot exhaustion during peak commit hours — by far the most frequent; (2) a maxRunners ceiling that wasn't raised as the org grew; (3) registry cache backend throttling or an unintended full-cache invalidation; (4) an actual GitHub Actions service incident (check the GitHub status page before spending an hour debugging your own cluster).

7. Hands-on Lab

Goal: stand up ARC with a gha-runner-scale-set on a local kind cluster, run a real job against it, and validate BuildKit registry caching on a second run.

# 1. Create a local cluster
kind create cluster --name arc-lab

# 2. Install the ARC controller
helm install arc \
  --namespace arc-systems --create-namespace \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

# 3. Install a runner scale set, pointed at a real repo
# (requires a GitHub App or PAT with appropriate Actions permissions)
helm install arc-runner-set \
  --namespace arc-runners --create-namespace \
  --set githubConfigUrl="https://github.com/<org>/<repo>" \
  --set githubConfigSecret.github_token="<token>" \
  --set maxRunners=5 \
  --set minRunners=0 \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

# 4. Verify the listener is connected
kubectl get pods -n arc-runners
kubectl logs -n arc-runners -l app.kubernetes.io/component=runner-scale-set-listener

# 5. Trigger a workflow run in the target repo with:
#    runs-on: arc-runner-set
# then watch a runner pod appear
kubectl get ephemeralrunners -n arc-runners -w

# 6. In the workflow, add a BuildKit registry-cache build step:
cat <<'EOF' >> .github/workflows/build.yml
      - name: Build with registry cache
        run: |
          docker buildx build \
            --cache-from=type=registry,ref=ghcr.io/<org>/cache:build \
            --cache-to=type=registry,ref=ghcr.io/<org>/cache:build,mode=max \
            -t ghcr.io/<org>/app:latest --push .
EOF

# 7. Run the workflow twice; on the second run, confirm cache hits in the build log
#    (look for "CACHED" against unchanged layers, and compare wall-clock duration)

# Cleanup
helm uninstall arc-runner-set -n arc-runners
helm uninstall arc -n arc-systems
kind delete cluster --name arc-lab

What to validate: that the runner pod is created fresh per job and disappears after completion (kubectl get pods -n arc-runners shows nothing lingering post-job), that the second build run shows CACHED steps and a materially shorter duration than the first, and that scaling to zero (minRunners=0) actually drops the runner-pod count to zero during idle periods — confirming the cost model this whole design is built around.

8. Production Case Study

Large engineering organizations running thousands of repositories converge on the ARC-plus-remote-cache pattern for a structural reason: at that scale, CI compute has to be fungible with the rest of the platform's compute, not a separately-managed VM fleet with its own capacity planning, patching, and cost model. Companies operating at GitHub/Microsoft, Uber, and other high-repo-count engineering orgs have published variations of the same shape — self-hosted runners as Kubernetes pods, scheduled onto the same autoscaled node pools (often the same spot fleets) used for internal batch and build workloads, with build-artifact and layer caching pushed into a shared, high-throughput internal registry or object store rather than left on ephemeral local disk.

The consistent secondary pattern at this scale is treating CI capacity as a cost center with the same FinOps rigor as production infrastructure — spot-first scheduling for the overwhelming majority of jobs, GPU-tier runners scaled strictly to demand rather than provisioned statically, and cache-hit-rate tracked as a first-class metric because it directly and measurably drives both build latency and compute spend. Organizations that skip this discipline consistently report CI compute cost growing faster than headcount or repo count, because idle self-hosted capacity and cold-cache rebuilds both silently compound as the org scales.

9. Architecture Review

Strengths: ephemeral-by-default runners eliminate the most common self-hosted-runner security finding (persistent cross-job state) without giving up self-hosted flexibility (GPU access, VPC-internal network reach, custom images); decoupling pod-level scaling (ARC) from node-level scaling (Karpenter) lets each scale independently and reuses infrastructure the platform team already operates for production workloads; registry-backed build caching solves the "ephemeral runners have no disk to cache into" problem without requiring any change to the ephemerality model.

Weaknesses: the registry cache backend becomes a shared-fate dependency across every team's build pipeline — a registry outage or throttling event degrades build speed org-wide simultaneously, even though it degrades gracefully rather than failing hard; listener-pod-per-scale-set means the number of scale sets (and therefore listener pods and GitHub API connections) grows with organizational boundaries, which needs its own capacity/quota awareness against GitHub's API rate limits at very large repo counts.

What fails first at 10x scale: the registry cache backend's read throughput under concurrent high-volume pulls, and GitHub Actions API rate limits against a large number of simultaneous listener connections and JIT registration calls — not the Kubernetes-side pod scheduling, which scales close to linearly with available node capacity.

How it changes at very large (100M-user-product) scale: cache infrastructure likely federates — a regional/per-cluster pull-through cache in front of a central registry, mirroring the same hot/cold tiering pattern used for observability data elsewhere in this series — purely to keep cache-pull latency low without a single registry absorbing every cluster's traffic. Runner scale-set boundaries would likely map to organizational units with delegated maxRunners and cost budgets, similar to how large orgs delegate IAM policy authority, so a single team's traffic spike can't silently exhaust shared capacity or budget.

What would be redesigned: invest earlier in per-team/per-scale-set cost and cache-hit-rate dashboards rather than a single fleet-wide view, and build automated cache-key hygiene (detecting and alerting on cache keys that silently stop getting hits, e.g., after an undetected base-image bump) rather than relying on someone noticing job durations crept up.

10. Best Practices

Reliability on this platform means treating the listener and controller components with the same operational rigor as any other control-plane service — resource requests/limits sized from real load, restarts monitored, and GitHub API connectivity treated as an external dependency with its own health check, not assumed always-available. Scalability means separating the three independent scaling axes explicitly: pod count (ARC's minRunners/maxRunners), node capacity (Karpenter's NodePool limits and instance-type diversity), and cache backend throughput (registry capacity planning) — each needs its own headroom and its own alerting, because a limit hit on any one of the three produces the same symptom (slow CI) with a different root cause and a different fix.

On observability, track queue time and job duration as separate metrics (conflating them hides whether a slowdown is a capacity problem or a cache problem), and track cache hit rate per workflow/branch as a leading indicator of build-speed regressions before they show up as duration complaints. On security, scope every AutoscalingRunnerSet to the narrowest GitHub org/repo-group it needs, keep runners ephemeral without exception (no "just this one long-lived runner for a special case" — that exception is exactly how the ad-hoc VM problem in Section 2 starts), and apply NetworkPolicy to runner pods so a compromised job's reach is bounded. On cost, spot-first scheduling for the overwhelming majority of stateless CI work, scale-to-zero for GPU and other expensive specialized runner pools, and active cache-retention/GC policy rather than unbounded cache growth. On maintainability, manage AutoscalingRunnerSet, Karpenter NodePool, and cache-backend configuration as GitOps-managed code with the same review discipline as application deployments.

11. Common Production Mistakes

The most common mistake is under-provisioning maxRunners and never revisiting it as the org grows — teams set a reasonable ceiling at launch and forget it exists until queue times regress months later, at which point the fix (raise the ceiling, verify node capacity can actually support it) is trivial but the diagnosis time wasted getting there isn't. A close second is skipping the ephemerality discipline "just this once" for a team that needs a special long-lived runner for a legacy job — this single exception reintroduces exactly the cross-job persistent-state security exposure the whole ephemeral-runner design exists to eliminate, and it tends to quietly spread as other teams copy the pattern once they see it exists.

A third mistake is treating the BuildKit registry cache as zero-maintenance — cache keys that don't account for base-image or dependency-lockfile changes silently degrade to permanent cache misses (still functionally correct, just slow), and without hit-rate monitoring nobody notices until someone complains builds "got slow" with no obvious cause. A fourth is running CI node pools without taints/tolerations separating them from production workloads, which either lets CI burst starve production pods for resources during a busy morning, or — the opposite failure — lets production workloads land on cost-optimized spot CI nodes never designed for their reliability requirements. Finally, teams frequently under-invest in per-team cost attribution — a shared cluster-wide CI cost line item with no breakdown by team or repo makes it impossible to have a productive conversation with the one team whose test suite is disproportionately expensive, which is a conversation that becomes necessary at scale whether or not the tooling makes it easy to have.

12. Interview Preparation

Q: Why do ephemeral, one-job-per-pod self-hosted runners matter more for security than for performance? A: The core security property is that a runner pod has zero state carried over from any prior job — a compromised or malicious job cannot plant credentials, binaries, or modified tool configuration for the next job to inherit, whether that next job belongs to the same repo or a completely different one sharing the runner pool. Long-lived, statically-registered self-hosted runners are the most common real-world self-hosted-runner audit finding precisely because they violate this property. The performance angle (faster job start, resource efficiency) is a secondary benefit; the primary reason ephemerality is non-negotiable in a shared multi-team runner pool is blast-radius containment.

Q: How do you solve the "ephemeral runners have no persistent disk" caching problem? A: Move the cache off the runner entirely and into a registry-backed store — BuildKit's type=registry cache backend pushes intermediate build layers to a container registry via --cache-to and pulls matching layers via --cache-from on the next build, regardless of which ephemeral pod runs it. This decouples cache lifetime from runner-pod lifetime; the cache survives every individual pod's death because it never lived on the pod's disk in the first place.

Q: Explain the difference between what ARC scales and what Karpenter (or Cluster Autoscaler) scales in this architecture, and why that separation matters. A: ARC's listener and EphemeralRunnerSet/EphemeralRunner controllers decide how many runner pods should exist, based on the queued-job count reported by GitHub's Actions API — this is pod-level scaling driven by CI-specific signal. Karpenter decides how many nodes should exist to satisfy the resource requests of whatever pods (runner pods included) are currently unschedulable — this is node-level scaling driven by generic Kubernetes scheduling pressure, with no CI-specific knowledge at all. Keeping these separate means either half can be swapped (a different node autoscaler, a different runner controller) without redesigning the other, and each can be capacity-planned and alerted on independently.

Q: A team complains their CI got dramatically slower overnight with no code changes. Walk through your diagnosis. A: First, separate queue-time regression from job-duration regression — they point at different subsystems. If queue time is up, check for unschedulable runner pods (node-capacity/spot-exhaustion problem, Karpenter's domain) versus no pods being created at all (ARC listener/controller problem, check its connection health to GitHub's API). If job duration itself is up with queue time normal, suspect the build cache — check whether CACHED steps disappeared from build logs, which usually traces to an unintended full-cache invalidation (a base-image bump, a lockfile change that widened the cache-key scope more than intended) or a registry-side throttling/outage event. Cross-reference against recent GitOps changes to ARC, Karpenter, or cache-backend configuration before assuming an upstream incident.

Q: How would you contain the blast radius of a compromised CI job in this architecture? A: Multiple independent layers: ephemerality itself bounds temporal blast radius (the compromised pod is deleted the moment the job ends, with nothing surviving to affect the next job); per-team namespace and scale-set scoping bounds organizational blast radius (a compromised job in team A's pipeline has no path to team B's secrets, cache, or runner pool); NetworkPolicy on runner pods bounds network blast radius (no route to the Kubernetes API server or internal services beyond what the job legitimately needs); and scoping the registry cache per team means a compromised job can at worst poison its own team's cache, not the whole org's.

13. Latest Industry Updates

ARC's 0.14.0 release (March 2026) is the most consequential recent change for teams operating this at scale: it deprecates the old internal scaling client in favor of a standalone, publicly available Go package — the same client that powers ARC's own scaling decisions is now usable directly by platform teams and infrastructure vendors to build custom autoscaling logic against GitHub's scale-set API, without requiring Kubernetes at all. This matters because it opens the door to non-Kubernetes self-hosted runner autoscaling (relevant for teams on Nomad, ECS, or bare-metal fleets) using the exact same scaling semantics ARC validates in production, rather than everyone reimplementing scale-set polling logic independently. The same release adds configuration-staleness handling (runners exiting with code 7 now fully pause autoscaling for that runner set until the new config is confirmed healthy), directly closing a rollout-safety gap where stale runner images could keep spinning up mid-config-change.

On the caching side, the broader 2026 trend is registry-backed remote caching solidifying as the default recommendation over local-disk or actions/cache-style approaches specifically because ephemeral, autoscaled runners have made "cache lives on the runner" structurally unworkable — every team running self-hosted runners on Kubernetes at meaningful scale is converging on the same registry-cache pattern this session describes, which is worth watching because it signals the ecosystem has settled on an answer rather than still exploring alternatives. Expect continued convergence between CI compute scheduling and the same GPU-aware, spot-first autoscaling patterns already mainstream for application and AI inference workloads, as more organizations run GPU-dependent test suites (ML model validation, inference correctness checks) as a routine part of CI rather than a special case.

14. Summary & Cheat Sheet

Key concepts: self-hosted CI on Kubernetes solves two independent problems — compute elasticity (ARC scaling ephemeral runner pods, Karpenter scaling nodes underneath them) and cache locality (registry-backed BuildKit caching, since ephemeral pods have no persistent disk to cache into). Ephemerality is a security property first, a performance property second — it's what prevents shared runner pools from becoming a cross-job/cross-team lateral-movement vector.

Architecture in one line: GitHub queues a job → ARC listener scales an EphemeralRunnerSet → Karpenter provisions nodes for unschedulable runner pods → job runs, pulling/pushing BuildKit cache against a registry → runner pod deleted, no state retained.

ARC vs. GitHub-hosted runners:

Key commands:

kubectl get ephemeralrunners -n arc-runners -o wide      # runner pod status
kubectl logs -n arc-runners -l app.kubernetes.io/component=runner-scale-set-listener  # listener health
docker buildx build --cache-from=type=registry,ref=<repo>:cache --cache-to=type=registry,ref=<repo>:cache,mode=max
helm upgrade arc-runner-set oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set --set maxRunners=<n>

Best-practice checklist:

• Ephemeral runners without exception — no long-lived "special case" runner in a shared pool.
• Separate scaling axes and alerting: pod count (ARC), node capacity (Karpenter), cache throughput (registry).
• Per-team namespace, scale-set, and cache-repo scoping to bound blast radius.
• Spot-first for stateless jobs; scale-to-zero for GPU/specialized pools.
• Monitor cache hit rate per workflow as a leading indicator, not just job duration after the fact.

Troubleshooting checklist for a CI slowdown:

1. Separate queue-time regression from job-duration regression — they point at different subsystems.
2. Check for Pending/unschedulable runner pods (node-capacity problem) versus no pods created (ARC/listener problem).
3. Check Karpenter logs for spot exhaustion or instance-type constraints.
4. Check build logs for missing CACHED steps and registry health/throttling if duration (not queue time) regressed.
5. Correlate against recent GitOps changes to maxRunners, NodePool limits, or cache-backend config before assuming an upstream incident.