title: "Ray on Kubernetes: KubeRay, Gang Scheduling, and Distributed Training/Serving at Scale" date: 2026-09-11 tags: [Kubernetes, Ray, KubeRay, GPU, AI Infrastructure, Distributed Training, LLMOps, Kueue] cover: ../images/ray-kuberay-distributed-training-cover.png

Ray on Kubernetes: KubeRay, Gang Scheduling, and Distributed Training/Serving at Scale

1. Topic of the Day

Ray is the distributed execution substrate that most production ML platforms now sit on top of — not because it's fashionable, but because it solves a problem Kubernetes was never designed to solve: fine-grained, stateful, gang-scheduled distributed computation across a fleet of GPUs, with a single Python process model that spans hundreds of nodes.

Kubernetes gives you pod scheduling, bin-packing, and self-healing at the container level. It has no concept of "these 64 pods are one logical job and must all be Running before any of them does useful work," no concept of a distributed object store shared across processes, and no concept of an actor that needs to survive a node failure with its state intact. Ray fills that gap: a head node runs the Global Control Store (GCS) and scheduler, worker nodes run raylets that manage local object stores and actor/task execution, and the whole thing is exposed to Kubernetes via the KubeRay operator, which reconciles RayCluster, RayJob, and RayService custom resources into head/worker pod groups, handles autoscaling via the Ray autoscaler sidecar, and integrates with Kueue or the KAI Scheduler for gang admission control.

Where it shows up in enterprise production: distributed pretraining and fine-tuning of LLMs (Ray Train + PyTorch FSDP/DeepSpeed), RLHF pipelines (Ray Train + Ray Serve co-located for the actor/reward-model loop), large-scale batch inference and embedding generation (Ray Data), hyperparameter sweeps (Ray Tune), and increasingly, multi-model LLM serving (Ray Serve fronting vLLM replicas with autoscaling per model). OpenAI, Anthropic, Uber, ByteDance, Pinterest, Instacart, and most GPU-cloud providers (CoreWeave, Together AI, Anyscale itself) run Ray on Kubernetes as the orchestration layer underneath training and serving. If you've touched KServe or vLLM in this series already, Ray is the layer below that handles the multi-node coordination KServe assumes already exists.

2. Real Business Problem

Scenario: A platform team runs a shared EKS cluster with 64 H100 GPUs (8 nodes × 8 GPUs) for a research org fine-tuning a 70B-parameter model with FSDP across 32 GPUs (4 nodes). The job is submitted as a RayJob. Here's what goes wrong in week one of production use:

• Partial admission / straggler hang. The Kubernetes cluster autoscaler provisions nodes one at a time. Ray's own autoscaler requests 4 worker pods; 3 come up in 90 seconds, the 4th is stuck 6 minutes waiting on an EBS-backed AMI pull for the CUDA base image. FSDP's collective all-gather at step 0 blocks indefinitely because 3 of 4 ranks are ready and one isn't — no timeout, no signal, just a silently hung job burning 3 nodes' worth of H100 cost for 6 minutes doing nothing.
• No gang semantics. A second, higher-priority job needs the same 4 nodes. Because Kubernetes schedules pod-by-pod with no notion of "this job needs all-or-nothing," the scheduler happily starts admitting the second job's pods into the same node pool, causing resource contention and, eventually, evictions of the first job's already-running workers mid-all-reduce — the training step corrupts, gradients are garbage, and the job crashes 40 minutes in with a NCCL timeout, not a clean error.
• GPU exhaustion with no backpressure. Three teams submit RayJobs concurrently. There's no queueing layer, so all three jobs get partially admitted, no single job gets its full gang, and the cluster autoscaler thrashes trying to satisfy three simultaneous fractional demands, scaling nodes up and down every few minutes ($$ in on-demand churn).
• Silent GCS single point of failure. The Ray head node (running GCS) crashes under memory pressure from a Ray Data job materializing too many objects in the head's object store. Every worker in every RayCluster in the namespace loses its control plane simultaneously — a multi-tenant blast radius nobody scoped for.

This is the class of incident senior platform engineers get paged for: not "the model diverged," but "the platform doesn't know how to gang-schedule distributed jobs, and GPUs are too expensive to leave this to chance."

3. Production Architecture

Architecture image: blogs/architecture/ray-kuberay-distributed-training-architecture.png

The design below is what actually holds up in production at multi-team, multi-hundred-GPU scale:

Control plane layering

• KubeRay Operator (cluster-scoped Deployment) watches RayCluster, RayJob, RayService CRDs and reconciles them into head/worker Pod groups, per-cluster Services, and (for RayService) a zero-downtime rolling upgrade controller.
• Kueue sits in front of the operator as an admission/queueing layer. Teams submit RayJob wrapped in a Kueue Workload; Kueue holds it in Suspended state until it can guarantee gang capacity (all pods, not partial), then flips suspend: false and the operator creates pods atomically from the scheduler's perspective.
• Cluster Autoscaler / Karpenter provisions GPU node pools (tainted nvidia.com/gpu, labeled by instance family — p5.48xlarge for H100, g6 for L4 inference) below Kueue's ResourceFlavors, so capacity requests map 1:1 to real node shapes instead of generic CPU/memory bin-packing.
• NVIDIA GPU Operator + DCGM exporter on every GPU node for device plugin, driver management, and per-GPU utilization/temperature/ECC metrics into Prometheus.

Data plane

• RayCluster (training): 1 head pod (CPU-only or small GPU, running GCS + dashboard + autoscaler sidecar) + N worker pods (1 pod = 1 node = 8 GPUs, NCCL over EFA/RoCE for cross-node collectives). Ray Train wraps PyTorch FSDP; checkpoints stream to S3/GCS via ray.train.Checkpoint with async upload so a node loss doesn't stall the whole job past the last checkpoint interval.
• RayService (serving): head + autoscaling worker pool running Ray Serve deployments, each deployment fronting a model replica (often a vLLM engine per replica for LLM serving, or a plain Torch model for smaller models). RayService's in-place upgrade mechanism spins up a fully healthy new RayCluster before flipping the Kubernetes Service selector, giving you blue/green at the cluster level for free.
• Ray Data jobs (batch inference, embedding generation) run as ephemeral RayJobs against object storage, reading/writing Parquet directly, scaling workers independently of the training/serving clusters.
• Object storage / distributed object store: Ray's in-memory + spilled-to-disk object store per node (NVMe-backed) for intra-job data; S3/GCS for checkpoints, datasets, and model artifacts, accessed via IRSA/Workload Identity — no static credentials on pods.

Networking & security boundaries

• GPU worker pods run in a dedicated node pool with a NetworkPolicy default-deny except: GCS port (head↔worker), object manager port (worker↔worker for spill/pull), NCCL ports (worker↔worker, same-namespace only), and metrics scrape from the monitoring namespace.
• Ray Dashboard and Jobs API are never exposed directly; fronted by an internal ingress with OIDC (Istio/Envoy or Azure AD-integrated ingress), separate from the model-serving ingress which goes through the AI Gateway covered in an earlier session.
• Multi-tenancy: one RayCluster (and one Kubernetes namespace) per team/workload class, never shared GCS across teams — the head-node blast radius from the incident above is a hard boundary, not a soft one.

HA / DR

• Head node runs with GCS fault tolerance backed by an external Redis (GCS FT mode) so a head pod restart doesn't lose in-flight actor/task state for long-running RayServices.
• Training checkpoints are the real recovery unit — RayJob retries re-hydrate from the last checkpoint in object storage, not from GCS state, which is why checkpoint interval tuning (covered in the Deep Technical Walkthrough) matters more than head HA for training workloads.
• Multi-region: RayService replicas run active-active in two regions behind the AI Gateway with health-based routing; RayCluster training jobs are region-pinned (GPU capacity + data locality dominate) with cross-region checkpoint replication as the DR path, not live failover.

Why this shape: Kueue in front of KubeRay is the single highest-leverage decision — it converts Kubernetes' pod-at-a-time scheduling into job-at-a-time admission, which is the actual unit GPU capacity needs to be reasoned about in. Everything else (NCCL networking, checkpoint cadence, RayService blue/green) is standard distributed-systems hygiene once admission is correct. As this scales from 64 to 1,000+ GPUs, the head node's GCS becomes the next bottleneck (it's single-threaded for many control operations), pushing toward sharding workloads across multiple smaller RayClusters rather than one giant cluster — a trade-off explored in Section 9.

4. Solution Design

Design decisions and alternatives:

Scalability: horizontal scaling is bounded first by GCS throughput (control-plane operations: actor creation, task scheduling RPCs), typically comfortable to ~2,000 nodes per RayCluster before you should be sharding into multiple clusters fronted by a job router. Data-plane scaling (NCCL collectives) is bounded by network fabric — EFA/RoCE at 400Gbps+ is table stakes past 64 GPUs for training throughput to not be network-bound.

Cost implications: GPU idle time from partial-gang hangs is the dominant cost driver, not compute efficiency — a 5-minute stall on 32 H100s costs roughly the same as 2.5 GPU-hours; at $2-4/GPU-hour that's real money multiplied across every job submission without gang scheduling. Kueue's admission gating is a cost-avoidance feature before it's a scheduling feature.

Security implications: Ray's default Dashboard/Jobs API has no built-in auth — it must never be exposed without a fronting auth layer. Object store spill-to-disk on shared nodes means multi-tenant isolation must happen at the node-pool level (dedicated nodes per tenant), not just namespace level, if data sensitivity requires it.

Performance implications: checkpoint frequency is a direct trade against both recovery time and steady-state throughput (async checkpointing to object storage costs bandwidth and briefly stalls the training loop); tuning this is covered in depth below.

5. Deep Technical Walkthrough

Request/job flow for a RayJob submission:

1. User submits RayJob (or kubectl apply -f rayjob.yaml), which wraps a Kueue Workload via the kueue.x-k8s.io/queue-name label.
2. Kueue's admission controller intercepts the workload, checks the target ClusterQueue's quota against its ResourceFlavor (e.g., nvidia-h100: 32), and either admits immediately or holds it Suspended in a priority-ordered queue.
3. On admission, Kueue unsuspends the workload; KubeRay's operator reconciles the RayCluster spec into a head Pod and N worker Pods simultaneously submitted to the Kubernetes API — this is the gang property: all pod specs exist at once, so the K8s scheduler (or a gang-aware scheduler plugin) can bin-pack them together rather than admitting a subset.
4. Pods bind to nodes; if the underlying node pool lacks capacity, Karpenter/Cluster Autoscaler provisions new nodes matching the ResourceFlavor's node selector/taints. Kueue's waitForPodsReady gate holds the workload's actual "Running" signal until all pods reach Ready — this is what fixes the straggler-hang incident, because the training script itself doesn't start until every rank is genuinely ready, not just scheduled.
5. Ray's GCS on the head pod builds the cluster membership table as raylets on worker pods register. Ray Train's TorchTrainer launches one training worker process per GPU via Ray actors, each wrapping a PyTorch DistributedDataParallel/FSDP rank; Ray sets MASTER_ADDR/RANK/WORLD_SIZE env vars per actor to bootstrap the underlying torch.distributed process group over NCCL — Ray does not replace NCCL, it orchestrates the processes that then talk NCCL directly to each other over the pod network (hence the NetworkPolicy allowances for NCCL ports).
6. Training proceeds with the standard collective pattern (forward → backward → all-reduce/all-gather for FSDP-sharded params → optimizer step). Ray Train's Checkpoint API triggers on a configurable interval; the checkpoint upload to object storage is async relative to the next training step where possible (overlap compute with I/O) but does introduce a synchronization barrier across ranks to agree "this is a consistent checkpoint."
7. On job completion (or failure), Ray Train reports final metrics to the driver; RayJob's controller marks the K8s Job succeeded/failed and (if configured) tears down the RayCluster to release GPUs back to the pool — critical for cost, since idle GPU-holding clusters after job completion are a common source of wasted spend.

Control plane vs. data plane interaction: GCS is purely control plane — cluster membership, actor placement decisions, task scheduling metadata. It is never on the critical path for the actual tensor data movement during all-reduce; that happens raylet-to-raylet (data plane) over NCCL/RDMA directly. This separation is why GCS restarts (with FT enabled) don't necessarily kill in-flight collectives — but in practice, a raylet losing contact with GCS for too long will self-terminate as a safety measure, which does kill the collective. This is the subtlety that catches teams: GCS FT protects actor/task metadata continuity, not in-flight collective operations, which remain vulnerable to any single-node loss.

Failure scenarios and recovery:

• Single worker node preempted (spot): raylet on that node dies, GCS detects it, in-flight NCCL collective involving that rank times out (default ~10 min, should be tuned to ~2-3 min in production to fail fast). Ray Train's FailureConfig triggers a retry that re-launches all worker actors and restores from the last checkpoint — meaning your effective throughput loss is (time since last checkpoint) + (cluster re-provision time), which is the concrete argument for checkpoint interval tuning, not just a best practice.
• Head node loss (no GCS FT): total cluster loss, RayJob controller marks it failed, Kueue releases the quota, a fresh RayCluster must be created — expensive, hence GCS FT with external Redis for anything long-running.
• Network partition between worker pods (common with misconfigured CNI/security groups on multi-AZ node pools): NCCL collective hangs until timeout rather than failing fast, which is why NCCL_TIMEOUT and Ray's own RAY_gcs_rpc_server_reconnect_timeout_s need explicit, aggressive tuning in production — the defaults are tuned for correctness/patience, not for fast-fail cost control.

Performance bottlenecks: at scale, three things dominate: (1) network fabric bandwidth for cross-node all-reduce — this is why H100/A100 clusters pay for EFA/InfiniBand rather than standard VPC networking; (2) checkpoint I/O contention when many jobs checkpoint to the same object storage bucket simultaneously — shard by job/team prefix and watch S3 request-rate partitioning; (3) GCS RPC latency under high actor-churn workloads (e.g., Ray Data with thousands of short-lived tasks) — mitigated by tuning num_cpus_for_gcs and, past a few thousand nodes, sharding into multiple RayClusters.

6. Production Troubleshooting

Symptom: Training job hangs at step 0, no progress, no error, GPU utilization flatlines at 0% on some nodes.

Investigation path a senior SRE follows:

1. Check Kueue workload state first, before touching Ray at all:

   kubectl get workloads -n ml-training
   kubectl describe workload <name> -n ml-training
   

   Look for status.conditions — Admitted=False with reason Pending means it's queued behind quota, not a Ray problem at all. This single check eliminates half of "job hangs" tickets.

2. If Admitted=True, check pod readiness gang-wide:

   kubectl get pods -n ml-training -l ray.io/cluster=<cluster-name> -o wide
   

   Any pod stuck in Pending/ContainerCreating (usually image pull or node provisioning) means the gang isn't actually complete — Ray's autoscaler thinks it asked for N workers but K8s hasn't delivered them, and depending on waitForPodsReady config, the training script may have started prematurely against a partial world size.

3. Check the Ray Dashboard (port-forward, never public) or ray status inside the head pod:

   kubectl exec -it <head-pod> -n ml-training -- ray status
   

   This shows live cluster resource summary and pending/failed actor placements — a mismatch between "expected workers: 4" and "alive workers: 3" confirms partial gang.

4. Check NCCL debug output (set NCCL_DEBUG=INFO on the training pods ahead of time — retrofit-adding it requires a restart, so it should be default-on in any GPU training image):

   kubectl logs <worker-pod> -n ml-training | grep -i nccl
   

   NCCL INFO Timeout with a specific rank identifies exactly which node dropped out of the collective — cross-reference with kubectl get events for that node around the same timestamp (spot reclaim, OOM-kill, node NotReady).

5. Metrics/dashboards: Grafana panel on DCGM DCGM_FI_DEV_GPU_UTIL per node — a node at 0% while siblings are pegged at 95%+ confirms the straggler; Prometheus alert should already exist on ray_train_num_workers_healthy < expected sustained for >2 minutes, paging before the user even notices.

6. Root cause classification and fix:

   • Image pull slowness → pre-warm node images via DaemonSet cache-puller, or use a smaller base image with layers cached in the node AMI.
   • Spot reclaim mid-collective → move training node pool to on-demand/reserved for the training tier, keep spot for the batch-inference/Ray Data tier where task-level (not collective-level) retry is cheap.
   • NetworkPolicy blocking NCCL ports after a security tightening change → this is the classic "someone locked down the namespace and broke training" incident; verify with kubectl exec ... -- nc -zv <peer-pod-ip> <nccl-port>.

Configuration fix example — tightening timeouts and enabling waitForPodsReady so this fails fast instead of hanging:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: gpu-training-queue
spec:
  waitForPodsReady:
    enable: true
    timeout: 5m          # fail the whole workload if not gang-ready in 5 min
    recoveryTimeout: 3m
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu"]
    flavors:
    - name: nvidia-h100
      resources:
      - name: "nvidia.com/gpu"
        nominalQuota: 32

7. Hands-on Lab

Runs on a local kind/minikube cluster (CPU-only, for mechanics) or a real GPU cluster (for the full path). Validation and cleanup included.

# 1. Create a cluster with GPU-simulated node labels (kind) or use your real GPU cluster
kind create cluster --name ray-lab

# 2. Install KubeRay operator via Helm
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.1 -n kuberay-system --create-namespace

# 3. Install Kueue for gang scheduling
kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.10.1/manifests.yaml

# 4. Define a minimal ClusterQueue + LocalQueue + ResourceFlavor
cat <<'EOF' | kubectl apply -f -
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: default-flavor
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: cluster-queue
spec:
  namespaceSelector: {}
  resourceGroups:
  - coveredResources: ["cpu", "memory"]
    flavors:
    - name: default-flavor
      resources:
      - name: "cpu"
        nominalQuota: 8
      - name: "memory"
        nominalQuota: 16Gi
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: team-queue
  namespace: default
spec:
  clusterQueue: cluster-queue
EOF

# 5. Submit a RayJob wired to the Kueue LocalQueue
cat <<'EOF' | kubectl apply -f -
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: ray-sample-job
  labels:
    kueue.x-k8s.io/queue-name: team-queue
spec:
  entrypoint: python /home/ray/samples/sample_code.py
  rayClusterSpec:
    rayVersion: '2.58.0'
    headGroupSpec:
      rayStartParams: {}
      template:
        spec:
          containers:
          - name: ray-head
            image: rayproject/ray:2.58.0
            resources:
              requests: { cpu: "1", memory: "2Gi" }
              limits: { cpu: "1", memory: "2Gi" }
    workerGroupSpecs:
    - replicas: 2
      groupName: small-group
      rayStartParams: {}
      template:
        spec:
          containers:
          - name: ray-worker
            image: rayproject/ray:2.58.0
            resources:
              requests: { cpu: "1", memory: "2Gi" }
              limits: { cpu: "1", memory: "2Gi" }
EOF

# 6. Validate: watch the Workload get admitted, cluster stand up, job run
kubectl get workloads
kubectl get raycluster
kubectl get rayjob ray-sample-job -o jsonpath='{.status.jobStatus}'

# 7. Cleanup
kubectl delete rayjob ray-sample-job
helm uninstall kuberay-operator -n kuberay-system
kind delete cluster --name ray-lab

For the GPU path, swap the container image for rayproject/ray-ml:2.58.0-gpu, add nvidia.com/gpu: 1 to resource requests/limits, and set the ClusterQueue resource group to nvidia.com/gpu with a real ResourceFlavor node selector matching your GPU node pool's labels.

8. Production Case Study

OpenAI runs distributed training coordination across tens of thousands of GPUs spanning multiple Kubernetes clusters, with custom controllers handling GPU failure detection and node replacement to sustain high fleet utilization — the same class of problem this session addresses at a smaller scale: partial-gang and straggler handling matter more as fleet size grows, not less, because the probability of some node failing during a long collective operation approaches certainty at thousands-of-GPU scale.

Google (GKE + KubeRay + Kueue, published reference architecture): Google's own guidance for running Ray on GKE explicitly pairs KubeRay with Kueue for exactly the gang-scheduling and quota-fairness reasons covered here, treating it as the default pattern for multi-team GPU sharing rather than an advanced option — validating that this isn't a niche configuration but the expected production baseline.

Anyscale (the company founded by Ray's creators) built a commercial platform specifically because operating KubeRay's autoscaling, multi-cluster GCS sharding, and spot-instance interruption handling reliably at scale is nontrivial enough that many organizations prefer to buy it rather than build it — a useful signal for a build-vs-buy conversation with your own leadership when scoping a platform team's roadmap.

Uber's Michelangelo and similar internal ML platforms (Pinterest, Instacart, ByteDance's internal training infra) converge on the same shape independently: a queueing/admission layer in front of Ray or an equivalent executor, checkpoint-to-object-storage as the primary DR mechanism, and strict tenant isolation at the cluster (not namespace) level for GPU workloads — this is strong evidence the architecture in Section 3 isn't one team's opinion but a converged industry pattern.

9. Architecture Review

Strengths: Kueue-gated admission eliminates the highest-cost failure mode (partial-gang GPU idle burn). Checkpoint-based recovery decouples training resilience from any single component's HA story. Per-team RayCluster isolation contains the GCS blast radius. RayService's cluster-level blue/green gives safe serving upgrades without a custom controller.

Weaknesses: GCS remains a single-threaded control-plane bottleneck within one RayCluster — fine to ~2,000 nodes, but teams running one enormous shared cluster for "simplicity" will hit control-plane RPC latency under high actor churn well before GPU capacity is the limiting factor. NCCL timeout tuning is a constant tension between fail-fast (better cost control) and false-positive job kills on transient network blips — there's no universally correct value, only a value tuned to your specific network fabric's tail latency. Kueue's fair-share algorithm optimizes for quota fairness, not for minimizing fragmentation, so a cluster with many small jobs and a few huge ones can still end up with GPUs held idle waiting for a big job's gang while smaller jobs that would fit are queued behind it (head-of-line blocking) — cohort/borrowing configuration mitigates but doesn't eliminate this.

What fails first at 10x scale (640 GPUs → 6,400 GPUs): GCS RPC throughput within a single RayCluster; the fix is workload sharding across multiple RayClusters with a routing/queueing layer above KubeRay (which is exactly what Kueue's multi-ClusterQueue cohorts are for, plus likely a custom job router). Checkpoint I/O to a single object storage bucket also becomes a throughput wall — needs prefix sharding and possibly regional bucket splitting.

What changes for 100 million end users (serving side, not training): The RayService/vLLM serving path needs to shift from "autoscale within a cluster" to "route across many regional RayService clusters behind a global AI Gateway with per-region capacity reservations," because a single RayCluster's head node and Ray Serve router, however well-tuned, is not designed to be the sole ingress point for planet-scale QPS — it's designed to manage a bounded pool of replicas well, not to be a global load balancer itself.

What I'd redesign: Move from "one Kueue ClusterQueue per resource type" to cohort-based borrowing across team queues from day one — retrofitting fair-share cohorts after teams have gotten used to hard quota isolation is a much harder organizational conversation than starting with it. I'd also bake NCCL/Ray timeout tuning into the platform's golden training image rather than leaving it to each team, since it's exactly the kind of setting nobody tunes until after their first expensive incident.

10. Best Practices

Reliability comes from gang-admission (Kueue) plus aggressive, tuned collective timeouts, not from head-node HA alone — training resilience is fundamentally a checkpoint-cadence problem. Scalability means treating GCS throughput as a first-class capacity metric alongside GPU count, and sharding RayClusters proactively rather than reactively. Observability requires Ray's own metrics (actor/task state, object store spill rate) exported to Prometheus alongside DCGM GPU metrics and Kueue's workload-queue-depth metrics — any one alone under-diagnoses incidents. Security means Ray Dashboard/Jobs API behind authenticated ingress always, GPU node pools with dedicated NetworkPolicies for NCCL ports, and IRSA/Workload Identity for all object storage access — never static credentials baked into training images. Cost optimization means tearing down RayClusters immediately on RayJob completion (don't let idle GPU-holding clusters linger), using spot for Ray Data/batch-inference tiers where task-level retry is cheap, and reserving on-demand/committed capacity for long-running training where mid-collective preemption is expensive. Performance means matching checkpoint interval to your acceptable "lost work on failure" window, not defaulting to whatever the framework ships with. Maintainability means one golden Ray training image per org with NCCL/timeout/logging defaults baked in, versioned alongside the KubeRay operator version. Operational excellence means the Kueue workload state, not the Ray dashboard, is the first thing an on-call engineer checks for a "job won't start" ticket.

11. Common Production Mistakes

Running one giant shared RayCluster across teams "to keep it simple" — this is the single most common anti-pattern, and it's the direct cause of the GCS blast-radius incident in Section 2; experienced platform teams isolate at the cluster level even though it means more operational surface area. Leaving Ray's default NCCL/GCS timeouts untouched — they're tuned for correctness in a lab, not cost control in production, and the first time a team discovers this is usually via a five-figure GPU bill from a hung job. Submitting RayJobs with no Kueue (or equivalent) gating "because it worked fine in dev with 2 nodes" — this scales terribly the moment more than one team shares the GPU pool. Treating GCS fault tolerance as sufficient training-job resilience — it protects actor/task metadata, not in-flight collectives, so teams that skip explicit checkpoint tuning discover the gap during their first real node failure, usually on the most expensive job running that week. Not tearing down RayClusters after RayJob completion, leaving GPUs held by an idle cluster — a quiet, continuous cost leak that doesn't show up until someone reconciles the cloud bill against actual utilization.

12. Interview Preparation

Q: Why do you need Kueue (or similar) in front of KubeRay instead of relying on Kubernetes' native scheduler? A: Kubernetes schedules pods independently; it has no concept of a job requiring all-or-nothing admission. Without gang scheduling, a distributed training job can get partially admitted, hang on a collective operation waiting for stragglers, or get preempted mid-collective by a competing job — all of which waste expensive GPU time. Kueue holds the entire job's pod set as a Workload and only admits it when full capacity is guaranteed, converting the unit of scheduling from "pod" to "job," which matches how distributed training actually needs to be reasoned about.

Q: What's the difference between GCS fault tolerance and training-job fault tolerance in Ray? A: GCS FT (backed by external Redis) preserves cluster control-plane state — actor/task metadata — across a head node restart, which matters for long-running services like RayService. It does not protect in-flight collective operations (NCCL all-reduce/all-gather); a worker node loss mid-collective still corrupts that step regardless of GCS state. Training resilience comes from checkpoint-and-restart, not from GCS HA — these are separate mechanisms solving separate failure classes.

Q: How would you scale a Ray platform from 500 GPUs to 5,000 GPUs? A: First, instrument GCS RPC latency and actor-churn rate as capacity metrics, not just GPU utilization — a single RayCluster's control plane becomes the bottleneck well before GPU count does, typically in the low thousands of nodes. Shard workloads across multiple RayClusters fronted by a job router (or Kueue cohorts with borrowing across ClusterQueues) rather than growing one cluster indefinitely. Second, audit network fabric — cross-node NCCL bandwidth requirements grow with model/cluster size, and what was adequate VPC networking at 500 GPUs is often a throughput wall at 5,000. Third, revisit checkpoint I/O patterns since a single object storage bucket/prefix under 10x more concurrent training jobs will hit request-rate limits.

Q: A distributed training job is hanging with 0% GPU utilization on some nodes but not others. Walk through your debugging process. A: Check Kueue/queue admission state first — confirm the job is actually fully admitted, not partially queued. If admitted, check pod readiness across the full gang via kubectl get pods — any non-Ready pod means the world size the training script sees doesn't match what's actually up. If all pods are Ready, check ray status on the head node for actor placement mismatches, then grep NCCL debug logs for timeout messages identifying the specific stalled rank, and cross-reference with kubectl get events for that node around the same time (spot reclaim, OOM-kill, NotReady) plus DCGM utilization dashboards to visually confirm which node is the straggler.

Q: When would you choose RayService over KServe for model serving? A: RayService is the right choice when the model itself needs Ray's distributed execution — multi-GPU tensor-parallel serving, or a serving pipeline that includes non-trivial Python business logic/pre/post-processing that benefits from Ray's actor model, or when you're already running Ray Train for the same models and want operational continuity between training and serving. KServe is the right choice for more standardized, framework-native model serving (ONNX, standard PyTorch/TF, or vLLM directly) where you want tighter integration with Knative-style scale-to-zero and a lighter operational footprint than running full RayClusters for serving.

13. Latest Industry Updates

KubeRay has matured through the 1.6.x line (1.6.0 and 1.6.1) with improved RayJob/RayService coordination for head/worker topology management, and Ray itself is iterating rapidly through the 2.5x release series (2.58.0 referenced in current docs), with active development on Kueue integration for RayJob and RayCluster gang scheduling, plus emerging support for the NVIDIA-donated KAI Scheduler as an alternative to Kueue that adds first-class GPU sharing/fractional-allocation semantics — worth evaluating if your workloads need sub-GPU multi-tenancy rather than whole-GPU gang scheduling. Google Cloud has published KubeRay+Kueue as a reference architecture for GKE, cementing the queueing-layer pattern as the default rather than an advanced configuration. The broader 2026 AI-infrastructure-on-Kubernetes landscape (vLLM, KServe, Kueue, Ray together) is increasingly discussed as a converged "stack," not competing point solutions — the practical implication is that platform teams should plan for all four to coexist and integrate (Ray for training/complex serving pipelines, vLLM as the inference engine inside either Ray Serve or KServe, Kueue as the shared admission layer across both) rather than picking one to the exclusion of the others. This matters in production because teams that adopted Ray or KServe in isolation eighteen months ago are now doing integration work to bring Kueue in front of both, which is a strong argument for platform teams starting new builds today to design the queueing layer in from the beginning.

Sources consulted: KubeRay GitHub (ray-project/kuberay), Ray documentation (docs.ray.io) on Kueue and KAI Scheduler integration, Google Cloud blog on KubeRay+Kueue on GKE, and 2026 industry roundups on Kubernetes AI infrastructure production patterns.

14. Summary & Cheat Sheet

Key concepts: Ray = distributed execution engine (GCS control plane + raylet data plane per node); KubeRay = Kubernetes operator reconciling RayCluster/RayJob/RayService CRDs; Kueue = admission-layer gang scheduling that converts pod-at-a-time K8s scheduling into job-at-a-time queueing.

Architecture at a glance: Kueue (admission/quota) → KubeRay operator (reconciliation) → RayCluster (head=GCS+dashboard, workers=raylets+GPUs) → Ray Train (FSDP/DeepSpeed over NCCL) for training, Ray Serve/RayService (often fronting vLLM) for serving → checkpoints/artifacts to S3/GCS via IRSA.

Key commands:

kubectl get workloads                      # Kueue admission state — check FIRST
kubectl get raycluster / rayjob / rayservice
kubectl exec <head-pod> -- ray status      # live cluster resource + actor state
kubectl logs <worker-pod> | grep -i nccl    # collective timeout diagnosis

Best-practice defaults: per-team RayCluster isolation (never shared GCS across tenants); Kueue waitForPodsReady with a tuned timeout (2-5 min, not the multi-minute default); NCCL_TIMEOUT tuned to your fabric's real tail latency, not left at framework defaults; checkpoint interval sized to your acceptable lost-work window; RayCluster teardown on RayJob completion, always.

Troubleshooting checklist: (1) Kueue Workload admitted? (2) All gang pods Ready? (3) ray status actor/worker count matches expected? (4) NCCL debug logs show a specific stalled rank? (5) kubectl get events correlate with node-level cause (spot reclaim, OOM, NetworkPolicy)? (6) DCGM/Grafana confirm which node is the straggler?

Design patterns to reuse: admission-layer gating (Kueue) before any distributed scheduler; checkpoint-to-object-storage as the primary DR mechanism, independent of control-plane HA; cluster-level (not namespace-level) tenant isolation for anything with a shared single-threaded control plane like GCS.