ReplicaSet Fails To Scale GPU Pods After Benchmark Load

Q: Debugging and Investigation Steps 1. Inspect ReplicaSet status and pod events

kubectl get rs benchmark-rs -o wide kubectl describe rs benchmark-rs kubectl get pods -l app=benchmark -o wide kubectl describe pod <pending-pod-name> Typical output snippet:

Q: 2. Verify GPU resource availability on each node

kubectl get nodes -L nvidia.com/gpu kubectl describe node gpu-node-1 | grep -i gpu Expected output when GPUs are fully allocated:

Q: 3. Check device plugin health

kubectl -n gpu-operator get pods -l app=nvidia-device-plugin kubectl logs -n gpu-operator <device-plugin-pod> Look for log lines such as:

Q: 4. Examine scheduler cache and preemption settings

kubectl get schedulerconfig default -o yaml kubectl describe priorityclass high-priority If no PriorityClass is defined, the scheduler cannot pre‑empt lower‑priority pods.

Q: 5. Review node‑autoscaler limits (if enabled)

kubectl get machinedeployment -n kube-system kubectl describe machinedeployment <gpu-nodegroup> Check that the max size has not been reached, which would prevent new GPU nodes from being provisioned (see the incident where “NodeAffinity mismatch” occurred).

Q: 3. Define a high‑priority class and enable preemption

# PriorityClass definition apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high-priority value: 1000000 globalDefault: false description: "Priority for benchmark workloads that must pre‑empt other GPU pods." Reference the class in the pod template:

In this article

Table of Contents

Problem: ReplicaSet Fails to Scale GPU Pods During Benchmark Load

During a performance benchmark of a deep‑learning workload, a ReplicaSet that requests nvidia.com/gpu resources stalls after a subset of pods become running. The remaining pods stay in Pending with scheduler events such as:

0/5 nodes are available: 5 Insufficient nvidia.com/gpu

Typical impact includes:

Benchmark run never reaches the intended concurrency level.
Cluster metrics show a spike in pending pod count while GPU utilisation appears saturated.
Subsequent jobs are blocked because the scheduler cannot free GPU resources.

Root Cause Analysis

The failure is usually a combination of three interacting factors:

GPU resource exhaustion – Each pod requests 1 nvidia.com/gpu. After the first four pods are scheduled on the four‑GPU nodes, the scheduler reports Insufficient nvidia.com/gpu for the remaining pods (see Kubernetes GPU scheduling docs).
Device plugin state loss under load – The NVIDIA device plugin can restart or report zero GPUs when the driver or runtime is stressed (see device plugin API reference and GitHub issue #453). When the plugin restarts, the scheduler’s cache temporarily loses GPU capacity information, causing newly created pods to be marked unschedulable.
Missing preemption configuration – In multi‑tenant clusters, higher‑priority jobs may pre‑empt lower‑priority GPU pods. If PriorityClass objects are not defined, the scheduler cannot pre‑empt existing pods, leading to FailedScheduling events with reason PreemptionFailed (see GitHub issue #106123).

Debugging and Investigation Steps

1. Inspect ReplicaSet status and pod events

kubectl get rs benchmark-rs -o wide
kubectl describe rs benchmark-rs
kubectl get pods -l app=benchmark -o wide
kubectl describe pod <pending-pod-name>

Typical output snippet:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Normal   Scheduled         2m    default-scheduler  Successfully assigned gpu-node-1 to pod/benchmark-5
  Warning  FailedScheduling  1m    default-scheduler  0/5 nodes are available: 5 Insufficient nvidia.com/gpu.

2. Verify GPU resource availability on each node

kubectl get nodes -L nvidia.com/gpu
kubectl describe node gpu-node-1 | grep -i gpu

Expected output when GPUs are fully allocated:

Name:               gpu-node-1
Capacity:
  nvidia.com/gpu:    4
Allocatable:
  nvidia.com/gpu:    0

3. Check device plugin health

kubectl -n gpu-operator get pods -l app=nvidia-device-plugin
kubectl logs -n gpu-operator <device-plugin-pod>

Look for log lines such as:

Device plugin started with 4 GPUs

or error lines indicating a restart:

Failed to list devices: driver not ready, retrying…

4. Examine scheduler cache and preemption settings

kubectl get schedulerconfig default -o yaml
kubectl describe priorityclass high-priority

If no PriorityClass is defined, the scheduler cannot pre‑empt lower‑priority pods.

5. Review node‑autoscaler limits (if enabled)

kubectl get machinedeployment -n kube-system
kubectl describe machinedeployment <gpu-nodegroup>

Check that the max size has not been reached, which would prevent new GPU nodes from being provisioned (see the incident where “NodeAffinity mismatch” occurred).

Resolution

1. Adjust ReplicaSet pod template to respect actual GPU capacity

Replace a fixed replica count with a calculated value or use a HorizontalPodAutoscaler that respects nvidia.com/gpu metrics.

# Before – static replica count
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: benchmark-rs
spec:
  replicas: 8
  selector:
    matchLabels:
      app: benchmark
  template:
    metadata:
      labels:
        app: benchmark
    spec:
      containers:
      - name: trainer
        image: myorg/trainer:latest
        resources:
          limits:
            nvidia.com/gpu: 1

# After – use HPA with GPU metric (requires custom metrics adapter)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: benchmark-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: ReplicaSet
    name: benchmark-rs
  minReplicas: 1
  maxReplicas: 8
  metrics:
  - type: External
    external:
      metric:
        name: nvidia_gpu_utilization
      target:
        type: AverageValue
        averageValue: "70"

2. Ensure the NVIDIA device plugin remains healthy under load

Apply the following configuration to the GPU Operator to increase plugin watchdog timeout and enable automatic restarts:

# gpu-operator values.yaml snippet
devicePlugin:
  config:
    watchdogTimeoutSeconds: 300
    enablePreemptible: true

Redeploy the operator:

helm upgrade --install gpu-operator nvidia/gpu-operator -f values.yaml

3. Define a high‑priority class and enable preemption

# PriorityClass definition
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "Priority for benchmark workloads that must pre‑empt other GPU pods."

Reference the class in the pod template:

spec:
  priorityClassName: high-priority
  containers:
  - name: trainer
    ...

4. Expand GPU node pool or adjust autoscaler limits

If the benchmark exceeds the physical GPU capacity, increase the node group max size:

# Example for AWS EKS managed node group
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: gpu-cluster
  region: us-west-2
nodeGroups:
  - name: gpu-ng
    instanceType: p3.8xlarge
    desiredCapacity: 4
    maxSize: 8   # increase from 4
    labels:
      nvidia.com/gpu: "true"

Verification

Confirm that the ReplicaSet reaches the expected replica count:

kubectl get rs benchmark-rs

Expected output:

NAME           DESIRED   CURRENT   READY   AGE
benchmark-rs   8         8         8       10m

Check that each pod is scheduled on a node with a free GPU:

kubectl get pods -l app=benchmark -o wide

All pods should show a node name and the nvidia.com/gpu capacity should be non‑zero.

Validate GPU utilisation via the metrics server or Prometheus:

curl -s http://prometheus:9090/api/v1/query?query=nvidia_gpu_utilization

Utilisation should be distributed across the eight pods without any “Pending” spikes.

Inspect device plugin logs for stability:

kubectl logs -n gpu-operator -l app=nvidia-device-plugin

No “Failed to list devices” errors should appear during the benchmark window.

Prevention and Best Practices

Capacity planning: Keep a buffer of at least one free GPU per node to accommodate scheduler cache latency.
GPU‑aware autoscaling: Pair the GPU Operator with a cluster‑autoscaler that respects the nvidia.com/gpu label and can provision new GPU nodes when pending pods exceed capacity.
Priority and preemption: Define PriorityClass objects for latency‑sensitive workloads and enable the scheduler’s preemption plugin (--enable-preemptible-pods=true).
Device plugin health checks: Use liveness probes on the device plugin DaemonSet and configure a watchdog timeout to auto‑restart on driver stalls.
Monitoring and alerts: Alert on events containing “Insufficient nvidia.com/gpu” or “FailedScheduling” with reason “PreemptionFailed”.
Version alignment: Ensure the NVIDIA driver, container runtime, and device plugin versions are compatible (see GPU Operator docs).

FAQ

Why do pods stay Pending even though the node shows free GPU capacity?
Because the device plugin restarted and the scheduler cache was stale. Until the plugin re‑registers the GPUs, the scheduler believes the node has zero allocatable GPUs.
Can I use a NodeSelector instead of a label like nvidia.com/gpu?
A NodeSelector that matches a custom label (e.g., gpu=true) works, but you must also request nvidia.com/gpu so the scheduler accounts for the actual GPU resource.
How does preemption interact with GPU pods?
Preemption only works if pods have a PriorityClass. The scheduler will evict lower‑priority GPU pods to free nvidia.com/gpu resources for higher‑priority ones.
Is there a way to force the scheduler to ignore stale cache entries?
Restarting the scheduler or clearing its cache (e.g., kubectl delete pod -n kube-system -l component=kube-scheduler) forces a refresh, but the preferred solution is to keep the device plugin stable.
Do GPU‑aware scheduler plugins improve scaling under load?
Yes, plugins such as NodeResourcesFit with GPU awareness reduce scheduling latency, but they must be enabled and configured correctly (see scheduler‑plugins issue #212).

Related Topic Hub: GPU Infrastructure Troubleshooting Hub

Kubernetes ReplicaSet scaling issue with GPU resources available

ReplicaSet fails to scale GPU pods after benchmark load