red and blue cargo ship on sea during daytime

Kubernetes ReplicaSet scaling issue with GPU resources available

by

Problem Description

In a Docker‑based AI training workload orchestrated by Kubernetes, a ReplicaSet that should run multiple pods—each requesting a set of GPUs—stops scaling after the first replica. The cluster has sufficient free GPUs (e.g., a node with 8 × NVIDIA A100), but subsequent pods remain in Pending with scheduler messages such as:


0/5 nodes are available: 5 Insufficient nvidia.com/gpu

Symptoms observed:

  • Only the initial replica reaches Running state.
  • Additional replicas stay Pending indefinitely.
  • Node GPU utilization never exceeds the GPUs allocated to the first pod, leading to under‑utilization and longer training runs.
  • Pod events show FailedScheduling or FailedCreatePodSandBox errors.

Root Cause Analysis

The issue originates from a mismatch between how Docker, the NVIDIA device plugin, and the Kubernetes scheduler account for GPU resources.

  • GPU resource accounting: Kubernetes treats nvidia.com/gpu as a scalar resource. When a pod requests 2 GPUs, the scheduler subtracts that amount from the node’s allocatable count. If the device plugin reports the GPUs as already allocated (e.g., because the first pod’s containers are still being created with the default runc runtime), the scheduler sees no free GPUs.
  • Docker runtime configuration: The NVIDIA Container Toolkit must be registered as a runtime (e.g., nvidia) and referenced in the pod spec via runtimeClassName: nvidia or the --gpus flag. If the runtime is not configured, Docker falls back to runc, which cannot expose GPUs. This causes the first pod to succeed (because the device plugin still reports free GPUs) but subsequent pods fail binding, as seen in the Azure AKS incident where FailedCreatePodSandBox was logged.
  • Device plugin version skew: An outdated NVIDIA device plugin (or a kubelet version that does not support the plugin’s API) can report GPUs as “Allocated” after the first bind, even though the node still has free devices. This is documented in the GitHub issue kubernetes#102123.
  • Node selector / topology constraints: When a ReplicaSet uses a nodeSelector that matches only a subset of GPU nodes, the scheduler may repeatedly try the same node, exhausting its reported free GPUs and never considering other nodes.

In short, the scheduler believes there are no free GPUs because either the device plugin reports them as allocated or Docker fails to expose them to the container runtime.

Investigation and Debugging Steps

1. Verify GPU capacity and allocation on each node


kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.capacity.nvidia\.com/gpu}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'

Expected output (example):


gpu-node-1    8    8
gpu-node-2    8    8

2. Inspect the device plugin logs


kubectl -n kube-system logs -l app=nvidia-device-plugin -c nvidia-device-plugin

Look for lines such as Device plugin reported 0 GPUs or “Allocated” status changes.

3. Check pod events for scheduling failures


kubectl describe pod replicaset-pod-name

Typical event snippet:


Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  2m    default-scheduler  0/5 nodes are available: 5 Insufficient nvidia.com/gpu

4. Confirm Docker runtime configuration

On a node, inspect /etc/docker/daemon.json for the NVIDIA runtime entry:


{
  "runtimes": {
    "nvidia": {
      "path": "nvidia-container-runtime",
      "runtimeArgs": []
    }
  },
  "default-runtime": "nvidia"
}

Validate that the NVIDIA Container Toolkit is installed (see NVIDIA documentation install guide).

5. Review the ReplicaSet manifest for GPU request syntax

Common mistake: mixing --gpus flag in the Dockerfile with Kubernetes resource requests, leading to duplicate or conflicting specifications.

Solution

1. Align Docker runtime with Kubernetes GPU requests

Update the node’s Docker daemon to use the NVIDIA runtime as the default, or explicitly reference it via runtimeClassName in the pod spec.


# /etc/docker/daemon.json
{
  "runtimes": {
    "nvidia": {
      "path": "nvidia-container-runtime",
      "runtimeArgs": []
    }
  },
  "default-runtime": "nvidia"
}

Restart Docker:


systemctl restart docker

2. Correct the ReplicaSet manifest

Before (incorrect):


apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: ai-trainer
spec:
  replicas: 4
  selector:
    matchLabels:
      app: trainer
  template:
    metadata:
      labels:
        app: trainer
    spec:
      containers:
      - name: trainer
        image: myregistry/ai-trainer:latest
        resources:
          limits:
            nvidia.com/gpu: "2"
        # Incorrect: using Docker CLI flag in pod spec
        command: ["python", "train.py", "--gpus", "all"]

After (correct):


apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: ai-trainer
spec:
  replicas: 4
  selector:
    matchLabels:
      app: trainer
  template:
    metadata:
      labels:
        app: trainer
    spec:
      runtimeClassName: nvidia               # <-- ensures NVIDIA runtime
      containers:
      - name: trainer
        image: myregistry/ai-trainer:latest
        resources:
          limits:
            nvidia.com/gpu: "2"               # request 2 GPUs per pod
        command: ["python", "train.py"]
        env:
        - name: CUDA_VISIBLE_DEVICES
          value: "0,1"                         # optional explicit device ordering

3. Upgrade the NVIDIA device plugin and kubelet

Ensure both are at compatible versions (e.g., device plugin v0.12.0+ and kubelet ≥1.22). Deploy the latest plugin manifest:


kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.12.0/nvidia-device-plugin.yml

4. Remove overly restrictive nodeSelector or add topologySpreadConstraints if needed

Example of a more permissive selector:


spec:
  template:
    spec:
      nodeSelector:
        accelerator: nvidia-a100

Or distribute pods across nodes:


topologySpreadConstraints:
- maxSkew: 1
  topologyKey: kubernetes.io/hostname
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: trainer

Verification

After applying the changes, run the following checks:

  1. Scale the ReplicaSet and observe pod status:

kubectl scale rs ai-trainer --replicas=4
kubectl get pods -l app=trainer -o wide

All four pods should reach Running and show the node’s GPU capacity reduced accordingly.

  1. Confirm GPU allocation per pod:

kubectl exec -it ai-trainer-xxxx -- nvidia-smi

Output should list two GPUs per pod.

  1. Check node allocatable values again; they should reflect the consumed GPUs (e.g., 8 total – 8 allocated = 0 free).

kubectl describe node gpu-node-1 | grep -A2 "Allocated resources"

Prevention and Best Practices

  • Declare GPU resources only via Kubernetes limits—do not rely on Docker CLI flags inside container commands.
  • Standardize the NVIDIA runtime across all GPU nodes; enforce it with a DaemonSet that validates /etc/docker/daemon.json at node boot.
  • Version‑pin the NVIDIA device plugin and keep kubelet versions aligned; automate upgrades with a CI pipeline.
  • Monitor GPU allocation metrics (e.g., kube_node_status_allocatable_gpu in Prometheus) and set alerts for “Insufficient nvidia.com/gpu” events.
  • Use topology spread constraints or PodAntiAffinity to avoid concentrating many GPU‑heavy pods on a single node.
  • Validate the runtime before deployment with a health‑check pod that runs nvidia-smi and exits 0 only when GPUs are visible.

FAQ

  • Why does the first replica start but subsequent ones stay pending?
    Because the first pod consumes GPUs using the default Docker runtime, causing the device plugin to mark the GPUs as allocated. The scheduler then sees no free GPUs for additional pods.
  • Can I use the --gpus flag in the container command?
    No. GPU allocation must be expressed via Kubernetes resource limits (nvidia.com/gpu) and the appropriate runtime. Mixing CLI flags leads to undefined behavior.
  • What error indicates a missing NVIDIA runtime?
    FailedCreatePodSandBox: failed to allocate GPU or “rpc error: code = Unknown desc = failed to allocate GPU” from Docker.
  • How do I verify which GPU devices a pod sees?
    Execute nvidia-smi inside the pod. The output lists the GPUs exposed to that container.
  • Is there a way to force the scheduler to ignore GPU allocation state?
    No. The scheduler relies on the device plugin’s reported allocatable resources. The correct approach is to fix the plugin/runtime configuration.

Related Topic Hub: Distributed Systems Troubleshooting Hub

Related Articles