Amd ryzen 6000 series processor on circuit board background

PVC remains pending on AMD GPU nodes during training job launch

by

Problem Description

In an AMD‑GPU‑enabled Kubernetes cluster, AI training jobs launched by an event‑driven pipeline (e.g., Kubeflow + Argo Events) fail to start because the associated PersistentVolumeClaim (PVC) stays in the Pending state. The pod is scheduled onto a GPU node, but the volume never binds, leading to errors such as:


persistentvolumeclaim "my-pvc" is pending – event: "no volume plugin found for claim".
Failed to bind volume: no persistent volumes available for claim "default/my-pvc".
MountVolume.SetUp failed for volume "pvc-1234abcd": could not attach volume: device not found

Operational impact includes:

  • Training jobs never start, causing pipeline stalls.
  • Auto‑scaled GPU node pools waste resources while waiting for storage.
  • Multi‑tenant inference services report “PVC not bound” errors, breaking SLA.

Root Cause Analysis

The PVC remains pending because the Kubernetes volume binding controller cannot find a PersistentVolume (PV) that satisfies the claim’s storageClassName, nodeSelector, and tolerations constraints on AMD GPU nodes. The underlying reasons, documented in the AMD ROCm Kubernetes Integration Guide and the AMD GPU Operator Documentation, are:

  1. Missing or mismatched StorageClass: The claim references storageClassName: amd-gpu-sc, but the operator has created a class named amd-rocm-sc. This mismatch triggers the “StorageClass not found” error.
  2. Node taints without matching tolerations: GPU nodes are tainted with amd.com/gpu=NoSchedule (as required by the GPU Operator). Pods that do not declare the corresponding toleration cannot be scheduled, and the PVC binding controller discards those nodes when searching for a suitable PV.
  3. Node selector mismatch: The PVC (or the pod’s volumeClaimTemplates) includes nodeSelector: { "amd.com/gpu": "true" }, but the actual node label is amd.com/gpu.present=true. This discrepancy is highlighted in the GitHub issue rocm/k8s-device-plugin #312.
  4. Race condition during auto‑scaling: When a new GPU node is provisioned, the storage class controller may not have created a PV yet. The PVC is created first, leading to a temporary “no PV available” state that persists if the pod is scheduled before the PV appears (see the real incident with CI/CD auto‑scaling).

Collectively, these factors prevent the binding controller from locating a compatible PV, leaving the PVC in Pending.

Investigation and Debugging Steps

Follow the sequence below to isolate the exact failure point.

1. Inspect PVC status and events

kubectl get pvc my-pvc -n training -o wide
kubectl describe pvc my-pvc -n training

Typical output:


Name:          my-pvc
Namespace:     training
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
StorageClass:  amd-gpu-sc
Status:        Pending
Events:
  Type    Reason                Age   From                         Message
  ----    ------                ----  ----                         -------
  Normal  ProvisioningFailed    2m    persistentvolume-controller  no persistent volumes available for claim "training/my-pvc"
  Normal  FailedBinding         2m    persistentvolume-controller  no volume plugin found for claim

2. Verify StorageClass existence

kubectl get sc

If amd-gpu-sc is missing, create or rename it to match the claim.

3. Check node taints and labels

kubectl get nodes -L amd.com/gpu -o wide
kubectl describe node <gpu-node>

Example snippet:


Name:               gpu-node-01
Labels:             amd.com/gpu.present=true
Taints:             amd.com/gpu=NoSchedule:NoExecute

4. Review pod spec for tolerations and nodeSelector

kubectl get pod training-job-abcde -n training -o yaml

Key sections to look for:


spec:
  nodeSelector:
    amd.com/gpu: "true"
  tolerations:
  - key: "amd.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"

If the selector or tolerations do not align with the node’s labels/taints, the pod cannot be placed, and the PVC binding fails.

5. Examine the storage provisioner logs

kubectl logs -n kube-system -l app=rook-ceph-operator
# or, if using a CSI driver:
kubectl logs -n kube-system -l app=amd-gpu-csi-provisioner

Look for messages such as “no matching node for PVC” or “failed to create volume for claim”.

6. Detect race conditions

If the PVC is created before the GPU node finishes provisioning, the controller may log:


Failed to bind volume: no persistent volumes available for claim "training/my-pvc"

Confirm by checking the node’s Ready condition timestamp versus PVC creation time.

Resolution

Apply the fixes that address each identified cause. Below are the most common corrective actions.

1. Align StorageClass name

Before (incorrect PVC spec):


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  storageClassName: amd-gpu-sc   # does not exist
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi

After (matching existing class):


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  storageClassName: amd-rocm-sc   # created by GPU Operator
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi

2. Add required tolerations to the pod (or Job) spec

Before (missing toleration):


spec:
  containers:
  - name: trainer
    image: mytrainer:latest
  nodeSelector:
    amd.com/gpu.present: "true"

After (toleration added):


spec:
  containers:
  - name: trainer
    image: mytrainer:latest
  nodeSelector:
    amd.com/gpu.present: "true"
  tolerations:
  - key: "amd.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"

3. Correct nodeSelector label

Replace the outdated selector with the actual node label used by the GPU Operator.


# Incorrect
nodeSelector:
  amd.com/gpu: "true"

# Correct
nodeSelector:
  amd.com/gpu.present: "true"

4. Guard against race conditions

Introduce a initContainer that waits for the PV to become Bound before the main training container starts.


apiVersion: batch/v1
kind: Job
metadata:
  name: training-job
spec:
  template:
    spec:
      initContainers:
      - name: wait-for-pvc
        image: bitnami/kubectl:latest
        command: ["sh", "-c", "
          while [ $(kubectl get pvc my-pvc -n training -o jsonpath='{.status.phase}') != 'Bound' ]; do
            echo 'Waiting for PVC to bind...';
            sleep 5;
          done
        "]
        volumeMounts:
        - name: pvc
          mountPath: /mnt
      containers:
      - name: trainer
        image: mytrainer:latest
        volumeMounts:
        - name: pvc
          mountPath: /data
      volumes:
      - name: pvc
        persistentVolumeClaim:
          claimName: my-pvc
      restartPolicy: Never

Validation

After applying the changes, verify each step:

  1. PVC binding 
    kubectl get pvc my-pvc -n training
# Expected output:
NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
my-pvc   Bound    pvc-1234abcd-5678-efgh-9012-ijklmnopqrst   200Gi      RWO            amd-rocm-sc    1m
  2. Pod scheduling 
    kubectl get pod -n training -l job-name=training-job -o wide
# Pod should be on a node with the GPU taint and show READY=1/1.
  3. Training container logs 
    kubectl logs training-job-abcde -n training
# Look for successful data mount messages, e.g.:
Mounting /data (200Gi) on /mnt/training-data
Training started...
  4. Metrics – confirm that the storage I/O metrics (e.g., kubelet_volume_stats_used_bytes) are reporting non‑zero values for the PVC.

Prevention and Best Practices

  • Standardize StorageClass naming: Use a cluster‑wide convention (e.g., amd-rocm-sc) and enforce it via a ValidatingAdmissionWebhook.
  • Label and taint consistency: Document the exact node labels (amd.com/gpu.present) and taints (amd.com/gpu=NoSchedule) introduced by the AMD GPU Operator. Ensure all workload manifests reference them.
  • Automated PVC health checks: Deploy a Prometheus rule that fires when a PVC in the training namespace stays Pending longer than 2 minutes.
  • Graceful node provisioning: In auto‑scale pipelines, add a postStart hook that waits for the node’s Ready condition and the storage provisioner to report the expected PV before creating the PVC.
  • Use CSI topology constraints: If the CSI driver supports topologyKeys, configure the StorageClass to include topology.kubernetes.io/zone or amd.com/gpu so the controller only creates PVs on compatible nodes.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub

FAQ

  1. Why does the PVC stay pending only on newly added GPU nodes?
    Because the GPU Operator adds a amd.com/gpu=NoSchedule taint. Pods without the matching toleration cannot be scheduled, and the binding controller discards those nodes when searching for a PV.
  2. How can I confirm which StorageClass a PVC is using?
    Run kubectl get pvc <name> -o jsonpath='{.spec.storageClassName}'. Compare the result with kubectl get sc to ensure the class exists.
  3. Can a PVC bind to a node that does not have a GPU?
    Yes, if the PVC does not specify a nodeSelector or topologyConstraints. Adding a selector that matches the GPU node label forces the binding controller to consider only GPU‑enabled nodes.
  4. What is the recommended way to handle the race between node provisioning and PVC creation?
    Either delay PVC creation until the node reports Ready (e.g., via a Kubernetes Event listener) or use an initContainer that waits for the PVC to reach Bound before the main container starts.
  5. Why do I see “no volume plugin found for claim” in the events?
    This indicates that the cluster’s volume provisioner could not locate a PV matching the claim’s parameters—most often caused by a missing or mismatched StorageClass, or by node selector/taint constraints that exclude all available nodes.