AMD GPU Kubernetes PVC Stuck In Pending State

Q: Debug – Investigation Steps 1. Verify Node Labels

$ kubectl get nodes -L amd.com/gpu,amd.com/rocm NAME STATUS ROLES AGE VERSION AMD.COM/GPU AMD.COM/ROCM gpu-node-1 Ready 12d v1.27.3 true gpu-node-2 Ready 12d v1.27.3 true If the label column is empty, the ROCm operator may not have applied the label; reinstall or upgrade the AMD GPU Operator.

Q: 2. Inspect the StorageClass

$ kubectl get sc amd-gpu-sc -o yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: amd-gpu-sc provisioner: rook-ceph.rbd.csi.ceph.com parameters: pool: gpu-pool imageFormat: "2" imageFeatures: layering # allowedTopologies missing → likely cause 3. Check Provisioner Logs $ kubectl logs -n rook-ceph $(kubectl get pod -n rook-ceph -l app=rook-ceph-csi-rbd-provisioner -o name) | grep topology 2024-04-12T08:15:42Z error provisioning volume: topology selector mismatch (required: amd.com/gpu) 4. Confirm Storage Capacity $ ceph df POOL ID USED %USED MAX AVAIL gpu-pool 5 0B 0% 10TiB If the pool is exhausted, the error will be “insufficient storage”. This scenario matches the AI startup incident (2024) where the NFS backend ran out of free space.

Q: Solution – Align StorageClass Topology with AMD GPU Node Labels Step 1 – Create a GPU‑aware StorageClass

Include an allowedTopologies field that references the correct label key (amd.com/gpu or amd.com/rocm depending on driver version).

Q: Step 2 – Delete the Stuck PVC (or edit to use the new class)

$ kubectl delete pvc my-model-pvc $ kubectl apply -f pvc.yaml # pvc.yaml now references storageClassName: amd-gpu-sc Step 3 – Verify that the PVC binds $ kubectl get pvc my-model-pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE my-model-pvc Bound pvc-1234abcd-5678-efgh-9012-ijklmnopqr 500Gi RWO amd-gpu-sc 10s Alternative – Update Existing StorageClass (if no other workloads depend on it) $ kubectl patch sc amd-gpu-sc -p ' { "allowedTopologies": [ { "matchLabelExpressions": [ { "key": "amd.com/gpu", "values": ["true"] } ] } ] }' After patching, delete and recreate the pending PVCs.

In this article

Table of Contents

Problem – PVC Stuck in Pending on an AMD GPU‑Enabled Cluster

In a Kubernetes cluster where nodes are provisioned with AMD GPUs (ROCm), AI workloads that require GPU‑accelerated storage often create a PersistentVolumeClaim (PVC). Operators observe that the PVC never transitions to Bound and remains in the Pending phase despite the presence of storage back‑ends such as Ceph RBD or NFS.

Typical console output:


$ kubectl get pvc my-model-pvc
NAME           STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
my-model-pvc   Pending                                 amd-gpu-sc      2m

Relevant error messages from the controller manager:


persistentvolumeclaim "my-model-pvc" is not bound: no persistent volumes available for claim
Failed to provision volume with StorageClass "amd-gpu-sc": rpc error: code = InvalidArgument desc = "topology selector mismatch"
node affinity mismatch: node(s) didn't match pod's node selector "amd.com/gpu"
Provisioning failed for claim "my-model-pvc": insufficient storage: requested storage 500Gi, available 0Gi

Root Cause – Mismatch Between StorageClass Topology and AMD GPU Node Labels

AMD GPU nodes are labeled by the ROCm operator (see AMD ROCm “Deploying ROCm on Kubernetes”) with a topology key such as amd.com/gpu (or, after driver upgrades, amd.com/rocm). A StorageClass that is intended for GPU‑aware provisioning must declare an allowedTopologies selector that matches these node labels.

When the StorageClass omits the selector or uses an outdated key, the dynamic provisioner cannot find a suitable node to place the volume, leading the PVC to stay pending. This was the exact failure observed in the FinTech AI platform incident (Q2 2024) and the University research lab case (2023), both documented in community reports.

Debug – Investigation Steps

1. Verify Node Labels


$ kubectl get nodes -L amd.com/gpu,amd.com/rocm
NAME          STATUS   ROLES    AGE   VERSION   AMD.COM/GPU   AMD.COM/ROCM
gpu-node-1    Ready       12d   v1.27.3   true
gpu-node-2    Ready       12d   v1.27.3   true

If the label column is empty, the ROCm operator may not have applied the label; reinstall or upgrade the AMD GPU Operator.

2. Inspect the StorageClass


$ kubectl get sc amd-gpu-sc -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: amd-gpu-sc
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  pool: gpu-pool
  imageFormat: "2"
  imageFeatures: layering
# allowedTopologies missing → likely cause

3. Check Provisioner Logs


$ kubectl logs -n rook-ceph $(kubectl get pod -n rook-ceph -l app=rook-ceph-csi-rbd-provisioner -o name) | grep topology
2024-04-12T08:15:42Z error provisioning volume: topology selector mismatch (required: amd.com/gpu)

4. Confirm Storage Capacity


$ ceph df
POOL         ID  USED  %USED  MAX AVAIL
gpu-pool     5   0B    0%     10TiB

If the pool is exhausted, the error will be “insufficient storage”. This scenario matches the AI startup incident (2024) where the NFS backend ran out of free space.

Solution – Align StorageClass Topology with AMD GPU Node Labels

Step 1 – Create a GPU‑aware StorageClass

Include an allowedTopologies field that references the correct label key (amd.com/gpu or amd.com/rocm depending on driver version).


apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: amd-gpu-sc
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  pool: gpu-pool
  imageFormat: "2"
  imageFeatures: layering
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: amd.com/gpu
    values:
    - "true"

Step 2 – Delete the Stuck PVC (or edit to use the new class)


$ kubectl delete pvc my-model-pvc
$ kubectl apply -f pvc.yaml   # pvc.yaml now references storageClassName: amd-gpu-sc

Step 3 – Verify that the PVC binds


$ kubectl get pvc my-model-pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
my-model-pvc   Bound    pvc-1234abcd-5678-efgh-9012-ijklmnopqr   500Gi      RWO            amd-gpu-sc     10s

Alternative – Update Existing StorageClass (if no other workloads depend on it)


$ kubectl patch sc amd-gpu-sc -p '
{
  "allowedTopologies": [
    {
      "matchLabelExpressions": [
        {
          "key": "amd.com/gpu",
          "values": ["true"]
        }
      ]
    }
  ]
}'

After patching, delete and recreate the pending PVCs.

Verify – Confirm End‑to‑End Functionality

PVC Status: kubectl get pvc shows Bound.
Pod Scheduling: Pod that consumes the PVC should schedule on a GPU node without “node affinity mismatch” events.
Volume Mount: kubectl describe pod should not contain “unmounted volumes” errors.
Metrics: Ceph RBD CSI metrics (e.g., csi_rbd_provision_success_total) increment.
Application Test: Run a short training job that writes to the PVC; verify data persists after pod restart.

Prevent – Guardrails for Future Deployments

Guardrail	Implementation
Enforce StorageClass topology policy	Include `allowedTopologies` in all GPU‑specific StorageClasses; CI lint rule checks for missing field.
Label consistency after driver upgrades	Automate a post‑upgrade job that re‑applies `amd.com/gpu=true` (or `amd.com/rocm`) to all GPU nodes.
Capacity monitoring	Alert on `CephPoolFull` or NFS free‑space < 10% using Prometheus.
Pod‑PVC topology validation	Use `kubectl wait --for=condition=Ready pod/<name>` in CI pipelines; fail if PVC remains pending beyond 30s.

FAQ – Common Follow‑Up Questions

Why does the PVC bind only after I delete and recreate it? The existing PVC was already bound (or attempted to bind) with a topology selector that no longer matches any node. Updating the PVC’s storageClassName does not retroactively change the selector, so a fresh PVC is required.
Can I use the generic gp2 StorageClass for GPU workloads? No. Generic classes lack the required allowedTopologies and will cause “topology selector mismatch” errors on AMD GPU nodes, as seen in the FinTech incident.
My nodes are labeled amd.com/rocm after an upgrade—how do I adapt? Update all GPU‑specific StorageClasses to reference the new key, or add both keys in allowedTopologies to maintain compatibility.
Is volumeBindingMode: Immediate acceptable for GPU storage? For GPU‑aware storage, WaitForFirstConsumer is recommended. It defers provisioning until the pod’s node affinity is known, preventing topology mismatches.
How do I troubleshoot “insufficient storage” errors on GPU PVCs? Check the underlying pool or NFS export capacity, verify quota settings, and ensure no stale PVs are holding space. Use ceph df or df -h on the NFS server.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub