PersistentVolumeClaim Pending After Deploying LLaMA Multi‑GPU Training Pod
Problem
When launching a Kubernetes pod that runs a multi‑GPU training job for the Meta LLaMA model, the pod remains in Pending because its associated PersistentVolumeClaim (PVC) never binds to a PersistentVolume (PV). The pod logs show:
Warning FailedScheduling 12s (x3 over 30s) default-scheduler
0/5 nodes are available: 5 node(s) had insufficient cpu, 5 node(s) had insufficient memory, 5 node(s) had no volume node affinity.
and the PVC status is:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
llama-data-pvc Pending fast-ssd 2m
The training job cannot start, causing a cascade of resource under‑utilisation on the GPU nodes.
Root Cause
The PVC remains pending because the cluster’s volume binding mode is set to WaitForFirstConsumer (the default for most dynamic provisioners). With this mode, the scheduler must first place the pod on a node before the PV is provisioned. The pod requests a nodeSelector that restricts scheduling to GPU nodes with the label accelerator=nvidia. However, the storage class fast-ssd is provisioned only on a subset of nodes that have the label ssd=true. No node in the cluster satisfies both accelerator=nvidia and ssd=true, so the scheduler cannot find a node with the required volume node affinity. Consequently, the PVC stays in Pending and the pod never schedules.
Debug
- Inspect PVC details
kubectl describe pvc llama-data-pvcKey output:
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Provisioning 1m pvc-controller External provisioner is provisioning volume Warning ProvisioningFailed 30s pvc-controller failed to provision volume with StorageClass "fast-ssd": no suitable node found for volume node affinity - Check storage class definition
kubectl get sc fast-ssd -o yamlapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast-ssd provisioner: ebs.csi.aws.com volumeBindingMode: WaitForFirstConsumer parameters: type: gp3 iopsPerGB: "10" encrypted: "true" allowedTopologies: - matchLabelExpressions: - key: ssd values: - "true" - Verify node labels
kubectl get nodes -L accelerator,ssdNAME STATUS ROLES AGE VERSION ACCELERATOR SSD gpu-node-1 Ready45d v1.27.3 nvidia false gpu-node-2 Ready 45d v1.27.3 nvidia false ssd-node-1 Ready 30d v1.27.3 true ssd-node-2 Ready 30d v1.27.3 true There is no node where
accelerator=nvidiaandssd=truecoexist. - Examine pod spec
kubectl get pod llama-train -o yamlapiVersion: v1 kind: Pod metadata: name: llama-train spec: nodeSelector: accelerator: nvidia containers: - name: trainer image: meta/llama:latest resources: limits: nvidia.com/gpu: 4 volumeMounts: - name: data mountPath: /data volumes: - name: data persistentVolumeClaim: claimName: llama-data-pvc
Solution
Two practical approaches resolve the affinity mismatch:
Option 1 – Align node labeling with the storage class
Add the ssd=true label to the GPU nodes (or attach SSD‑backed storage to them) so that they satisfy the storage class topology constraints.
# Label GPU nodes
kubectl label node gpu-node-1 ssd=true
kubectl label node gpu-node-2 ssd=true
After labeling, the scheduler can place the pod on a GPU node that also matches the PV topology, allowing the dynamic provisioner to create the volume.
Option 2 – Remove topology constraints from the storage class
If SSD locality is not a strict requirement, edit the storage class to drop allowedTopologies and optionally change volumeBindingMode to Immediate so the PV is provisioned before scheduling.
# Export current SC, edit, and apply
kubectl get sc fast-ssd -o yaml > fast-ssd.yaml
# fast-ssd.yaml (before)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast-ssd
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
- key: ssd
values:
- "true"
# fast-ssd.yaml (after)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast-ssd
provisioner: ebs.csi.aws.com
volumeBindingMode: Immediate
# allowedTopologies removed
kubectl replace -f fast-ssd.yaml
Re‑create the PVC (or delete and let the pod recreate it) so the new storage class definition takes effect.
Option 3 – Use a separate PVC per GPU node pool
Create two storage classes: one for GPU nodes (fast-ssd-gpu) without topology constraints, and another for pure SSD nodes. Update the pod to reference the appropriate PVC.
Verification
- Confirm node labels (if Option 1 was chosen):
kubectl get nodes -L accelerator,ssdOutput should show
ssd=trueon the GPU nodes. - Check PVC binding:
kubectl get pvc llama-data-pvcExpected status:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE llama-data-pvc Bound pvc-1234abcd-5678-efgh-9012-ijklmnopqrst 500Gi RWO fast-ssd 3m - Verify pod scheduling:
kubectl get pod llama-train -o widePod should transition to
Runningand be assigned to a GPU node. - Inspect pod logs for successful start:
kubectl logs llama-train -c trainer[INFO] Initializing LLaMA trainer... [INFO] Loading dataset from /data [INFO] Distributed training started on 4 GPUs
Prevention
- Consistent node labeling: Enforce a policy (via
kubectl labelautomation or a mutating webhook) that any node intended for GPU workloads also carries the storage topology labels required by the default storage class. - Explicit storage class selection: Define separate storage classes for GPU‑enabled nodes and for generic compute nodes, and reference the appropriate class in PVC manifests.
- Validate topology compatibility in CI: Include a step that runs
kubectl get nodes -L <labels>and checks that every combination ofnodeSelectorandallowedTopologieshas at least one matching node before merging manifests. - Prefer
Immediatebinding for fast‑provisioned volumes when locality is not a performance bottleneck; this eliminates the “wait‑for‑consumer” deadlock. - Monitoring alerts: Create a Prometheus rule that fires when a PVC stays in
Pendinglonger than 2 minutes, attaching the PVC name and namespace to the alert payload.
FAQ
- Why does the PVC stay pending only after adding
nodeSelectorfor GPUs?
Because the storage class restricts provisioning to nodes withssd=true. The node selector limits scheduling to nodes without that label, leaving no node that satisfies both constraints. - Can I keep
WaitForFirstConsumerand still make the PVC bind?
Yes, if you add the required topology label to the GPU nodes (or adjust the storage class to include the GPU label in itsallowedTopologies). - Is changing
volumeBindingModetoImmediatesafe for multi‑GPU training?
It is safe as long as the underlying storage does not need to be physically attached to a specific node before the pod starts. For cloud‑managed block storage (e.g., AWS EBS), the volume can be attached to the node at pod start time. - How do I know which storage class a PVC is using?
Runkubectl get pvc <name> -o jsonpath='{.spec.storageClassName}'. If the field is empty, the cluster default storage class is applied. - What if I need both SSD locality and GPU affinity but have no nodes that satisfy both?
Provision a new node pool that includes GPUs and SSD‑backed storage, label those nodes accordingly, and schedule the training workload onto that pool.
Related Topic Hub: LLM Systems Troubleshooting Hub