PersistentVolumeClaim pending after deploying LLaMA multi‑GPU training pod

PersistentVolumeClaim Pending After Deploying LLaMA Multi‑GPU Training Pod

Problem

When launching a Kubernetes pod that runs a multi‑GPU training job for the Meta LLaMA model, the pod remains in Pending because its associated PersistentVolumeClaim (PVC) never binds to a PersistentVolume (PV). The pod logs show:


Warning  FailedScheduling  12s (x3 over 30s)  default-scheduler  
  0/5 nodes are available: 5 node(s) had insufficient cpu, 5 node(s) had insufficient memory, 5 node(s) had no volume node affinity.

and the PVC status is:


NAME            STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
llama-data-pvc  Pending                                   fast-ssd        2m

The training job cannot start, causing a cascade of resource under‑utilisation on the GPU nodes.

Root Cause

The PVC remains pending because the cluster’s volume binding mode is set to WaitForFirstConsumer (the default for most dynamic provisioners). With this mode, the scheduler must first place the pod on a node before the PV is provisioned. The pod requests a nodeSelector that restricts scheduling to GPU nodes with the label accelerator=nvidia. However, the storage class fast-ssd is provisioned only on a subset of nodes that have the label ssd=true. No node in the cluster satisfies both accelerator=nvidia and ssd=true, so the scheduler cannot find a node with the required volume node affinity. Consequently, the PVC stays in Pending and the pod never schedules.

Debug

  1. Inspect PVC details
    
    kubectl describe pvc llama-data-pvc
    

    Key output:

    
    Events:
      Type    Reason                Age   From                         Message
      ----    ------                ----  ----                         -------
      Normal  Provisioning          1m    pvc-controller               External provisioner is provisioning volume
      Warning ProvisioningFailed    30s   pvc-controller               failed to provision volume with StorageClass "fast-ssd": no suitable node found for volume node affinity
    
  2. Check storage class definition
    
    kubectl get sc fast-ssd -o yaml
    
    
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: fast-ssd
    provisioner: ebs.csi.aws.com
    volumeBindingMode: WaitForFirstConsumer
    parameters:
      type: gp3
      iopsPerGB: "10"
      encrypted: "true"
    allowedTopologies:
    - matchLabelExpressions:
      - key: ssd
        values:
        - "true"
    
  3. Verify node labels
    
    kubectl get nodes -L accelerator,ssd
    
    
    NAME          STATUS   ROLES    AGE   VERSION   ACCELERATOR   SSD
    gpu-node-1    Ready       45d   v1.27.3   nvidia        false
    gpu-node-2    Ready       45d   v1.27.3   nvidia        false
    ssd-node-1    Ready       30d   v1.27.3           true
    ssd-node-2    Ready       30d   v1.27.3           true
    

    There is no node where accelerator=nvidia and ssd=true coexist.

  4. Examine pod spec
    
    kubectl get pod llama-train -o yaml
    
    
    apiVersion: v1
    kind: Pod
    metadata:
      name: llama-train
    spec:
      nodeSelector:
        accelerator: nvidia
      containers:
      - name: trainer
        image: meta/llama:latest
        resources:
          limits:
            nvidia.com/gpu: 4
        volumeMounts:
        - name: data
          mountPath: /data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: llama-data-pvc
    

Solution

Two practical approaches resolve the affinity mismatch:

Option 1 – Align node labeling with the storage class

Add the ssd=true label to the GPU nodes (or attach SSD‑backed storage to them) so that they satisfy the storage class topology constraints.


# Label GPU nodes
kubectl label node gpu-node-1 ssd=true
kubectl label node gpu-node-2 ssd=true

After labeling, the scheduler can place the pod on a GPU node that also matches the PV topology, allowing the dynamic provisioner to create the volume.

Option 2 – Remove topology constraints from the storage class

If SSD locality is not a strict requirement, edit the storage class to drop allowedTopologies and optionally change volumeBindingMode to Immediate so the PV is provisioned before scheduling.


# Export current SC, edit, and apply
kubectl get sc fast-ssd -o yaml > fast-ssd.yaml

# fast-ssd.yaml (before)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: ssd
    values:
    - "true"

# fast-ssd.yaml (after)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
volumeBindingMode: Immediate
# allowedTopologies removed

kubectl replace -f fast-ssd.yaml

Re‑create the PVC (or delete and let the pod recreate it) so the new storage class definition takes effect.

Option 3 – Use a separate PVC per GPU node pool

Create two storage classes: one for GPU nodes (fast-ssd-gpu) without topology constraints, and another for pure SSD nodes. Update the pod to reference the appropriate PVC.

Verification

  1. Confirm node labels (if Option 1 was chosen):
    
    kubectl get nodes -L accelerator,ssd
    

    Output should show ssd=true on the GPU nodes.

  2. Check PVC binding:
    
    kubectl get pvc llama-data-pvc
    

    Expected status:

    
    NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    llama-data-pvc  Bound    pvc-1234abcd-5678-efgh-9012-ijklmnopqrst   500Gi      RWO            fast-ssd       3m
    
  3. Verify pod scheduling:
    
    kubectl get pod llama-train -o wide
    

    Pod should transition to Running and be assigned to a GPU node.

  4. Inspect pod logs for successful start:
    
    kubectl logs llama-train -c trainer
    
    
    [INFO] Initializing LLaMA trainer...
    [INFO] Loading dataset from /data
    [INFO] Distributed training started on 4 GPUs
    

Prevention

  • Consistent node labeling: Enforce a policy (via kubectl label automation or a mutating webhook) that any node intended for GPU workloads also carries the storage topology labels required by the default storage class.
  • Explicit storage class selection: Define separate storage classes for GPU‑enabled nodes and for generic compute nodes, and reference the appropriate class in PVC manifests.
  • Validate topology compatibility in CI: Include a step that runs kubectl get nodes -L <labels> and checks that every combination of nodeSelector and allowedTopologies has at least one matching node before merging manifests.
  • Prefer Immediate binding for fast‑provisioned volumes when locality is not a performance bottleneck; this eliminates the “wait‑for‑consumer” deadlock.
  • Monitoring alerts: Create a Prometheus rule that fires when a PVC stays in Pending longer than 2 minutes, attaching the PVC name and namespace to the alert payload.

FAQ

  1. Why does the PVC stay pending only after adding nodeSelector for GPUs?
    Because the storage class restricts provisioning to nodes with ssd=true. The node selector limits scheduling to nodes without that label, leaving no node that satisfies both constraints.
  2. Can I keep WaitForFirstConsumer and still make the PVC bind?
    Yes, if you add the required topology label to the GPU nodes (or adjust the storage class to include the GPU label in its allowedTopologies).
  3. Is changing volumeBindingMode to Immediate safe for multi‑GPU training?
    It is safe as long as the underlying storage does not need to be physically attached to a specific node before the pod starts. For cloud‑managed block storage (e.g., AWS EBS), the volume can be attached to the node at pod start time.
  4. How do I know which storage class a PVC is using?
    Run kubectl get pvc <name> -o jsonpath='{.spec.storageClassName}'. If the field is empty, the cluster default storage class is applied.
  5. What if I need both SSD locality and GPU affinity but have no nodes that satisfy both?
    Provision a new node pool that includes GPUs and SSD‑backed storage, label those nodes accordingly, and schedule the training workload onto that pool.

Related Topic Hub: LLM Systems Troubleshooting Hub