A person holding a cell phone in their hand

DeepSeek training job PVC pending after node upgrade on A100 cluster

by

Problem

After upgrading the GPU nodes in the A100/H100‑based training cluster, DeepSeek training jobs are created but never start. The associated PersistentVolumeClaim (PVC) stays in the Pending phase indefinitely, preventing the pod from mounting the required dataset and model checkpoint storage.

Root Cause

The node upgrade introduced a change in the kubelet volumeBindingMode for the StorageClass used by DeepSeek (deepseek-gpu-storage). The previous version used WaitForFirstConsumer, which defers volume provisioning until a pod is scheduled. After the upgrade the default switched to Immediate. Because the GPU nodes are tainted (nvidia.com/gpu=present:NoSchedule) and the scheduler cannot place the pod on a node without satisfying the GPU request, the PVC is bound to a node that lacks the required GPU resources. The scheduler then marks the PVC as Pending and never retries the binding, resulting in a deadlock:

  • Pod requests nvidia.com/gpu: 1 (via resourceLimits).
  • PVC is bound to a non‑GPU node because the storage class provisioner allocated the volume immediately.
  • Pod cannot be scheduled on a GPU node because the volume is already attached to a different node.

Debug

1. Observe the PVC status

kubectl get pvc -n deepseek -l app=deepseek-training
NAME                STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
deepseek-data-pvc   Pending   <none>                                      500Gi      RWO            deepseek-gpu-storage  2m

2. Check the StorageClass definition

kubectl get sc deepseek-gpu-storage -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: deepseek-gpu-storage
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: Immediate   # <-- changed after node upgrade

3. Verify node taints and GPU resources

kubectl get nodes -l node-type=gpu -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'
ip-10-0-1-12    [{"key":"nvidia.com/gpu","value":"present","effect":"NoSchedule"}]
ip-10-0-1-13    [{"key":"nvidia.com/gpu","value":"present","effect":"NoSchedule"}]

kubectl describe node ip-10-0-1-12 | grep -i "nvidia.com/gpu"
Capacity:
  nvidia.com/gpu: 8
Allocatable:
  nvidia.com/gpu: 8

4. Scheduler events for the pod

kubectl describe pod -n deepseek deepseek-trainer-abc123
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  1m    default-scheduler  0/2 nodes are available: 1 Insufficient cpu, 1 node(s) had volume node affinity conflict.

The “volume node affinity conflict” line confirms that the PVC is bound to a node that does not satisfy the pod’s GPU node selector.

Solution

Restore the original volumeBindingMode to WaitForFirstConsumer so that the provisioner defers volume creation until the pod is scheduled on a GPU‑compatible node.

Step‑by‑step fix

  1. Patch the StorageClass:
kubectl patch storageclass deepseek-gpu-storage -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl edit storageclass deepseek-gpu-storage
# Change the line:
# volumeBindingMode: Immediate
# to:
# volumeBindingMode: WaitForFirstConsumer

Before

volumeBindingMode: Immediate

After

volumeBindingMode: WaitForFirstConsumer

Save and exit the editor. The change forces the CSI driver to wait for the pod’s node affinity before provisioning the EBS volume.

2. Delete the stuck PVCs (optional)

If PVCs are already in Pending and bound to the wrong node, delete them so that they are recreated with the corrected binding mode.

kubectl delete pvc -n deepseek deepseek-data-pvc
# The training controller will recreate the PVC automatically.

3. Restart the training job

kubectl delete pod -n deepseek deepseek-trainer-abc123
# The controller will launch a new pod that will now bind the PVC to a GPU node.

Verify

1. PVC should become Bound

kubectl get pvc -n deepseek deepseek-data-pvc
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
deepseek-data-pvc   Bound    pvc-1234abcd-5678-efgh-9012-ijklmnopqrst  500Gi      RWO            deepseek-gpu-storage  1m

2. Pod should schedule successfully

kubectl get pod -n deepseek -l app=deepseek-training
NAME                READY   STATUS    RESTARTS   AGE
deepseek-trainer-xyz 1/1     Running   0          30s

3. Confirm volume attachment on a GPU node

kubectl describe pod -n deepseek deepseek-trainer-xyz | grep -i "VolumeMounts"
VolumeMounts:
  /data from deepseek-data-pvc (rw)

Prevent

  • Lock the StorageClass configuration. Add an annotation or a policy that prevents accidental changes to volumeBindingMode during cluster upgrades.
  • Upgrade validation checklist. Include a step that verifies the volumeBindingMode of all storage classes used by GPU workloads before applying node‑level changes.
  • Automated tests. Deploy a minimal training job in a staging environment after each upgrade and assert that PVCs reach Bound within a defined SLA.
  • Node taint awareness. Ensure that any new StorageClass defaults to WaitForFirstConsumer when the cluster contains tainted GPU nodes.
  • Monitoring. Add a Prometheus alert on kube_persistentvolumeclaim_status_phase{phase="Pending"} filtered by the DeepSeek namespace, with a firing threshold of >5 minutes.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  1. Why does the PVC become pending only after a node upgrade?
    The upgrade replaced the default StorageClass parameters, switching volumeBindingMode from WaitForFirstConsumer to Immediate. Immediate provisioning attaches the volume to the first node the controller contacts, which is often a non‑GPU node, causing a node‑affinity conflict.
  2. Can I keep Immediate mode and still run GPU jobs?
    Only if you add explicit nodeAffinity rules to the PVC that match GPU nodes, or you remove the GPU taint from non‑GPU nodes. This adds complexity and is error‑prone; WaitForFirstConsumer is the recommended approach for heterogeneous clusters.
  3. What if I need dynamic provisioning for non‑GPU workloads on the same StorageClass?
    Create a separate StorageClass (e.g., deepseek-gpu-storage-wffc) with WaitForFirstConsumer for GPU jobs, and keep the existing Immediate class for CPU‑only workloads. Use the appropriate class in the pod spec.
  4. Is there a way to force a re‑binding of an already provisioned volume?
    Kubernetes does not support moving an existing volume between nodes. The typical workflow is to delete the PVC (and any dependent pod) and let the controller recreate it, which triggers a fresh provisioning cycle respecting the current volumeBindingMode.
  5. How can I detect this issue before it impacts production?
    Add a pre‑upgrade validation script that queries all StorageClasses for volumeBindingMode and fails the CI pipeline if any GPU‑related class is set to Immediate. Combine this with a smoke‑test that launches a short‑lived GPU pod and checks PVC status.