Weaviate Pods Pending After GPU Request Exceeds Node Capacity

Q: 1. Update the NVIDIA device plugin configuration

Set maxPodsPerGPU to 0 (unlimited) or a value that matches the expected pod density.

Q: 4. Simplify affinity when topology is not a hard requirement

If NCCL can operate across nodes (e.g., using nccl-p2p over RDMA), relax the affinity to allow the scheduler to split the request across multiple nodes using PodTopologySpread or PreferredDuringSchedulingIgnoredDuringExecution. Example:

Q: 5. Deploy the corrected pod definition

kubectl apply -f weaviate-trainer-pod.yaml Verification Confirm that the node now reports the correct GPU count: kubectl get node gpu-node-1 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}' # Expected output: 4 Check that the pod transitions to Running: kubectl get pod weaviate-trainer -w # Output should show STATUS: Running Inspect the trainer logs for successful NCCL initialization: kubectl logs weaviate-trainer -c trainer | grep NCCL # Expected line: INFO NCCL: Initialized communication across 4 GPUs. Validate GPU utilization with nvidia-smi on the node: kubectl exec -it weaviate-trainer -- nvidia-smi # Should list 4 processes using the GPUs. Prevention Automated capacity checks: Add a pre‑deployment script that queries kubectl get nodes -o jsonpath for allocatable.nvidia.com/gpu and aborts if the requested count exceeds the maximum per node. Version lock the NVIDIA device plugin: Use a known‑good version (e.g., v0.14.0) and pin the maxPodsPerGPU flag to 0 in the daemonset. Label hygiene: Enforce a CI/CD policy that validates required node labels exist on all GPU nodes before merging changes to pod specs. Topology‑aware scheduling: When strict colocation is required, use the TopologySpreadConstraints feature or the NodeResourceTopology API (via the topology-aware-scheduling feature gate) to make the scheduler aware of GPU NUMA topology. Monitoring alerts: Create a Prometheus alert on kube_pod_status_phase{phase="Pending"} combined with a label selector for nvidia.com/gpu to catch future pending‑GPU‑pod incidents within 5 minutes. Related Topic Hub: Vector Databases Troubleshooting Hub

In this article

Table of Contents

Problem

When deploying Weaviate for multi‑GPU training workloads, the pods remain in Pending state indefinitely. The Kubernetes scheduler reports that the GPU resource request cannot be satisfied on any node, even though the cluster contains nodes with multiple GPUs.

Typical pod description:

apiVersion: v1
kind: Pod
metadata:
  name: weaviate-trainer
spec:
  containers:
  - name: trainer
    image: weaviate/trainer:latest
    resources:
      limits:
        nvidia.com/gpu: 4
    env:
    - name: NODE_SELECTOR
      value: "gpu-node"
  nodeSelector:
    accelerator: nvidia
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values: ["us-east-1a"]

Observed scheduler events:

Warning  FailedScheduling  2m31s (x2 over 3m)  default-scheduler  0/3 nodes are available: 3 Insufficient nvidia.com/gpu.

Result: the Weaviate training job never starts, causing downstream pipelines to stall.

Root Cause

The failure stems from a mismatch between the GPU resource request (nvidia.com/gpu: 4) and the actual GPU capacity exposed by the node’s device plugin. Several factors contribute:

Node capacity vs. request: Each GPU‑enabled node in the cluster reports only 2 GPUs to the scheduler (e.g., allocatable: nvidia.com/gpu: 2), so a request for 4 GPUs cannot be placed.
Node affinity constraints: The pod explicitly restricts placement to nodes with accelerator: nvidia and a specific zone. If only a subset of nodes carry those labels, the scheduler’s view of available GPUs shrinks further.
GPU topology awareness: Weaviate’s training code expects all requested GPUs to be on the same physical node for NCCL communication. The scheduler, however, only sees aggregate GPU counts and does not enforce colocation beyond the node selector.
Device plugin version drift: Some nodes run an older NVIDIA device plugin that reports a lower allocatable count due to a mis‑configured maxPodsPerGPU flag.

Consequently, the scheduler cannot find any node that simultaneously satisfies the resource request, node selector, and affinity rules, leading to the Pending state.

Debugging the Issue

Inspect node GPU capacity:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'

Expected output (healthy node):

gpu-node-1    4
gpu-node-2    4

If the output shows 2 or 0, the node is under‑reporting.

Review device plugin logs on a problematic node:

kubectl logs -n kube-system -l app=nvidia-device-plugin -c nvidia-device-plugin -f

Typical error:

Failed to allocate GPU resources: maxPodsPerGPU set to 1, but node has 2 GPUs; only 2 pods can be scheduled.

Check pod events for scheduling failures:
```
kubectl describe pod weaviate-trainer
```
Look for the FailedScheduling messages that list the exact reason (e.g., Insufficient nvidia.com/gpu).
Validate node labels and affinity:
```
kubectl get nodes --show-labels | grep accelerator
```
If the label accelerator=nvidia is missing on some GPU nodes, they will be excluded.
Confirm NCCL topology requirements by inspecting the Weaviate trainer’s start‑up logs (if any pod reaches Running):
```
2024-09-01 12:34:56,789 INFO NCCL: Detected 2 GPUs, but 4 were requested.
```

Solution

The fix consists of aligning the scheduler’s view of GPU resources with the actual hardware, and relaxing unnecessary placement constraints.

1. Update the NVIDIA device plugin configuration

Set maxPodsPerGPU to 0 (unlimited) or a value that matches the expected pod density.

Before (daemonset.yaml snippet):

containers:
- name: nvidia-device-plugin
  image: nvcr.io/nvidia/k8s-device-plugin:latest
  args:
    - "--max-pods-per-gpu=1"

After:

containers:
- name: nvidia-device-plugin
  image: nvcr.io/nvidia/k8s-device-plugin:latest
  args:
    - "--max-pods-per-gpu=0"

Apply the updated daemonset and restart the plugin pods:

kubectl apply -f daemonset.yaml
kubectl rollout restart daemonset/nvidia-device-plugin -n kube-system

2. Correct node labeling

Ensure every GPU node carries the expected labels:

kubectl label node gpu-node-1 accelerator=nvidia topology.kubernetes.io/zone=us-east-1a --overwrite
kubectl label node gpu-node-2 accelerator=nvidia topology.kubernetes.io/zone=us-east-1a --overwrite

3. Adjust the pod spec to request GPUs that actually exist

If a node only has 4 GPUs, a request for 4 is fine, but you must avoid requesting more than the node can provide.

Before (weaviate-trainer-pod.yaml):

resources:
  limits:
    nvidia.com/gpu: 8

After (request 4 GPUs, which matches node capacity):

resources:
  limits:
    nvidia.com/gpu: 4

4. Simplify affinity when topology is not a hard requirement

If NCCL can operate across nodes (e.g., using nccl-p2p over RDMA), relax the affinity to allow the scheduler to split the request across multiple nodes using PodTopologySpread or PreferredDuringSchedulingIgnoredDuringExecution. Example:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution: null
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 1
      preference:
        matchExpressions:
        - key: accelerator
          operator: In
          values: ["nvidia"]

5. Deploy the corrected pod definition

kubectl apply -f weaviate-trainer-pod.yaml

Verification

Confirm that the node now reports the correct GPU count:

kubectl get node gpu-node-1 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}'
# Expected output: 4

Check that the pod transitions to Running:

kubectl get pod weaviate-trainer -w
# Output should show STATUS: Running

Inspect the trainer logs for successful NCCL initialization:

kubectl logs weaviate-trainer -c trainer | grep NCCL
# Expected line:
INFO NCCL: Initialized communication across 4 GPUs.

Validate GPU utilization with nvidia-smi on the node:

kubectl exec -it weaviate-trainer -- nvidia-smi
# Should list 4 processes using the GPUs.

Prevention

Automated capacity checks: Add a pre‑deployment script that queries kubectl get nodes -o jsonpath for allocatable.nvidia.com/gpu and aborts if the requested count exceeds the maximum per node.
Version lock the NVIDIA device plugin: Use a known‑good version (e.g., v0.14.0) and pin the maxPodsPerGPU flag to 0 in the daemonset.
Label hygiene: Enforce a CI/CD policy that validates required node labels exist on all GPU nodes before merging changes to pod specs.
Topology‑aware scheduling: When strict colocation is required, use the TopologySpreadConstraints feature or the NodeResourceTopology API (via the topology-aware-scheduling feature gate) to make the scheduler aware of GPU NUMA topology.
Monitoring alerts: Create a Prometheus alert on kube_pod_status_phase{phase="Pending"} combined with a label selector for nvidia.com/gpu to catch future pending‑GPU‑pod incidents within 5 minutes.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Why does the pod stay pending even though the node has enough GPUs?
Because the node’s device plugin reports a lower allocatable GPU count (often due to maxPodsPerGPU or version mismatch), so the scheduler believes the resources are insufficient.
Can I request GPUs across multiple nodes for a single Weaviate training job?
Weaviate’s default training code expects all GPUs to be on the same node for optimal NCCL performance. To span nodes you must enable multi‑node NCCL (e.g., via --nccl-enable-p2p) and adjust the pod spec to use PodTopologySpreadConstraints instead of a single-node request.
What is the difference between limits and requests for nvidia.com/gpu?
For GPU resources, Kubernetes treats requests and limits as identical; the scheduler uses the value to decide placement. Omitting requests defaults to the limits value.
How do I verify which version of the NVIDIA device plugin is running?
Run kubectl get daemonset nvidia-device-plugin -n kube-system -o yaml | grep image. The image tag corresponds to the plugin version.
Is there a way to automatically adjust the GPU request based on node capacity?
Yes. Use a MutatingAdmissionWebhook that patches incoming pod specs: it can read the node’s allocatable.nvidia.com/gpu via the Node API and cap the request to the node’s maximum.