Haystack Scheduler Pod Placement Failure Due To Insufficient GPU

Q: Root Cause Analysis

The scheduler pod requests a specific number of nvidia.com/gpu resources and may also define nodeSelector or affinity rules that restrict placement to particular GPU types. The Kubernetes scheduler evaluates three independent constraints:

Q: Fix 3 – Add tolerations for GPU taints

If nodes are tainted with nvidia.com/gpu=present:NoSchedule, add a matching toleration.

In this article

Table of Contents

Problem Description

In a Haystack deployment that runs multi‑GPU training jobs, the scheduler pod remains Pending with events such as:


0/5 nodes are available: 5 Insufficient gpu


FailedScheduling: pod has unschedulable: no nodes match node selector

These failures prevent the haystack-scheduler from launching training workers, effectively blocking any multi‑GPU training pipeline.

Root Cause Analysis

The scheduler pod requests a specific number of nvidia.com/gpu resources and may also define nodeSelector or affinity rules that restrict placement to particular GPU types. The Kubernetes scheduler evaluates three independent constraints:

GPU capacity: Nodes must report enough nvidia.com/gpu capacity via the NVIDIA device plugin (see Kubernetes GPU scheduling docs).
Node selectors / affinity: The pod’s nodeSelector (e.g., cloud.google.com/gke-accelerator=nvidia-tesla-v100) must match a node label (see Haystack scheduler configuration reference).
Taints & tolerations: Nodes that are tainted with nvidia.com/gpu=present:NoSchedule require matching tolerations on the pod (see production incident with tainted GPU nodes).

Typical mismatches observed in the field:

Requested GPU count exceeds the per‑node limit (e.g., job requests 8 GPUs while each node only has 4).
Namespace‑wide ResourceQuota caps total GPUs (e.g., quota of 4 GPUs, job asks for 8).
Node selector specifies a GPU model that does not exist in the cluster (e.g., selector for V100 while only A100 nodes are present).
Missing tolerations for the nvidia.com/gpu=present:NoSchedule taint.

These conditions trigger the “Insufficient gpu” or “no nodes match node selector” messages reported in kubectl describe pod and the scheduler logs (see GitHub issue #2157).

Investigation and Debugging Steps

Inspect pod events:


kubectl describe pod haystack-scheduler-xxxx -n haystack

Typical output:


Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  2m    default-scheduler  0/5 nodes are available: 5 Insufficient gpu

Check node GPU capacity:


kubectl get nodes -o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu

Example output:


NAME            GPU
gpu-node-1      4
gpu-node-2      4

Verify device plugin health (NVIDIA GPU Operator):
```
kubectl logs -n gpu-operator nvidia-driver-daemonset-xxxxx
```
Look for messages such as “Failed to allocate GPU resources: insufficient GPUs”.

Validate node labels and taints:


kubectl get nodes --show-labels
kubectl describe node gpu-node-1 | grep -i taint

Example mismatched selector:


Labels: cloud.google.com/gke-accelerator=nvidia-tesla-a100

But the scheduler pod requests cloud.google.com/gke-accelerator=nvidia-tesla-v100.

Inspect namespace ResourceQuota:


kubectl get quota -n haystack

Sample output indicating a limit:


Name:            gpu-quota
Hard:
  limits.nvidia.com/gpu:  4
Used:
  limits.nvidia.com/gpu:  0

Review scheduler configuration (Haystack scheduler.yaml):


apiVersion: v1
kind: ConfigMap
metadata:
  name: haystack-scheduler-config
data:
  scheduler.yaml: |
    resources:
      limits:
        nvidia.com/gpu: "8"
    nodeSelector:
      cloud.google.com/gke-accelerator: nvidia-tesla-v100
    tolerations:
      - key: "nvidia.com/gpu"
        operator: "Exists"
        effect: "NoSchedule"

Resolution

Apply the fixes that address the specific constraint that caused the failure.

Fix 1 – Align GPU request with node capacity

If a training job needs 8 GPUs but each node only has 4, either:

Split the job across two nodes using PodGroup or job parallelism, or
Reduce the request to the per‑node limit.

Before (requesting 8 GPUs on a 4‑GPU node):


resources:
  limits:
    nvidia.com/gpu: "8"

After (requesting 4 GPUs per pod):


resources:
  limits:
    nvidia.com/gpu: "4"

Fix 2 – Correct node selector / affinity

Update the selector to match the actual node labels.

Before:


nodeSelector:
  cloud.google.com/gke-accelerator: nvidia-tesla-v100

After (cluster has A100 nodes):


nodeSelector:
  cloud.google.com/gke-accelerator: nvidia-tesla-a100

Fix 3 – Add tolerations for GPU taints

If nodes are tainted with nvidia.com/gpu=present:NoSchedule, add a matching toleration.

Before (no tolerations):


# empty tolerations block

After:


tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"

Fix 4 – Adjust namespace ResourceQuota

Increase the GPU quota or move the training job to a namespace with a higher quota.

Before (quota of 4 GPUs):


apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
spec:
  hard:
    limits.nvidia.com/gpu: "4"

After (quota of 12 GPUs):


apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
spec:
  hard:
    limits.nvidia.com/gpu: "12"

Validation

Redeploy the scheduler configuration:


kubectl apply -f scheduler-config.yaml
kubectl rollout restart deployment/haystack-scheduler -n haystack

Confirm pod status:


kubectl get pod -n haystack -l app=haystack-scheduler

Expected output:


NAME                         READY   STATUS    RESTARTS   AGE
haystack-scheduler-xxxx      1/1     Running   0          30s

Check that the training job acquires the requested GPUs:
```
kubectl describe pod haystack-trainer-xxxx -n haystack | grep -i gpu
```
Should show allocated GPU count matching the request.
Monitor GPU utilization via NVIDIA DCGM or Prometheus:
```
kubectl top pod -n haystack
```

Operational Experience & Lessons Learned

Misleading symptom: The “Insufficient gpu” message can also appear when a ResourceQuota is the bottleneck, not just node capacity.
Common wrong assumption: Operators often assume that setting nvidia.com/gpu in limits automatically adds a toleration for the GPU taint; it does not.
Edge case in upgrades: After upgrading the NVIDIA driver, nodes may temporarily stop reporting nvidia.com/gpu capacity, causing a cluster‑wide “no nodes available” condition (see production incident with driver upgrade).
Production tip: Keep a separate namespace for training workloads with a higher GPU quota and explicit node selectors to avoid accidental cross‑contamination with inference services.

Best Practices and Prevention

Enable node-exporter metrics for nvidia.com/gpu and set alerts on kube_node_status_capacity{resource="nvidia.com/gpu"} falling below expected levels.
Document the exact GPU model labels used by the cluster and version‑lock them in the scheduler config.
Apply tolerations for the NVIDIA taint at the deployment level to future‑proof against node‑level taint changes.
Regularly audit ResourceQuota objects to ensure they reflect the intended capacity for training workloads.
Validate GPU reporting after driver or GPU Operator upgrades with a quick kubectl get nodes -o yaml check for allocatable.nvidia.com/gpu.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

Why does the scheduler pod fail only when requesting more than 4 GPUs?
The cluster’s nodes each expose 4 nvidia.com/gpu resources. The scheduler pod requests a single pod with 8 GPUs, which exceeds per‑node capacity, triggering the “Insufficient gpu” error.
How can I confirm which GPU model a node is advertising?
Run kubectl get node <name> --show-labels and look for the cloud.google.com/gke-accelerator (or vendor‑specific) label.
Do I need to add tolerations manually for the NVIDIA taint?
Yes. The NVIDIA GPU Operator adds the nvidia.com/gpu=present:NoSchedule taint to GPU nodes. Pods that need GPUs must declare a matching toleration.
Can a ResourceQuota cause “Insufficient gpu” even if nodes have free GPUs?
Exactly. The quota limits the total number of GPUs that can be allocated in the namespace. If the quota is lower than the pod’s request, the scheduler rejects the pod before checking node capacity.
Is it safe to remove the node selector and rely solely on resource requests?
Removing the selector allows the scheduler to place the pod on any node with sufficient GPUs, but you lose control over GPU model selection. If your training code depends on a specific GPU architecture, keep the selector.