Google Gemini Pods Pending Despite Free A100/H100 GPUs In Kubernetes

Q: 2. Mismatched Node Labels and Affinity

Gemini pods request a custom node selector gpu-type: h100 (or a100) to ensure the correct hardware. In a real incident, the cluster was labeled only with gpu-type: a100 on all nodes, even the H100 ones (Kubeflow discussion). Consequently, pods that required gpu-type: h100 could not find a matching node, producing the affinity error shown above.

Q: 3. Taints Without Corresponding Tolerations

During scheduled maintenance, a taint gpu=reserved was applied to all H100 nodes. Gemini pods lacked a matching toleration, causing the scheduler to filter those nodes out (Gemini Scheduler documentation).

Q: 4. Custom CRD Desynchronization

Some deployments introduced a custom resource definition gemini.com/gpu that mirrors nvidia.com/gpu. The controller that synchronizes the two stopped updating after a rollout, leaving the Gemini scheduler with an outdated view of GPU availability (Vertex AI GPU configuration guide).

Q: Investigation and Debugging Steps Step 1 – Verify Node GPU Capacity

kubectl get nodes -L gpu-type -o custom-columns=NAME:.metadata.name,GPU_TYPE:.metadata.labels.gpu-type,ALLOCATABLE_GPU:.status.allocatable.nvidia\.com/gpu Expected output (healthy cluster):

Q: Step 2 – Inspect Device Plugin Logs

kubectl logs -n kube-system -l app=nvidia-device-plugin -c nvidia-device-plugin Typical stale‑capacity warning:

Q: Step 3 – Check Pod Events for Affinity/Taint Errors

kubectl describe pod gemini-train-xyz Look for lines such as:

Q: Step 4 – Validate Taints on Nodes

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}' Sample output showing the problematic taint:

Q: Step 5 – Examine Custom CRD Synchronization

kubectl get crd gemini.com/gpu -o yaml | grep -i status If the status.capacity field is zero while the underlying node reports GPUs, the CRD is out of sync.

In this article

Table of Contents

Problem: Gemini Training Pods Remain Pending Despite Free A100/H100 GPUs

In a production Vertex AI cluster that mixes NVIDIA A100 and H100 nodes, engineers observed that new gemini training pods never transition to Running. The scheduler reports:


FailedScheduling: 0/6 nodes are available: 6 Insufficient nvidia.com/gpu.

Other observed symptoms include:

Pod events showing pod didn't match node affinity: required node label "gpu-type" not present on any node.
Node status reports where capacity for nvidia.com/gpu is non‑zero, yet the scheduler treats it as zero.
Occasional taint warnings such as node(s) had taint {gpu=reserved}: that the pod didn't tolerate.

Despite kubectl get nodes -o jsonpath='{.status.allocatable.nvidia\.com/gpu}' showing available GPU counts on both A100 and H100 nodes, the Gemini scheduler never places the pods.

Root Cause Analysis

1. Inaccurate Node Resource Reporting

The NVIDIA device plugin caches node capacity. After kernel upgrades or node reboots, the plugin sometimes fails to refresh the nvidia.com/gpu metric, leading to stale capacity entries (GitHub issue nvidia/k8s-device-plugin#1023). The Gemini scheduler, which relies on the standard nvidia.com/gpu resource, therefore believes the node is out of GPUs.

2. Mismatched Node Labels and Affinity

Gemini pods request a custom node selector gpu-type: h100 (or a100) to ensure the correct hardware. In a real incident, the cluster was labeled only with gpu-type: a100 on all nodes, even the H100 ones (Kubeflow discussion). Consequently, pods that required gpu-type: h100 could not find a matching node, producing the affinity error shown above.

3. Taints Without Corresponding Tolerations

During scheduled maintenance, a taint gpu=reserved was applied to all H100 nodes. Gemini pods lacked a matching toleration, causing the scheduler to filter those nodes out (Gemini Scheduler documentation).

4. Custom CRD Desynchronization

Some deployments introduced a custom resource definition gemini.com/gpu that mirrors nvidia.com/gpu. The controller that synchronizes the two stopped updating after a rollout, leaving the Gemini scheduler with an outdated view of GPU availability (Vertex AI GPU configuration guide).

Investigation and Debugging Steps

Step 1 – Verify Node GPU Capacity


kubectl get nodes -L gpu-type -o custom-columns=NAME:.metadata.name,GPU_TYPE:.metadata.labels.gpu-type,ALLOCATABLE_GPU:.status.allocatable.nvidia\.com/gpu

Expected output (healthy cluster):


NAME        GPU_TYPE   ALLOCATABLE_GPU
node-a1     a100       8
node-a2     a100       8
node-h1     h100       4
node-h2     h100       4

Step 2 – Inspect Device Plugin Logs


kubectl logs -n kube-system -l app=nvidia-device-plugin -c nvidia-device-plugin

Typical stale‑capacity warning:


time="2024-08-12T03:21:45Z" level=error msg="Failed to update node status: node capacity for nvidia.com/gpu is outdated"

Step 3 – Check Pod Events for Affinity/Taint Errors


kubectl describe pod gemini-train-xyz

Look for lines such as:


Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  2m    default-scheduler  0/6 nodes are available: 6 Insufficient nvidia.com/gpu, 6 pod didn't match node affinity: required node label "gpu-type" not present on any node

Step 4 – Validate Taints on Nodes


kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'

Sample output showing the problematic taint:


node-h1    [{"key":"gpu","value":"reserved","effect":"NoSchedule"}]
node-h2    [{"key":"gpu","value":"reserved","effect":"NoSchedule"}]

Step 5 – Examine Custom CRD Synchronization


kubectl get crd gemini.com/gpu -o yaml | grep -i status

If the status.capacity field is zero while the underlying node reports GPUs, the CRD is out of sync.

Resolution

1. Refresh NVIDIA Device Plugin Capacity

Restart the device plugin daemonset to force a capacity refresh:


kubectl rollout restart daemonset nvidia-device-plugin -n kube-system

After restart, verify the logs no longer contain the stale‑capacity error and that kubectl describe node shows correct capacity values.

2. Align Node Labels with Pod Affinity Requirements

Apply the missing gpu-type label to H100 nodes:


kubectl label nodes node-h1 gpu-type=h100 --overwrite
kubectl label nodes node-h2 gpu-type=h100 --overwrite

If the label already exists but is misspelled (e.g., gpu-type: H100), correct it to the exact lower‑case value expected by the Gemini pod spec.

3. Add Tolerations to Gemini Pods (or Remove Taint)

Option A – Remove the maintenance taint after work is done:


kubectl taint nodes node-h1 gpu=reserved-
kubectl taint nodes node-h2 gpu=reserved-

Option B – Add a toleration to the pod template:


apiVersion: batch/v1
kind: Job
metadata:
  name: gemini-train
spec:
  template:
    spec:
      tolerations:
      - key: "gpu"
        operator: "Equal"
        value: "reserved"
        effect: "NoSchedule"
      containers:
      - name: trainer
        image: us-central1-docker.pkg.dev/...
        resources:
          limits:
            nvidia.com/gpu: "4"
        nodeSelector:
          gpu-type: h100

4. Synchronize Custom CRD with Native GPU Metric

Deploy a simple controller that copies nvidia.com/gpu capacity into gemini.com/gpu:


apiVersion: apps/v1
kind: Deployment
metadata:
  name: gemini-gpu-sync
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gemini-gpu-sync
  template:
    metadata:
      labels:
        app: gemini-gpu-sync
    spec:
      serviceAccountName: gemini-gpu-sync
      containers:
      - name: sync
        image: python:3.11-slim
        command: ["python", "-u", "/app/sync.py"]
        volumeMounts:
        - name: kubeconfig
          mountPath: /root/.kube
      volumes:
      - name: kubeconfig
        hostPath:
          path: /etc/kubernetes/kubelet.conf

Inside sync.py (simplified):


import os
from kubernetes import client, config

config.load_kube_config()
v1 = client.CoreV1Api()
custom = client.CustomObjectsApi()

while True:
    nodes = v1.list_node().items
    for node in nodes:
        gpu_cap = int(node.status.capacity.get('nvidia.com/gpu', 0))
        body = {"status": {"capacity": gpu_cap}}
        try:
            custom.patch_cluster_custom_object_status(
                group="gemini.com",
                version="v1",
                plural="gpus",
                name=node.metadata.name,
                body=body,
            )
        except Exception as e:
            print(f"Sync error: {e}")
    time.sleep(30)

After deploying, the Gemini scheduler sees up‑to‑date GPU capacity.

Validation

1. Confirm Pod Scheduling


kubectl apply -f gemini-train.yaml
kubectl get pod -w

Pod should transition from Pending to Running within seconds.

2. Verify Node Capacity Consistency


kubectl get node node-h1 -o yaml | grep -A2 "capacity"

Output should show matching values for both nvidia.com/gpu and gemini.com/gpu.

3. Check Scheduler Events


kubectl get events --field-selector involvedObject.kind=Pod,involvedObject.name=gemini-train-xyz

No FailedScheduling events should appear.

Operational Experience and Lessons Learned

Label drift is common. When new node types are added (e.g., H100), automated provisioning scripts must also apply the corresponding gpu-type label. A single missing label can block an entire class of jobs.
Device plugin cache staleness. After kernel or driver upgrades, the NVIDIA plugin may retain old capacity values. A rolling restart of the daemonset after any driver change eliminates this risk.
Custom CRDs add complexity. Mirroring native resources is only worthwhile when additional semantics are needed. If used, ensure a reconciliation loop runs with a short interval.
Taint management. Maintenance taints should be paired with a clear process for adding tolerations to affected workloads, otherwise production jobs will be silently unschedulable.

Best Practices and Prevention

Automate node labeling as part of the provisioning pipeline; validate labels with a post‑provisioning script.
Include a health check that compares kubectl get nodes -o jsonpath='{.items[*].status.capacity.nvidia.com/gpu}' against the device plugin’s internal metrics.
Deploy a watchdog DaemonSet that restarts the NVIDIA device plugin if log entries contain “Failed to update node status”.
Prefer native nvidia.com/gpu in Gemini pod specs unless a strong justification exists for a custom CRD.
Document taint usage and ensure every workload that may need those nodes includes the appropriate tolerations.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does the Gemini scheduler report “Insufficient nvidia.com/gpu” when the node shows free GPUs?
The device plugin’s cached capacity is stale, causing the scheduler to think the node has zero allocatable GPUs. Restarting the plugin daemonset forces a refresh.
My pods request gpu-type: h100 but the nodes are labeled gpu-type: H100. Will case matter?
Yes. Node selector matching is case‑sensitive. Ensure the label value matches exactly the string used in the pod spec (lower‑case is recommended).
How can I verify which GPU type a node actually hosts?
Run kubectl describe node <name> | grep -i 'model' after installing the NVIDIA driver; the nvidia.com/gpu.product_name annotation will contain “A100” or “H100”.
Do I need a custom CRD for Gemini GPU allocation?
Not unless you require additional metadata. The built‑in nvidia.com/gpu resource is fully supported by the Gemini scheduler (official docs).
What is the correct way to schedule across mixed‑type GPU nodes?
Use node affinity with a label that distinguishes the type (e.g., gpu-type: a100 or gpu-type: h100) and ensure all nodes carry the appropriate label. Combine with topologySpreadConstraints if you need balanced distribution.