Amd ryzen 6000 series processor on circuit board background

Kubernetes scheduler failing to place AI pods on AMD GPU edge nodes

by

Kubernetes Scheduler Failing to Place AI Pods on AMD GPU Edge Nodes

Problem Description

In a remote edge site the AI inference workload never leaves the Pending state. Typical scheduler messages include:


0/5 nodes are available: 5 node(s) didn't match node selector.
Insufficient amd.com/gpu: 0/5 nodes are available: 5 node(s) have insufficient resources.
scheduler: no nodes match pod's node affinity

These errors indicate that the scheduler cannot find a node that satisfies the amd.com/gpu resource request or the node selector/affinity defined in the pod spec.

Root Cause Analysis

The underlying reasons fall into three categories, each documented in the evidence package:

  1. Incorrect node labeling. The AMD GPU Operator expects nodes to carry the label amd.com/gpu (see AMD GPU Operator guide). In one incident the label was manually set to gpu=amd, causing the selector amd.com/gpu to miss all nodes.
  2. Device plugin registration failure. If the ROCm driver stack does not expose GPUs to the kubelet, the device plugin reports amd.com/gpu: 0 as allocatable (see the ROCm documentation). Kernel updates or missing firmware can break this path, resulting in “Insufficient amd.com/gpu”.
  3. Resource name mismatch across clusters. In a federation scenario the control plane used rocm.com/gpu while edge nodes reported amd.com/gpu, leading to unschedulable pods (see the multi‑cluster incident).

All three failures share a common symptom: the scheduler sees either no matching label or zero allocatable GPU resources.

Investigation and Debugging Steps

Follow this checklist on the edge node(s) and the control plane.

1. Verify node labels

kubectl get nodes -L amd.com/gpu

Expected output (healthy node):


NAME          STATUS   ROLES    AGE   VERSION   AMD.COM/GPU
edge-gpu-01   Ready       12d   v1.27.3   2

If the column is empty or shows a different key (e.g., gpu=amd), the label is wrong.

2. Inspect device plugin registration

kubectl logs -n kube-system $(kubectl get pods -n kube-system -l app=amd-gpu-device-plugin -o jsonpath='{.items[0].metadata.name}')

Look for messages such as:


Failed to register device plugin: connection timed out
Failed to get device plugin resources: AMD GPU not found

These correspond to the “Failed to register device plugin” errors in the evidence.

3. Check allocatable resources

kubectl describe node edge-gpu-01 | grep -A3 "Allocated resources"

Sample problematic output:


Allocated resources:
  cpu:                4
  memory:             8192Mi
  amd.com/gpu:        0

If amd.com/gpu is zero, the kubelet is not seeing any GPUs.

4. Validate ROCm driver and firmware

rocm-smi -i

Expected output lists each GPU with its health status. Missing entries indicate a driver or firmware issue (see the “Edge node reports 0 AMD GPUs after a kernel update” incident).

5. Confirm pod spec resource requests

apiVersion: v1
kind: Pod
metadata:
  name: ai-inference
spec:
  containers:
  - name: model
    image: myregistry/ai-model:latest
    resources:
      limits:
        amd.com/gpu: 1
  nodeSelector:
    amd.com/gpu: "present"

Ensure the resource name matches the node label exactly.

Resolution

Apply the fixes that correspond to the identified cause.

Fix A – Correct Node Labeling

Remove the incorrect label and apply the required one.

# Remove wrong label
kubectl label node edge-gpu-01 gpu- amd

# Add correct label (GPU count is optional; presence is enough)
kubectl label node edge-gpu-01 amd.com/gpu=present --overwrite

After relabeling, the scheduler will match the pod’s nodeSelector or nodeAffinity.

Fix B – Restore ROCm Device Plugin Functionality

Typical steps after a driver regression:

  1. Reinstall the ROCm driver matching the kernel version (e.g., rocm-dkms-5.4).
  2. Ensure firmware packages (amdgpu-firmware) are present.
  3. Restart kubelet to trigger device plugin registration:
systemctl restart kubelet

Validate with the kubectl describe node command; amd.com/gpu should now show the correct count.

Fix C – Align Resource Names Across Clusters

If using federation or a central CI pipeline, standardize on the official name amd.com/gpu everywhere.

# Example patch for a Deployment manifest
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference
spec:
  template:
    spec:
      containers:
      - name: model
        resources:
          limits:
            amd.com/gpu: 1   # changed from rocm.com/gpu

Verification

After applying the appropriate fix, run the following checks:

  • Pod scheduling: kubectl get pods -w – the pod should transition to Running.
  • Node resource view: kubectl get nodes -L amd.com/gpu – should display a non‑zero GPU count.
  • Device plugin logs: No “Failed to register device plugin” entries.
  • GPU visibility inside the container:
kubectl exec -it ai-inference -- rocm-smi -i

Output should list the GPU(s) with health OK.

Prevention and Best Practices

  • Automated label enforcement: Use a DaemonSet that runs kubectl label on every node after the GPU Operator completes, ensuring the label is always present.
  • Driver version pinning: Keep the ROCm driver version locked to a known‑good release in the edge node OS image; test kernel upgrades in a staging environment before rollout.
  • Health monitoring: Export amd.com/gpu allocatable metrics to Prometheus and alert when the value drops to zero.
  • Device plugin readiness probe: Add a livenessProbe to the GPU Operator pod that checks the plugin socket (/var/lib/kubelet/device-plugins/amd.com/gpu.sock).
  • Consistent resource naming: Store the GPU resource name in a ConfigMap (e.g., gpu-resource-name: amd.com/gpu) and reference it in all manifests via {{ .Values.gpuResourceName }} to avoid mismatches.

FAQ

  1. Why does the pod stay Pending even though the node has a GPU?
    Because the node either lacks the required amd.com/gpu label or reports zero allocatable GPUs due to a failed device plugin registration.
  2. Can I use a generic gpu label instead of amd.com/gpu?
    The AMD GPU Operator and Kubernetes device plugin expose the resource under the fully qualified name amd.com/gpu. Using a different key will not be recognized by the scheduler.
  3. How do I know which ROCm version is compatible with my kernel?
    Consult the ROCm release notes (e.g., ROCm Documentation) for the supported kernel range. Pin the driver package to that version on edge nodes.
  4. What if I need to run multiple AI pods on the same node?
    Request the exact number of GPUs in each pod (amd.com/gpu: 1) and ensure the node’s allocatable count reflects the total GPUs. The scheduler will pack pods until the count is exhausted.
  5. Is there a way to debug intermittent device plugin registration failures?
    Enable verbose logging for the GPU Operator (--v=4) and capture kubelet logs (journalctl -u kubelet -f). Look for timeout or “plugin not found” messages that correlate with network latency spikes to the edge node.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub