GPT-4o Inference Scheduler Pod Placement Failure After Node Affinity Change

Q: Problem – Scheduler Pod Fails to Schedule After Node‑Affinity Change

The GPT‑4o inference scheduler pod remains in Pending state despite the cluster having free capacity. The issue appeared after a change to the pod’s nodeAffinity rules intended to isolate inference workloads on GPU‑enabled nodes.

Q: Root Cause – Conflicting Affinity and Resource Requests

The new nodeAffinity limited the scheduler to nodes labelled gpu=true. Those nodes already host multiple inference pods that each request cpu: 8, memory: 32Gi, and nvidia.com/gpu: 1. The scheduler pod also requests the same resources, but the affinity rules inadvertently excluded the newly‑provisioned GPU nodes because the label was applied after the node joined the cluster, causing a temporary mismatch.

Q: 2. Introduce a ResourceQuota and adjust pod requests

Reduce the scheduler’s memory request to 24Gi and CPU to 6, reflecting actual usage measured during load testing. Add a LimitRange to prevent future pods from over‑requesting resources.

In this article

Table of Contents

Problem – Scheduler Pod Fails to Schedule After Node‑Affinity Change

The GPT‑4o inference scheduler pod remains in Pending state despite the cluster having free capacity. The issue appeared after a change to the pod’s nodeAffinity rules intended to isolate inference workloads on GPU‑enabled nodes.

Observed symptoms

Pod status: Pending
Events from kubectl describe pod gpt4o-scheduler-xxxx show “0/12 nodes are available: 1 node(s) didn’t match node selector, 11 node(s) had insufficient memory”.
Cluster autoscaler does not trigger new node creation.
Other services continue to run; only the scheduler is affected.

Root Cause – Conflicting Affinity and Resource Requests

The new nodeAffinity limited the scheduler to nodes labelled gpu=true. Those nodes already host multiple inference pods that each request cpu: 8, memory: 32Gi, and nvidia.com/gpu: 1. The scheduler pod also requests the same resources, but the affinity rules inadvertently excluded the newly‑provisioned GPU nodes because the label was applied after the node joined the cluster, causing a temporary mismatch.

Two intertwined problems emerged:

Resource saturation: The combined resource requests exceeded the capacity of the existing GPU nodes.
Affinity timing: The nodeAffinity expression used requiredDuringSchedulingIgnoredDuringExecution with matchExpressions that required the label gpu=true. Nodes that were just added by the autoscaler did not have the label at the moment the scheduler evaluated pod placement, so they were filtered out.

Consequently, the scheduler pod could not find any node satisfying both the affinity and the resource constraints, leading to perpetual Pending state.

Debugging the Scheduler Placement

Step‑by‑step investigation:

# 1. Inspect pod status and events
kubectl get pod gpt4o-scheduler-xxxx -n inference -o wide
kubectl describe pod gpt4o-scheduler-xxxx -n inference | grep -A5 Events

# Sample output
Events:
  Type    Reason                     Age   From               Message
  ----    ------                     ----  ----               -------
  Normal  Scheduled                  2m    default-scheduler  Successfully assigned inference/gpt4o-scheduler-xxxx to node-3
  Warning FailedScheduling           2m    default-scheduler  0/12 nodes are available: 1 node(s) didn't match node selector, 11 node(s) had insufficient memory.

2. Verify node labels and resources:

# List GPU nodes and their labels
kubectl get nodes -l gpu=true -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia.com/gpu,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory

# Sample output
NAME        GPU   CPU   MEM
node-3      4     64    256Gi
node-7      4     64    256Gi

3. Check current resource consumption on those nodes:

# Show pod resource requests per node
kubectl top pod -n inference --containers | grep node-3

# Sample output
gpt4o-inference-abc   8      32Gi
gpt4o-inference-def   8      32Gi
gpt4o-scheduler-xxxx  8      32Gi   <-- pending

4. Examine the affinity definition in the scheduler deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpt4o-scheduler
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: gpu
                operator: In
                values:
                - "true"
      containers:
      - name: scheduler
        resources:
          requests:
            cpu: "8"
            memory: "32Gi"
            nvidia.com/gpu: "1"

5. Confirm that newly created nodes receive the gpu=true label after provisioning:

# Simulate node creation (autoscaler)
kubectl get nodes -o wide | grep -i pending

# Check label propagation delay
kubectl get node node-9 -o jsonpath='{.metadata.labels.gpu}'

Result: the label is missing for a few seconds, during which the scheduler pod attempts placement.

Solution – Align Affinity with Autoscaler and Adjust Resource Requests

Two corrective actions were applied:

1. Change affinity to `preferredDuringSchedulingIgnoredDuringExecution`

This makes the scheduler treat the GPU label as a soft preference, allowing placement on newly added nodes before the label is applied.

# Before (hard affinity)
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: gpu
          operator: In
          values:
          - "true"

# After (soft affinity)
affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      preference:
        matchExpressions:
        - key: gpu
          operator: In
          values:
          - "true"

2. Introduce a `ResourceQuota` and adjust pod requests

Reduce the scheduler’s memory request to 24Gi and CPU to 6, reflecting actual usage measured during load testing. Add a LimitRange to prevent future pods from over‑requesting resources.

# Before (original requests)
resources:
  requests:
    cpu: "8"
    memory: "32Gi"
    nvidia.com/gpu: "1"

# After (optimized requests)
resources:
  requests:
    cpu: "6"
    memory: "24Gi"
    nvidia.com/gpu: "1"

Apply the updated deployment:

kubectl apply -f deployment-scheduler.yaml

Result: the scheduler pod transitions to Running on node‑7, and the autoscaler successfully provisions an additional GPU node when overall GPU utilization exceeds 80%.

Verification – Confirming Successful Scheduling

Run the following checks after the deployment rollout:

# Verify pod state
kubectl get pod -l app=gpt4o-scheduler -n inference -o wide

# Expected output
NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE
gpt4o-scheduler-abc123   1/1     Running   0          2m    10.1.2.5     node-7

# Confirm node has GPU label
kubectl get node node-7 -o jsonpath='{.metadata.labels.gpu}'
# Expected output: true

# Check resource allocation
kubectl top pod -n inference gpt4o-scheduler-abc123
# Expected output: CPU 6, Memory 24Mi, GPU 1

# Ensure autoscaler reacted (if load is high)
kubectl get nodes -l gpu=true
# Should show an additional node if utilization >80%

Prevention – Guardrails for Future Affinity Changes

Label propagation delay awareness: Use preferredDuringScheduling for newly introduced labels or add a NodeProblemDetector rule to enforce immediate label assignment.
Resource request auditing: Implement a CI check that validates pod requests against historical usage (e.g., kubectl top pod data) before merging changes.
Node‑affinity policy: Store affinity rules in a ConfigMap and reference them via envFrom to ensure consistency across deployments.
Cluster autoscaler tuning: Set expander=least-waste and balance-similar-node-groups=true so that new GPU nodes receive the required labels during provisioning.
Monitoring alerts: Create a Prometheus rule that fires when kube_pod_status_phase{phase="Pending"} for the scheduler exceeds 30 seconds, linking to a runbook that checks affinity and resource saturation.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why did the pod stay pending even though the cluster had free GPU capacity?
The hard requiredDuringScheduling affinity filtered out newly provisioned GPU nodes that had not yet received the gpu=true label, leaving only saturated nodes as candidates.
Can I keep the hard affinity and still solve the issue?
Yes, but you must ensure the label is applied atomically with node creation (e.g., via a node-problem-detector webhook) or add a preemptionPolicy that evicts lower‑priority pods to free resources.
How do I determine the optimal resource requests for the scheduler?
Collect runtime metrics with kubectl top pod or Prometheus, calculate the 95th‑percentile usage, and set requests slightly above that value to provide headroom without over‑provisioning.
Will changing to soft affinity affect performance?
Soft affinity only influences placement preference; if a node without the GPU label is chosen, the pod will fail to start because the GPU device plugin cannot be satisfied, causing the scheduler to retry until a suitable node appears. In practice, the autoscaler quickly adds correctly labelled nodes, so impact is negligible.
How can I prevent similar scheduling failures for other services?
Adopt a standard pattern: define node‑affinity in a shared ConfigMap, enforce resource request limits via LimitRange, and integrate a pre‑deployment validation step that runs kubectl dry-run with --field-manager to catch unsatisfiable constraints.