Gemini scheduler pod placement failure after adding new model microservice

Gemini Scheduler Pod Placement Failure After Adding New Model Microservice

Problem

The Google Gemini scheduler repeatedly reports FailedScheduling for pods belonging to the newly introduced model‑service microservice. Deployments stall, and the kubectl get pods output shows pods stuck in Pending state:

NAME                                 READY   STATUS      RESTARTS   AGE
model-service-7c9f5d9d9b-abcde       0/1     Pending     0          2m30s

Cluster-wide impact includes:

• Increased latency for downstream services awaiting model inference.
• Autoscaler churn as the scheduler repeatedly attempts placement.
• Alert fatigue from PodPending alerts.

Root Cause

The failure originates from two interacting constraints:

1. Resource Exhaustion – The new model microservice requests cpu: 4 and memory: 8Gi. Existing workloads already consume ~85% of the node pool’s allocatable resources, leaving insufficient headroom for the additional request.
2. Affinity Rule Conflict – The deployment includes a podAntiAffinity rule that forces each model-service pod to avoid nodes already running any model-service pod. Combined with the resource shortage, the scheduler cannot find a node that satisfies both the resource request and the anti‑affinity constraint.

Both constraints are enforced by the default kube-scheduler predicates Fit and PodAntiAffinity. When no node satisfies the predicate set, the scheduler emits a FitError with detailed reasons, which appear in the pod events:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  1m    default-scheduler  0/5 nodes are available: 5 Insufficient cpu, 5 Insufficient memory, 5 pod anti-affinity rule conflicts.

Debug

Step‑by‑step investigation:

1. Inspect pod events

kubectl describe pod model-service-7c9f5d9d9b-abcde

2. Check node resource availability

kubectl top nodes

NAME            CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node-a          28           93%    112Gi           92%
node-b          27           90%    108Gi           89%
...

3. Validate affinity rules

kubectl get deployment model-service -o yaml | grep -A5 affinity

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - model-service
      topologyKey: kubernetes.io/hostname

4. Simulate scheduler fit using kubectl dry-run and kube-scheduler dry‑run mode (available in v1.27+).

kube-scheduler --dry-run=client \
  --policy-configmap=default/kube-scheduler-policy \
  --kubeconfig=$KUBECONFIG \
  --scheduler-name=default-scheduler \
  --pod=model-service-7c9f5d9d9b-abcde.yaml

The dry‑run output confirms both Insufficient cpu and pod anti-affinity rule conflicts as reasons.

Solution

Address the two constraints independently, then verify combined effect.

1. Adjust Resource Requests

Reduce the pod’s request to values that fit within the current node pool, or provision a dedicated node pool with larger instance types.

Before:

resources:
  requests:
    cpu: "4"
    memory: "8Gi"
  limits:
    cpu: "8"
    memory: "16Gi"

After (scaled‑down request):

resources:
  requests:
    cpu: "2"
    memory: "4Gi"
  limits:
    cpu: "4"
    memory: "8Gi"

2. Relax Anti‑Affinity or Use Preferred Anti‑Affinity

If strict isolation is not required, switch to preferredDuringSchedulingIgnoredDuringExecution so the scheduler can place pods on the same node when necessary.

Before (required):

podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
  - labelSelector:
      matchExpressions:
      - key: app
        operator: In
        values:
        - model-service
    topologyKey: kubernetes.io/hostname

After (preferred):

podAntiAffinity:
  preferredDuringSchedulingIgnoredDuringExecution:
  - weight: 80
    podAffinityTerm:
      labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - model-service
      topologyKey: kubernetes.io/hostname

3. Deploy a Dedicated Node Pool (optional)

If the model service must retain its original resource profile, create a new node pool with sufficient capacity and node‑selector labels.

apiVersion: v1
kind: Node
metadata:
  labels:
    model-node: "true"

spec:
  template:
    spec:
      nodeSelector:
        model-node: "true"

Verify

After applying the changes, run the following checks:

1. Confirm pods transition to Running:

kubectl get pods -l app=model-service -o wide

NAME                                 READY   STATUS    RESTARTS   AGE   NODE
model-service-7c9f5d9d9b-abcde       1/1     Running   0          3m    node-c

Validate that resource usage stays within node limits:

kubectl top nodes node-c

NAME    CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node-c  12           45%    48Gi            55%

Check that the anti‑affinity rule is respected (if still required):

kubectl get pods -o wide | grep model-service

All pods should be on distinct hosts unless the scheduler had to relax the rule, which is reflected by the PreferredDuringScheduling weight.

Prevent

Implement safeguards to avoid recurrence:

• Capacity Planning: Integrate kubectl top nodes metrics into CI pipelines; fail PRs if new deployments would exceed 80% of node allocatable resources.
• Affinity Policy Review: Enforce a lint rule (e.g., kube-linter) that flags requiredDuringScheduling anti‑affinity on high‑request pods unless justified.
• Pod Disruption Budgets: Define PDBs for model services to ensure graceful scaling without saturating the cluster.
• Cluster Autoscaler Tuning: Set scale-up-from-zero and max-node-provision-time thresholds to allow rapid provisioning of larger nodes when needed.
• Observability: Add alerts on scheduler_attempts_total{result="unschedulable"} and on node‑level cpu/allocatable and memory/allocatable thresholds.

FAQ

1. Why does the scheduler report both “Insufficient cpu” and “pod anti‑affinity rule conflicts”?
   Both predicates are evaluated independently. If any node fails any predicate, the scheduler aggregates the reasons. In this case, every node lacked enough free CPU *and* violated the required anti‑affinity, so both messages appear.
2. Can I keep the original resource requests and still schedule the pods?
   Yes, by adding a dedicated node pool sized for the model workload and using a nodeSelector or nodeAffinity to target those nodes. This isolates the high‑resource pods from the rest of the cluster.
3. Is “preferredDuringSchedulingIgnoredDuringExecution” safe for production?
   It is safe when strict isolation is not a hard requirement. The scheduler will still try to spread pods, but will fall back to co‑location if no suitable node exists, preventing unschedulable states.
4. How do I know if a future microservice will cause similar placement failures?
   Run a “dry‑run” scheduler simulation (as shown in the Debug section) against a snapshot of current node resources, or use kubectl explain pod --recursive to validate resource requests against cluster capacity before merging.
5. What role does the Cluster Autoscaler play in this scenario?
   If the autoscaler is enabled and the pod requests exceed current capacity, it can provision new nodes. However, the anti‑affinity rule prevented any existing node from being a fit, so the autoscaler never triggered. Relaxing or removing the required anti‑affinity allows the autoscaler to act.

Related Topic Hub: LLM Systems Troubleshooting Hub