Problem Description
The Claude inference service experiences a cascade of pending pods during traffic spikes. Typical symptoms observed in the kubectl get pods -n claude-prod output are:
NAME READY STATUS RESTARTS AGE
claude-infer-7f9c9d8c9f-abcde 0/1 Pending 0 2m30s
claude-infer-7f9c9d8c9f-fghij 0/1 Pending 0 2m30s
...Relevant scheduler events (extracted from kubectl describe pod <pod>) contain the following error messages, matching the Common errors evidence:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m default-scheduler 0/5 nodes are available: 5 Insufficient cpu, 5 node(s) didn't match node selector
Warning FailedScheduling 2m default-scheduler pod didn't fit in node: node(s) had taints {key=spot, effect=NoSchedule} that the pod didn't tolerate
Warning FailedScheduling 1m default-scheduler Failed to schedule pod: pod has node affinity that conflicts with available nodes
Impact:
- Inference latency rises by 30‑40% as requests queue.
- CPU and GPU utilization spikes on existing nodes, causing throttling.
- Service-level agreements (SLAs) are breached during autoscaling events.
Root Cause Analysis
The failure stems from three interacting misconfigurations:
- Node selector mismatch: The Claude deployment specifies
nodeSelector: {instance-type: m5.large}(see Anthropic Claude Deployment Guide – node selector configuration). When the Cluster Autoscaler provisions GPU‑enabledp3.2xlargenodes, they lack theinstance-type: m5.largelabel, so the scheduler cannot place new pods. - Missing tolerations for spot node taints: Newly added spot node groups carry the taint
spot: true:NoSchedule(as described in the Cluster Autoscaler docs for mixed node groups). Inference pods do not declare a matching toleration, leading to the “node(s) had taints … that the pod didn’t tolerate” event. - Autoscaler scaling limits: The high‑memory node pool is capped at
maxSize: 10. During a rolling upgrade the pool hit its limit, and the scheduler reported “0/10 nodes are available: 10 Insufficient memory” (see the real incident logs). This prevented the autoscaler from provisioning additional memory‑rich nodes fast enough.
Combined, these issues cause the scheduler to back‑off, leaving pods pending until an operator manually relabels nodes or adjusts taints, matching the discussion in GitHub issue #2541 and the Stack Overflow solution on nodeSelector/taints.
Debugging Steps
Follow these commands to reproduce the failure and collect evidence:
# 1. List pending pods and their events
kubectl get pods -n claude-prod -o wide | grep Pending
kubectl describe pod $(kubectl get pods -n claude-prod -o name | grep Pending) | grep -A5 Events
# 2. Inspect node labels and taints
kubectl get nodes --show-labels | grep instance-type
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'
# 3. Verify Cluster Autoscaler decisions
kubectl -n kube-system logs deployment/cluster-autoscaler | grep -i "scale up"
# 4. Check resource requests of the Claude pod spec
kubectl get deployment claude-infer -n claude-prod -o yaml | grep -A5 resources
Expected outputs:
# Example node list
ip-10-0-1-23 instance-type=m5.large,region=us-east-2
ip-10-0-2-45 instance-type=p3.2xlarge,region=us-east-2
# Example taints
ip-10-0-2-45 [{"key":"spot","value":"true","effect":"NoSchedule"}]
# Autoscaler log snippet
2024-05-30T12:45:12Z scale-up: node group "gpu-pool" size increased from 2 to 3
2024-05-30T12:45:15Z scale-up: node group "high-memory-pool" max size reached (10)
Solution
Apply the following changes to align pod specifications with autoscaling behavior.
1. Update the deployment to use flexible node selectors and add required tolerations.
# Before (claude-deployment.yaml)
apiVersion: apps/v1
kind: Deployment
metadata:
name: claude-infer
namespace: claude-prod
spec:
replicas: 12
selector:
matchLabels:
app: claude-infer
template:
metadata:
labels:
app: claude-infer
spec:
nodeSelector:
instance-type: m5.large
containers:
- name: infer
image: anthropic/claude-infer:latest
resources:
requests:
cpu: "4"
memory: "16Gi"
limits:
cpu: "8"
memory: "32Gi"
# After
apiVersion: apps/v1
kind: Deployment
metadata:
name: claude-infer
namespace: claude-prod
spec:
replicas: 12
selector:
matchLabels:
app: claude-infer
template:
metadata:
labels:
app: claude-infer
spec:
# Use a label set that matches all supported node types
nodeSelector:
role: inference
tolerations:
- key: spot
operator: Exists
effect: NoSchedule
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: instance-type
operator: In
values: ["m5.large", "p3.2xlarge", "r5.4xlarge"]
containers:
- name: infer
image: anthropic/claude-infer:latest
resources:
requests:
cpu: "4"
memory: "16Gi"
limits:
cpu: "8"
memory: "32Gi"
2. Label all node groups with a common role=inference label.
# Example: add label to an existing node group via cloud‑provider tooling
aws autoscaling update-auto-scaling-group --auto-scaling-group-name high-memory-pool \
--tags Key=role,Value=inference,PropagateAtLaunch=true
# For newly created nodes, ensure the launch template includes:
{
"TagSpecifications": [
{
"ResourceType": "instance",
"Tags": [
{"Key": "role", "Value": "inference"}
]
}
]
}
3. Raise the max size of the high‑memory pool (or enable multiple node groups).
# Edit the autoscaler config map
kubectl edit configmap cluster-autoscaler-status -n kube-system
# Example snippet
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-autoscaler-status
namespace: kube-system
data:
config.yaml: |
...
nodeGroups:
- name: high-memory-pool
minSize: 2
maxSize: 20 # increased from 10
- name: gpu-pool
minSize: 1
maxSize: 15
Apply the updated deployment:
kubectl apply -f claude-deployment.yaml
Verification
After applying the changes, confirm that new pods schedule without delay.
# 1. Watch pod status
kubectl get pods -n claude-prod -w
# Expected output (pods transition to Running quickly)
NAME READY STATUS RESTARTS AGE
claude-infer-7f9c9d8c9f-abcde 1/1 Running 0 30s
claude-infer-7f9c9d8c9f-fghij 1/1 Running 0 30s
# 2. Verify scheduler events no longer contain errors
kubectl describe pod claude-infer-7f9c9d8c9f-abcde | grep -i "FailedScheduling" || echo "No scheduling errors"
# 3. Confirm autoscaler can scale both pools
kubectl -n kube-system logs deployment/cluster-autoscaler | grep "scale-up"
# Sample successful scale‑up log
2024-05-30T13:02:04Z scale-up: node group "high-memory-pool" size increased from 12 to 13
Additionally, monitor latency metrics (e.g., via Prometheus) to ensure they return to baseline (< 100 ms average inference latency).
Prevention
Implement these safeguards to avoid recurrence:
- Unified role label: Enforce a
role=inferencelabel on every node group that may host Claude pods. Use IAM policies or Terraform modules to embed the label in launch templates. - Node affinity over static nodeSelector: Prefer
nodeAffinitywith anInoperator to accommodate future node types (e.g., newer GPU families) without redeploying. - Taints & tolerations policy: Document required tolerations for all inference workloads and apply them automatically via a mutating admission webhook.
- Autoscaler capacity planning: Set
maxSizefor each node group based on peak load forecasts from the Claude API limits. Periodically review CloudWatch metrics for CPU/memory pressure. - Alerting: Create Prometheus alerts for events matching
FailedSchedulingwith reasons “Insufficient cpu”, “node(s) didn’t match node selector”, or “taints”. Example alert rule:alert: ClaudePodPending expr: kube_pod_status_phase{phase="Pending"} > 0 for: 2m labels: severity: critical annotations: summary: "Claude inference pods pending" description: "Check node selectors, taints, and autoscaler limits."
Related Topic Hub: LLM Systems Troubleshooting Hub
FAQ
Q: Why does the scheduler keep reporting “Insufficient cpu” even after the autoscaler adds new nodes?
A: The new nodes lack the label required by the pod’s nodeSelector, so the scheduler ignores them. Adding a broader nodeAffinity or ensuring the label exists on all node groups resolves the mismatch.
Q: Can I keep the existing nodeSelector: instance-type: m5.large and still use GPU nodes?
A: No. nodeSelector is an exact match. Replace it with a set‑based affinity that includes both CPU‑only and GPU instance types, or use separate Deployments for CPU and GPU workloads.
Q: How do I make the Cluster Autoscaler react faster to pending pods?
A: Reduce the scale-up-delay parameter (default 10 s) in the autoscaler config, and ensure the maxSize of each node group is high enough for peak demand. Also, avoid “max‑size reached” errors by monitoring node group limits.
Q: Do I need to add tolerations for every spot node taint I create?
A: Yes. Spot node groups are typically tainted with spot: true:NoSchedule. Add a matching toleration to the pod spec or use a MutatingWebhook to inject it automatically.
Q: What is the recommended way to label nodes for inference workloads?
A: Use a stable label such as role=inference applied at launch time. This decouples pod placement from specific instance types and lets you add new hardware without changing the deployment.