Pod Eviction Under Memory Pressure During Large Model Evaluation On Azure

Q: Why large model evaluation triggers the condition Checkpoint loading: Loading a 12‑GB transformer model into a Python process often doubles the resident set size due to duplicated tensors, temporary buffers, and the Python interpreter overhead. On an NC6s_v3 node with 112 GB RAM, a single evaluation job can consume > 100 GB, leaving insufficient headroom for system daemons and other pods. Insufficient pod resource requests: Many evaluation pods are launched without explicit resources.requests.memory. In the absence of a request, the pod receives a BestEffort QoS class, making it the first candidate for eviction even if the container’s limit is high enough to avoid OOMKilled. Node‑pool sizing & autoscaling misconfiguration: A single‑node pool (minSize=1) cannot absorb spikes from concurrent evaluations. When several jobs start simultaneously, the aggregate memory demand exceeds the node’s capacity, causing immediate pressure 3. Local staging of model artifacts: In hybrid clusters, model checkpoints are copied to the node’s /tmp before the container starts. The staging step itself can temporarily double the memory footprint, pushing the node over the limit before the pod even begins execution 4. Collectively, these factors cause the kubelet to set MemoryPressure=True and invoke the eviction manager, resulting in the observed “Evicted: The node was low on resource: memory” events. Debugging and Investigation 1. Verify node pressure condition

kubectl get node $(kubectl get pods -l app=evaluator -o jsonpath='{.items[0].spec.nodeName}') -o jsonpath='{.status.conditions[?(@.type=="MemoryPressure")].status}' Expected output: True during the spike.

Q: 2. Inspect kubelet logs for eviction thresholds

journalctl -u kubelet | grep -i "eviction" Sample snippet:

Q: 3. Correlate memory usage with pod lifecycle

kubectl top node $(kubectl get pods -l app=evaluator -o jsonpath='{.items[0].spec.nodeName}') kubectl top pod -A --containers | grep evaluator Look for sudden jumps from ~30Mi to > 100Gi at the moment the model is loaded.

Q: 4. Check pod QoS classification

kubectl get pod evaluator-abc123 -o jsonpath='{.status.qosClass}' If the output is BestEffort, the pod lacks a memory request.

Q: 5. Review node pool configuration

az aks nodepool show \ --resource-group myRG \ --cluster-name myAKS \ --name np-cpu \ --query "{minCount:minCount, maxCount:maxCount, vmSize:vmSize}" Confirm that minCount is not limiting scaling during bursts.

Problem: Pods Evicted Under Memory Pressure During Large‑Scale Model Evaluation on Azure

During batch evaluation of transformer‑based models (12‑20 GB checkpoint size) on Azure virtual machines managed by AKS, operators observed repeated pod evictions:


Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Killing    5m                kubelet            Killing container with id docker://evaluator:Need to kill pod
  Warning  Evicted    4m30s             kubelet            Evicted: The node was low on resource: memory.
  Warning  OOMKilled  4m30s             kubelet            Container evaluator terminated due to OOMKilled

Typical impact includes incomplete evaluation runs, pipeline failures, and costly re‑queues. The issue surfaces on a variety of Azure VM sizes (e.g., NC6s_v3, Standard_E8s_v3) and is reproduced across both GPU‑enabled and CPU‑only node pools.

Root Cause Analysis

Memory‑pressure eviction flow in AKS

AKS nodes run the Kubernetes eviction manager. When the node’s MemoryPressure condition becomes True, the kubelet ranks pods by QoS class and priority, then evicts the lowest‑ranked pods until the node’s availableMemoryBytes falls below the evictionHard threshold (default memory.available<100Mi) ¹. The eviction event “The node was low on resource: memory” is emitted directly from the kubelet log ².

Why large model evaluation triggers the condition

Checkpoint loading: Loading a 12‑GB transformer model into a Python process often doubles the resident set size due to duplicated tensors, temporary buffers, and the Python interpreter overhead. On an NC6s_v3 node with 112 GB RAM, a single evaluation job can consume > 100 GB, leaving insufficient headroom for system daemons and other pods.
Insufficient pod resource requests: Many evaluation pods are launched without explicit resources.requests.memory. In the absence of a request, the pod receives a BestEffort QoS class, making it the first candidate for eviction even if the container’s limit is high enough to avoid OOMKilled.
Node‑pool sizing & autoscaling misconfiguration: A single‑node pool (minSize=1) cannot absorb spikes from concurrent evaluations. When several jobs start simultaneously, the aggregate memory demand exceeds the node’s capacity, causing immediate pressure ³.
Local staging of model artifacts: In hybrid clusters, model checkpoints are copied to the node’s /tmp before the container starts. The staging step itself can temporarily double the memory footprint, pushing the node over the limit before the pod even begins execution ⁴.

Collectively, these factors cause the kubelet to set MemoryPressure=True and invoke the eviction manager, resulting in the observed “Evicted: The node was low on resource: memory” events.

Debugging and Investigation

1. Verify node pressure condition


kubectl get node $(kubectl get pods -l app=evaluator -o jsonpath='{.items[0].spec.nodeName}') -o jsonpath='{.status.conditions[?(@.type=="MemoryPressure")].status}'

Expected output: True during the spike.

2. Inspect kubelet logs for eviction thresholds


journalctl -u kubelet | grep -i "eviction"

Sample snippet:


I0905 12:34:56.789012    1234 eviction_manager.go:215] Node memory pressure: eviction manager is evicting pods
I0905 12:34:56.789045    1234 eviction_manager.go:237] Evicting pod default/evaluator-abc123 (qos=BestEffort, priority=0) due to memory pressure

3. Correlate memory usage with pod lifecycle


kubectl top node $(kubectl get pods -l app=evaluator -o jsonpath='{.items[0].spec.nodeName}')
kubectl top pod -A --containers | grep evaluator

Look for sudden jumps from ~30Mi to > 100Gi at the moment the model is loaded.

4. Check pod QoS classification


kubectl get pod evaluator-abc123 -o jsonpath='{.status.qosClass}'

If the output is BestEffort, the pod lacks a memory request.

5. Review node pool configuration


az aks nodepool show \
  --resource-group myRG \
  --cluster-name myAKS \
  --name np-cpu \
  --query "{minCount:minCount, maxCount:maxCount, vmSize:vmSize}"

Confirm that minCount is not limiting scaling during bursts.

Solution: Preventing Eviction During Large Model Evaluation

1. Define explicit memory requests and limits

Set requests to the realistic baseline memory the container needs (e.g., 8 Gi) and limits to a safe ceiling (e.g., 16 Gi). This promotes a Burstable QoS class, protecting the pod from premature eviction.


apiVersion: v1
kind: Pod
metadata:
  name: evaluator
spec:
  containers:
  - name: evaluator
    image: myregistry.azurecr.io/evaluator:latest
    resources:
      requests:
        memory: "8Gi"
      limits:
        memory: "16Gi"

2. Use pod priority and preemption

Assign a high priority class to evaluation jobs so that, if eviction is unavoidable, lower‑priority system pods are removed first.


apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: eval-high
value: 1000000
globalDefault: false
description: "High priority for model evaluation jobs"


apiVersion: v1
kind: Pod
metadata:
  name: evaluator
spec:
  priorityClassName: eval-high
  ...

3. Resize node pools or enable burstable VM SKUs

Choose VM sizes with sufficient RAM for the worst‑case memory consumption. For a 20 GB BERT model, a Standard_E16s_v3 (128 GB RAM) or a GPU‑enabled NC6s_v3 (112 GB) with a dedicated nodepool for evaluation is recommended.

Model Checkpoint	Estimated Peak RAM	Recommended VM Size
12 GB transformer	≈ 90 Gi	Standard_E8s_v3 (64 Gi) – not sufficient, use Standard_E16s_v3
20 GB BERT	≈ 130 Gi	NC6s_v3 (112 Gi) – borderline, use NC12s_v3 (224 Gi)

4. Adjust eviction thresholds (advanced)

If temporary spikes are unavoidable, increase the hard eviction threshold on the node via the kubelet --eviction-hard flag (e.g., memory.available<500Mi). This requires custom node pool VMSS extensions or a self‑managed node pool.

5. Separate staging from evaluation

Mount a dedicated Azure Files share or use emptyDir with medium: "Memory" for checkpoint staging, then delete the temporary copy after loading to free memory.


apiVersion: v1
kind: Pod
metadata:
  name: evaluator
spec:
  volumes:
  - name: checkpoint
    emptyDir:
      medium: Memory
  containers:
  - name: evaluator
    image: ...
    volumeMounts:
    - name: checkpoint
      mountPath: /mnt/checkpoint

6. Configure autoscaling with appropriate min/max

Set minCount to at least 2 for evaluation node pools so that concurrent jobs are spread across nodes, reducing per‑node memory pressure.


az aks nodepool update \
  --resource-group myRG \
  --cluster-name myAKS \
  --name np-eval \
  --enable-cluster-autoscaler \
  --min-count 2 \
  --max-count 10

Verification: Confirming the Fix

Deploy the updated pod manifest with requests, limits, and priority.
Trigger a model evaluation run that previously caused eviction.
Monitor node pressure:


kubectl get node $(kubectl get pod evaluator -o jsonpath='{.spec.nodeName}') -w -o jsonpath='{.status.conditions[?(@.type=="MemoryPressure")].status}'

Expected output should remain False throughout the run.

Check that kubectl describe pod evaluator no longer shows Evicted events.
Validate memory usage stays below the node’s allocatable value:


kubectl top node $(kubectl get pod evaluator -o jsonpath='{.spec.nodeName}')

Peak usage should be comfortably under the node’s total RAM (e.g., ~95Gi on a 128 Gi node).

Prevention: Operational Guardrails

Monitoring & alerts: Use Azure Monitor for containers to create alerts on memoryPressure and containerMemoryWorkingSetBytes thresholds. Example alert query:


InsightsMetrics
| where Namespace == "kubelet"
| where Name == "memoryPressure"
| where Val == 1
| summarize count() by Computer, bin(TimeGenerated, 5m)

Resource quota enforcement: Define a ResourceQuota per namespace that caps total memory requests, preventing accidental over‑commitment.

Pod disruption budgets: For long‑running evaluation jobs, set a PodDisruptionBudget with maxUnavailable: 0 to avoid voluntary evictions during node upgrades.

Continuous profiling: Integrate tools like py-spy or gperftools to track memory growth of the inference process and catch leaks before they affect the node.

Automated testing: Include a memory‑stress test in CI that loads the largest model on a test node and asserts that MemoryPressure never becomes True.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub

FAQ

Why does the pod show OOMKilled after it has already been evicted?
When the kubelet evicts a pod, the container runtime still attempts to terminate the process. If the container’s memory limit is lower than the actual usage, the runtime logs an OOMKilled reason. The eviction event is the primary cause; OOMKilled is a secondary symptom.
Can I rely on kubectl top pod alone to detect memory pressure?
kubectl top reports container working set memory, not the node’s overall pressure. A node can be under pressure even if individual pods appear below their limits, because system daemons and kernel buffers also consume RAM. Always check the node condition MemoryPressure and kubelet logs.
Do GPU‑enabled NC series VMs have a different eviction behavior?
The eviction manager operates the same across VM SKUs. However, NC series allocate a portion of RAM for GPU drivers, reducing the effective memory available to pods. This can make a 12 GB model exceed the usable RAM on an NC6s_v3 even though the VM advertises 112 GB.
Is increasing the hard eviction threshold safe?
Raising --eviction-hard=memory.available<500Mi gives pods more breathing room during spikes, but it also delays the kubelet’s protection against OOM on the node itself. Use this only as a temporary measure while you adjust requests, limits, and node sizing.
How do I prevent non‑GPU pods from being evicted when GPU pods load large checkpoints?
Apply a nodeSelector or taint/toleration strategy so that checkpoint staging occurs on dedicated GPU nodes. Keep CPU‑only node pools free of large model artifacts, or use separate emptyDir volumes with medium: "Memory" that are deleted after the GPU pod finishes.