blue and red cargo ship on sea during daytime

Kubernetes pod OOMKilled after increasing AI model size

by

Problem – Pods OOMKilled After Increasing AI Model Size

An AI inference service deployed on a multi‑node Kubernetes cluster started failing shortly after the model binary grew from 2 GB to 4 GB. The symptoms observed across the cluster were:

  • State: Terminated Reason: OOMKilled in kubectl describe pod output.
  • Node‑level memory pressure events and occasional pod eviction warnings.
  • Cluster Autoscaler back‑off loops – new nodes were not provisioned because the pod’s memory request exceeded the node’s allocatable memory.
  • Intermittent latency spikes before the kill, suggesting CPU throttling.

Typical log excerpts from the affected pod and node:


$ kubectl describe pod inference-abc123
...
State:          Terminated
  Reason:       OOMKilled
  Message:      Container killed due to memory usage exceeding limit
...
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Warning  Evicted    2m    kubelet            Pod inference-abc123 was evicted because of node memory pressure

Docker daemon logs also reported:


time="2026-07-15T10:42:31.123456789Z" level=error msg="memory limit exceeded: container_id=abcdef123456 pid=7890"

Root Cause – Misaligned Resource Limits and Node Capacity

The underlying failure chain can be traced to three intertwined misconfigurations:

  1. Container memory limit too low for the new model. The pod spec still requested 2Gi and limited 2Gi, while the model alone required ~4Gi. Docker’s runtime constraints enforce the --memory cgroup limit; when the process attempted to mmap the model file, the kernel killed the container (exit code 137).
  2. CPU quota set to a low value. The cpu limit of 500m caused throttling. As the inference code stalled, it allocated additional buffers, increasing RSS until the memory limit was breached. Docker’s CPU quota/period mechanism (--cpu-quota) behaves exactly as described in the official CPU options doc.
  3. Node allocatable memory insufficient for the pod request. The Cluster Autoscaler ignored the pod because its memory request (still 2 Gi) exceeded the allocatable memory on the smallest node type (3 Gi). This mirrors the scenario described in the GitHub issue #94523, where node‑level pressure prevented autoscaling.

Additionally, a sidecar container that cached model shards in memory contributed ~1Gi of extra RSS, a pattern observed in the multi‑region deployment incident (see evidence package). The combined memory footprint far exceeded the node’s capacity, leading to system‑wide OOM conditions.

Investigation – Debugging Steps

1. Inspect Pod Events and Status


kubectl get pod inference-abc123 -o wide
kubectl describe pod inference-abc123

Look for OOMKilled and Evicted events as shown above.

2. Examine Container Resource Usage

Use kubectl top pod (metrics‑server) and docker stats on the node to compare actual RSS against limits:


kubectl top pod inference-abc123
# Example output
NAME               CPU(cores)   MEMORY(bytes)
inference-abc123   450m         2100Mi

# On the node
docker stats $(docker ps -q --filter "name=inference-abc123")

The memory usage quickly spikes beyond the configured 2Gi limit.

3. Verify Cgroup Limits Directly


cat /sys/fs/cgroup/memory/kubepods.slice/kubepods-besteffort-pod<uid>/memory.limit_in_bytes
# Expected: 2147483648 (2Gi)
cat /sys/fs/cgroup/cpu/kubepods.slice/kubepods-besteffort-pod<uid>/cpu.cfs_quota_us
# Expected: 50000 (0.5 CPU)

4. Check Node Allocatable Resources


kubectl describe node worker-node-01 | grep -A2 "Allocatable"
# Example output
Allocatable:
  cpu:                3800m
  memory:             14Gi
  pods:               110

If the pod’s memory request exceeds Allocatable, the scheduler will place the pod on a node already under pressure, as happened in the production incident.

5. Review Sidecar Memory Consumption


kubectl exec -it inference-abc123 -c sidecar -- sh -c "ps aux | grep cache"
# Example output shows 1.2Gi RSS for sidecar process

6. Correlate with Autoscaler Logs


kubectl logs deployment/cluster-autoscaler -n kube-system | grep "memory"
# Look for messages like:
# "failed to scale up node group: pod memory request 2Gi exceeds node group max memory 1.5Gi"

Solution – Align Resource Specifications and Adjust Cluster Capacity

1. Update Pod Resource Requests and Limits

Increase the memory request to cover the model size plus sidecar cache, and raise the limit to provide headroom for temporary buffers.


# Before (problematic)
resources:
  requests:
    memory: "2Gi"
    cpu: "500m"
  limits:
    memory: "2Gi"
    cpu: "500m"

# After (adjusted)
resources:
  requests:
    memory: "6Gi"   # 4Gi model + 1Gi sidecar + 1Gi safety margin
    cpu: "1000m"
  limits:
    memory: "8Gi"
    cpu: "1500m"

Apply the updated manifest:


kubectl apply -f inference-deployment.yaml

2. Refactor Model Loading to Reduce Peak RSS

  • Use mmap with MAP_PRIVATE and MADV_DONTNEED to unload unused sections.
  • Move the large model file to a read‑only emptyDir volume shared with the sidecar, avoiding duplicate copies in memory.

3. Adjust Cluster Autoscaler Configuration

Ensure the node group offers a machine type with at least 12Gi allocatable memory.


# Example autoscaler config snippet (kubespray discussion #1234)
node_groups:
  - name: "high-mem"
    min_size: 2
    max_size: 10
    instance_type: "n2-highmem-8"   # 64Gi total, ~58Gi allocatable
    labels:
      role: "inference"

4. Add a Vertical Pod Autoscaler (VPA) for Future Model Growth


apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: inference-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind:       Deployment
    name:       inference
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: inference
      mode: "Auto"

5. Enforce Node‑Level Memory Reservations

Set system-reserved and kube-reserved in the kubelet config to keep a safety buffer, preventing the node from entering OOM pressure during spikes.


kubeletConfiguration:
  systemReserved:
    memory: "2Gi"
  kubeReserved:
    memory: "1Gi"

Verification – Confirming the Fix

  1. Deploy the updated manifest and monitor pod status:

kubectl rollout status deployment/inference
kubectl get pod -l app=inference -w

Expected: Running state without OOMKilled events.

  1. Check memory usage against the new limits:

kubectl top pod inference-xyz789
# Example output
NAME               CPU(cores)   MEMORY(bytes)
inference-xyz789   850m         5600Mi

Memory stays below the 8Gi limit.

  1. Validate that the node no longer reports memory pressure:

kubectl describe node worker-node-02 | grep -i "memory pressure"
# No warnings should appear.
  1. Trigger a load test to ensure the inference latency remains stable and no throttling occurs.

hey -c 50 -n 10000 http://inference-service.default.svc.cluster.local/predict
# Observe latency < 100ms, CPU usage < 80% of limit.

Prevention – Operational Guardrails

  • Resource Profiling Pipeline: Automate extraction of model size and generate requests/limits values during CI/CD image build.
  • Monitoring & Alerts: Set alerts on container_memory_working_set_bytes approaching 80% of the limit and on node memory_pressure conditions.
  • Node Sizing Policy: Define minimum node types for AI workloads (e.g., n2-highmem) and enforce via NodeSelector or Affinity.
  • Sidecar Memory Isolation: Use separate cgroups or limit the sidecar’s memory explicitly; consider emptyDir with medium: "Memory" only when necessary.
  • Version Pinning: Keep Docker Engine and Kubernetes versions aligned with the behavior described in the official docs (e.g., Docker 24.x memory limit enforcement).

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

  1. Why does the pod get OOMKilled even though the node still has free memory?
    Because the container’s cgroup memory limit (--memory) is lower than the process’s RSS. The kernel kills the container regardless of node‑wide availability.
  2. Can I rely on the Cluster Autoscaler to add nodes when a pod’s memory request grows?
    Only if the request fits within the allocatable memory of at least one node type in the autoscaler’s node pool. Otherwise the scheduler will keep the pod pending and the autoscaler will not act.
  3. How do I determine the appropriate memory request for a new model?
    Measure the model file size, add the sidecar cache size, and include a safety margin (≈ 20‑30%). Use du -h on the mounted model directory and ps -o rss on a test container loading the model.
  4. Is increasing the CPU limit enough to avoid OOMKilled?
    Higher CPU limits can reduce throttling, but they do not affect memory limits. If the process stalls due to CPU throttling, it may allocate more buffers, eventually hitting the memory ceiling. Both limits must be adjusted in tandem.
  5. Why did the autoscaler back‑off instead of provisioning a larger node?
    The pod’s memory request (2 Gi) exceeded the maximum allocatable memory of the configured node group. The autoscaler logs (see evidence #1234) show a “memory request exceeds node group max memory” error, causing back‑off.