Azure VM OOM Error When Launching AI Inference After DR Failover

Q: 4. Adjust kernel swappiness for balanced behavior

sysctl -w vm.swappiness=10 echo 'vm.swappiness=10' >> /etc/sysctl.d/99-swap.conf 5. Re‑deploy the inference service kubectl rollout restart deployment/inference The service now starts, loads the model, and begins serving requests without OOM events.

In this article

Table of Contents

Problem – Azure VM OOM error when launching AI inference after DR failover

During a planned disaster‑recovery (DR) failover from the primary Azure region to a secondary region, the AI inference service fails to start. The container runtime reports a memory allocation failure and the process is terminated by the Linux OOM killer:


2024-09-15T08:12:34.567Z ERROR InferenceService[pid=3124]: Failed to allocate 2.1GiB for model weights
2024-09-15T08:12:34.572Z INFO  kernel: Out of memory: Kill process 3124 (inference) score 987 or sacrifice child
2024-09-15T08:12:34.573Z INFO  kernel: Killed process 3124 (inference) total-vm:2156788kB, anon-rss:2097152kB, file-rss:0kB

The same Docker image and inference code run without issue in the primary region, indicating that the failure is environment‑specific rather than a code defect.

Root Cause – Mismatched VM SKU and memory‑related configuration drift

Investigation revealed three intertwined factors that together caused the out‑of‑memory (OOM) condition:

VM size downgrade during failover. The primary region used Standard_ND40rs_v2 (40 vCPU, 672 GiB RAM, 8 × NVIDIA Tesla V100). The secondary region could not provision that SKU due to capacity constraints, so the DR script fell back to Standard_NC24s_v3 (24 vCPU, 224 GiB RAM, 4 × Tesla V100). The inference model requires ~250 GiB of RAM when loaded in FP16 precision.
Container memory limit. The Kubernetes pod spec in the DR manifest set resources.limits.memory: "256Gi". On the smaller VM the cgroup limit is enforced, but the kernel still reserves ~10 GiB for the OS and GPU driver, leaving <≈214 GiB> usable, which is insufficient for the model.
Swap disabled. The secondary VMs were created from a custom image that disables swap (vm.swappiness=0) to improve latency. Without swap the kernel has no fallback when the cgroup limit is exceeded, so it immediately invokes the OOM killer.

Collectively, the reduced physical memory, strict cgroup limit, and lack of swap caused the inference process to exceed available memory during model loading, triggering the OOM kill.

Debug – Step‑by‑step investigation

Confirm VM SKU and available memory.


# Azure CLI
az vm show -g rg-dr -n ai-inference-vm --query "hardwareProfile.vmSize" -o tsv
# Expected: Standard_ND40rs_v2
# Actual:   Standard_NC24s_v3

# Inside the VM
free -h
# Output
              total        used        free      shared  buff/cache   available
Mem:           224Gi       212Gi        2Gi       0.1Gi        9Gi        5Gi

Inspect cgroup memory limits for the pod.


kubectl get pod inference-0 -o jsonpath='{.spec.containers[0].resources.limits.memory}'
# Output: 256Gi

# Inside the container
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
# Output: 274877906944   # 256Gi

Check kernel OOM logs.


journalctl -k | grep -i "out of memory"
# Sample output
Sep 15 08:12:34 vmkernel kernel: Out of memory: Kill process 3124 (inference) score 987 or sacrifice child

Validate swap status.


swapon --show
# No output – swap is disabled

cat /proc/sys/vm/swappiness
# 0

Compare DR deployment manifest with production.

Field Production DR

VM size Standard_ND40rs_v2 (672 GiB) Standard_NC24s_v3 (224 GiB)

Pod memory limit 512Gi 256Gi

Swap Enabled (2 GiB) Disabled

Field	Production	DR
VM size	Standard_ND40rs_v2 (672 GiB)	Standard_NC24s_v3 (224 GiB)
Pod memory limit	512Gi	256Gi
Swap	Enabled (2 GiB)	Disabled

Solution – Align resources and adjust memory policies

1. Ensure SKU parity or provide a fallback with sufficient RAM

Update the Azure Site Recovery (ASR) failover policy to request the same SKU or a minimum of Standard_ND40rs_v2. If capacity is unavailable, pre‑allocate a reserved VM of the required size in the secondary region.


// ARM template snippet (before)
{
  "type": "Microsoft.Compute/virtualMachines",
  "name": "[parameters('vmName')]",
  "properties": {
    "hardwareProfile": {
      "vmSize": "Standard_NC24s_v3"
    }
  }
}

// After – enforce SKU parity
{
  "type": "Microsoft.Compute/virtualMachines",
  "name": "[parameters('vmName')]",
  "properties": {
    "hardwareProfile": {
      "vmSize": "Standard_ND40rs_v2"
    }
  }
}

2. Raise the pod memory limit to match available RAM

Modify the Kubernetes deployment used for DR to request a limit that leaves a safety margin for the OS and GPU driver.


// deployment.yaml (before)
resources:
  limits:
    memory: "256Gi"
  requests:
    memory: "256Gi"

// after – increase to 480Gi (fits within 672Gi VM)
resources:
  limits:
    memory: "480Gi"
  requests:
    memory: "480Gi"

3. Enable a small swap file as a safety net

Even with ample RAM, a swap file prevents immediate OOM termination during transient spikes.


# Inside the VM (run as root)
fallocate -l 8G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile

# Persist across reboots
echo '/swapfile none swap sw 0 0' >> /etc/fstab

# Verify
swapon --show
# /swapfile  size=8G   used=0   priority=-2

4. Adjust kernel swappiness for balanced behavior


sysctl -w vm.swappiness=10
echo 'vm.swappiness=10' >> /etc/sysctl.d/99-swap.conf

5. Re‑deploy the inference service


kubectl rollout restart deployment/inference

The service now starts, loads the model, and begins serving requests without OOM events.

Verify – Confirm that the issue is resolved

Check pod status and logs.


kubectl get pod -l app=inference -o wide
kubectl logs inference-0
# Expected: "Model loaded successfully, ready to serve"

Validate memory consumption.


kubectl exec inference-0 -- free -h
# Output should show ~460Gi used, ~200Gi free

Monitor Azure VM metrics.


az monitor metrics list \
  --resource /subscriptions/xxxx/resourceGroups/rg-dr/providers/Microsoft.Compute/virtualMachines/ai-inference-vm \
  --metric "AvailableMemoryBytes" "Percentage CPU" \
  --interval PT1M

AvailableMemoryBytes should stay above 100 GiB during normal load.

Run a functional inference test.


curl -X POST https://inference-dr.example.com/v1/predict \
  -H "Content-Type: application/json" \
  -d '{"input": [1,2,3,4]}'
# Expected: JSON response with prediction

Prevent – Practices to avoid recurrence

Enforce SKU parity in DR plans. Use Azure Policy or ASR custom scripts to abort failover if the target SKU does not meet a minimum memory threshold.
Include memory safety margins. Set pod requests at ≤80 % of the VM’s total RAM to leave headroom for OS, drivers, and burst traffic.
Automate swap provisioning. Add a VM extension or cloud‑init script that creates a swap file on all DR VMs.
Integrate memory‑related alerts. Configure Azure Monitor alerts on AvailableMemoryBytes and on OOM killer events (LinuxSyslog with keyword “Out of memory”).
Run regular DR drills. Verify that the secondary region can provision the exact VM size and that the deployment manifests are identical.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub

FAQ

Why does the OOM error appear even though the pod limit is higher than the VM’s physical RAM?
The cgroup limit is enforced regardless of physical RAM. When the limit exceeds the actual available memory, the kernel cannot satisfy allocation requests and triggers the OOM killer. Aligning the limit with the VM’s capacity prevents this mismatch.
Can I keep the smaller VM size and still run the model?
Only if you reduce the model’s memory footprint (e.g., quantize to INT8, use model sharding, or lower batch size). Otherwise the model exceeds the hardware limits.
Is disabling swap advisable for AI inference workloads?
Disabling swap improves latency but removes a safety net for sudden memory spikes. A modest swap file (4‑8 GiB) provides protection without noticeable latency impact.
How do I detect that a DR failover has fallen back to a lower‑spec VM?
Include a post‑failover validation step that queries az vm show for hardwareProfile.vmSize and compares it against a whitelist. Fail the DR runbook if the size does not match the expected SKU.
Will increasing the pod’s requests.memory without changing the limit help?
No. The limit caps the maximum usage; the request only influences scheduling. Both must be set to a value that the underlying VM can sustain.