GPU OOM in inference service after blue‑green deployment on Kubernetes

Problem

The AI inference microservice runs on GPU‑enabled nodes in a Kubernetes cluster and consumes a high‑throughput Kafka topic. After a blue‑green deployment rollout, the new (green) pods start reporting CUDA out of memory (OOM) errors, causing request failures and a cascade of consumer lag in Kafka.

Typical log excerpt from a failing pod:

2026-09-28 14:32:07,123 ERROR inference_worker.py:210 - CUDA error: out of memory
Traceback (most recent call last):
  File "/app/inference_worker.py", line 198, in run_batch
    outputs = model(input_tensor)
  File "/usr/local/lib/python3.9/site-packages/torch/cuda/__init__.py", line 123, in _lazy_init
    torch._C._cuda_init()
RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB (GPU 0; 11.00 GiB total capacity; 9.45 GiB already allocated; 0.12 GiB free; 9.55 GiB reserved in total)

Impact includes:

• Increased end‑to‑end latency (seconds per inference).
• Kafka consumer group lag spikes (>10 minutes).
• Pod restarts due to OOM kill (observed in kubectl get pods).

Root Cause

The blue‑green rollout introduced two changes that together exhausted GPU memory:

1. Increased consumer concurrency. The green deployment raised max.poll.records from 500 to 2000 and added three additional consumer threads per pod to improve throughput. Each thread creates its own model instance (due to a lazy‑load pattern), resulting in three copies of the model residing simultaneously on the same GPU.
2. Missing GPU memory limits. The original (blue) deployment relied on the default limits: { nvidia.com/gpu: 1 } without an explicit memory limit. The green pod spec added a requests entry but omitted limits.memory, allowing the container runtime to allocate beyond the GPU’s physical memory.

Combined, the three model copies (≈3 GiB each) plus the input batch buffers exceeded the 11 GiB GPU capacity, triggering the OOM error.

Debug

1. Verify pod status and OOM events

kubectl get pods -n ai-inference -l app=inference -o wide
kubectl describe pod inference-green-5d9f8c9c9-7kz9l -n ai-inference
# Look for OOMKilled in the Events section

2. Inspect GPU utilization inside the pod

kubectl exec -it inference-green-5d9f8c9c9-7kz9l -n ai-inference -- nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap| Memory-Usage | GPU-Util  Compute M. |
|                               |                  |               |
| 0    Tesla V100  Off          | 00000000:3B:00.0 Off | 0                  |
| N/A   32C    P0    45W / 250W |  9500MiB / 16160MiB |  78%      Default   |
+-----------------------------------------------------------------------------+

3. Correlate consumer configuration with model loading

# Inference service config (ConfigMap)
apiVersion: v1
kind: ConfigMap
metadata:
  name: inference-config
data:
  KAFKA_MAX_POLL_RECORDS: "2000"
  CONSUMER_THREADS: "3"
  MODEL_PATH: "/models/resnet50.pt"

4. Check for multiple model instances

# Grep for torch.load calls in the container image
kubectl exec -it inference-green-5d9f8c9c9-7kz9l -n ai-inference -- grep -R "torch.load" -n /app

5. Review deployment manifest differences (blue vs. green)

Field	Blue (working)	Green (failing)
resources.requests.cpu	2	2
resources.limits.cpu	4	4
resources.requests.memory	8Gi	8Gi
resources.limits.memory	12Gi	8Gi (missing)
resources.limits.nvidia.com/gpu	1	1
env.KAFKA_MAX_POLL_RECORDS	500	2000
env.CONSUMER_THREADS	1	3

Solution

1. Enforce a single model instance per GPU

Refactor the inference worker to load the model once at process start and share it across consumer threads.

Before (lazy load per thread):

def worker_thread():
    # Each thread loads the model independently
    model = torch.load(os.getenv("MODEL_PATH")).to('cuda')
    while True:
        batch = consumer.poll()
        run_inference(model, batch)

After (singleton model):

# Global model loaded once
model = torch.load(os.getenv("MODEL_PATH")).to('cuda')
model.eval()
model_lock = threading.Lock()

def worker_thread():
    while True:
        batch = consumer.poll()
        with model_lock:
            outputs = model(batch)
        # post‑process outputs

2. Add explicit memory limits and enable nvidia.com/gpu memory enforcement

Update the deployment spec to set limits.memory and use the NVIDIA device plugin’s memory resource (if supported) or a initContainer that pre‑allocates a memory pool.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: inference
  template:
    metadata:
      labels:
        app: inference
    spec:
      containers:
      - name: inference
        image: registry.example.com/inference:green
        resources:
          requests:
            cpu: "2"
            memory: "8Gi"
            nvidia.com/gpu: "1"
          limits:
            cpu: "4"
            memory: "12Gi"
            nvidia.com/gpu: "1"
        env:
        - name: KAFKA_MAX_POLL_RECORDS
          value: "500"
        - name: CONSUMER_THREADS
          value: "1"

3. Reduce Kafka poll size and throttle consumer throughput

Revert KAFKA_MAX_POLL_RECORDS to a value that keeps the per‑batch GPU memory footprint below 4 GiB, and configure max.poll.interval.ms to allow back‑pressure.

4. Implement graceful shutdown during blue‑green switch

Ensure the green pods finish processing in‑flight batches before termination:

# Add preStop hook
lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "python /app/drain.py && sleep 30"]

Verify

1. Confirm single model instance:

   
   kubectl exec -it inference-green-5d9f8c9c9-7kz9l -n ai-inference -- ps -ef | grep python
   # Should show only one python process per pod
   

2. Check GPU memory after rollout:

   
   kubectl exec -it inference-green-5d9f8c9c9-7kz9l -n ai-inference -- nvidia-smi
   # Expect Memory-Usage around 4–5GiB, free > 6GiB
   

3. Validate Kafka lag:

   
   kubectl exec -it kafka-0 -n kafka -- kafka-consumer-groups.sh \
     --bootstrap-server localhost:9092 \
     --describe --group inference-consumer
   # Lag should be near zero after a few minutes
   

4. Monitor pod restarts:

   
   kubectl get pods -n ai-inference -l app=inference -w
   # No OOMKilled events for at least 30 minutes
   

Prevent

• Resource Guardrails: Enforce limits.memory and limits.nvidia.com/gpu via a LimitRange policy.
• GPU Memory Metrics: Export nvidia_smi_memory_used_bytes to Prometheus and set alerts when usage exceeds 80 % of capacity.
• Canary with Throttled Throughput: Deploy green pods with a reduced KAFKA_MAX_POLL_RECORDS and a lower CONSUMER_THREADS count; only increase after confirming stable memory usage.
• Model Loading Guard: Use a singleton pattern or a dedicated model server (e.g., Triton Inference Server) to avoid accidental multiple copies.
• Blue‑Green Coordination: Add a post‑deployment hook that checks GPU memory before scaling green pods to full replica count.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

1. Why did the OOM only appear after the blue‑green rollout? The green version increased both the number of consumer threads and the Kafka poll size, causing multiple model instances to be loaded concurrently. The original blue deployment used a single thread and a smaller poll size, keeping GPU usage within limits.
2. Can I keep multiple consumer threads without OOM? Yes, but you must share a single model instance across threads (e.g., using a global model with a lock) or allocate additional GPUs per pod. Avoid loading the model per thread.
3. Is there a way to limit GPU memory per container? The NVIDIA device plugin supports nvidia.com/gpu count limits. For finer‑grained memory control, use CUDA_VISIBLE_DEVICES with gpu_memory_limit in recent driver versions or run a memory‑capped container runtime such as runsc with --memory flags.
4. How should I size KAFKA_MAX_POLL_RECORDS for GPU inference? Estimate the per‑record GPU memory (model size + input tensor). Multiply by the desired batch size and keep the total below ~70 % of the GPU capacity to leave headroom for framework overhead.
5. What monitoring alerts are most useful for early detection? Alert on:
   • GPU memory usage > 80 % (Prometheus query on nvidia_smi_memory_used_bytes).
   • Pod OOMKilled events.
   • Kafka consumer lag > 5 minutes.
   • Sudden increase in process_resident_memory_bytes for the inference container.