CUDA OOM During Long Sequence Inference On NVIDIA GPU

Q: Solution – Dynamic KV‑Cache Eviction

The robust fix is to replace the static, monolithic KV buffer with a sliding‑window eviction strategy that:

Q: Before: Static Allocation (Problematic)

# Simplified PyTorch inference loop max_seq_len = 8192 # static kv_cache = torch.empty( (batch_size, max_seq_len, hidden_dim * 2), device="cuda", dtype=torch.float16, ) for token_idx in range(prompt_len + generation_len): # write into kv_cache[token_idx] ... When prompt_len + generation_len > max_seq_len, the write overruns the allocated tensor, causing the CUDA OOM shown in the logs.

Q: After: Sliding‑Window Eviction (Recommended)

class SlidingKVCache: def __init__(self, batch_size, hidden_dim, max_total_tokens, device): self.batch_size = batch_size self.hidden_dim = hidden_dim self.max_total_tokens = max_total_tokens # e.g., 12_000 self.device = device # Allocate once with the maximum possible size self.buffer = torch.empty( (batch_size, max_total_tokens, hidden_dim * 2), device=device, dtype=torch.float16, ) self.start = 0 # index of the oldest token self.end = 0 # index after the newest token def append(self, new_kv): new_len = new_kv.shape[1] if self.end + new_len > self.max_total_tokens: # Evict oldest tokens to make room evict_len = (self.end + new_len) - self.max_total_tokens self.start += evict_len # Shift remaining data to the front to keep it contiguous self.buffer[:, :self.end-self.start] = self.buffer[:, self.start:self.end] self.end = self.end - self.start self.start = 0 # Write new KV at the tail self.buffer[:, self.end:self.end+new_len] = new_kv self.end += new_len def get_cache(self): return self.buffer[:, :self.end] Integration into the inference loop:

In this article

Table of Contents

Problem Description

On‑premise LLM inference nodes equipped with NVIDIA GPUs (e.g., A100 40 GiB, RTX 3090 24 GiB) crash when processing prompts that exceed the pre‑allocated key‑value (KV) cache size. The failure manifests as a CUDA out‑of‑memory (OOM) exception during the attention kernel launch.

Typical log excerpts:


[2026-08-10 14:32:07] ERROR - torch.cuda.OutOfMemoryError: CUDA out of memory. 
 Tried to allocate 2.00 GiB for tensor shape [1, 8192, 4096] on device 0
[2026-08-10 14:32:08] ERROR - CUBLAS_STATUS_ALLOC_FAILED during attention kernel launch
kv cache overflow: required cache size (tokens * hidden_dim) exceeds allocated buffer; aborting kernel execution.

Operational impact includes:

Immediate termination of the serving process.
Loss of in‑flight requests and degraded throughput.
Memory fragmentation that prevents reuse of freed buffers, causing repeated OOM spikes.

Root Cause Analysis

The transformer decoder maintains a KV cache that stores the projected key and value tensors for every processed token. In many production deployments the cache buffer is allocated once per model instance with a fixed capacity (e.g., 8 k tokens) using torch.empty or cudaMalloc. This static allocation is documented in the NVIDIA CUDA Programming Guide – Memory Management and the TensorRT Dynamic Tensor Memory guidelines.

When a request exceeds the static capacity, the inference engine attempts to write beyond the allocated region. The CUDA runtime detects the over‑commit and aborts the kernel, surfacing the OOM error shown above. Additional contributing factors:

Multi‑tenant batch serving: concurrent requests multiply the KV footprint (e.g., 4×6 k tokens on RTX 3090 exceeded the 30 GiB usable pool, as reported in the real incident logs).
Memory fragmentation: Even after a request finishes, the freed KV buffers may be split into non‑contiguous blocks, preventing a large contiguous allocation for the next long sequence (observed in the DGX‑A100 intermittent spikes).
Assumption of monotonic growth: The inference pipeline assumes the cache only grows, never shrinks, which is invalid for sliding‑window generation or when early‑stop tokens are produced.

Thus, the root cause is a mismatch between the static KV‑cache size and the actual runtime context window, compounded by fragmentation and lack of eviction logic.

Investigation and Debugging

Below is a reproducible debugging workflow that mirrors the steps taken in the Nsight Systems profiling and the community GitHub issues (vLLM#124, huggingface/transformers#26031).

Inspect GPU memory usage before request:

nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 1

Expected output:

memory.used [MiB], memory.total [MiB]
22000, 24576

Capture per‑tensor allocation with PyTorch:
```
import torch
print(torch.cuda.memory_summary(device=0, abbreviated=False))
```
Look for a large block labeled kv_cache whose size equals max_seq_len × hidden_dim × 2 × dtype_size.
Profile the attention kernel: Use Nsight Systems to record the kernel launch timeline and verify the failure point.
```
nsys profile -t cuda -o kv_profile python serve.py --prompt "..."
```
The timeline will show a red‑highlighted cublasGemmEx or custom attention kernel with the status CUBLAS_STATUS_ALLOC_FAILED.

Validate static KV size configuration:

# Example from a HuggingFace pipeline
model.config.max_position_embeddings  # e.g., 8192
model.config.use_cache               # True
model.config.max_new_tokens          # 1024 (runtime)

If max_position_embeddings < prompt_len + max_new_tokens, the cache will overflow.

Check for fragmentation: After a failed request, run:
```
torch.cuda.empty_cache()
torch.cuda.memory_snapshot()
```
If the snapshot shows many small free blocks, fragmentation is present.

Solution – Dynamic KV‑Cache Eviction

The robust fix is to replace the static, monolithic KV buffer with a sliding‑window eviction strategy that:

Allocates a maximum buffer based on the worst‑case aggregate token count across all concurrent requests.
Evicts the oldest KV entries once the total token count exceeds a configurable high‑water mark.
Re‑uses freed memory blocks to avoid fragmentation.

Before: Static Allocation (Problematic)

# Simplified PyTorch inference loop
max_seq_len = 8192  # static
kv_cache = torch.empty(
    (batch_size, max_seq_len, hidden_dim * 2),
    device="cuda",
    dtype=torch.float16,
)

for token_idx in range(prompt_len + generation_len):
    # write into kv_cache[token_idx]
    ...

When prompt_len + generation_len > max_seq_len, the write overruns the allocated tensor, causing the CUDA OOM shown in the logs.

After: Sliding‑Window Eviction (Recommended)

class SlidingKVCache:
    def __init__(self, batch_size, hidden_dim, max_total_tokens, device):
        self.batch_size = batch_size
        self.hidden_dim = hidden_dim
        self.max_total_tokens = max_total_tokens  # e.g., 12_000
        self.device = device
        # Allocate once with the maximum possible size
        self.buffer = torch.empty(
            (batch_size, max_total_tokens, hidden_dim * 2),
            device=device,
            dtype=torch.float16,
        )
        self.start = 0   # index of the oldest token
        self.end = 0     # index after the newest token

    def append(self, new_kv):
        new_len = new_kv.shape[1]
        if self.end + new_len > self.max_total_tokens:
            # Evict oldest tokens to make room
            evict_len = (self.end + new_len) - self.max_total_tokens
            self.start += evict_len
            # Shift remaining data to the front to keep it contiguous
            self.buffer[:, :self.end-self.start] = self.buffer[:, self.start:self.end]
            self.end = self.end - self.start
            self.start = 0
        # Write new KV at the tail
        self.buffer[:, self.end:self.end+new_len] = new_kv
        self.end += new_len

    def get_cache(self):
        return self.buffer[:, :self.end]

Integration into the inference loop:

kv = SlidingKVCache(batch_size, hidden_dim, max_total_tokens=12000, device="cuda")

for step in range(total_steps):
    # Compute new key/value for the current step
    new_kv = model.compute_kv(input_ids[:, step:step+1])
    kv.append(new_kv)

    # Pass only the active portion to the attention module
    attn_output = model.attention(
        query,
        kv.get_cache(),
        ...
    )

This approach guarantees that the total allocated memory never exceeds the pre‑determined budget, while still preserving the most recent context needed for generation.

Validation

After deploying the sliding‑window cache, verify stability with the following steps:

Run a stress test that sequentially generates 15 k tokens on a single request.
Monitor GPU memory:

watch -n 0.5 nvidia-smi --query-gpu=memory.used,memory.total --format=csv

Memory should plateau near the configured max_total_tokens (e.g., ~30 GiB on A100) without spikes.

Check that no OOM errors appear in the application logs.
Confirm that the KV cache size reported by torch.cuda.memory_summary() matches the expected allocation.
Run functional tests to ensure generated text quality is unchanged (the eviction only removes the oldest context, which is acceptable for sliding‑window attention as per the Hugging Face KV‑cache handling docs).

Operational Best Practices and Prevention

Configure a high‑water/low‑water ratio: Allocate 10‑15 % headroom (e.g., set max_total_tokens = 1.1 × expected peak) to absorb bursty traffic.
Enable GPU memory profiling in production: Use nsight-systems or torch.cuda.memory_stats() to emit metrics to Prometheus. Alert when memory.used / memory.total > 0.85.
Isolate tenants with per‑model memory quotas: In multi‑tenant setups, instantiate a separate SlidingKVCache per tenant and enforce a token‑budget limit.
Periodically compact fragmented buffers: The append method already shifts active KV entries; schedule a background compaction if eviction frequency is low.
Prefer mixed‑precision (FP16/ BF16) for KV tensors: Reduces per‑token footprint by half, directly increasing the token capacity for a fixed VRAM budget.
Document the cache policy: Include the eviction strategy in model configuration files (e.g., cache.max_total_tokens) so that downstream services can query it.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub

FAQ

Why does the OOM only happen for long prompts but not for short ones?
The static KV cache is sized for a typical maximum (e.g., 8 k tokens). Short prompts stay within that bound; long prompts exceed it, forcing the runtime to allocate beyond the pre‑reserved region, which triggers the CUDA OOM.
Can I simply increase max_position_embeddings in the model config?
Increasing the config without adjusting the underlying GPU allocation leads to the same OOM, because the actual buffer is still allocated with the original size. You must also enlarge the CUDA buffer or switch to a dynamic eviction scheme.
Is torch.cuda.empty_cache() a reliable fix?
Calling empty_cache releases PyTorch’s memory pool back to the driver but does not shrink the static KV tensor. It may temporarily free enough space for a small request but does not solve the fundamental overflow problem.
How does sliding‑window eviction affect generation quality?
When the oldest context is evicted, the model loses information about tokens that are far behind the current generation window. For most LLMs this is acceptable because attention weights decay with distance, and the Hugging Face transformers library explicitly supports sliding windows for long‑sequence generation.
What monitoring metrics should I expose to detect future KV‑cache pressure?
Expose at least the following:
- gpu_memory_used_bytes
- kv_cache_current_tokens
- kv_cache_eviction_rate
- inference_requests_oom_total
Set alerts on sustained high memory usage or a rising eviction rate, which can indicate that the configured max_total_tokens is still insufficient.