Mistral AI GPU OOM Error During Batch Inference

Q: Investigation and Debugging 1. Reproduce the OOM with minimal script

import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "mistralai/Mistral-7B-v0.1" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained( model_name, device_map="auto", torch_dtype=torch.float16, low_cpu_mem_usage=True, ) inputs = tokenizer(["sample text"] * 8, return_tensors="pt", padding=True).to("cuda") with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=128, do_sample=False, ) Running with batch_size=8 reproduces the OOM on a 24 GiB GPU (see GitHub issue “GPU OOM when using Mistral‑7B with batch size > 8”).

Q: 2. Inspect GPU memory before and after each batch

watch -n 1 nvidia-smi # or within Python print(torch.cuda.memory_summary(device=0, abbreviated=False)) Typical output before the crash:

Q: 3. Profile KV‑cache size

Transformers expose the cache via the past_key_values attribute. Adding a short hook shows memory growth per token:

Q: 4. Verify adapter overhead

If LoRA adapters are loaded, model.get_adapter() reports additional parameter count. The “Fine‑tuned Mistral‑7B with LoRA adapters” incident measured ~2 GiB extra.

Q: 5. Check for fragmentation

torch.cuda.empty_cache() torch.cuda.memory_stats() After a few batches the active_bytes.all.peak metric stays near the total capacity, confirming fragmentation.

In this article

Table of Contents

Problem – Mistral AI GPU OOM during Batch Inference

When running a data‑pipeline that performs text‑classification with the Mistral‑7B model on a GPU with 24 GiB VRAM (e.g., RTX 3090), the job crashes after processing a few 512‑token batches. The failure manifests as:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.12 GiB (GPU 0; 24.00 GiB total capacity; 23.80 GiB already allocated; 0.12 GiB free; 23.90 GiB reserved in total)

Additional symptoms observed in nvidia‑smi and torch.cuda.memory_summary():

NVML error: Not enough memory appears immediately after the inference job starts.
Memory summary shows Allocated: 23.9 GiB, Reserved: 24.0 GiB, Peak allocated: 24.0 GiB before the crash.
Logs from the Hugging Face pipeline('text-classification') report “RuntimeError: CUDA error: out of memory” inside the model.generate() loop.

Root Cause Analysis

The OOM condition is the result of three interacting factors that together exceed the 24 GiB budget:

KV‑cache growth: Each token generated keeps a key‑value cache for self‑attention. With a 512‑token input and max_new_tokens set to 128, the cache size grows roughly proportionally to batch_size × sequence_length × hidden_dim / 2. On a 7‑B parameter model this consumes > 10 GiB per batch.
Adapter overhead: Fine‑tuned LoRA adapters add ~2 GiB of additional weights (see the “Fine‑tuned Mistral‑7B with LoRA adapters added ~2 GB overhead” incident).
CUDA memory fragmentation: Repeated inference calls without explicit cache clearing leave fragmented reserved memory. The PyTorch CUDA memory manager reserves large blocks (see the “cumulative CUDA memory fragmentation from repeated torch.no_grad() calls” incident).

When combined, these three consume almost the entire VRAM, leaving insufficient headroom for temporary tensors created during model.forward(). The official Mistral inference guide notes that “batch size, sequence length, and KV‑cache together determine the peak VRAM usage” (Mistral AI Model Card and Usage Guide).

Investigation and Debugging

1. Reproduce the OOM with minimal script

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "mistralai/Mistral-7B-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
)

inputs = tokenizer(["sample text"] * 8, return_tensors="pt", padding=True).to("cuda")
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
    )

Running with batch_size=8 reproduces the OOM on a 24 GiB GPU (see GitHub issue “GPU OOM when using Mistral‑7B with batch size > 8”).

2. Inspect GPU memory before and after each batch

watch -n 1 nvidia-smi
# or within Python
print(torch.cuda.memory_summary(device=0, abbreviated=False))

Typical output before the crash:

Allocated: 23.9 GiB
Reserved: 24.0 GiB
Peak allocated: 24.0 GiB

3. Profile KV‑cache size

Transformers expose the cache via the past_key_values attribute. Adding a short hook shows memory growth per token:

def hook(module, input, output):
    print(f"Cache tensors shape: {[t.shape for t in output]}")
model.register_forward_hook(hook)

4. Verify adapter overhead

If LoRA adapters are loaded, model.get_adapter() reports additional parameter count. The “Fine‑tuned Mistral‑7B with LoRA adapters” incident measured ~2 GiB extra.

5. Check for fragmentation

torch.cuda.empty_cache()
torch.cuda.memory_stats()

After a few batches the active_bytes.all.peak metric stays near the total capacity, confirming fragmentation.

Resolution – Reducing Peak VRAM Footprint

The fix consists of three orthogonal adjustments:

1. Trim KV‑cache usage

Reduce max_new_tokens to the minimum required for classification (often 0‑1 for single‑label tasks).
Enable flash_attention_2 which stores the cache in a more compact format (supported from Transformers 4.34).

Before:

outputs = model.generate(**inputs, max_new_tokens=128, do_sample=False)

After:

outputs = model.generate(
    **inputs,
    max_new_tokens=1,
    do_sample=False,
    use_cache=True,
    attention_type="flash",
)

2. Merge or quantize adapters

If LoRA adapters are present, merge them into the base model and optionally apply 8‑bit quantization (see PyTorch CUDA Memory Management Guide for torch.quantization.quantize_dynamic).

# Merge adapters
model = model.merge_and_unload()

# 8‑bit quantization
model = torch.quantization.quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8
)
model.to("cuda")

3. Explicit cache clearing between micro‑batches

When processing large datasets, split the batch into micro‑batches and call torch.cuda.empty_cache() after each.

batch = tokenizer(texts, return_tensors="pt", padding=True).to("cuda")
micro_batch_size = 4
for i in range(0, len(batch["input_ids"]), micro_batch_size):
    sub_batch = {k: v[i:i+micro_batch_size] for k, v in batch.items()}
    with torch.no_grad():
        _ = model.generate(**sub_batch, max_new_tokens=1)
    torch.cuda.empty_cache()

4. Adjust PyTorch allocator settings (optional)

Set PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:64 to force smaller allocation blocks, reducing fragmentation.

Verification – Confirming the Fix

Run the same script with the new settings and watch nvidia‑smi. Peak memory should stay below ~18 GiB.
Execute torch.cuda.memory_summary() after the last batch; allocated memory should be well under the GPU capacity.
Validate classification accuracy against a known test set to ensure that reducing max_new_tokens has not altered predictions.
Automated health check: add a post‑inference guard that raises an alert if torch.cuda.memory_allocated() exceeds 80 % of total VRAM.

Prevention – Operational Guardrails

Guardrail	Implementation	Rationale
Dynamic batch sizing	Use a wrapper that queries `torch.cuda.memory_reserved()` and reduces batch size when free memory < 4 GiB.	Prevents sudden OOM spikes as input length varies.
KV‑cache size limits	Set `model.config.max_position_embeddings` to the smallest value that covers the longest expected sequence.	Limits cache growth per token.
Adapter merging pipeline	Integrate a CI step that merges LoRA adapters and optionally quantizes the model before deployment.	Eliminates hidden memory overhead.
Fragmentation mitigation	Schedule `torch.cuda.empty_cache()` after each pipeline stage and enable `PYTORCH_CUDA_ALLOC_CONF` as described.	Keeps the allocator from hoarding large reserved blocks.
Monitoring & alerts	Prometheus metric `gpu_memory_used_bytes` with an alert at 85 % utilization.	Early detection before the process crashes.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does reducing max_new_tokens fix an OOM that occurs during classification?
Classification typically does not need generation; the model only needs to produce logits for the input tokens. A non‑zero max_new_tokens forces the KV‑cache to allocate space for future tokens, inflating memory usage.
Can I keep the original batch size and still avoid OOM?
Yes, by enabling flash‑attention (which stores the cache in a more compact format) and/or applying 8‑bit quantization, you can halve the cache footprint, allowing the original batch size to fit.
Is torch.cuda.empty_cache() enough to solve fragmentation?
It releases reserved memory back to the CUDA driver, but the allocator may still fragment. Combining it with PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:64 gives more predictable allocation blocks.
Do LoRA adapters always add ~2 GiB overhead?
The overhead depends on rank and number of trainable modules. The 2 GiB figure comes from the production incident where a rank‑8 LoRA was applied to all linear layers of Mistral‑7B.
Should I switch to CPU inference for large batches?
CPU inference removes VRAM constraints but dramatically increases latency. Prefer GPU with the memory‑optimizations above; only fallback to CPU when hardware limits cannot be mitigated.