LlamaIndex OOM Error During Kafka Batch Processing In Production

Q: LlamaIndex OOM error during Kafka batch processing in production Problem

In a high‑throughput event‑driven pipeline, a LlamaIndex worker consumes document batches from Kafka, creates embeddings using a transformer model, and stores the indexed vectors in a vector store. Under load the worker crashes with an out‑of‑memory (OOM) termination:

In this article

Table of Contents

LlamaIndex OOM error during Kafka batch processing in production

Problem

In a high‑throughput event‑driven pipeline, a LlamaIndex worker consumes document batches from Kafka, creates embeddings using a transformer model, and stores the indexed vectors in a vector store. Under load the worker crashes with an out‑of‑memory (OOM) termination:


2024-10-12 03:45:21,874 ERROR worker.py:124 -- Process terminated due to OOM
Traceback (most recent call last):
  File "/app/worker.py", line 87, in process_batch
    embeddings = index.embed_documents(docs)
  File "/usr/local/lib/python3.9/site-packages/llama_index/embeddings.py", line 212, in embed_documents
    vectors = model.encode(batch_texts, **self.encode_kwargs)
  File "/usr/local/lib/python3.9/site-packages/transformers/pipelines.py", line 456, in __call__
    outputs = self.model(**inputs)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in __call__
    return self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in forward
    raise RuntimeError('CUDA out of memory. Tried to allocate ...')
RuntimeError: CUDA out of memory. Tried to allocate 2.34 GiB (GPU 0; 7.80 GiB total capacity; 5.23 GiB already allocated; 1.12 GiB free; 0.00 GiB reserved in total)

The failure propagates to the Kafka consumer group, causing message re‑processing loops and increased latency.

Root Cause Analysis

The OOM condition originates from three interacting factors:

Unbounded batch size. The consumer fetches up to max.poll.records (default 500) and forwards the entire payload to LlamaIndex. Large documents (average 2 KB) multiplied by the model’s tokenization overhead cause the in‑memory representation to exceed GPU memory.
Embedding model loaded per request. The worker code instantiated a new SentenceTransformer for every batch, causing duplicate model copies in GPU memory before the previous copy was released.
Missing explicit memory management. The PyTorch tensors generated during tokenization and model inference were not detached or cleared, leaving references in the Python GC heap and preventing CUDA memory from being freed.

Combined, these issues lead to a rapid escalation of GPU allocation until the runtime aborts with a CUDA OOM error.

Investigation and Debugging Steps

Below is the step‑by‑step methodology that reproduced the issue and identified the root cause.

1. Reproduce with a controlled batch


# Simulate a single batch of 500 docs (2KB each)
python - <<'PY'
from llama_index import LlamaIndex
import random, string

def random_doc():
    return ''.join(random.choices(string.ascii_letters + ' ', k=2048))

docs = [random_doc() for _ in range(500)]
index = LlamaIndex()
index.embed_documents(docs)  # crashes with OOM
PY

2. Monitor GPU memory


$ watch -n1 nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap| Memory-Usage | GPU-Util  Compute M. |
|                               |                  |               |
|  0  Tesla T4            Off  | 00000000:00:1E.0 Off | 0                  |
| N/A   45C    P0    30W /  70W |  7200MiB / 15360MiB |  98%      Default |
+-------------------------------+----------------------+----------------------+

The memory usage spikes just before the crash, confirming GPU pressure.

3. Trace model instantiation


2024-10-12 03:44:58,112 DEBUG worker.py:45 -- Loading embedding model: sentence-transformers/all-MiniLM-L6-v2
2024-10-12 03:44:58,115 INFO  torch.cuda:125 -- CUDA device 0 initialized, total memory 7.80 GiB
2024-10-12 03:44:58,120 DEBUG worker.py:45 -- Model loaded, GPU memory usage: 1.12 GiB
2024-10-12 03:45:20,001 DEBUG worker.py:78 -- Processing next batch, reloading model
2024-10-12 03:45:20,003 DEBUG worker.py:45 -- Loading embedding model: sentence-transformers/all-MiniLM-L6-v2
...

Repeated model loads are evident in the logs.

4. Verify reference retention


>>> import gc, torch
>>> gc.collect()
>>> torch.cuda.memory_summary()
...
Allocated memory blocks:  12,345,678 bytes
Reserved memory blocks:   9,876,543 bytes
Active memory blocks:      2,469,135 bytes

Even after gc.collect(), a substantial amount of memory remains reserved, indicating lingering tensor references.

Solution

The fix consists of three coordinated changes:

1. Enforce a bounded batch size

Configure the Kafka consumer to limit max.poll.records to a safe value (e.g., 50) and split larger payloads into sub‑batches before invoking LlamaIndex.

# before (default)
consumer = KafkaConsumer(
    'documents',
    bootstrap_servers='kafka:9092',
    group_id='llama-index-workers',
    enable_auto_commit=False,
)

# after (bounded batch)
consumer = KafkaConsumer(
    'documents',
    bootstrap_servers='kafka:9092',
    group_id='llama-index-workers',
    enable_auto_commit=False,
    max_poll_records=50,               # limit Kafka fetch size
    fetch_max_bytes=5_000_000,         # optional size cap
)

2. Load the embedding model once per worker process

Instantiate the model at startup and reuse the same instance for all batches.

# before (per‑batch load)
def process_batch(batch):
    model = SentenceTransformer('all-MiniLM-L6-v2')
    index = LlamaIndex(embedding_model=model)
    embeddings = index.embed_documents(batch)

# after (singleton)
# global singleton initialized at module import
from sentence_transformers import SentenceTransformer
from llama_index import LlamaIndex

_EMBEDDING_MODEL = SentenceTransformer('all-MiniLM-L6-v2')
_INDEX = LlamaIndex(embedding_model=_EMBEDDING_MODEL)

def process_batch(batch):
    embeddings = _INDEX.embed_documents(batch)

3. Explicitly free GPU tensors after each batch

Detach tensors, clear the CUDA cache, and invoke the Python garbage collector.

import torch, gc

def process_batch(batch):
    embeddings = _INDEX.embed_documents(batch)
    # Use embeddings (e.g., store in vector DB) ...

    # Explicit cleanup
    del embeddings
    torch.cuda.empty_cache()
    gc.collect()

4. Optional: Enable mixed‑precision inference

Running the transformer in torch.float16 halves memory consumption.

# model initialization with fp16
_EMBEDDING_MODEL = SentenceTransformer(
    'all-MiniLM-L6-v2',
    device='cuda',
    torch_dtype=torch.float16
)

Verification

After applying the changes, repeat the load test and monitor GPU usage.


$ watch -n1 nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap| Memory-Usage | GPU-Util  Compute M. |
|                               |                  |               |
|  0  Tesla T4            Off  | 00000000:00:1E.0 Off | 0                  |
| N/A   42C    P0    12W /  70W |  2100MiB / 15360MiB |  23%      Default |
+-------------------------------+----------------------+----------------------+

Memory usage stays below 2 GiB even when processing 50‑document batches at peak load. No OOM events appear in the logs for a 24‑hour production run.

Prevention

Batch size guardrails. Enforce max.poll.records and, if necessary, split incoming payloads in the consumer before handing them to LlamaIndex.
Singleton model pattern. Load heavy models once per process; avoid per‑request instantiation.
Memory hygiene. After each inference call, detach tensors and clear the CUDA cache. Consider using torch.no_grad() context if gradients are not needed.
Monitoring. Set alerts on nvidia-smi metrics (e.g., GPU Memory Used > 80%) and on worker OOM exit codes.
Capacity planning. Benchmark the maximum safe batch size for the chosen model and hardware; document the limits in the runbook.

FAQ

Why does the OOM only appear under production load and not in local testing? Local tests usually process a single document or a tiny batch. Production consumes up to the default max.poll.records (500) and may receive larger documents, exceeding GPU capacity.
Can I keep the per‑batch model loading and still avoid OOM? Not reliably. Each load creates a new GPU context; even with torch.cuda.empty_cache() the previous context may linger, leading to fragmentation.
Is mixed‑precision safe for all embedding models? Most transformer‑based sentence encoders support float16 without loss of accuracy, but verify with a downstream validation set before enabling in production.
What if my vector store also runs on the same GPU? Co‑locate the vector store on CPU or a separate GPU. Sharing GPU memory between the embedding model and the vector store can cause contention and unexpected OOMs.
How can I detect hidden tensor references that prevent memory release? Use torch.cuda.memory_summary() after gc.collect(). Persistent “reserved” blocks indicate leaked tensors; tools like tracemalloc can help locate the source.

Related Topic Hub: RAG Systems Troubleshooting Hub