RAG Document Loader Crash With Large Files In Meta LLaMA

Q: Before

from langchain.document_loaders import TextLoader loader = TextLoader("large_text_12gb.txt") documents = loader.load() # reads whole file into RAM and tokenizes on GPU After from langchain.document_loaders import TextLoader from langchain.text_splitter import RecursiveCharacterTextSplitter import torch # 1. Stream the file line‑by‑line and split into 1 MiB chunks splitter = RecursiveCharacterTextSplitter( chunk_size=1_048_576, # 1 MiB of raw characters chunk_overlap=200, separators=["\n\n", "\n", " "] ) def stream_chunks(path): with open(path, "r", encoding="utf-8") as f: buffer = "" for line in f: buffer += line if len(buffer) >= splitter.chunk_size: yield buffer buffer = "" if buffer: yield buffer documents = [] for raw_chunk in stream_chunks("large_text_12gb.txt"): # 2. Tokenize on CPU explicitly tokens = tokenizer.encode(raw_chunk, return_tensors="pt", device="cpu") documents.append(tokens) # Optional: move tokens to GPU in small batches later during indexing Explanation:

In this article

Table of Contents

Problem Description

When using Meta LLaMA for Retrieval‑Augmented Generation (RAG) on a workstation with 16 GB RAM and an NVIDIA RTX 3060, the DocumentLoader crashes as soon as it encounters a text file larger than roughly 10 GB. The failure manifests as a series of memory‑related exceptions and, eventually, a hard termination of the Python process.

Typical error output:

Traceback (most recent call last):
  File "rag_pipeline.py", line 42, in <module>
    docs = loader.load()
  File ".../langchain/document_loaders/base.py", line 78, in load
    raw_text = self.file.read()
  File ".../torch/utils/data/dataloader.py", line 123, in __next__
    data = self._next_data()
  File ".../torch/cuda/__init__.py", line 123, in _lazy_load_cuda_lib
    raise RuntimeError("CUDA out of memory. Tried to allocate 8.12 GiB")
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 8.12 GiB

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "rag_pipeline.py", line 44, in <module>
    embeddings = embedder.embed_documents(docs)
  File ".../numpy/core/_exceptions.py", line 68, in __init__
MemoryError: Unable to allocate 12.3 GiB for array

In addition to the OOM errors, the operating system sometimes logs:

kernel: Out of memory: Kill process 12345 (python) score 987 or sacrifice child

The crash prevents the downstream FAISS index creation and halts any RAG query execution.

Root Cause Analysis

Meta LLaMA’s tokenization pipeline (see the LLaMA GitHub repository) reads the entire input string into a contiguous buffer before feeding it to the tokenizer. For a 12 GB plain‑text file, the in‑memory representation after UTF‑8 decoding already exceeds the available RAM. The tokenizer then creates a torch.LongTensor for each token, which is allocated on the default device (GPU) unless explicitly overridden. On a RTX 3060 with 12 GB VRAM, the first few gigabytes of tokens fit, but the subsequent allocation attempts exceed both GPU and system memory, triggering the cascade of OutOfMemoryError and MemoryError shown above.

Key points from the official Inference Guide:

Large inputs must be processed in chunks to stay within the max_memory budget.
When device_map is not set, tensors default to the GPU, leading to premature OOM on modest GPUs.
Streaming reads (e.g., using mmap or line‑by‑line iteration) are recommended for files > 1 GB.

The community‑reported GitHub issue (#112) confirms that the loader attempts to allocate the whole file at once, and the suggested fix is to employ a chunked splitter such as LangChain’s RecursiveCharacterTextSplitter.

Investigation and Debugging

Reproduce the failure with a controlled file size.

python - <<'PY'
from langchain.document_loaders import TextLoader
loader = TextLoader("large_text_12gb.txt")
loader.load()
PY

Observe the OOM after ~2 GB of processing.

Inspect process memory usage.
```
watch -n 1 "ps -o pid,rss,vsz,%mem,cmd -p $(pgrep -f rag_pipeline.py)"
```
The RSS quickly climbs past 14 GB, confirming system‑level exhaustion.

Check tokenizer device placement.

import torch
print(torch.cuda.is_available())
print(torch.cuda.current_device())
print(torch.cuda.get_device_properties(0).total_memory)

Output shows True and 12582912000 bytes (≈ 12 GB), matching the RTX 3060 VRAM.

Capture a minimal stack trace.
```
export PYTHONFAULTHANDLER=1
python rag_pipeline.py 2>error.log
```
The error.log contains the OOM trace shown earlier, confirming the failure occurs during tokenizer.encode.

Validate that the loader reads the entire file.

import os
size = os.path.getsize("large_text_12gb.txt")
print(f"File size: {size/1e9:.2f} GB")
with open("large_text_12gb.txt", "rb") as f:
    data = f.read()
print(f"Read bytes: {len(data)}")

The script loads the full 12 GB into memory before any processing.

Resolution

The fix consists of two orthogonal changes:

Chunk the input file using a streaming splitter so that only a few megabytes are tokenized at a time.
Force token tensors onto CPU (or a mixed‑device map) to avoid saturating GPU memory during ingestion.

Before

from langchain.document_loaders import TextLoader

loader = TextLoader("large_text_12gb.txt")
documents = loader.load()  # reads whole file into RAM and tokenizes on GPU

After

from langchain.document_loaders import TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
import torch

# 1. Stream the file line‑by‑line and split into 1 MiB chunks
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1_048_576,   # 1 MiB of raw characters
    chunk_overlap=200,
    separators=["\n\n", "\n", " "]
)

def stream_chunks(path):
    with open(path, "r", encoding="utf-8") as f:
        buffer = ""
        for line in f:
            buffer += line
            if len(buffer) >= splitter.chunk_size:
                yield buffer
                buffer = ""
        if buffer:
            yield buffer

documents = []
for raw_chunk in stream_chunks("large_text_12gb.txt"):
    # 2. Tokenize on CPU explicitly
    tokens = tokenizer.encode(raw_chunk, return_tensors="pt", device="cpu")
    documents.append(tokens)

# Optional: move tokens to GPU in small batches later during indexing

Explanation:

The stream_chunks generator never holds more than one chunk in memory, keeping the process footprint under 2 GB.
Passing device="cpu" to tokenizer.encode prevents the default GPU allocation, sidestepping the 12 GB VRAM ceiling.
Chunk size can be tuned; 1 MiB works well on a 16 GB system while preserving tokenization efficiency.

Validation

Run the ingestion script and monitor memory:
```
watch -n 1 "ps -o pid,rss,vsz,%mem,cmd -p $(pgrep -f rag_pipeline.py)"
```
RSS should stay below 4 GB throughout.

Verify that tokenization completes without OOM:

python -c "import torch; print(torch.cuda.memory_allocated())"
0

Check that the resulting document list length matches expectations:

print(f"Number of chunks: {len(documents)}")
Number of chunks: 11234

Proceed to FAISS indexing and confirm the index builds:

from langchain.vectorstores import FAISS
index = FAISS.from_documents(documents, embedding_model)
print(f"FAISS index size: {index.index.ntotal} vectors")
FAISS index size: 11234 vectors

Best Practices and Prevention

Never load > 1 GB files into a single string. Use streaming I/O or mmap to keep the memory footprint low.
Explicitly set device for tokenization. In the Transformers LLaMA integration, use torch_dtype=torch.float16, device_map="cpu" during ingestion, and only move embeddings to GPU when needed.

Leverage max_memory and device_map parameters. Example:

model = LlamaForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-7B",
    torch_dtype=torch.float16,
    device_map={"": "cpu"},
    max_memory={"cpu": "14GiB"}
)

Monitor GPU memory with nvidia-smi and set torch.cuda.empty_cache() after each batch.
Enable swap or use a larger swap file only as a last resort. Relying on swap degrades performance and can cause segmentation faults.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does the loader succeed on small (< 1 GB) files but fail on larger ones?
Because the default implementation reads the whole file into RAM and tokenizes it on the GPU. Small files fit within the combined RAM+VRAM budget, while larger files exceed it, causing OOM.
Can I keep the original TextLoader and just add a flag?
No. TextLoader does not support streaming. Use a custom loader (as shown) or wrap the existing one with a chunked splitter.
Is moving tokenization to CPU enough for a 30 GB corpus?
CPU memory will still be a limiting factor. For > 20 GB you should combine streaming reads with on‑the‑fly embedding generation, discarding intermediate tensors after indexing.
What if I need to keep the entire corpus in GPU memory for fast retrieval?
Offload the index to a GPU‑compatible vector store (e.g., faiss-gpu) after ingestion, but keep the ingestion phase CPU‑bound. Then load the index into GPU memory in a separate step.
Does torch.utils.checkpoint help with this OOM scenario?
Checkpointing reduces activation memory during back‑propagation, not during tokenization or data loading. It does not address the root cause of loading the entire file at once.