Hugging Face pipeline RuntimeError OOM during tokenization batch >256

Hugging Face Transformers Pipeline – RuntimeError OOM During Tokenization (Batch > 256)

Problem

During a nightly ETL job that streams millions of raw documents through a pipeline built on the PyTorch backend, the DataLoader aborts with a RuntimeError: CUDA out of memory as soon as the batch size exceeds 256 sequences. The failure occurs during the tokenization step, before any model inference is performed.


Traceback (most recent call last):
  File "run_etl.py", line 112, in 
    for batch in loader:
  File ".../torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File ".../torch/utils/data/dataloader.py", line 671, in _next_data
    return self._process_data(data)
  File ".../torch/utils/data/dataloader.py", line 735, in _process_data
    data = self.collate_fn(data)
  File ".../transformers/pipelines/base.py", line 274, in __call__
    inputs = self.tokenizer(batch, **self.tokenizer_kwargs)
  File ".../transformers/tokenization_utils_base.py", line 1245, in __call__
    return self._batch_encode_plus(
  File ".../transformers/tokenization_utils_base.py", line 1412, in _batch_encode_plus
    batch_outputs = self._encode_plus(
  File ".../transformers/tokenization_utils_base.py", line 1585, in _encode_plus
    inputs = self.prepare_for_model(
  File ".../transformers/tokenization_utils_base.py", line 1722, in prepare_for_model
    inputs = self._pad(
  File ".../transformers/tokenization_utils_base.py", line 1915, in _pad
    batch = torch.tensor(padded, dtype=self.dtype, device=self.device)
RuntimeError: CUDA out of memory. Tried to allocate 2.31 GiB (GPU 0; 11.00 GiB total capacity; 9.12 GiB already allocated; 0.31 GiB free; 9.23 GiB reserved in total)

The job runs fine with batch_size=256 or lower, but any increase triggers the same OOM error.

Root Cause

The tokenization pipeline creates a single large torch.Tensor that holds the padded token IDs for the entire batch. When the batch size grows, the tensor size grows linearly, but the padding strategy (padding='longest') forces every sequence to be padded to the length of the longest example in the batch. In the nightly data set, a small fraction of documents exceed 2048 tokens (the model’s maximum sequence length). Those outliers inflate the padded length for the whole batch, causing the tensor to exceed GPU memory.

Additional contributing factors:

  • Using return_tensors='pt' forces the tokenizer to allocate the tensor on the default device (GPU) immediately, before any memory‑saving tricks (e.g., moving to CPU for tokenization).
  • The pipeline’s default tokenizer_kwargs do not enable truncation=True, so long sequences are not clipped to the model’s max_length.
  • The DataLoader’s collate_fn is the pipeline’s own collator, which does not split the batch into sub‑batches for tokenization.

Debug & Investigation

Step‑by‑step diagnostics that revealed the cause:

  1. Inspect batch statistics before tokenization
def batch_stats(batch):
    lengths = [len(txt.split()) for txt in batch]
    return {
        "max_len": max(lengths),
        "mean_len": sum(lengths)/len(lengths),
        "outliers": sum(1 for l in lengths if l > 2048)
    }

print(batch_stats(sample_batch))
# {'max_len': 5789, 'mean_len': 312.4, 'outliers': 3}

The three outliers were responsible for the massive padding.

  1. Measure GPU memory consumption of the tokenization step
import torch, gc
torch.cuda.empty_cache()
gc.collect()
before = torch.cuda.memory_allocated()
_ = tokenizer(sample_batch, return_tensors='pt')
after = torch.cuda.memory_allocated()
print(f"Tokenization allocated { (after-before)/1e9:.2f } GiB")
# Tokenization allocated 2.31 GiB
  1. Reproduce the OOM with a minimal script
from transformers import pipeline, AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased").to("cuda")
pipe = pipeline("text-classification", model=model, tokenizer=tokenizer, device=0)

# Batch size 260 with a single long document
batch = ["short text"] * 259 + [" ".join(["word"]*6000)]
pipe(batch)  # raises OOM

The failure reproduced confirmed that the outlier length, not the batch size alone, triggered the OOM.

Solution

Apply three complementary changes:

  1. Enable truncation at the model’s maximum length
# Before
pipe = pipeline(
    "text-classification",
    model=model,
    tokenizer=tokenizer,
    device=0,
    batch_size=256
)

# After – enforce truncation and explicit max_length
pipe = pipeline(
    "text-classification",
    model=model,
    tokenizer=tokenizer,
    device=0,
    batch_size=256,
    tokenizer_kwargs={
        "truncation": True,
        "max_length": 512,          # or model.config.max_position_embeddings
        "padding": "longest",
        "return_tensors": "pt"
    }
)

Truncation caps every sequence at max_length, preventing a single outlier from inflating the padded tensor.

  1. Perform tokenization on CPU, then move tensors to GPU for inference
# Custom collate function
def cpu_tokenize(batch):
    # Tokenize on CPU
    inputs = tokenizer(
        batch,
        padding="longest",
        truncation=True,
        max_length=512,
        return_tensors="pt"
    )
    # Move only the tensors needed for the model to GPU
    inputs = {k: v.to("cuda") for k, v in inputs.items()}
    return inputs

pipe = pipeline(
    "text-classification",
    model=model,
    tokenizer=tokenizer,
    device=0,
    batch_size=256,
    collate_fn=cpu_tokenize
)

CPU tokenization avoids allocating the large padded tensor on GPU, freeing memory for the model forward pass.

  1. Split oversized batches dynamically
def dynamic_batch_loader(dataset, max_batch=256, max_tokens=128000):
    batch = []
    token_budget = 0
    for txt in dataset:
        token_len = len(tokenizer.encode(txt, add_special_tokens=False))
        if len(batch) + 1 > max_batch or token_budget + token_len > max_tokens:
            yield batch
            batch = []
            token_budget = 0
        batch.append(txt)
        token_budget += token_len
    if batch:
        yield batch

for batch in dynamic_batch_loader(raw_texts):
    results = pipe(batch)
    # store results …

The loader respects both a hard batch‑size limit and a token‑budget limit (max_tokens), ensuring the padded tensor never exceeds a predictable size.

Verification

After applying the three changes, the ETL job completed without OOM errors. Validation steps:

  • Run a full‑scale dry‑run with the production data set and monitor nvidia-smi:
$ nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 5
# Example output (steady state)
memory.used [MiB], memory.total [MiB]
  3120 MiB, 11264 MiB

Memory usage stayed below 4 GiB, well within the 11 GiB GPU.

  • Check that no documents were silently truncated beyond the intended length:
truncated = sum(1 for txt in raw_texts if len(tokenizer.encode(txt)) > 512)
print(f"Documents truncated: {truncated}")
# Documents truncated: 1274  (expected, all >512 are trimmed)
  • Confirm functional correctness by comparing a random sample of predictions before and after the fix (differences only due to truncation, not model mis‑behavior).
  • import random, numpy as np
    sample_idxs = random.sample(range(len(raw_texts)), 100)
    old_preds = old_pipe([raw_texts[i] for i in sample_idxs])
    new_preds = pipe([raw_texts[i] for i in sample_idxs])
    diff = np.mean([o['label'] != n['label'] for o, n in zip(old_preds, new_preds)])
    print(f"Label disagreement after fix: {diff:.2%}")
    # Label disagreement after fix: 0.00%
    

    Prevention & Operational Guardrails

    To avoid recurrence:

    • Enforce token‑length limits at data‑ingestion time. Reject or pre‑truncate records exceeding max_length before they reach the pipeline.
    • Instrument token‑length histograms. Emit a Prometheus metric such as tokenizer_batch_max_length and alert when it exceeds a configurable threshold.
    • Configure the pipeline to tokenize on CPU by default. Set return_tensors='np' or 'pt' with device='cpu' and move tensors manually.
    • Cap batch size and token budget. Use the dynamic loader pattern shown above, or rely on DataLoader(collate_fn=...) that respects a max_tokens budget.
    • Run a nightly memory‑profile job. Record torch.cuda.memory_allocated() before and after tokenization; raise an alert if the delta exceeds 2 GiB.

    FAQ

    1. Why does the OOM happen only when the batch size is > 256?
      Because with the default padding='longest', a single long document forces the padded length for the whole batch. At 256 items the total token count stays under the GPU’s allocation budget; adding a few more items pushes the padded tensor over the limit.
    2. Can I keep return_tensors='pt' on GPU and still avoid OOM?
      Yes, by enabling truncation=True and setting an explicit max_length. This guarantees the padded tensor size is bounded regardless of outliers.
    3. Is CPU tokenization slower enough to be a problem?
      In our nightly ETL the tokenization step accounts for ~15 % of total runtime. Moving it to CPU adds ~0.8 s per 256‑item batch, which is acceptable compared to the stability gain.
    4. What if I need the full text (no truncation) for downstream processing?
      Perform a two‑stage pipeline: first run a lightweight tokenizer with truncation=False on CPU to store the raw token IDs (e.g., in a DB); then feed the truncated version to the model for inference.
    5. How do I monitor token‑length distribution in production?
      Expose a histogram metric (e.g., tokenizer_input_length_bucket) using the prometheus_client library. Alert when the 99th percentile exceeds max_length * 0.9.

    Related Topic Hub: Model Serving Troubleshooting Hub