Hugging Face Transformers Pipeline – RuntimeError OOM During Tokenization (Batch > 256)
Problem
During a nightly ETL job that streams millions of raw documents through a pipeline built on the PyTorch backend, the DataLoader aborts with a RuntimeError: CUDA out of memory as soon as the batch size exceeds 256 sequences. The failure occurs during the tokenization step, before any model inference is performed.
Traceback (most recent call last):
File "run_etl.py", line 112, in
for batch in loader:
File ".../torch/utils/data/dataloader.py", line 628, in __next__
data = self._next_data()
File ".../torch/utils/data/dataloader.py", line 671, in _next_data
return self._process_data(data)
File ".../torch/utils/data/dataloader.py", line 735, in _process_data
data = self.collate_fn(data)
File ".../transformers/pipelines/base.py", line 274, in __call__
inputs = self.tokenizer(batch, **self.tokenizer_kwargs)
File ".../transformers/tokenization_utils_base.py", line 1245, in __call__
return self._batch_encode_plus(
File ".../transformers/tokenization_utils_base.py", line 1412, in _batch_encode_plus
batch_outputs = self._encode_plus(
File ".../transformers/tokenization_utils_base.py", line 1585, in _encode_plus
inputs = self.prepare_for_model(
File ".../transformers/tokenization_utils_base.py", line 1722, in prepare_for_model
inputs = self._pad(
File ".../transformers/tokenization_utils_base.py", line 1915, in _pad
batch = torch.tensor(padded, dtype=self.dtype, device=self.device)
RuntimeError: CUDA out of memory. Tried to allocate 2.31 GiB (GPU 0; 11.00 GiB total capacity; 9.12 GiB already allocated; 0.31 GiB free; 9.23 GiB reserved in total)
The job runs fine with batch_size=256 or lower, but any increase triggers the same OOM error.
Root Cause
The tokenization pipeline creates a single large torch.Tensor that holds the padded token IDs for the entire batch. When the batch size grows, the tensor size grows linearly, but the padding strategy (padding='longest') forces every sequence to be padded to the length of the longest example in the batch. In the nightly data set, a small fraction of documents exceed 2048 tokens (the model’s maximum sequence length). Those outliers inflate the padded length for the whole batch, causing the tensor to exceed GPU memory.
Additional contributing factors:
- Using
return_tensors='pt'forces the tokenizer to allocate the tensor on the default device (GPU) immediately, before any memory‑saving tricks (e.g., moving to CPU for tokenization). - The pipeline’s default
tokenizer_kwargsdo not enabletruncation=True, so long sequences are not clipped to the model’smax_length. - The DataLoader’s
collate_fnis the pipeline’s own collator, which does not split the batch into sub‑batches for tokenization.
Debug & Investigation
Step‑by‑step diagnostics that revealed the cause:
- Inspect batch statistics before tokenization
def batch_stats(batch):
lengths = [len(txt.split()) for txt in batch]
return {
"max_len": max(lengths),
"mean_len": sum(lengths)/len(lengths),
"outliers": sum(1 for l in lengths if l > 2048)
}
print(batch_stats(sample_batch))
# {'max_len': 5789, 'mean_len': 312.4, 'outliers': 3}
The three outliers were responsible for the massive padding.
- Measure GPU memory consumption of the tokenization step
import torch, gc
torch.cuda.empty_cache()
gc.collect()
before = torch.cuda.memory_allocated()
_ = tokenizer(sample_batch, return_tensors='pt')
after = torch.cuda.memory_allocated()
print(f"Tokenization allocated { (after-before)/1e9:.2f } GiB")
# Tokenization allocated 2.31 GiB
- Reproduce the OOM with a minimal script
from transformers import pipeline, AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased").to("cuda")
pipe = pipeline("text-classification", model=model, tokenizer=tokenizer, device=0)
# Batch size 260 with a single long document
batch = ["short text"] * 259 + [" ".join(["word"]*6000)]
pipe(batch) # raises OOM
The failure reproduced confirmed that the outlier length, not the batch size alone, triggered the OOM.
Solution
Apply three complementary changes:
- Enable truncation at the model’s maximum length
# Before
pipe = pipeline(
"text-classification",
model=model,
tokenizer=tokenizer,
device=0,
batch_size=256
)
# After – enforce truncation and explicit max_length
pipe = pipeline(
"text-classification",
model=model,
tokenizer=tokenizer,
device=0,
batch_size=256,
tokenizer_kwargs={
"truncation": True,
"max_length": 512, # or model.config.max_position_embeddings
"padding": "longest",
"return_tensors": "pt"
}
)
Truncation caps every sequence at max_length, preventing a single outlier from inflating the padded tensor.
- Perform tokenization on CPU, then move tensors to GPU for inference
# Custom collate function
def cpu_tokenize(batch):
# Tokenize on CPU
inputs = tokenizer(
batch,
padding="longest",
truncation=True,
max_length=512,
return_tensors="pt"
)
# Move only the tensors needed for the model to GPU
inputs = {k: v.to("cuda") for k, v in inputs.items()}
return inputs
pipe = pipeline(
"text-classification",
model=model,
tokenizer=tokenizer,
device=0,
batch_size=256,
collate_fn=cpu_tokenize
)
CPU tokenization avoids allocating the large padded tensor on GPU, freeing memory for the model forward pass.
- Split oversized batches dynamically
def dynamic_batch_loader(dataset, max_batch=256, max_tokens=128000):
batch = []
token_budget = 0
for txt in dataset:
token_len = len(tokenizer.encode(txt, add_special_tokens=False))
if len(batch) + 1 > max_batch or token_budget + token_len > max_tokens:
yield batch
batch = []
token_budget = 0
batch.append(txt)
token_budget += token_len
if batch:
yield batch
for batch in dynamic_batch_loader(raw_texts):
results = pipe(batch)
# store results …
The loader respects both a hard batch‑size limit and a token‑budget limit (max_tokens), ensuring the padded tensor never exceeds a predictable size.
Verification
After applying the three changes, the ETL job completed without OOM errors. Validation steps:
- Run a full‑scale dry‑run with the production data set and monitor
nvidia-smi:
$ nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 5
# Example output (steady state)
memory.used [MiB], memory.total [MiB]
3120 MiB, 11264 MiB
Memory usage stayed below 4 GiB, well within the 11 GiB GPU.
- Check that no documents were silently truncated beyond the intended length:
truncated = sum(1 for txt in raw_texts if len(tokenizer.encode(txt)) > 512)
print(f"Documents truncated: {truncated}")
# Documents truncated: 1274 (expected, all >512 are trimmed)
import random, numpy as np
sample_idxs = random.sample(range(len(raw_texts)), 100)
old_preds = old_pipe([raw_texts[i] for i in sample_idxs])
new_preds = pipe([raw_texts[i] for i in sample_idxs])
diff = np.mean([o['label'] != n['label'] for o, n in zip(old_preds, new_preds)])
print(f"Label disagreement after fix: {diff:.2%}")
# Label disagreement after fix: 0.00%
Prevention & Operational Guardrails
To avoid recurrence:
- Enforce token‑length limits at data‑ingestion time. Reject or pre‑truncate records exceeding
max_lengthbefore they reach the pipeline. - Instrument token‑length histograms. Emit a Prometheus metric such as
tokenizer_batch_max_lengthand alert when it exceeds a configurable threshold. - Configure the pipeline to tokenize on CPU by default. Set
return_tensors='np'or'pt'withdevice='cpu'and move tensors manually. - Cap batch size and token budget. Use the dynamic loader pattern shown above, or rely on
DataLoader(collate_fn=...)that respects amax_tokensbudget. - Run a nightly memory‑profile job. Record
torch.cuda.memory_allocated()before and after tokenization; raise an alert if the delta exceeds2 GiB.
FAQ
- Why does the OOM happen only when the batch size is > 256?
Because with the defaultpadding='longest', a single long document forces the padded length for the whole batch. At 256 items the total token count stays under the GPU’s allocation budget; adding a few more items pushes the padded tensor over the limit. - Can I keep
return_tensors='pt'on GPU and still avoid OOM?
Yes, by enablingtruncation=Trueand setting an explicitmax_length. This guarantees the padded tensor size is bounded regardless of outliers. - Is CPU tokenization slower enough to be a problem?
In our nightly ETL the tokenization step accounts for ~15 % of total runtime. Moving it to CPU adds ~0.8 s per 256‑item batch, which is acceptable compared to the stability gain. - What if I need the full text (no truncation) for downstream processing?
Perform a two‑stage pipeline: first run a lightweight tokenizer withtruncation=Falseon CPU to store the raw token IDs (e.g., in a DB); then feed the truncated version to the model for inference. - How do I monitor token‑length distribution in production?
Expose a histogram metric (e.g.,tokenizer_input_length_bucket) using theprometheus_clientlibrary. Alert when the 99th percentile exceedsmax_length * 0.9.
Related Topic Hub: Model Serving Troubleshooting Hub