Hugging Face Transformers Vision Token Sequence Exceeds Max_position

Q: Problem: Vision token sequence exceeds max_position_embeddings in Hugging Face Transformers

When processing high‑resolution images with VisionEncoderDecoderModel, ViTModel or CLIPModel, the image is split into fixed‑size patches. Each patch becomes a token, and the total token count must be ≤ the model's max_position_embeddings (e.g., 197 for ViT‑B/16, 512 for CLIP). In production pipelines that ingest variable‑resolution images, the following failures have been observed:

Q: Why the error appears only for some batches Variable‑resolution data sources (scanned documents, satellite imagery, video frames) produce a mix of sizes. Dynamic padding logic that pads to the largest image in the batch can unintentionally raise the sequence length for the whole batch. Some pipelines rely on the default ImageProcessor behaviour, which does not automatically resize unless size is set explicitly. Investigation and Debugging 1. Reproduce the failure locally

from transformers import ViTFeatureExtractor, ViTModel import torch, PIL.Image as Image extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224") model = ViTModel.from_pretrained("google/vit-base-patch16-224") img = Image.open("high_res_1024x1024.jpg") inputs = extractor(images=img, return_tensors="pt") print("patches:", inputs["pixel_values"].shape) # -> [1, 3, 1024, 1024] print("num_tokens:", (inputs["pixel_values"].shape[-2] // extractor.size["height"]) * (inputs["pixel_values"].shape[-1] // extractor.size["width"]) + 1) Typical log output:

Q: 2. Inspect the model configuration

print(model.config.max_position_embeddings) # 197 print(model.config.patch_size) # 16 3. Check the processor's resize settings print(extractor.size) # {'height': 224, 'width': 224} print(extractor.crop_size) # {'height': 224, 'width': 224} If size is None, the processor will keep the original resolution, which is the direct cause of the overflow (see GitHub issue #12457).

Q: 4. Capture batch‑level statistics

def batch_stats(batch_images): lengths = [] for img in batch_images: h, w = img.size n_patches = (h // extractor.size["height"]) * (w // extractor.size["width"]) lengths.append(n_patches + 1) return max(lengths), sum(lengths) / len(lengths) max_len, avg_len = batch_stats(images) print(f"max token length in batch: {max_len}") When max_len exceeds model.config.max_position_embeddings, the batch will raise the error.

Q: Verification 1. Confirm token length stays within limits

inputs = safe_resize_batch(images, processor) assert inputs["pixel_values"].shape[-2] // processor.patch_size * \ inputs["pixel_values"].shape[-1] // processor.patch_size + 1 \

In this article

Table of Contents

Problem: Vision token sequence exceeds `max_position_embeddings` in Hugging Face Transformers

When processing high‑resolution images with VisionEncoderDecoderModel, ViTModel or CLIPModel, the image is split into fixed‑size patches. Each patch becomes a token, and the total token count must be ≤ the model’s max_position_embeddings (e.g., 197 for ViT‑B/16, 512 for CLIP). In production pipelines that ingest variable‑resolution images, the following failures have been observed:

ValueError: Token indices sequence length is longer than the max_position_embeddings (197) for vision model
RuntimeError (CUDA OOM): Trying to allocate 12.3 GiB for tensor with shape [batch, 4096, 768]
Silent truncation warnings such as Truncating vision tokens from 1024 to max_position_embeddings=197, leading to mis‑aligned image‑text pairs.

These errors manifest during the preprocessing step where ImageProcessor/ViTFeatureExtractor converts an image to a dense token sequence, causing the batch job to crash or silently lose visual information.

Root Cause Analysis

How token length is derived

The token count for a vision model is computed as:

num_patches = (image_height // patch_size) * (image_width // patch_size)
sequence_length = num_patches + 1   # +1 for the CLS token

Both image_height and image_width are taken from the raw input unless the ImageProcessor is instructed to resize or crop. The ViTConfig.max_position_embeddings field defines the maximum sequence_length the positional embedding matrix can index.

When an image’s dimensions are larger than the product patch_size × sqrt(max_position_embeddings‑1), num_patches exceeds the allowed limit. For ViT‑B/16 with patch_size=16 and max_position_embeddings=197, the maximum supported image side is:

max_side = floor(sqrt(197‑1)) * 16 = floor(14) * 16 = 224 px

Any image larger than 224 × 224 (or any aspect ratio that yields >196 patches) will trigger the overflow.

Why the error appears only for some batches

Variable‑resolution data sources (scanned documents, satellite imagery, video frames) produce a mix of sizes.
Dynamic padding logic that pads to the largest image in the batch can unintentionally raise the sequence length for the whole batch.
Some pipelines rely on the default ImageProcessor behaviour, which does not automatically resize unless size is set explicitly.

Investigation and Debugging

1. Reproduce the failure locally

from transformers import ViTFeatureExtractor, ViTModel
import torch, PIL.Image as Image

extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
model = ViTModel.from_pretrained("google/vit-base-patch16-224")

img = Image.open("high_res_1024x1024.jpg")
inputs = extractor(images=img, return_tensors="pt")
print("patches:", inputs["pixel_values"].shape)   # -> [1, 3, 1024, 1024]
print("num_tokens:", (inputs["pixel_values"].shape[-2] // extractor.size["height"]) *
      (inputs["pixel_values"].shape[-1] // extractor.size["width"]) + 1)

Typical log output:

ValueError: Token indices sequence length is longer than the max_position_embeddings (197) for vision model

2. Inspect the model configuration

print(model.config.max_position_embeddings)   # 197
print(model.config.patch_size)                # 16

3. Check the processor’s resize settings

print(extractor.size)   # {'height': 224, 'width': 224}
print(extractor.crop_size)   # {'height': 224, 'width': 224}

If size is None, the processor will keep the original resolution, which is the direct cause of the overflow (see GitHub issue #12457).

4. Capture batch‑level statistics

def batch_stats(batch_images):
    lengths = []
    for img in batch_images:
        h, w = img.size
        n_patches = (h // extractor.size["height"]) * (w // extractor.size["width"])
        lengths.append(n_patches + 1)
    return max(lengths), sum(lengths) / len(lengths)

max_len, avg_len = batch_stats(images)
print(f"max token length in batch: {max_len}")

When max_len exceeds model.config.max_position_embeddings, the batch will raise the error.

Solution: Align image size with model positional capacity

Approach A – Explicit resizing in the `ImageProcessor`

Force the processor to resize every image to the size that matches the model’s patch grid.

from transformers import ViTFeatureExtractor, VisionEncoderDecoderModel

# 1️⃣ Create a processor that always resizes to 224×224 (ViT‑B/16 default)
processor = ViTFeatureExtractor.from_pretrained(
    "google/vit-base-patch16-224",
    size={"height": 224, "width": 224},
    do_resize=True,
    resample=Image.BILINEAR,
)

model = VisionEncoderDecoderModel.from_pretrained("nlpconnect/vit-gpt2-image-captioning")

def preprocess_batch(pil_images):
    # Returns a dict with pixel_values already resized
    return processor(images=pil_images, return_tensors="pt")

Why it works: The processor now guarantees that num_patches = (224/16)² = 196, so sequence_length = 197, which fits the positional embedding table.

Approach B – Adjust `patch_size` and `max_position_embeddings` for large images

If preserving higher resolution is required (e.g., satellite imagery), create a custom ViT configuration with a larger patch grid.

from transformers import ViTConfig, ViTModel, ViTFeatureExtractor

# 2️⃣ Define a model that can handle 1024×1024 images with 16‑pixel patches
custom_cfg = ViTConfig(
    image_size=1024,
    patch_size=16,
    num_hidden_layers=12,
    hidden_size=768,
    num_attention_heads=12,
    max_position_embeddings=(1024 // 16) ** 2 + 1,  # 4097
)

# Re‑initialize the model (weights can be loaded from a checkpoint if compatible)
model = ViTModel(custom_cfg)

processor = ViTFeatureExtractor(
    size={"height": 1024, "width": 1024},
    do_resize=False,          # we already have the correct size
    patch_size=16,
)

After training or fine‑tuning the new model, the positional embedding matrix will accommodate the longer sequence. This approach mirrors the community discussion in GitHub issue #13802.

Approach C – Dynamic per‑batch resizing with a safety guard

When image dimensions are unpredictable, compute the maximum safe size per batch and resize accordingly.

def safe_resize_batch(pil_images, processor):
    # Determine the largest side that stays within max_position_embeddings
    max_seq = processor.max_position_embeddings - 1   # exclude CLS token
    patch = processor.patch_size
    max_side = int((max_seq ** 0.5) * patch)

    resized = []
    for img in pil_images:
        w, h = img.size
        scale = min(max_side / w, max_side / h, 1.0)   # never upscale
        new_w, new_h = int(w * scale), int(h * scale)
        resized.append(img.resize((new_w, new_h), Image.BILINEAR))
    return processor(images=resized, return_tensors="pt")

This guard prevents accidental OOM while preserving as much resolution as possible.

Verification

1. Confirm token length stays within limits

inputs = safe_resize_batch(images, processor)
assert inputs["pixel_values"].shape[-2] // processor.patch_size * \
       inputs["pixel_values"].shape[-1] // processor.patch_size + 1 \
       <= processor.max_position_embeddings
print("Verification passed: token length within limit")

2. Run a forward pass and monitor memory

with torch.no_grad():
    outputs = model.vision_model(pixel_values=inputs["pixel_values"])
print("output shape:", outputs.last_hidden_state.shape)

Expected output (ViT‑B/16): torch.Size([batch, 197, 768]) – no CUDA OOM.

3. End‑to‑end functional test

Generate a caption (or any downstream task) and compare against a baseline to ensure that resizing has not introduced unacceptable quality loss.

Operational Experience & Lessons Learned

Misleading symptom: The exception often appears only when a single outlier image is present in a batch, leading engineers to suspect a GPU driver issue rather than token overflow.
Common incorrect assumption: “The processor automatically resizes to the model’s expected size.” In reality, do_resize defaults to True only when size is provided; otherwise the original resolution is kept (GitHub issue #12457).
Production edge case: Non‑square images with aspect ratios far from 1:1 generate extra padding tokens after the center_crop step, pushing the token count over the limit. Explicitly setting crop_size to a square that matches size eliminates the padding.
Lesson: Encode the maximum safe image dimension in a configuration file and enforce it early in the ETL pipeline; this prevents silent truncation that degrades downstream accuracy (see document‑understanding incident).

Best Practices & Prevention

Practice	Implementation Detail
Enforce a canonical image size	Set `size={'height': 224, 'width': 224}` (or the size matching your model) in the `ImageProcessor` and keep `do_resize=True`.
Validate token length before model call	Assert `seq_len <= config.max_position_embeddings` after preprocessing; raise a clear error if violated.
Monitor token‑length metrics	Expose `vision_token_length` as a Prometheus gauge; alert if > 0.9 × max_position_embeddings.
Dynamic resizing guard	Implement the `safe_resize_batch` function in the data loader; log the original vs. resized dimensions.
Version‑consistent configs	When swapping models (e.g., ViT‑L/14), update both `patch_size` and `max_position_embeddings` in the processor configuration.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why does the error only appear for some images? The token count depends on the image’s pixel dimensions divided by the patch size. Images larger than the model’s supported grid produce more patches, exceeding max_position_embeddings. Smaller images stay within the limit.
Can I increase max_position_embeddings without retraining? No. The positional embedding matrix is learned for a fixed sequence length. Changing the value without re‑initialising or fine‑tuning the model leads to mismatched dimensions and runtime errors.
Is cropping preferable to resizing? Cropping preserves the original resolution of the retained region but discards content. Resizing keeps the whole image at the cost of down‑sampling. Choose based on task requirements; for OCR, a centered crop may lose critical text.
How do I detect silent truncation? Enable the processor’s verbose=True or inspect the warning: Warning: Truncating vision tokens from N to max_position_embeddings=197. Logging this warning and treating it as an error prevents unnoticed information loss.
What if I need >197 tokens for a specific use‑case? Create a custom ViT configuration with a larger max_position_embeddings and train/fine‑tune from a checkpoint that matches the new architecture, or switch to a model designed for higher resolution (e.g., ViT‑L/14 with 384‑pixel inputs).

Problem: Vision token sequence exceeds max_position_embeddings in Hugging Face Transformers