Bumblebee the transformer is pictured in detail.

Hugging Face Transformers text image embedding mismatch

by

Problem – Misaligned Text and Image Embeddings in a Multimodal Hugging Face Model

When training a vision‑language model (e.g., CLIP, ViLT, BLIP) on a local workstation inside a Jupyter notebook, the downstream similarity or classification scores become meaningless after the first few steps. Typical symptoms include:

  • Cosine similarity between matching text‑image pairs dropping to ~0.0 after the first epoch.
  • Runtime errors such as ValueError: The number of tokens (32) does not match the number of image patches (196).
  • Shape mismatches when concatenating features: RuntimeError: size mismatch, m1: [batch, 512] vs m2: [batch, 768].
  • Log excerpts showing different dtype or batch_size for the two modalities.

These issues manifest despite using the same model instance for both get_text_features and get_image_features, leading to inaccurate downstream predictions.

Root Cause – Inconsistent Pre‑processing and State Management

Multimodal Transformers rely on two tightly coupled preprocessing pipelines:

  1. Tokenizer for text – produces input_ids, attention_mask, and a fixed embedding dimension defined by the model’s text projection head.
  2. Feature extractor for images – produces pixel_values (or image_embeddings) whose shape must match the image projection head.

When the two pipelines diverge, the model receives tensors of incompatible shape or dtype, causing the projection heads to output vectors of different dimensionality. The following real‑world incidents illustrate the same pattern:

  • In a data‑science team, the image feature extractor was re‑initialized on each notebook cell execution, unintentionally changing its internal scaling parameters and breaking alignment after the first epoch.
  • A researcher mixed AutoTokenizer from the CLIP checkpoint with a ViTFeatureExtractor from a different checkpoint, resulting in image_embeddings of size 768 while text_embeddings remained 512.
  • A Kaggle kernel omitted feature_extractor(image, return_tensors='pt'), feeding raw PIL objects to the model and triggering an AssertionError: Expected feature extractor output shape [batch, channels, height, width] but got [batch, channels].

Official documentation emphasizes that the tokenizer and feature extractor must be instantiated from the *same* pretrained checkpoint and that their padding and return_tensors settings must be synchronized (CLIP docs, feature‑extractor guide).

Investigation and Debugging Steps

1. Verify Tokenizer and Feature Extractor Pairing

from transformers import AutoTokenizer, AutoFeatureExtractor, CLIPModel

model_name = "openai/clip-vit-base-patch32"
tokenizer = AutoTokenizer.from_pretrained(model_name)
feature_extractor = AutoFeatureExtractor.from_pretrained(model_name)

If tokenizer and feature_extractor come from different checkpoints, the output dimensions will differ.

2. Check Tensor Shapes Before Model Call

import torch

texts = ["a photo of a cat", "a photo of a dog"]
images = [Image.open(p) for p in ["cat.jpg", "dog.jpg"]]

# Tokenizer
text_inputs = tokenizer(texts, padding="max_length", max_length=77,
                        truncation=True, return_tensors="pt")
print(text_inputs["input_ids"].shape)   # Expected: torch.Size([2, 77])

# Feature extractor
image_inputs = feature_extractor(images, return_tensors="pt")
print(image_inputs["pixel_values"].shape)  # Expected: torch.Size([2, 3, 224, 224])

Typical mismatch log:

ValueError: The number of tokens (77) does not match the number of image patches (196).

3. Ensure Consistent return_tensors and dtype

# Wrong: mixed dtype
text_inputs = tokenizer(texts, return_tensors="pt")          # default torch.float32
image_inputs = feature_extractor(images, return_tensors="np")  # returns numpy

# Correct: both as PyTorch tensors, same dtype
text_inputs = tokenizer(texts, return_tensors="pt")
image_inputs = feature_extractor(images, return_tensors="pt")

4. Detect Unintended Re‑initialization

In a notebook, each cell execution that calls AutoFeatureExtractor.from_pretrained creates a fresh instance. Verify that the same object is reused:

# Cell 1
feature_extractor = AutoFeatureExtractor.from_pretrained(model_name)

# Cell 2 (do NOT re‑run the line above)
image_inputs = feature_extractor(images, return_tensors="pt")

5. Examine Model Projection Heads

print(model.visual_projection.out_features)   # e.g., 512
print(model.text_projection.out_features)     # e.g., 512

If these values differ (e.g., 768 vs 512), the checkpoint mismatch is confirmed.

Solution – Align Pre‑processing Pipelines and Stabilize State

Step‑by‑step Fix

  1. Instantiate a single Processor when available. The CLIPProcessor bundles tokenizer and feature extractor with synchronized settings.
  2. Pin the same checkpoint for both modalities. Use the model name consistently across AutoTokenizer, AutoFeatureExtractor, and CLIPModel.
  3. Force identical padding and tensor return types. Set padding="max_length" and return_tensors="pt" for both calls.
  4. Cache the processor objects at notebook start. Avoid re‑instantiation in later cells.

Before (Problematic Code)

# Separate tokenizer and feature extractor from different checkpoints
tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32")
feature_extractor = AutoFeatureExtractor.from_pretrained("google/vit-base-patch16-224")

# Inconsistent padding
text_inputs = tokenizer(texts, padding=True, return_tensors="pt")
image_inputs = feature_extractor(images, return_tensors="pt")

After (Corrected Code)

# Use the bundled processor – guarantees alignment
from transformers import CLIPProcessor, CLIPModel

model_name = "openai/clip-vit-base-patch32"
processor = CLIPProcessor.from_pretrained(model_name)
model = CLIPModel.from_pretrained(model_name)

# Single call returns a unified dict
inputs = processor(text=texts, images=images,
                   padding="max_length", max_length=77,
                   return_tensors="pt")

# Forward pass – embeddings are now aligned
outputs = model(**inputs)
text_embeds = outputs.text_embeds          # shape: [batch, 512]
image_embeds = outputs.image_embeds        # shape: [batch, 512]

Why This Fix Works

The CLIPProcessor internally creates a tokenizer and a feature extractor from the same checkpoint, applies identical padding strategies, and returns tensors with matching dtype. Consequently, both text_embeds and image_embeds are projected to the same dimensionality (512 for the base CLIP model), eliminating the size‑mismatch errors and ensuring that cosine similarity reflects true semantic alignment.

Verification – Confirming Correct Alignment

1. Shape Assertions

assert text_embeds.shape == image_embeds.shape, \
       f"Shape mismatch: {text_embeds.shape} vs {image_embeds.shape}"
print("Embedding shapes aligned:", text_embeds.shape)

Expected output:

Embedding shapes aligned: torch.Size([2, 512])

2. Cosine Similarity Check

from torch.nn.functional import cosine_similarity

cos_sim = cosine_similarity(text_embeds, image_embeds, dim=-1)
print("Cosine similarity per pair:", cos_sim)

For matching pairs the similarity should be > 0.7 after a few training steps; mismatched pairs should be near 0.

3. Log Inspection

2026-06-30 14:02:11,023 INFO  processor - Padding strategy: max_length (77 tokens)
2026-06-30 14:02:11,025 INFO  processor - Image size: (224, 224), return_tensors: pt
2026-06-30 14:02:12,110 DEBUG model - text_embeds shape: torch.Size([32, 512])
2026-06-30 14:02:12,112 DEBUG model - image_embeds shape: torch.Size([32, 512])

Prevention – Guardrails for Future Development

  • Always use the bundled Processor (e.g., CLIPProcessor, ViLTProcessor) when available.
  • Freeze the preprocessing objects at notebook start. Store them in a dedicated cell and reference them elsewhere.
  • Add unit tests that assert matching output dimensions. Example:
def test_embedding_alignment():
    inputs = processor(text=["test"], images=[Image.new("RGB", (224,224))],
                       return_tensors="pt")
    out = model(**inputs)
    assert out.text_embeds.shape[-1] == out.image_embeds.shape[-1]
  • Monitor embedding similarity during training. Set up a simple metric that logs average cosine similarity for a validation batch each epoch.
  • Pin library versions. Incompatibilities between transformers and datasets can change default padding behavior. Use a requirements.txt with exact versions (e.g., transformers==4.40.0).

    • FAQ – Common Follow‑up Questions

    1. Why does the error only appear after the first epoch?
      Because the notebook re‑executed the cell that creates a new AutoFeatureExtractor, changing its internal weight scaling. The first epoch uses the original instance, but subsequent epochs see a different feature extractor, causing the projection heads to diverge.
    2. Can I still use separate tokenizer and feature extractor?
      Yes, but you must instantiate both from the same checkpoint and enforce identical padding and return_tensors settings. Using the processor is less error‑prone.
    3. What if my custom dataset returns images as NumPy arrays?
      Pass them through the feature extractor: feature_extractor(np_array, return_tensors="pt"). Directly feeding raw arrays bypasses the required normalization and reshaping, leading to shape assertions.
    4. How do I debug a “size mismatch, m1: [batch, 512] vs m2: [batch, 768]” error?
      Check that the tokenizer and feature extractor come from the same checkpoint. A mismatch usually means you mixed a CLIP tokenizer (512‑dim) with a ViT feature extractor (768‑dim).
    5. Is there a way to automatically detect misaligned embeddings?
      Add a sanity‑check after each forward pass that compares text_embeds.shape and image_embeds.shape. Raise an exception or log a warning if they differ.

    Related Topic Hub: Model Serving Troubleshooting Hub