a white square button with a blue v on it

vLLM multimodal inference mismatch between image and text

by

Problem Description

During multimodal inference on an on‑premises server, a custom‑trained vision‑language model integrated with vLLM produces captions and answers that do not correspond to the supplied image. The generated text is either generic (e.g., “a person standing in a room”) or completely unrelated to the visual content. The failure is reproducible across different images and manifests in both image‑captioning and vision‑based question‑answering endpoints.

Typical symptoms observed in logs and API responses:

  • API response contains only the textual part of the prompt, ignoring the image.
  • Log entry: vLLM.InferenceError: modality mismatch
  • Log entry: RuntimeError: size mismatch, expected 768 but got 1024
  • Log entry: ValueError: Unexpected number of image channels (1) when grayscale images are supplied.
  • Performance metrics show normal token generation latency, indicating the text decoder runs but receives empty or corrupted visual embeddings.

Root Cause Analysis

The mismatch originates from three tightly coupled components:

  1. Image preprocessing pipeline – vLLM expects vision encoders (e.g., CLIP, BLIP) to receive images normalized with CLIP’s mean/std ([0.48145466, 0.4578275, 0.40821073] and [0.26862954, 0.26130258, 0.27577711]) and resized to 224×224 (or model‑specific size). Using ImageNet normalization (mean [0.485, 0.456, 0.406], std [0.229, 0.224, 0.225]) shifts the distribution of pixel values, producing embeddings that do not align with the text decoder’s learned space. This is documented in the vLLM Multimodal Inference Guide and matches the real incident where “image preprocessing used ImageNet mean/std while vLLM’s default expects CLIP normalization” (Evidence Package).
  2. Vision encoder output dimensionality – The custom model’s vision head was exported to TorchScript with float16 tensors, while the text decoder expects float32 embeddings of size 768. The mismatch triggers the runtime error “expected 768 but got 1024”, indicating the encoder’s output dimension diverges from the decoder’s projection layer (Evidence Package).
  3. Request payload construction – Omitting the required image_id field or providing an unsupported image format forces vLLM to fall back to a cached text‑only prompt, yielding answers that ignore the visual input (Evidence Package).

Collectively, these issues break the alignment between visual embeddings and textual token embeddings, leading to the observed irrelevant outputs.

Investigation and Debugging

The following step‑by‑step process reproduces the diagnosis on a typical deployment.

1. Verify vLLM version and CUDA compatibility

python -c "import vllm, torch; print(vllm.__version__, torch.version.cuda)"
# Expected output, e.g.: 0.3.1 12.1

If the CUDA runtime reported by torch.version.cuda does not match the version used to build vLLM (see the On‑Premises Setup guide), rebuild vLLM with the correct CUDA toolkit.

2. Inspect the request payload

curl -X POST http://localhost:8000/v1/completions \\
  -H "Content-Type: application/json" \\
  -d '{
        "model": "custom-blip",
        "prompt": " Describe the scene.",
        "max_tokens": 64
      }'

Ensure the JSON contains a valid image_id that maps to a pre‑loaded image tensor. Missing this field triggers the “modality mismatch” error.

3. Check image preprocessing parameters

python - <<'PY'
from vllm.multimodal import MultimodalProcessor
proc = MultimodalProcessor.from_pretrained("custom-blip")
print("Mean:", proc.image_processor.mean)
print("Std :", proc.image_processor.std)
PY

Expected output (CLIP normalization):

Mean: [0.48145466, 0.4578275, 0.40821073]
Std : [0.26862954, 0.26130258, 0.27577711]

If the printed values differ, the processor was instantiated with the wrong configuration.

4. Validate vision encoder output shape and dtype

python - <<'PY'
import torch
from vllm.multimodal import VisionEncoder
encoder = VisionEncoder.from_pretrained("custom-blip-vision")
dummy = torch.randn(1, 3, 224, 224).to("cuda")
out = encoder(dummy)
print(out.shape, out.dtype)
PY

Expected: torch.Size([1, 768]) torch.float32. A shape of [1, 1024] or dtype float16 indicates a projection mismatch.

5. Examine server logs for preprocessing failures

journalctl -u vllm.service -f | grep -i "image"
# Sample log snippet:
2026-06-21 14:03:12,845 WARN  vllm.multimodal.processor - Failed to decode image tensor: unsupported image format or corrupted file
2026-06-21 14:03:12,847 ERROR vllm.inference - vLLM.InferenceError: modality mismatch

6. Reproduce the issue with a minimal script

python - <<'PY'
import requests, base64, json, pathlib
img_path = pathlib.Path("sample.jpg").read_bytes()
payload = {
    "model": "custom-blip",
    "prompt": f" What is in the picture?",
    "max_tokens": 32
}
r = requests.post("http://localhost:8000/v1/completions", json=payload)
print(r.json())
PY

If the response contains only the textual prompt without any visual grounding, the problem likely lies in preprocessing or encoder‑decoder alignment.

Resolution

Apply the following fixes in the order presented. Each block shows the configuration before the change and the corrected version after.

1. Align image normalization with CLIP defaults

Before (incorrect ImageNet normalization):

from torchvision import transforms
transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225]),
])

After (CLIP normalization as required by vLLM):

from torchvision import transforms
transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.48145466, 0.4578275, 0.40821073],
                         std=[0.26862954, 0.26130258, 0.27577711]),
])

Re‑run the preprocessing pipeline and reload the model.

2. Ensure vision encoder output dimension matches text decoder

If the encoder was exported with an incompatible projection layer, re‑export using the same dtype and projection size as the decoder.

Re‑export script (TorchScript, float32, 768‑dim output):

import torch
from model import VisionEncoder

encoder = VisionEncoder()
encoder.eval()
example = torch.randn(1, 3, 224, 224)
traced = torch.jit.trace(encoder, example)
traced.save("vision_encoder_fp32.pt")

Replace the old checkpoint and restart the vLLM engine.

3. Correct request payload construction

Include the image_id field and ensure the image is base64‑encoded.

# Incorrect payload (missing image_id)
{
  "model": "custom-blip",
  "prompt": "Describe the image."
}
# Correct payload
{
  "model": "custom-blip",
  "prompt": " Describe the image."
}

4. Rebuild vLLM for the target CUDA version

If the environment uses CUDA 12.1 while the binary was compiled for CUDA 11.8, rebuild:

# From the vLLM source directory
export CUDA_HOME=/usr/local/cuda-12.1
pip install -e .  # rebuilds native extensions against CUDA 12.1

Validation

After applying the fixes, perform the following checks:

  1. Functional test – Send a known image and verify the caption matches the content.
    2. curl -X POST http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"custom-blip","prompt":" Caption this image.","max_tokens":32}'

    Expected output (example):

    {
  "id":"cmpl-...",
  "choices":[{"text":"A red sports car parked in front of a modern building."}]
}
  2. Log inspection – No “modality mismatch” or size‑mismatch errors should appear.
  3. Tensor shape verification – Run the diagnostic script from the investigation section; it should print torch.Size([1, 768]) torch.float32.
  4. Performance sanity check – Latency should remain within the baseline (< 200 ms for 32 tokens on the test hardware).

Prevention and Best Practices

  • Standardize image preprocessing across all pipelines. Store the mean/std values in a configuration file and reference them in both training and inference scripts.
  • Pin the vision encoder and text decoder to the same dtype and projection dimension. Include a CI test that asserts encoder.output_dim == decoder.hidden_size.
  • Validate request payloads with a schema validator (e.g., jsonschema) before forwarding to vLLM to catch missing image_id fields early.
  • Maintain a reproducible build environment: use the same CUDA/cuDNN versions for training, exporting, and serving. Record these versions in requirements.txt and container tags.
  • Enable vLLM’s built‑in monitoring (see the Multimodal Inference Guide) to emit metrics for vision_encoder_latency_ms and modality_mismatch_errors. Alert on non‑zero counts.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

  1. Why does the model generate correct text when I disable the image input?
    Without an image, vLLM falls back to a text‑only prompt, so the decoder produces generic completions based solely on the language model. The visual pathway is simply bypassed.
  2. Can I use ImageNet normalization if my model was trained with it?
    Yes, but you must also train the text decoder on embeddings generated with the same normalization. Mixing CLIP‑normalized encoders with ImageNet‑normalized inputs will cause the alignment error described above.
  3. What does the error “size mismatch, expected 768 but got 1024” indicate?
    The vision encoder’s final projection layer outputs a vector of length 1024, while the text decoder’s cross‑attention expects a 768‑dimensional embedding. Re‑export the encoder with the correct projection size or add a linear adaptor to reshape the vector.
  4. My images are grayscale; how should I handle them?
    Convert them to three‑channel RGB before preprocessing. vLLM raises ValueError: Unexpected number of image channels (1) for single‑channel inputs.
  5. Is it safe to run the vision encoder in float16 to save memory?
    Only if the downstream text decoder also operates in float16 and the model was fine‑tuned with mixed‑precision. Mixing float16 encoder outputs with a float32 decoder leads to degraded embeddings and mismatched dimensions.