Multimodal Alignment Error In Prometheus After Model Update

Q: 1. Verify endpoint response and dimensionality

curl -X POST https://prometheus.example.com/v1/evaluate \ -H "Content-Type: application/json" \ -d '{ "model_version": "clip-v2.4.1", "inputs": [ {"type":"text","data":"A dog playing in the park."}, {"type":"image","data":"s3://bucket/image1.jpg"} ] }' | jq . Sample output:

Q: 2. Inspect configuration flags

cat /etc/prometheus/evaluation.yaml # Current configuration alignment_threshold: 0.05 embedding_normalization: false #

Q: 4. Check tokenizer version mismatch

prometheus-cli get-dataset-schema --name coco_caption # Output snippet { "text_tokenizer_version": "v1.3", "image_preprocess_version": "v1.2" } The text tokenizer version (v1.3) does not match the image preprocessing version (v1.2), confirming the schema drift.

Q: 4. Deploy the corrected pipeline

kubectl rollout restart deployment/prometheus-evaluator kubectl wait --for=condition=available deployment/prometheus-evaluator --timeout=180s Validation After applying the fixes, run the same evaluation request and compare metrics.

Problem

After upgrading the CLIP‑like vision‑language model to the version shipped with Prometheus v2.4.1, evaluation runs on multimodal benchmarks (e.g., COCO caption, Flickr30k) began producing divergent scores between the text and image modalities. Typical symptoms include:

Cross‑modal retrieval MAP dropped ~15 % while text‑only metrics stayed flat (Production run on the COCO caption benchmark).
~3 % of text‑image pairs return NaN similarity scores, triggering MultimodalAlignmentException.
Metric calculation failures such as “Division by zero encountered while computing cross‑modal Recall@K”.

Log entries like:


2026-07-12 14:03:27,812 ERROR prometheus.evaluator - MultimodalAlignmentException: Text and image embeddings are not comparable – cosine similarity matrix contains NaN values.
2026-07-12 14:03:28,019 WARN  prometheus.pipeline - EmbeddingDimensionMismatchError: Expected embedding size 512 but received 768 for modality 'image'.

Root Cause Analysis

The failure traces back to three intertwined changes introduced in the v2.4.1 release (see Prometheus Release Notes – v2.4.1 update notes describing changes to the multimodal embedding pipeline):

Embedding dimension shift: The new vision encoder outputs 768‑dimensional vectors, while the text encoder still produces 512‑dimensional embeddings. The evaluation service assumes a shared dimensionality for cosine similarity, leading to EmbeddingDimensionMismatchError and NaNs after the subsequent normalization step.
Default embedding_normalization reset: The release unintentionally reverted the embedding_normalization flag to false. Without L2‑normalization, the magnitude disparity between 512‑D and 768‑D vectors amplifies, causing the similarity variance to exceed the configured alignment_threshold (0.05) – see Prometheus Configuration Manual – alignment_threshold and embedding_normalization settings.
Tokenizer‑encoder schema drift: A schema change in the benchmark dataset introduced a newer tokenizer version for text inputs while the image encoder kept the older tokenization pipeline. This mismatch creates divergent embedding spaces, as reported in the internal audit (“text tokenizer version differed from the image encoder”).

Collectively, these issues break the alignment guarantees described in the Prometheus AI Platform Documentation – Multimodal Model Evaluation Guide (section on embedding alignment and metric calculation), which expects:

Identical embedding dimensionality across modalities.
Consistent L2‑normalization before similarity computation.
Matching preprocessing pipelines for text and image encoders.

Investigation and Debugging

The following steps reproduced the misalignment and isolated the root causes.

1. Verify endpoint response and dimensionality

curl -X POST https://prometheus.example.com/v1/evaluate \
  -H "Content-Type: application/json" \
  -d '{
        "model_version": "clip-v2.4.1",
        "inputs": [
          {"type":"text","data":"A dog playing in the park."},
          {"type":"image","data":"s3://bucket/image1.jpg"}
        ]
      }' | jq .

Sample output:

{
  "embeddings": {
    "text": {"vector": [0.12, -0.03, ...], "dim": 512},
    "image": {"vector": [0.07, 0.22, ...], "dim": 768}
  },
  "error": null
}

The mismatch in dim confirms the dimension shift.

2. Inspect configuration flags

cat /etc/prometheus/evaluation.yaml
# Current configuration
alignment_threshold: 0.05
embedding_normalization: false   # <-- should be true

3. Reproduce NaN similarity

python - <<'PY'
import numpy as np
text = np.random.rand(512)
image = np.random.rand(768)
# Simulate normalization step that expects same shape
try:
    from sklearn.preprocessing import normalize
    text_n = normalize(text.reshape(1, -1))[0]
    image_n = normalize(image.reshape(1, -1))[0]
    # Cosine similarity (dot product) will raise shape error
    sim = np.dot(text_n, image_n)
    print(sim)
except Exception as e:
    print("Error:", e)
PY

Output:

Error: shapes (1,512) and (1,768) not aligned: 512 (dim 0) != 768 (dim 0)

4. Check tokenizer version mismatch

prometheus-cli get-dataset-schema --name coco_caption
# Output snippet
{
  "text_tokenizer_version": "v1.3",
  "image_preprocess_version": "v1.2"
}

The text tokenizer version (v1.3) does not match the image preprocessing version (v1.2), confirming the schema drift.

Resolution

Fixes are applied at three levels: model compatibility, configuration, and dataset schema alignment.

1. Align embedding dimensions

Two approaches are viable:

Projection layer: Insert a linear projection to map the 768‑D image embeddings to 512‑D before similarity calculation.
Model downgrade: Pin the vision encoder to the previous 512‑D checkpoint.

We chose the projection layer for minimal disruption.

Before (model inference code):

def encode_image(img):
    return vision_encoder(img)  # returns 768‑D vector

After (added projection):

import torch.nn as nn

proj = nn.Linear(768, 512, bias=False)

def encode_image(img):
    raw = vision_encoder(img)          # 768‑D
    return proj(raw)                   # 512‑D aligned with text

2. Re‑enable embedding normalization

Update the evaluation configuration to restore the default normalization behavior.

# /etc/prometheus/evaluation.yaml
alignment_threshold: 0.05
embedding_normalization: true   # changed from false

3. Synchronize tokenizer and preprocessing versions

Regenerate the benchmark schema so that both modalities use the same preprocessing pipeline version (v1.3).

prometheus-cli update-dataset-schema \
  --name coco_caption \
  --text-tokenizer-version v1.3 \
  --image-preprocess-version v1.3

4. Deploy the corrected pipeline

kubectl rollout restart deployment/prometheus-evaluator
kubectl wait --for=condition=available deployment/prometheus-evaluator --timeout=180s

Validation

After applying the fixes, run the same evaluation request and compare metrics.

1. Endpoint sanity check

curl -s -X POST https://prometheus.example.com/v1/evaluate \
  -H "Content-Type: application/json" \
  -d '{"model_version":"clip-v2.4.1","inputs":[{"type":"text","data":"A dog playing in the park."},{"type":"image","data":"s3://bucket/image1.jpg"}]}' | jq .

Expected output (dimensions now match):

{
  "embeddings": {
    "text": {"vector": [0.12, -0.03, ...], "dim": 512},
    "image": {"vector": [0.09, 0.21, ...], "dim": 512}
  },
  "error": null
}

2. Metric regression

Re‑run the COCO caption benchmark.

prometheus-cli evaluate --benchmark coco_caption --model clip-v2.4.1 --output results.json

Key figures (pre‑fix vs post‑fix):

Metric	Before	After
Image‑to‑Text MAP	0.42 (‑15 %)	0.61 (restored)
Text‑to‑Image Recall@1	0.48	0.73
NaN similarity rate	3 %	0 %

3. Health check

curl -f http://prometheus.example.com/healthz || echo "Unhealthy"

Output:

OK

Operational Experience

Misleading symptom: The text‑only metrics appeared healthy, leading some teams to overlook the multimodal drift.
Assumption trap: It was assumed that a model version bump would preserve embedding size; the release notes only mentioned “improved visual encoder” without explicit dimension change.
Edge case: The embedding_normalization flag reset only affected new deployments; older pods continued to use the correct setting, causing inconsistent behavior across a rolling upgrade.
Lesson learned: Always pin both alignment_threshold and embedding_normalization in the configuration file rather than relying on defaults.

Best Practices and Prevention

Include embedding_dim validation in CI pipelines:

def test_embedding_dim():
    txt = encode_text("sample")
    img = encode_image("sample.jpg")
    assert txt.shape[-1] == img.shape[-1], "Embedding dimension mismatch"

Monitor alignment_threshold violations via Prometheus alerts:

ALERT MultimodalAlignmentDrift
  IF prometheus_alignment_variance > 0.05
  FOR 5m
  LABELS {severity="critical"}
  ANNOTATIONS {
    summary = "Cross‑modal similarity variance exceeds threshold",
    description = "Investigate embedding dimension or normalization changes."
  }

Version‑lock tokenizer and encoder pipelines together using a manifest file (e.g., model_manifest.yaml).
Automate schema synchronization after any dataset version bump.
Run a smoke‑test that computes cosine similarity for a small random batch after each deployment; fail fast on NaNs.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

Why does the error appear only after a model upgrade?
Because v2.4.1 introduced a 768‑D image encoder while the text encoder stayed at 512 D, breaking the assumption of equal dimensionality required for cosine similarity.
Can I keep the 768‑D embeddings without a projection?
Only if you also upgrade the text encoder to produce 768‑D vectors and adjust downstream metrics accordingly. Otherwise, a projection layer is the simplest fix.
What does EmbeddingDimensionMismatchError indicate?
It signals that the evaluator received embeddings of different sizes for the two modalities, which prevents the dot‑product used in cosine similarity.
How does the embedding_normalization flag affect alignment?
When disabled, vectors retain their raw magnitudes; the resulting scale mismatch inflates cosine distances and can push similarity variance beyond the alignment_threshold, causing MultimodalAlignmentException.
Is there a way to detect tokenizer‑encoder drift automatically?
Yes—store the tokenizer and image preprocessing versions in the benchmark schema and add a CI check that asserts they are identical before evaluation runs.