Weaviate Multimodal Embedding Mismatch Across AWS Regions

Q: Weaviate Multimodal Embedding Mismatch Across AWS Regions Problem Description (Symptoms and Impact)

In a multi‑region Weaviate deployment (e.g., us-east-1 and eu-west-1) with cross‑region replication enabled, identical multimodal objects (image + text) produce divergent search results after replication. The most common observable artifacts are:

Q: Root Cause Analysis

Several independent incidents point to a common theme: model version drift. The underlying causes are:

In this article

Table of Contents

Weaviate Multimodal Embedding Mismatch Across AWS Regions

Problem Description (Symptoms and Impact)

In a multi‑region Weaviate deployment (e.g., us-east-1 and eu-west-1) with cross‑region replication enabled, identical multimodal objects (image + text) produce divergent search results after replication. The most common observable artifacts are:

Search recall drops up to 15 % for image‑text queries after a regional failover (see Incident 2024‑03‑12).

Log entries such as:

WARN: model version conflict – local node using text2vec‑multimodal v1.3, remote node using v1.4; vectors may not be comparable.

Replication sync failures with messages like:

ERROR: vector dimensions mismatch – expected 512, got 768 (replication sync failure).

Inconsistent ranking of the same image across regions, leading to SLA breaches for recommendation services.

Technical Background

Weaviate’s text2vec‑multimodal module typically wraps OpenAI’s CLIP model. The module performs deterministic preprocessing (image resizing, tokenization) and then produces a fixed‑size vector (512 dimensions for the default CLIP‑ViT‑B/32). In a multi‑region setup, Weaviate replicates the objects table and the associated vectors column across clusters (see Weaviate Documentation – Replication & Multi‑Region Deployment). Consistency is guaranteed only if the vectorizer produces identical outputs on every node.

Root Cause Analysis

Several independent incidents point to a common theme: model version drift. The underlying causes are:

Unpinned module versions. Nodes pull the latest text2vec‑multimodal Docker image at startup. If one region auto‑updates to v1.4 (which ships CLIP‑ViT‑L/14, 768‑dim vectors) while another stays on v1.3, vectors become incomparable (GitHub Issue #3124).
Separate model caches. Each region downloads the CLIP weights into its local cache directory. A partial download or a cached older checkpoint results in subtle preprocessing differences (e.g., different image resize algorithm) (Stack Overflow 823456).
Serialization bug. The replication serializer historically truncated CLIP embeddings when the source and target dimensions differed, leading to corrupted vectors (GitHub Discussion #2871).
Inconsistent environment configuration. The WEAVIATE_MODULES_TEXT2VEC_MULTIMODAL_MODEL_VERSION env var was set only in one region, causing the other region to fall back to the default.

Because Weaviate’s replication layer treats vectors as opaque byte arrays, any deviation in size or content causes either a sync error (dimension mismatch) or silent drift (same size, different semantics).

Investigation and Debugging Steps

Follow this checklist to isolate the mismatch:

Verify module versions on each node.
```
kubectl exec -it weaviate-0 -- weaviate-modules version
```
Expected output:
```
text2vec-multimodal: v1.3.0 (CLIP model: ViT-B/32, 512d)
```
If regions report different versions, you have a version drift.

Inspect the vector dimensions stored in the database.

curl -X GET "https:///v1/objects/_search" -H "Content-Type: application/json" -d '{
  "class": "MultimodalItem",
  "limit": 1,
  "includeVector": true
}' | jq ".objects[0].vector | length"

Compare the numeric output between regions. A mismatch (512 vs 768) confirms the problem.

Check model cache timestamps.
```
ls -l /var/weaviate/models/clip/
```
Look for differing timestamps or missing files.
Capture a sample image vector on each region.
```
curl -X POST "https://us-east-1.example.com/v1/vectors" -H "Content-Type: application/json" -d '{
  "image": "s3://bucket/sample.jpg",
  "text": "a red sports car"
}' | jq ".vector"
```
Do the same against eu-west-1 and diff the JSON arrays. Even a single differing element indicates a preprocessing or model version mismatch.
Review replication logs for serialization warnings.
```
2024-06-05T12:34:56Z WARN replication: vector dimension mismatch – expected 512, got 768
```
If such warnings appear, the serializer is rejecting the payload.

Resolution (Fix Implementation)

The fix consists of three coordinated actions: pin the module version, centralize model storage, and ensure consistent environment variables.

1. Pin the multimodal module version

Update the Helm values (or Kubernetes manifest) for every region to use the same image tag.

# before (different tags per region)
image:
  repository: semitechnologies/weaviate
  tag: latest   # <-- leads to auto‑upgrade

# after (explicit version)
image:
  repository: semitechnologies/weaviate
  tag: 1.21.0   # matches text2vec‑multimodal v1.3.0

2. Share a single model artifact bucket

Configure the module to load CLIP weights from an S3 bucket that all regions mount read‑only.

# weaviate.conf (common to all regions)
modules:
  text2vec-multimodal:
    image: s3://weaviate-models/clip/ViT-B-32.pt
    modelVersion: "v1.3"

Deploy the bucket and set the IAM role accordingly. This eliminates divergent downloads.

3. Align environment variables

Ensure the following env vars are identical across clusters:

WEAVIATE_MODULES_TEXT2VEC_MULTIMODAL_MODEL_VERSION=v1.3
WEAVIATE_MODULES_TEXT2VEC_MULTIMODAL_MODEL_PATH=/models/clip/ViT-B-32.pt
WEAVIATE_REPLICATION_CONSISTENCY_LEVEL=QUORUM

4. Restart nodes to reload the pinned model

kubectl rollout restart deployment/weaviate -n weaviate

5. Re‑synchronize existing vectors (optional but recommended)

If vectors were already stored with mismatched dimensions, re‑ingest them:

# Export objects from the source region
weaviate-export --endpoint https://us-east-1.example.com --class MultimodalItem --output /tmp/export.json

# Import into target region after model alignment
weaviate-import --endpoint https://eu-west-1.example.com --input /tmp/export.json --recreate

Verification (Validation Steps)

Re‑run the vector dimension check. Both regions should now report 512.

Execute a deterministic query.

curl -X POST "https://us-east-1.example.com/v1/graphql" -H "Content-Type: application/json" -d '{
  "query": "{ Get { MultimodalItem(where: {operator: Equal, path: [\"image\"], valueString: \"s3://bucket/sample.jpg\"}) { _additional { distance } } } }"
}'

The distance values returned from us-east-1 and eu-west-1 should be identical (within floating‑point tolerance).

Monitor replication health.
```
kubectl logs weaviate-0 -n weaviate | grep "replication"
```
No warnings about dimension mismatch or model version conflict should appear.
Run a recall benchmark. Compare pre‑ and post‑fix results; the 15 % drop reported in Incident 2024‑03‑12 should disappear.

Operational Experience (Lessons Learned)

Version drift is silent until you query across regions. Because Weaviate does not embed module version metadata in the vector payload, the system assumes compatibility.
Auto‑updates of Docker images are dangerous in multi‑region setups. A single node pulling latest can corrupt the whole replica set.
Model cache directories must be part of your IaC. Treat them as immutable assets and mount them from a shared object store.
Replication lag can mask drift. In the early minutes after a failover, stale vectors may still be served, leading to intermittent inconsistencies.

Best Practices and Prevention

Pin text2vec‑multimodal module versions in Helm values or Docker compose files.
Store model binaries in a versioned S3 bucket and reference them via absolute paths.
Enable WEAVIATE_REPLICATION_CONSISTENCY_LEVEL=QUORUM to ensure writes are acknowledged by a majority of regions before returning success.
Instrument a health check that compares a known “golden” image vector across regions every 5 minutes; alert on any distance > 1e‑5.
Include a CI step that validates vector dimension consistency after any module upgrade.

FAQ (Related Questions)

Why do vectors sometimes have 768 dimensions instead of 512?
The default CLIP model (ViT‑B/32) produces 512‑dim vectors. Upgrading to CLIP‑ViT‑L/14 (bundled with text2vec‑multimodal v1.4) changes the output size to 768. If one region runs v1.4 while another runs v1.3, the dimensions diverge.
Can I use different model versions in different regions if I need higher accuracy?
Not without breaking vector comparability. The only safe approach is to keep all replicas on the same model version or to create separate classes per model and query them independently.
How do I confirm which CLIP model a node is actually using?
Run weaviate-modules info text2vec‑multimodal on the node; the output includes the model file checksum and version. Compare the checksum across regions.
What should I do if I see “failed to deserialize vector” errors after a failover?
These errors usually indicate corrupted payloads caused by a serialization bug when dimensions differ. Re‑synchronize the affected class after aligning model versions, as described in the “Resolution” section.
Is there a way to automatically enforce identical module versions across clusters?
Yes. Use a ConfigMap that stores the desired module version and mount it into every Weaviate pod. Combine this with an admission controller that rejects deployments with mismatched image tags.

Related Topic Hub: Vector Databases Troubleshooting Hub