RAG embedding model dimension mismatch after deployment update

Problem Description

The RAG (Retrieval‑Augmented Generation) service deployed on a Kubernetes cluster began returning vector dimension mismatch errors after a routine rolling update. The symptoms observed across multiple pods were:

• FAISS index load failures: ValueError: Expected embedding dimension 768 but got 1024
• Cosine‑similarity runtime errors: RuntimeError: shape mismatch for dot product: (batch, 768) vs (batch, 1024)
• Intermittent faiss.IndexFlatL2: dimension mismatch logs during batch query processing
• Overall retrieval precision dropped from ~92 % to < 60 % as measured by downstream QA tests

All pods are stateless inference containers that mount a persistent volume holding the vector database (FAISS, Chroma, or Milvus). The deployment controller uses the imagePullPolicy: Always setting and the Docker image tag myorg/embedding-service:latest.

Root Cause Analysis

The mismatch originates from three tightly coupled factors:

1. Unpinned model image tag – Docker’s Tagging and versioning images guidance recommends pinning immutable tags to avoid accidental drift. Using :latest allowed the CI pipeline to push a new checkpoint (a 1024‑dimensional BERT model) while the existing index was built with a 768‑dimensional SBERT checkpoint.
2. Rolling update without index migration – Kubernetes performed a rolling update, pulling the new image on a subset of pods. Because the vector store resides on a persistent volume, the newly started pods began serving queries with the 1024‑dim model against an index still containing 768‑dim vectors, triggering the FAISS and HNSW errors reported in the logs.
3. Image build non‑determinism – The Dockerfile relied on ADD to copy the model directory from a remote URL at build time. According to Dockerfile reference, COPY should be used for deterministic artifacts; otherwise the image can embed a different checkpoint each build, making it impossible to guarantee dimensionality consistency.

These points are echoed in community reports: the LangChain issue #8421 and Milvus issue #12634 both describe identical failures after a model upgrade, and the Haystack discussion #3849 highlights mixed‑version pods causing cosine‑similarity failures.

Investigation and Debugging

Below is a reproducible debugging workflow that was used to pinpoint the root cause.

1. Gather pod logs

kubectl logs -l app=rag-embedding -c inference --tail=100 | grep -i "dimension"

Sample output:

2026-07-28T14:12:03.112Z INFO Loading model from /app/model.pt – unexpected hidden size 1024 (expected 768)
2026-07-28T14:12:04.237Z ERROR faiss.IndexFlatL2: dimension mismatch (expected 768, got 1024)
2026-07-28T14:12:04.239Z TRACE Query embedding shape: torch.Size([32, 1024])
2026-07-28T14:12:04.240Z TRACE Index embedding shape: torch.Size([?, 768])

2. Inspect the Docker image tag used by each pod

kubectl get pods -l app=rag-embedding -o jsonpath="{range .items[*]}{.metadata.name}{'\\t'}{.spec.containers[0].image}{'\\n'}{end}"

Result showed a mix of myorg/embedding-service:latest and myorg/embedding-service:2024.07.15 after the update.

3. Verify model dimensions directly inside a running container

kubectl exec -it $(kubectl get pod -l app=rag-embedding -o name | head -n1) -- bash -c "
python - <<'PY'
import torch, transformers
model = torch.load('/app/model.pt')
print('hidden size:', model.config.hidden_size)
PY"

Output:

hidden size: 1024

4. Check the index metadata on the persistent volume

kubectl exec -it $(kubectl get pod -l app=vector-db -o name | head -n1) -- bash -c "
python - <<'PY'
import faiss, json, os
index_path = '/data/faiss.index'
if os.path.exists(index_path):
    idx = faiss.read_index(index_path)
    print('index dimension:', idx.d)
else:
    print('index not found')
PY"

Output:

index dimension: 768

5. Confirm image build process

cat Dockerfile
# Bad example from the broken image
FROM python:3.11-slim
ADD https://model-repo.com/checkpoints/bert_latest.pt /app/model.pt
RUN pip install -r requirements.txt

The ADD instruction pulls the latest checkpoint at build time, violating deterministic builds.

Resolution

The fix consists of three coordinated actions: enforce immutable image tags, make the Docker build deterministic, and re‑index the vector store to match the new model (or downgrade the model to the previous dimension).

1. Pin the model image tag

Update the deployment manifest to reference a specific tag and set imagePullPolicy: IfNotPresent to avoid pulling unintended versions.

# Before (broken)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rag-embedding
spec:
  replicas: 3
  selector:
    matchLabels:
      app: rag-embedding
  template:
    metadata:
      labels:
        app: rag-embedding
    spec:
      containers:
      - name: inference
        image: myorg/embedding-service:latest
        imagePullPolicy: Always
        volumeMounts:
        - name: vectordb
          mountPath: /data
      volumes:
      - name: vectordb
        persistentVolumeClaim:
          claimName: vectordb-pvc

# After (fixed)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rag-embedding
spec:
  replicas: 3
  selector:
    matchLabels:
      app: rag-embedding
  template:
    metadata:
      labels:
        app: rag-embedding
    spec:
      containers:
      - name: inference
        image: myorg/embedding-service:2024.07.15   # immutable tag
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: vectordb
          mountPath: /data
      volumes:
      - name: vectordb
        persistentVolumeClaim:
          claimName: vectordb-pvc

2. Make the Docker build deterministic

Replace ADD with COPY and bake the exact checkpoint into the image during CI.

# Dockerfile (recommended)
FROM python:3.11-slim

# Copy a specific, version‑controlled model artifact
COPY models/bert_sbert_768.pt /app/model.pt

RUN pip install -r requirements.txt

The models/bert_sbert_768.pt file should be stored in the repository (or a version‑controlled artifact store) and referenced by a hash in CI to guarantee consistency.

3. Re‑index or migrate the vector database

If the new model is intentionally larger (e.g., 1024‑dim), the existing index must be rebuilt. Otherwise, revert to the previous model and keep the existing index.

• Option A – Re‑index with new dimensions

  # Re‑index script (run once)
  python index_documents.py \
    --model /app/model.pt \
    --output /data/faiss.index \
    --dimension 1024
  

• Option B – Downgrade model to match existing index

  Replace the model artifact with the 768‑dim checkpoint and redeploy without rebuilding the index.

Verification

After applying the fixes, perform the following checks:

1. Pod image consistency

   kubectl get pods -l app=rag-embedding -o jsonpath="{.items[*].spec.containers[0].image}" | tr ' ' '\n' | sort | uniq

   Expected output: a single line with myorg/embedding-service:2024.07.15.

2. Index load test

   kubectl exec -it $(kubectl get pod -l app=vector-db -o name | head -n1) -- bash -c "
   python - <<'PY'
   import faiss, numpy as np
   idx = faiss.read_index('/data/faiss.index')
   print('Loaded index dim:', idx.d)
   # Dummy query vector matching the model dimension
   q = np.random.random((1, idx.d)).astype('float32')
   D, I = idx.search(q, k=5)
   print('Search succeeded, distances:', D)
   PY"
   

   Should print Loaded index dim: 1024 (or 768, depending on chosen option) and no dimension‑mismatch errors.

3. End‑to‑end query test

   curl -X POST http://rag-embedding.default.svc.cluster.local/query \
     -H "Content-Type: application/json" \
     -d '{"question":"What is the capital of France?"}' | jq .answer

   Answer should be correct and latency within expected SLA.

4. Monitoring alerts

   Confirm that the alert VectorDimensionMismatch (if configured) is in a OK state in Prometheus/Grafana.

Operational Best Practices and Prevention

Practice	Why it matters	Implementation tip
Immutable image tags	Prevents accidental drift when latest moves.	Use CI to generate tags like v2024.07.15‑sha1abc123 and reference them in manifests.
Deterministic Docker builds	Ensures the same checkpoint is baked every time.	Prefer COPY from a version‑controlled directory; avoid ADD from remote URLs.
Schema versioning for vector stores	Detects dimension mismatches before runtime.	Store dimension as a metadata field; reject queries if mismatch detected.
Rolling‑update coordination	Guarantees all pods run the same model before traffic resumes.	Use maxUnavailable: 0 and a pre‑hook that validates the new model dimension against the stored index.
Automated re‑indexing pipeline	Reduces manual effort when model dimensions change.	Trigger a re‑index job in CI when a model version with a new hidden_size is published.

FAQ

1. Why does the error appear only after a rolling update and not during the initial deployment?
   Because the initial deployment used the same model version that built the index. The rolling update introduced a newer model image while the persistent index remained at the old dimension, causing the mismatch only when the new pods started handling queries.
2. Can I keep using imagePullPolicy: Always safely?
   Yes, but only if the tag you pull is immutable (e.g., v2024.07.15) and the CI pipeline guarantees that the artifact behind that tag never changes. Otherwise, IfNotPresent with pinned tags is the safer default.
3. How do I detect a dimension mismatch before it crashes the service?
   Add a startup health check that loads the vector store, reads index.d, and compares it to model.config.hidden_size. Fail the pod if they differ, allowing the orchestrator to roll back.
4. Is there a way to migrate an existing FAISS index to a new dimension without full re‑indexing?
   FAISS does not support changing the dimensionality of an existing index. The only reliable path is to rebuild the index with embeddings generated from the new model, optionally using a background re‑index job to avoid downtime.
5. What monitoring metric should I alert on to catch this early?
   Expose a gauge such as vector_store_dimension{model="bert_sbert_768"} = 768 and another gauge for the loaded model dimension. Alert when the two differ, e.g., vector_store_dimension != model_embedding_dimension.

Related Topic Hub: Distributed Systems Troubleshooting Hub

Related Articles

• RAG embedding dimension mismatch in PostgreSQL after model update