RAG pipeline dimension error after updating embedding model in Docker Compose

Problem

The Retrieval‑Augmented Generation (RAG) service, deployed with Docker Compose, fails during inference with a dimension mismatch error. The query encoder produces vectors of a different size than the document encoder, causing the retrieval component to raise an exception and abort the request.

2026-10-31 14:12:07,842 ERROR rag_service.inference - RuntimeError: Expected embedding dimension 768 but got 1024
Traceback (most recent call last):
  File "/app/rag_service/inference.py", line 112, in handle_query
    query_vec = self.query_encoder.encode(query)
  File "/app/encoders/query_encoder.py", line 45, in encode
    return self.model.encode(text)
  File "/usr/local/lib/python3.10/site-packages/transformers/pipelines.py", line 312, in __call__
    embeddings = self.model(**inputs)[0]
RuntimeError: Expected embedding dimension 768 but got 1024

Impact:

• All incoming queries return HTTP 500.
• Monitoring alerts fire on “RAG retrieval failure”.
• No new documents can be indexed because the document encoder also emits 1024‑dimensional vectors.

Root Cause

The failure originates from an uncoordinated upgrade of the embedding model in the query_encoder container. The Docker image was rebuilt with sentence‑transformers/all‑MiniLM‑L6‑v2 (output dimension 384) while the document_encoder still uses the older sentence‑transformers/all‑mpnet‑base‑v2 (output dimension 768). The RAG retrieval layer is hard‑coded to expect matching dimensions (default 768). Because the two encoders now emit vectors of different sizes, the FAISS index construction fails, resulting in the observed RuntimeError.

Why it happened:

• The docker-compose.yml referenced a floating tag sentence-transformers:latest for the query encoder.
• During a routine CI run, latest resolved to a newer model version with a different hidden size.
• No version pinning or validation step existed to guarantee dimensional parity between the two encoders.

Debug

Step‑by‑step investigation that reproduced the mismatch:

1. Inspect container logs – the error above clearly shows the expected vs. actual dimensions.
2. Check the model files inside each container:

# Inside query_encoder container
docker exec -it rag_query_encoder bash -c "python - <<'PY'
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('/models')
print('Query encoder dim:', model.get_sentence_embedding_dimension())
PY"
Query encoder dim: 384

# Inside document_encoder container
docker exec -it rag_doc_encoder bash -c "python - <<'PY'
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('/models')
print('Document encoder dim:', model.get_sentence_embedding_dimension())
PY"
Document encoder dim: 768

Verify the Docker Compose definition that pulls the model:

# docker-compose.yml (original)
services:
  query_encoder:
    image: myorg/rag-encoder:latest
    environment:
      - MODEL_NAME=sentence-transformers:latest   # ← problematic
  document_encoder:
    image: myorg/rag-encoder:1.3.0
    environment:
      - MODEL_NAME=sentence-transformers/all-mpnet-base-v2

Confirm the RAG configuration (usually config.yaml) expects 768‑dimensional vectors:

retriever:
  embedding_dim: 768
  index_type: faiss

Solution

Align both encoders to the same model version and dimension, and add explicit version pinning.

Before (problematic) configuration

# docker-compose.yml
services:
  query_encoder:
    image: myorg/rag-encoder:latest
    environment:
      - MODEL_NAME=sentence-transformers:latest
  document_encoder:
    image: myorg/rag-encoder:1.3.0
    environment:
      - MODEL_NAME=sentence-transformers/all-mpnet-base-v2

After (fixed) configuration

# docker-compose.yml
services:
  query_encoder:
    image: myorg/rag-encoder:1.3.0
    environment:
      - MODEL_NAME=sentence-transformers/all-mpnet-base-v2   # pinned to 768‑dim model
  document_encoder:
    image: myorg/rag-encoder:1.3.0
    environment:
      - MODEL_NAME=sentence-transformers/all-mpnet-base-v2

Additional steps:

• Rebuild the encoder image with the correct model bundled (or let it download at startup).
• Update config.yaml if a different dimension is desired, e.g., switching both to a 384‑dim model, but ensure the value matches the actual model.
• Run docker compose up --build -d to redeploy.

Verify

After redeployment, confirm that both encoders report the same dimension and that a sample query succeeds.

# Verify dimensions
docker exec rag_query_encoder python -c "
from sentence_transformers import SentenceTransformer
print('Query dim:', SentenceTransformer('/models').get_sentence_embedding_dimension())
"
Query dim: 768

docker exec rag_doc_encoder python -c "
from sentence_transformers import SentenceTransformer
print('Doc dim:', SentenceTransformer('/models').get_sentence_embedding_dimension())
"
Doc dim: 768

# Run a health‑check request
curl -s -X POST http://localhost:8080/rag/query -d '{"query":"What is the capital of France?"}' | jq .
{
  "answer": "The capital of France is Paris."
}

Check logs for absence of the previous RuntimeError and confirm that the FAISS index loads without warnings.

Prevent

Implement safeguards to avoid future dimension mismatches:

Guardrail	Implementation
Pin model versions in Docker images	Use immutable tags (e.g., sentence-transformers/all-mpnet-base-v2@sha256:…) instead of :latest.
Validate embedding dimension at container start‑up	Add a start‑up script that loads the model and exits with non‑zero status if model.get_sentence_embedding_dimension() differs from the value in config.yaml.
CI test matrix	Run an integration test that encodes a dummy query and a dummy document, asserts equal vector sizes, and fails the pipeline if they diverge.
Monitoring alert	Expose the embedding dimension as a Prometheus gauge (e.g., rag_encoder_embedding_dim{service="query"} 768) and alert on mismatched values.
Documentation lock	Record the approved model name and dimension in the repository’s README and enforce via code review checklists.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

1. Why does the error only appear after a Docker image rebuild?
   Because the rebuild pulled the latest tag, which resolved to a newer model with a different hidden size. The document encoder image was not rebuilt, so the two containers diverged.
2. Can I use different models for query and document encoders?
   Only if they share the exact same output dimension and are compatible with the retriever’s index type. Otherwise the retrieval layer cannot compute similarity scores.
3. How do I discover the embedding dimension of an arbitrary SentenceTransformer model?
   Load the model in Python and call model.get_sentence_embedding_dimension(). This is the value that must match the embedding_dim field in the RAG config.
4. Is there a way to make the RAG service auto‑adjust to dimension changes?
   You could add a dynamic re‑initialization step that rebuilds the FAISS index with the new dimension, but this adds latency and complexity. Pinning versions is the simpler, more reliable approach.
5. What should I do if I must upgrade to a higher‑dimensional model?
   Update config.yaml to the new embedding_dim, rebuild both encoder containers with the same model, and re‑index all documents so the index matches the new vector size.