RAG Embedding Dimension Mismatch After Switching Fine‑tune Model In DDP

Q: 1. Inspect training logs

[Rank 0] INFO 2024-10-12 10:03:12,345 Retriever initialized with embedding_dim=768 [Rank 2] ERROR 2024-10-12 10:03:13,112 ValueError: Expected embedding dimension 768, got 1024 [Rank 3] WARNING 2024-10-12 10:03:13,115 Retrieval fallback to empty context 2. Verify embedding model configuration # retriever.py (before change) EMBEDDING_MODEL = "text-embedding-ada-002" # 768‑dim # training script (after change) model_name = "gpt4-large" # expects 1024‑dim embeddings 3. Check FAISS index metadata >>> index = faiss.read_index("faiss.index") >>> index.d 768 4. Confirm query embedding shape >>> query = encoder.encode("What is RAG?") >>> query.shape torch.Size([1, 1024]) 5. Reproduce on a single GPU Running the same script with CUDA_VISIBLE_DEVICES=0 eliminated the race condition, confirming that the issue is specific to DDP initialization order.

Q: 1. Centralize embedding model selection

Introduce a single source of truth in config.yaml and read it in both the fine‑tuning script and the retriever.

Q: 2. Update retriever.py to load the model from config

# before EMBEDDING_MODEL = "text-embedding-ada-002" # after import yaml with open("config.yaml") as f: cfg = yaml.safe_load(f) EMBEDDING_MODEL = cfg["embedding"]["model_name"] EMBEDDING_DIM = cfg["embedding"]["dimension"] 3. Re‑create FAISS index with the correct dimension # before (incorrect) index = faiss.IndexFlatL2(768) # after (correct) index = faiss.IndexFlatL2(1024) faiss.write_index(index, "faiss.index") 4. Enforce deterministic model loading in DDP Wrap the retriever initialization in a torch.distributed.barrier() so that rank 0 creates the index first and all other ranks wait.

Q: 1. Index dimension sanity

>>> index = faiss.read_index("faiss.index") >>> index.d 1024 2. Query embedding shape >>> query = encoder.encode("Explain RAG.") >>> query.shape torch.Size([1, 1024]) 3. Retrieval hit rate [Rank 0] INFO Retrieval hit rate: 0.87 (previously 0.12) [Rank 2] INFO Retrieval latency: 12ms avg 4. Generation quality metrics BLEU: 31.4 → 45.7 ROUGE-L: 38.2 → 52.1 5. End‑to‑end test script #!/usr/bin/env python3 from retriever import init_retriever from generator import load_model retriever = init_retriever() generator = load_model("gpt4-large") question = "What are the benefits of using RAG?" docs = retriever.retrieve(question, k=5) answer = generator.generate(question, context=docs) print("Answer:", answer) Prevent To avoid recurrence of embedding mismatches in distributed training:

Problem

During a multi‑GPU Distributed Data Parallel (DDP) fine‑tuning run of a GPT‑4 based generator, the Retrieval‑Augmented Generation (RAG) pipeline started returning irrelevant documents and the generated answers degraded noticeably. The root symptom was an embedding dimension mismatch between the vector store used for retrieval and the embedding model that was implicitly selected during the fine‑tuning step.

Typical observations:

Retrieval latency spikes after the first epoch.
Log entries such as ValueError: Expected embedding dimension 768, got 1024.
Down‑stream generation quality drops (BLEU, ROUGE scores fall by ~30%).
Only a subset of GPU nodes report the error; others continue with the original dimension.

Root Cause

The DDP launch script was updated to switch the fine‑tuning checkpoint from gpt4-base (which uses a 768‑dimensional OpenAI embedding model) to gpt4-large (which expects a 1024‑dimensional embedding model). The RAG retrieval component, however, continued to load the original text-embedding-ada-002 model (768‑dim) because the embedding model name is hard‑coded in the retriever.py module.

In a DDP setting, each rank loads its own copy of the model. The rank that first instantiated the Retriever cached the 768‑dimensional encoder, while the later ranks attempted to re‑initialize it with the new 1024‑dimensional encoder, triggering a dimension mismatch error during the torch.nn.Linear weight load.

Consequences:

Vector store (FAISS) was built with 768‑dim vectors.
Fine‑tuned model produced 1024‑dim query embeddings.
FAISS returned no nearest neighbours for most queries, falling back to random retrieval or empty results.
Generation quality suffered because the context window was effectively empty.

Debug

Step‑by‑step investigation that reproduced the failure:

1. Inspect training logs


[Rank 0] INFO 2024-10-12 10:03:12,345 Retriever initialized with embedding_dim=768
[Rank 2] ERROR 2024-10-12 10:03:13,112 ValueError: Expected embedding dimension 768, got 1024
[Rank 3] WARNING 2024-10-12 10:03:13,115 Retrieval fallback to empty context

2. Verify embedding model configuration


# retriever.py (before change)
EMBEDDING_MODEL = "text-embedding-ada-002"   # 768‑dim

# training script (after change)
model_name = "gpt4-large"   # expects 1024‑dim embeddings

3. Check FAISS index metadata


>>> index = faiss.read_index("faiss.index")
>>> index.d
768

4. Confirm query embedding shape


>>> query = encoder.encode("What is RAG?")
>>> query.shape
torch.Size([1, 1024])

5. Reproduce on a single GPU

Running the same script with CUDA_VISIBLE_DEVICES=0 eliminated the race condition, confirming that the issue is specific to DDP initialization order.

Solution

The fix consists of aligning the embedding model across the entire pipeline and ensuring deterministic initialization across DDP ranks.

1. Centralize embedding model selection

Introduce a single source of truth in config.yaml and read it in both the fine‑tuning script and the retriever.

# config.yaml
embedding:
  model_name: "text-embedding-3-large"
  dimension: 1024

2. Update `retriever.py` to load the model from config


# before
EMBEDDING_MODEL = "text-embedding-ada-002"

# after
import yaml
with open("config.yaml") as f:
    cfg = yaml.safe_load(f)

EMBEDDING_MODEL = cfg["embedding"]["model_name"]
EMBEDDING_DIM = cfg["embedding"]["dimension"]

3. Re‑create FAISS index with the correct dimension


# before (incorrect)
index = faiss.IndexFlatL2(768)

# after (correct)
index = faiss.IndexFlatL2(1024)
faiss.write_index(index, "faiss.index")

4. Enforce deterministic model loading in DDP

Wrap the retriever initialization in a torch.distributed.barrier() so that rank 0 creates the index first and all other ranks wait.


# retriever_init.py
def init_retriever():
    if torch.distributed.get_rank() == 0:
        retriever = Retriever(EMBEDDING_MODEL, EMBEDDING_DIM)
        torch.save(retriever.state_dict(), "retriever.pt")
    torch.distributed.barrier()
    if torch.distributed.get_rank() != 0:
        retriever = Retriever(EMBEDDING_MODEL, EMBEDDING_DIM)
        retriever.load_state_dict(torch.load("retriever.pt"))
    return retriever

5. Deploy the corrected pipeline

Run the DDP launch with the updated configuration:


torchrun --nproc_per_node=4 \
    --master_addr=10.0.0.1 --master_port=29500 \
    train_rag.py --config=config.yaml

Verify

After applying the changes, perform the following checks:

1. Index dimension sanity


>>> index = faiss.read_index("faiss.index")
>>> index.d
1024

2. Query embedding shape


>>> query = encoder.encode("Explain RAG.")
>>> query.shape
torch.Size([1, 1024])

3. Retrieval hit rate


[Rank 0] INFO Retrieval hit rate: 0.87 (previously 0.12)
[Rank 2] INFO Retrieval latency: 12ms avg

4. Generation quality metrics


BLEU: 31.4 → 45.7
ROUGE-L: 38.2 → 52.1

5. End‑to‑end test script


#!/usr/bin/env python3
from retriever import init_retriever
from generator import load_model

retriever = init_retriever()
generator = load_model("gpt4-large")

question = "What are the benefits of using RAG?"
docs = retriever.retrieve(question, k=5)
answer = generator.generate(question, context=docs)

print("Answer:", answer)

Prevent

To avoid recurrence of embedding mismatches in distributed training:

Single source of truth: Store embedding model name and dimension in a version‑controlled config file; load it everywhere.
Automated sanity check: Add a startup assertion that retriever.embedding_dim == generator.expected_dim and abort if they differ.
Versioned FAISS indexes: Include the embedding dimension in the index filename (e.g., faiss_1024.index) and validate on load.
DDP initialization guard: Use torch.distributed.barrier() around any shared resource creation.
CI pipeline test: Spin up a minimal 2‑GPU DDP job that runs a retrieval‑generation round‑trip and asserts non‑empty results.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does the error appear only on some ranks? The rank that first instantiated the Retriever cached the old embedding dimension. Subsequent ranks attempted to overwrite the model with a different dimension, causing a mismatch that manifested as a ValueError on those ranks.
Can I keep using the original 768‑dim embedding model with gpt4-large? No. The generator expects query embeddings that match its internal projection layer (1024‑dim). Using a lower‑dimensional encoder forces an implicit linear projection that degrades retrieval relevance.
Is it safe to rebuild the FAISS index during training? Rebuilding is safe if you pause retrieval calls or use a separate index version. In DDP, ensure only one rank performs the rebuild and broadcast the new index path to others.
How do I detect a silent dimension mismatch before it affects generation? Add a health‑check endpoint that compares retriever.embedding_dim with generator.embedding_dim. Emit a non‑zero exit code if they differ.
What monitoring metrics help catch this issue early? Track retrieval_hit_rate and retrieval_latency_ms. A sudden drop in hit rate combined with unchanged latency often signals an embedding incompatibility.