Qwen RAG Query Decomposition Failure After Embedding Model Change

Q: 2. Make decomposition robust to multi‑chunk embeddings

Modify decompose_query to aggregate embeddings from all chunks before passing them to the LLM that generates sub‑queries.

In this article

Table of Contents

Problem

The RAG (Retrieval‑Augmented Generation) pipeline in the Qwen service started returning incomplete answers after the embedding model was upgraded from text‑embedding‑ada‑002 to qwen‑embedding‑v2. The most visible symptom was that the query decomposition stage produced either malformed sub‑queries or no sub‑queries at all, causing downstream vector search to return empty result sets. The issue manifested only in the event‑driven path where messages are consumed from the rag‑requests Kafka topic and processed asynchronously.

Typical log excerpt:


2026-08-22 14:03:12,845 INFO  [rag-worker-7] DecomposeQuery - Starting decomposition for request_id=42f1c9e3
2026-08-22 14:03:12,851 WARN  [rag-worker-7] DecomposeQuery - Embedding model returned empty vector for input chunk.
2026-08-22 14:03:12,852 ERROR DecomposeQuery - Failed to generate sub‑queries: IndexError: list index out of range
2026-08-22 14:03:12,853 INFO  [rag-worker-7] RequestHandler - Sending failure response for request_id=42f1c9e3

Impact:

End‑users receive truncated or “I don’t know” answers.
Service latency spikes because the worker retries the decomposition step.
Alerting threshold on rag.decomposition.errors breaches.

Root Cause

The new embedding model changed two critical behaviours:

Vector dimensionality: text‑embedding‑ada‑002 produced 1536‑dimensional vectors, whereas qwen‑embedding‑v2 returns 1024 dimensions. The downstream faiss index was still initialised for 1536 dimensions, causing the IndexFlatL2 constructor to reject incoming vectors and return an empty list.
Chunk‑level tokenisation: The model now splits input text into sub‑chunks based on a different token limit (256 vs 512 tokens). The decomposition logic expects exactly one embedding per user query; when multiple chunks are emitted, the code that selects the first embedding (e.g., embeddings[0]) throws an IndexError because the list is empty after the failed index insertion.

Both changes broke the implicit contract between the EmbeddingService and the DecomposeQuery microservice, leading to malformed sub‑queries and retrieval failures.

Debug

Step‑by‑step investigation performed in the production environment:

Confirm embedding dimensions:

curl -X POST http://embedding-service.local/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen-embedding-v2","input":"What is the capital of France?"}'

Expected output (new model):

{
  "data": [{"embedding": [0.12, -0.03, ... 1024 values ...]}],
  "model": "qwen-embedding-v2",
  "usage": {"prompt_tokens": 12, "total_tokens": 12}
}

Inspect FAISS index initialisation (container rag-indexer):

docker exec -it rag-indexer python - <<'PY'
import faiss, json, os
cfg = json.load(open('/app/config.json'))
print('index dim:', cfg['faiss']['dimension'])
PY

Output showed dimension: 1536, confirming mismatch.

Reproduce the IndexError by invoking the decomposition function locally:

python - <<'PY'
from rag.decompose import decompose_query
try:
    decompose_query("Explain quantum tunnelling and its applications.")
except Exception as e:
    print('Exception:', e)
PY

Result:

Exception: list index out of range

Check tokenisation behaviour using the model’s tokenizer:

python - <<'PY'
from qwen_tokenizer import QwenTokenizer
tok = QwenTokenizer.from_pretrained('qwen-embedding-v2')
chunks = tok.chunk("Explain quantum tunnelling and its applications.", max_tokens=256)
print('chunks:', len(chunks))
PY

Output: chunks: 2 (previous model would have returned chunks: 1).

Solution

The fix consists of three coordinated changes:

1. Align FAISS index dimension with the new embedding model

Update the index initialisation configuration and re‑create the index.

# config.json (before)
{
  "faiss": {
    "dimension": 1536,
    "index_type": "FlatL2"
  }
}

# config.json (after)
{
  "faiss": {
    "dimension": 1024,
    "index_type": "FlatL2"
  }
}

Re‑create the index (run once during deployment):

docker exec rag-indexer python - <<'PY'
import faiss, json, os
cfg = json.load(open('/app/config.json'))
dim = cfg['faiss']['dimension']
index = faiss.IndexFlatL2(dim)
faiss.write_index(index, '/data/faiss.index')
PY

2. Make decomposition robust to multi‑chunk embeddings

Modify decompose_query to aggregate embeddings from all chunks before passing them to the LLM that generates sub‑queries.

# rag/decompose.py (before)
def decompose_query(user_query):
    embedding = embedding_service.encode(user_query)[0]  # assumes single vector
    sub_queries = llm.generate_subqueries(embedding)
    return sub_queries

# rag/decompose.py (after)
def decompose_query(user_query):
    # Tokenise and embed each chunk
    chunks = tokenizer.chunk(user_query, max_tokens=256)
    embeddings = embedding_service.encode_many(chunks)  # returns list[vector]
    # Concatenate embeddings (e.g., mean pooling)
    aggregated = np.mean(np.stack(embeddings), axis=0)
    sub_queries = llm.generate_subqueries(aggregated)
    return sub_queries

Key changes:

Use encode_many to handle variable chunk count.
Apply mean pooling to produce a fixed‑size vector compatible with the LLM.
Add defensive checks for empty embedding lists, returning a clear error instead of an uncaught exception.

3. Deploy updated services with versioned Docker images

Update the CI/CD pipeline to tag the new images:


docker build -t registry.internal/qwen-rag:decompose-v2 .
docker push registry.internal/qwen-rag:decompose-v2

Roll out using a blue‑green strategy to avoid downtime.

Verify

After deployment, run the following validation suite:

Unit test for multi‑chunk handling:

pytest tests/test_decompose_multi_chunk.py -q

Expected output:

..                                                                  [100%]
2 passed in 0.34s

Integration test against the live index:

curl -X POST http://rag-service.local/v1/query \
  -H "Content-Type: application/json" \
  -d '{"query":"Describe the impact of climate change on marine biodiversity"}'

Response should contain a sub_queries array with at least three items and a non‑empty retrieved_documents field.

Metrics check (Prometheus):

# HELP rag_decomposition_errors Total number of decomposition errors
# TYPE rag_decomposition_errors counter
rag_decomposition_errors 0
# HELP rag_subqueries_generated Total sub‑queries generated per request
# TYPE rag_subqueries_generated gauge
rag_subqueries_generated{request_id="a1b2c3"} 4

Log sanity – ensure no IndexError entries appear in the last 15 minutes:
```
journalctl -u rag-worker -u rag-indexer --since "15 minutes ago" | grep -i error
```
Output should be empty.

Prevent

To avoid similar regressions when swapping models:

Schema validation: Store model metadata (dimension, token limit) in a central model‑registry service and have each microservice validate compatibility at start‑up.
Automated contract tests: Include a CI job that fetches a sample embedding from the configured model and asserts that the downstream index dimension matches.
Versioned configuration: Keep separate config files per model version (e.g., faiss_1536.json, faiss_1024.json) and reference them via an environment variable.
Graceful degradation: If embedding_service.encode_many returns an empty list, return a structured error (HTTP 422) instead of propagating an exception.
Observability: Add a Prometheus gauge embedding_vector_dimensionality and alert when it diverges from the expected index dimension.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why did the decomposition work with the old model but not with the new one?
The old model produced a single 1536‑dimensional vector per query, matching the FAISS index. The new model emits multiple 1024‑dimensional vectors, breaking the assumption of a single embedding and causing dimension mismatches.
Can I keep the old FAISS index and just pad the new vectors?
Padding would increase vector size to 1536 but would degrade similarity search quality because the added dimensions are zero. Re‑creating the index with the correct dimension is the recommended approach.
What if my downstream LLM expects a specific embedding size?
Aggregate the chunk embeddings (mean, max, or weighted pooling) to produce a fixed‑size vector that matches the LLM’s expectation, as shown in the solution.
How do I detect future model changes automatically?
Implement a health‑check endpoint in the embedding service that returns its vector_dim and max_tokens. The RAG orchestrator should compare these values against its configuration at start‑up and refuse to start if they differ.
Is there a performance impact when encoding multiple chunks?
Yes, encoding overhead grows roughly linearly with chunk count. Mitigate by increasing the max token limit if the model permits, or by caching embeddings for repeated queries.