LangChain Hybrid Search Relevance Mismatch After Autoscaling Compute Instances
Problem
In a production LangChain service that uses hybrid search (vector similarity + keyword matching), the returned relevance scores become inconsistent after the compute layer scales out. Documents that should be highly relevant are ranked below unrelated entries, causing downstream applications to surface incorrect answers.
Root Cause
The anomaly originates from two interacting factors:
- Stateless vector store client caching: When a new compute instance starts, it creates a fresh
FAISS/Milvusclient without sharing thedocstoremetadata cache that holdsdoc_id → metadatamappings. The hybrid retriever then falls back to a defaultscore=0.0for missing metadata, skewing the combined score. - Non‑deterministic
lambdaweighting: TheHybridRetrievercomputesfinal_score = alpha * vector_score + (1 - alpha) * keyword_score. Autoscaling triggers a race condition where the environment variableHYBRID_ALPHAis read at module import time on each instance. Some instances load the default0.5while others respect the overridden0.8set in the deployment manifest, leading to divergent score calculations across the fleet.
Debug
Below is a step‑by‑step investigation that isolates the problem.
1. Observe the symptom in logs
2026-11-13 10:12:04,321 INFO hybrid_retriever.py:112 - Query: "data lake best practices"
2026-11-13 10:12:04,322 DEBUG hybrid_retriever.py:115 - Vector scores: [(doc_42, 0.92), (doc_7, 0.87), (doc_19, 0.85)]
2026-11-13 10:12:04,323 DEBUG hybrid_retriever.py:118 - Keyword scores: [(doc_7, 0.78), (doc_19, 0.80), (doc_42, 0.60)]
2026-11-13 10:12:04,324 INFO hybrid_retriever.py:124 - Combined scores (alpha=0.5): [(doc_42, 0.76), (doc_7, 0.825), (doc_19, 0.825)]
2026-11-13 10:12:04,325 WARN request_handler.py:45 - Result ordering mismatch: expected doc_42 top, got doc_7
After scaling out, the same query logs on a newly created instance:
2026-11-13 10:15:27,089 INFO hybrid_retriever.py:112 - Query: "data lake best practices"
2026-11-13 10:15:27,090 DEBUG hybrid_retriever.py:115 - Vector scores: [(doc_42, 0.92), (doc_7, 0.87), (doc_19, 0.85)]
2026-11-13 10:15:27,091 DEBUG hybrid_retriever.py:118 - Keyword scores: [(doc_7, 0.78), (doc_19, 0.80), (doc_42, 0.60)]
2026-11-13 10:15:27,092 INFO hybrid_retriever.py:124 - Combined scores (alpha=0.8): [(doc_42, 0.864), (doc_7, 0.822), (doc_19, 0.828)]
2026-11-13 10:15:27,093 INFO request_handler.py:45 - Result ordering correct.
The differing alpha values point to inconsistent configuration loading.
2. Verify environment variable propagation
$ kubectl exec -it $(kubectl get pod -l app=langchain -o jsonpath="{.items[0].metadata.name}") -- printenv | grep HYBRID_ALPHA
HYBRID_ALPHA=0.5
$ kubectl exec -it $(kubectl get pod -l app=langchain -o jsonpath="{.items[1].metadata.name}") -- printenv | grep HYBRID_ALPHA
HYBRID_ALPHA=0.8
3. Check vector store client cache state
>>> from langchain.vectorstores import FAISS
>>> store = FAISS.load_local("index_path")
>>> store.docstore._search_index # internal cache
{...} # populated on warm instance, empty on fresh instance
Missing entries cause the keyword retriever to return score=0.0 for those documents, which reduces the combined score.
Solution
The fix consists of two parts: enforce a single source of truth for alpha and make the vector store client share metadata across autoscaled instances.
1. Centralise configuration with a ConfigMap and load at runtime
Replace the module‑level read with a function call that queries the ConfigMap on each request.
Before
# hybrid_retriever.py
import os
ALPHA = float(os.getenv("HYBRID_ALPHA", "0.5"))
def _combine_scores(v_score, k_score):
return ALPHA * v_score + (1 - ALPHA) * k_score
After
# hybrid_retriever.py
import os
from kubernetes import client, config
def _load_alpha():
# Load once per process, but fallback to live ConfigMap if missing
try:
return float(os.getenv("HYBRID_ALPHA"))
except (TypeError, ValueError):
config.load_incluster_config()
v1 = client.CoreV1Api()
cm = v1.read_namespaced_config_map("langchain-config", "default")
return float(cm.data.get("HYBRID_ALPHA", "0.5"))
ALPHA = _load_alpha()
def _combine_scores(v_score, k_score):
return ALPHA * v_score + (1 - ALPHA) * k_score
Deploy the ConfigMap with the desired value and ensure the pod spec mounts it as environment variables.
2. Use a shared Redis‑backed metadata cache
Configure the vector store to persist docstore entries in Redis, which is reachable from every compute node.
Before
# vector_store_setup.py
vector_store = FAISS.from_documents(docs, embeddings)
vector_store.save_local("index_path")
# No external metadata store
After
# vector_store_setup.py
import redis
from langchain.docstore import InMemoryDocstore
from langchain.vectorstores import FAISS
redis_client = redis.Redis(host="redis-service", port=6379, db=0)
# Load or create docstore backed by Redis
if redis_client.exists("docstore"):
docstore = InMemoryDocstore.from_json(redis_client.get("docstore"))
else:
docstore = InMemoryDocstore({doc.metadata["id"]: doc for doc in docs})
redis_client.set("docstore", docstore.to_json())
vector_store = FAISS.from_documents(docs, embeddings, docstore=docstore)
vector_store.save_local("index_path")
All autoscaled instances now read the same metadata, eliminating the score=0.0 fallback.
Verify
After applying the changes, run the same query on multiple pods and compare the combined scores.
$ for pod in $(kubectl get pods -l app=langchain -o name); do
echo "=== $pod ==="
kubectl exec $pod -- python -c "
from hybrid_retriever import retrieve
print(retrieve('data lake best practices'))"
done
Expected unified output (example):
=== pod/langchain-0 ===
[('doc_42', 0.864), ('doc_19', 0.828), ('doc_7', 0.822)]
=== pod/langchain-1 ===
[('doc_42', 0.864), ('doc_19', 0.828), ('doc_7', 0.822)]
Additionally, monitor the metric hybrid_retriever_combined_score_stddev – it should drop to near zero across the fleet.
Prevent
- Store all mutable hyper‑parameters (e.g.,
HYBRID_ALPHA) in a single ConfigMap or feature flag service and reload on pod restart. - Externalise any stateful caches (docstore, keyword index) to a shared datastore (Redis, DynamoDB, Cloud Memorystore) rather than relying on in‑process memory.
- Include a health‑check endpoint that validates that
vector_scoreandkeyword_scoreare both present for a sample query. - Add a Prometheus alert on
hybrid_retriever_combined_score_stddev > 0.05to catch future divergence early. - When scaling, ensure the pod startup script runs a short “warm‑up” routine that pre‑loads the docstore cache.
Related Topic Hub: RAG Systems Troubleshooting Hub
FAQ
- Why does the relevance mismatch appear only after autoscaling? New instances start without the in‑process docstore cache and may load a different
HYBRID_ALPHAvalue if the environment variable is not propagated uniformly. - Can I keep the original in‑process docstore and still avoid the issue? Only if you implement a side‑car that synchronises the cache across pods, which adds complexity. Using a shared external cache is simpler and more reliable.
- What if I need per‑tenant
alphavalues? Move the weighting logic into the request handler and fetch the tenant‑specific value from a configuration service (e.g., Consul, etcd) at request time. - Does this fix affect query latency? Adding Redis lookups adds ~1‑2 ms per request, which is negligible compared to vector similarity computation. Cache warm‑up may add a few seconds on pod start‑up.
- How do I test the hybrid scoring logic locally? Run the retriever with a mocked docstore that returns deterministic metadata and set
HYBRID_ALPHAexplicitly before importing the module.