LangChain hybrid search relevance mismatch after autoscaling compute instances

LangChain Hybrid Search Relevance Mismatch After Autoscaling Compute Instances

Problem

In a production LangChain service that uses hybrid search (vector similarity + keyword matching), the returned relevance scores become inconsistent after the compute layer scales out. Documents that should be highly relevant are ranked below unrelated entries, causing downstream applications to surface incorrect answers.

Root Cause

The anomaly originates from two interacting factors:

  1. Stateless vector store client caching: When a new compute instance starts, it creates a fresh FAISS/Milvus client without sharing the docstore metadata cache that holds doc_id → metadata mappings. The hybrid retriever then falls back to a default score=0.0 for missing metadata, skewing the combined score.
  2. Non‑deterministic lambda weighting: The HybridRetriever computes final_score = alpha * vector_score + (1 - alpha) * keyword_score. Autoscaling triggers a race condition where the environment variable HYBRID_ALPHA is read at module import time on each instance. Some instances load the default 0.5 while others respect the overridden 0.8 set in the deployment manifest, leading to divergent score calculations across the fleet.

Debug

Below is a step‑by‑step investigation that isolates the problem.

1. Observe the symptom in logs


2026-11-13 10:12:04,321 INFO  hybrid_retriever.py:112 - Query: "data lake best practices"
2026-11-13 10:12:04,322 DEBUG hybrid_retriever.py:115 - Vector scores: [(doc_42, 0.92), (doc_7, 0.87), (doc_19, 0.85)]
2026-11-13 10:12:04,323 DEBUG hybrid_retriever.py:118 - Keyword scores: [(doc_7, 0.78), (doc_19, 0.80), (doc_42, 0.60)]
2026-11-13 10:12:04,324 INFO  hybrid_retriever.py:124 - Combined scores (alpha=0.5): [(doc_42, 0.76), (doc_7, 0.825), (doc_19, 0.825)]
2026-11-13 10:12:04,325 WARN  request_handler.py:45 - Result ordering mismatch: expected doc_42 top, got doc_7

After scaling out, the same query logs on a newly created instance:


2026-11-13 10:15:27,089 INFO  hybrid_retriever.py:112 - Query: "data lake best practices"
2026-11-13 10:15:27,090 DEBUG hybrid_retriever.py:115 - Vector scores: [(doc_42, 0.92), (doc_7, 0.87), (doc_19, 0.85)]
2026-11-13 10:15:27,091 DEBUG hybrid_retriever.py:118 - Keyword scores: [(doc_7, 0.78), (doc_19, 0.80), (doc_42, 0.60)]
2026-11-13 10:15:27,092 INFO  hybrid_retriever.py:124 - Combined scores (alpha=0.8): [(doc_42, 0.864), (doc_7, 0.822), (doc_19, 0.828)]
2026-11-13 10:15:27,093 INFO  request_handler.py:45 - Result ordering correct.

The differing alpha values point to inconsistent configuration loading.

2. Verify environment variable propagation


$ kubectl exec -it $(kubectl get pod -l app=langchain -o jsonpath="{.items[0].metadata.name}") -- printenv | grep HYBRID_ALPHA
HYBRID_ALPHA=0.5

$ kubectl exec -it $(kubectl get pod -l app=langchain -o jsonpath="{.items[1].metadata.name}") -- printenv | grep HYBRID_ALPHA
HYBRID_ALPHA=0.8

3. Check vector store client cache state


>>> from langchain.vectorstores import FAISS
>>> store = FAISS.load_local("index_path")
>>> store.docstore._search_index  # internal cache
{...}  # populated on warm instance, empty on fresh instance

Missing entries cause the keyword retriever to return score=0.0 for those documents, which reduces the combined score.

Solution

The fix consists of two parts: enforce a single source of truth for alpha and make the vector store client share metadata across autoscaled instances.

1. Centralise configuration with a ConfigMap and load at runtime

Replace the module‑level read with a function call that queries the ConfigMap on each request.

Before


# hybrid_retriever.py
import os
ALPHA = float(os.getenv("HYBRID_ALPHA", "0.5"))

def _combine_scores(v_score, k_score):
    return ALPHA * v_score + (1 - ALPHA) * k_score

After


# hybrid_retriever.py
import os
from kubernetes import client, config

def _load_alpha():
    # Load once per process, but fallback to live ConfigMap if missing
    try:
        return float(os.getenv("HYBRID_ALPHA"))
    except (TypeError, ValueError):
        config.load_incluster_config()
        v1 = client.CoreV1Api()
        cm = v1.read_namespaced_config_map("langchain-config", "default")
        return float(cm.data.get("HYBRID_ALPHA", "0.5"))

ALPHA = _load_alpha()

def _combine_scores(v_score, k_score):
    return ALPHA * v_score + (1 - ALPHA) * k_score

Deploy the ConfigMap with the desired value and ensure the pod spec mounts it as environment variables.

2. Use a shared Redis‑backed metadata cache

Configure the vector store to persist docstore entries in Redis, which is reachable from every compute node.

Before


# vector_store_setup.py
vector_store = FAISS.from_documents(docs, embeddings)
vector_store.save_local("index_path")
# No external metadata store

After


# vector_store_setup.py
import redis
from langchain.docstore import InMemoryDocstore
from langchain.vectorstores import FAISS

redis_client = redis.Redis(host="redis-service", port=6379, db=0)

# Load or create docstore backed by Redis
if redis_client.exists("docstore"):
    docstore = InMemoryDocstore.from_json(redis_client.get("docstore"))
else:
    docstore = InMemoryDocstore({doc.metadata["id"]: doc for doc in docs})
    redis_client.set("docstore", docstore.to_json())

vector_store = FAISS.from_documents(docs, embeddings, docstore=docstore)
vector_store.save_local("index_path")

All autoscaled instances now read the same metadata, eliminating the score=0.0 fallback.

Verify

After applying the changes, run the same query on multiple pods and compare the combined scores.


$ for pod in $(kubectl get pods -l app=langchain -o name); do
    echo "=== $pod ==="
    kubectl exec $pod -- python -c "
from hybrid_retriever import retrieve
print(retrieve('data lake best practices'))"
done

Expected unified output (example):


=== pod/langchain-0 ===
[('doc_42', 0.864), ('doc_19', 0.828), ('doc_7', 0.822)]
=== pod/langchain-1 ===
[('doc_42', 0.864), ('doc_19', 0.828), ('doc_7', 0.822)]

Additionally, monitor the metric hybrid_retriever_combined_score_stddev – it should drop to near zero across the fleet.

Prevent

  • Store all mutable hyper‑parameters (e.g., HYBRID_ALPHA) in a single ConfigMap or feature flag service and reload on pod restart.
  • Externalise any stateful caches (docstore, keyword index) to a shared datastore (Redis, DynamoDB, Cloud Memorystore) rather than relying on in‑process memory.
  • Include a health‑check endpoint that validates that vector_score and keyword_score are both present for a sample query.
  • Add a Prometheus alert on hybrid_retriever_combined_score_stddev > 0.05 to catch future divergence early.
  • When scaling, ensure the pod startup script runs a short “warm‑up” routine that pre‑loads the docstore cache.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

  1. Why does the relevance mismatch appear only after autoscaling? New instances start without the in‑process docstore cache and may load a different HYBRID_ALPHA value if the environment variable is not propagated uniformly.
  2. Can I keep the original in‑process docstore and still avoid the issue? Only if you implement a side‑car that synchronises the cache across pods, which adds complexity. Using a shared external cache is simpler and more reliable.
  3. What if I need per‑tenant alpha values? Move the weighting logic into the request handler and fetch the tenant‑specific value from a configuration service (e.g., Consul, etcd) at request time.
  4. Does this fix affect query latency? Adding Redis lookups adds ~1‑2 ms per request, which is negligible compared to vector similarity computation. Cache warm‑up may add a few seconds on pod start‑up.
  5. How do I test the hybrid scoring logic locally? Run the retriever with a mocked docstore that returns deterministic metadata and set HYBRID_ALPHA explicitly before importing the module.