Nginx routing to old embedding model causing RAG retrieval mismatch

Problem Description

In a cloud‑managed deployment the Retrieval‑Augmented Generation (RAG) service began returning unrelated documents. The downstream application logged an “Embedding dimension mismatch” error and occasional 502 Bad Gateway responses such as:

2024/09/20 12:34:56 [error] 27#27: *1023 upstream prematurely closed connection while reading response header from upstream, client: 10.2.3.45, server: rag-api.example.com, request: "POST /embed HTTP/1.1", upstream: "http://embedding-v1:8000/embed", host: "rag-api.example.com"

Metrics showed a 15‑20 % drop in retrieval relevance (Mean Reciprocal Rank fell from 0.78 to 0.42) after a recent model rollout. The symptom set matches the “old embedding model routing” pattern described in the LangChain GitHub issue #8425 and the LlamaIndex discussion #317.

Root Cause Analysis

The Nginx reverse‑proxy was still directing /embed requests to the embedding‑v1 upstream, which hosts the deprecated model (e.g., 384‑dim vectors). The new model (embedding‑v2, 768‑dim) is exposed on a different service name (embedding‑v2:8000) and is selected by the application via a Model-Version header.

• Configuration drift: The upstream block for embedding_backend was edited in a ConfigMap, but the Nginx Ingress controller never reloaded, leaving the old DNS entry cached.
• Map directive misuse: A map $http_model_version $backend block incorrectly fell back to the default upstream when the header value was missing or malformed, causing the request to hit the stale backend.
• Cache key collision: The default proxy_cache_key omitted the Model-Version header, so cached responses from the old model were served to queries intended for the new model, as documented in the Nginx caching guide.

These factors together produced the observed dimension mismatch and relevance degradation.

Investigation and Debugging

1. Verify upstream resolution

# Resolve the service name used in the upstream block
nslookup embedding-v1
# Expected: points to the old pod IPs
nslookup embedding-v2
# Expected: points to the new pod IPs

If both names resolve to the same IP, the DNS cache inside Nginx is stale.

2. Inspect Nginx configuration

# Dump the effective config from the running container
nginx -T

Key sections to locate:

upstream embedding_backend {
    server embedding-v1:8000;   # <-- should be updated to embedding-v2
}

map $http_model_version $backend {
    default         embedding_backend;
    "v2"            embedding-v2:8000;
}

3. Check request headers and cache key

curl -i -H "Model-Version: v2" https://rag-api.example.com/embed -d '{"text":"sample"}'

Expected response header:

X-Proxy-Cache: MISS

If X-Proxy-Cache: HIT appears and the body contains 384‑dim vectors, the cache key is not varying by Model-Version.

4. Review Nginx error log for upstream errors

journalctl -u nginx -f | grep -E "upstream|502|504"

Typical entries from the incident:

2024/09/20 12:34:56 [error] 27#27: *1023 upstream prematurely closed connection while reading response header from upstream, client: 10.2.3.45, server: rag-api.example.com, request: "POST /embed HTTP/1.1", upstream: "http://embedding-v1:8000/embed", host: "rag-api.example.com"
2024/09/20 12:35:02 [error] 27#27: *1045 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.2.3.45, server: rag-api.example.com, request: "POST /embed HTTP/1.1", upstream: "http://embedding-v1:8000/embed", host: "rag-api.example.com"

Resolution

Configuration update

Replace the stale upstream definition and make the routing map explicit. Also include the Model-Version header in the cache key.

Before (faulty config snippet):

upstream embedding_backend {
    server embedding-v1:8000;
}

map $http_model_version $backend {
    default         embedding_backend;
    "v2"            embedding-v2:8000;
}

proxy_cache_key "$scheme$request_method$host$request_uri";

After (corrected config):

# Updated upstream points to the new model service
upstream embedding_v2 {
    server embedding-v2:8000;
}

# Explicit map – any unknown version falls back to v2 (current production)
map $http_model_version $backend {
    default         embedding_v2;
    "v1"            embedding-v1:8000;
    "v2"            embedding_v2;
}

# Cache key now varies by model version to avoid stale responses
proxy_cache_key "$scheme$request_method$host$request_uri$model_version:$http_model_version";

# Preserve the header when proxying
proxy_set_header Model-Version $http_model_version;

Apply changes and reload

# For a Kubernetes Ingress controller
kubectl apply -f nginx-configmap.yaml
kubectl rollout restart deployment/nginx-ingress-controller

# Verify reload
kubectl logs -l app=nginx-ingress -c nginx --tail=20 | grep "configuration file /etc/nginx/nginx.conf test is successful"

Validation

• Functional test: Send a request with Model-Version: v2 and verify the response vector length is 768.
• Cache behavior: Repeat the same request; the first should be MISS, subsequent should be HIT with the same 768‑dim payload.
• Metrics: Observe the RAG relevance metric (MRR) returning to pre‑incident levels (≈0.78) within a few minutes.
• Log sanity check: No longer see “upstream prematurely closed connection” or “upstream timed out” entries for /embed.

Prevention and Best Practices

Practice	Why it helps
Include version‑specific headers in proxy_cache_key	Avoids serving stale embeddings from a previous model.
Use map with an explicit default that points to the current production model	Prevents accidental fallback to deprecated services.
Automate Nginx reload on ConfigMap or Ingress changes	Ensures DNS updates and upstream modifications take effect immediately (see Nginx docs on proxy_pass).
Monitor $upstream_addr and $proxy_host variables	Detects unexpected upstream routing early via Prometheus alerts.
Validate embedding dimensions at the API gateway	Early rejection of mismatched vectors prevents downstream pipeline failures.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

1. Why does the 502 error appear only for some queries?
   The error occurs when the request includes a Model-Version header that the map directive does not match, causing Nginx to fall back to the default upstream (embedding‑v1). Queries without the header hit the correct backend, leading to intermittent failures.
2. How can I confirm which upstream address handled a request?
   Enable logging of the $upstream_addr variable in the access log:

   log_format upstream '$remote_addr - $remote_user [$time_local] "$request" '
                           '$status $body_bytes_sent "$http_referer" '
                           '"$http_user_agent" upstream=$upstream_addr';

3. What if the DNS entry for the new model service changes frequently?
   Use the resolver directive with a low valid time (e.g., resolver 10.96.0.10 valid=30s;) so Nginx re‑resolves the service name on each request.
4. Can I avoid cache key collisions without modifying proxy_cache_key?
   Yes, you can set proxy_cache_use_stale error timeout updating; and purge the cache after each model rollout using curl -X PURGE http://nginx/cache/purge, but including the version header in the key is the more robust approach.
5. Is the if directive safe for routing based on headers?
   The Nginx documentation warns that if inside location blocks can lead to unpredictable behavior. Prefer the map module for header‑based routing, as demonstrated in the solution.