a golden statue of a man and a woman on a boat

RAG retrieval pipeline silent failures after embedding model update

by

Problem – Silent Retrieval Failures After Embedding Model Update

The Retrieval‑Augmented Generation (RAG) pipeline started returning empty result sets while query latency dropped dramatically. No HTTP error was propagated to callers, and the existing Prometheus alerts did not fire. The symptoms appeared during a rolling deployment of a new sentence‑transformer model that changed the embedding dimension from 768 to 1024. The ingestion workers began indexing 1024‑dimensional vectors, but the query service continued to issue searches against a FAISS index that expected 768 dimensions.

Typical log excerpts:


ERROR: Vector dimension mismatch – expected 768, got 1024 (FAISS index loader)
WARN: Embedding size inconsistency detected between ingestion and query services – skipping retrieval
INFO: Query latency dropped to 12 ms, results count = 0 – possible silent failure due to dimension error

Because the query path returned 0 hits without raising an exception, the API gateway reported a successful 200 response with an empty documents array. Prometheus recorded a lower latency series, which stayed under the alert thresholds defined in the alerting rules (Prometheus Documentation – Alerting Rules).

Root Cause – Uncoordinated Embedding Model Version Rollout

The deployment model used asynchronous workers that discover each other via Prometheus service discovery (Prometheus Documentation – Service Discovery for Dynamic Worker Pools). When the new model was rolled out:

  • Ingestion workers fetched the new sentence-transformers checkpoint, generated 1024‑dim vectors and wrote them to the FAISS index.
  • Query workers, still running the older model binary, loaded the existing index metadata (dimension = 768) and rejected any search request whose query vector did not match that size.
  • The rejection path logged a WARN but did not surface an error to the client, so the failure remained silent.
  • Prometheus scrapes from the query workers timed out intermittently during the rollout (Prometheus Documentation – Remote Write & High‑Throughput Scraping), causing the latency metric series to miss the spikes that would have triggered an alert.

This pattern matches several real incidents:

  • Uber Michelangelo (2023‑09) – ingestion wrote 768‑dim vectors while queries expected 512, leading to empty results and a 30 % latency dip.
  • Airbnb Search Service outage (2024‑02) – multilingual encoder upgrade caused dimension mismatch across shards.
  • Netflix Recommendation pipeline (2024‑05) – canary release of a 1024‑dim model broke queries that still used a 768‑dim index.

Debug – Investigation and Diagnostics

1. Verify Model Versions Across Workers


# On an ingestion worker
curl -s http://localhost:8501/v1/models/embedding | jq .model_version
# Expected: "v2.1.0" (1024‑dim)

# On a query worker
curl -s http://localhost:8502/v1/models/embedding | jq .model_version
# Expected: "v2.0.0" (768‑dim)

If the versions differ, the root cause is confirmed.

2. Inspect FAISS Index Metadata


python - <<'PY'
import faiss, json, pathlib
index_path = pathlib.Path("/data/faiss/index")
meta = json.loads((index_path / "meta.json").read_text())
print(f"Dimension stored: {meta['dimension']}")
PY

Output showing dimension: 768 while newly ingested vectors are 1024.

3. Examine Prometheus Scrape Health


# Prometheus UI query
up{job="rag-query-worker"} == 0
# Returns a series of missing scrapes during the rollout window.

Missing up series explains why latency alerts did not fire.

4. Search for Specific Log Patterns


journalctl -u rag-query-service -g "dimension mismatch"
# Example output:
# 2024-08-13T10:42:17.123Z WARN Embedding size inconsistency detected between ingestion and query services – skipping retrieval

Solution – Coordinated Model Rollout and Guardrails

1. Enforce Versioned Index Buckets

Separate indexes by model version to avoid cross‑dimension contamination.


# Ingestion configuration (before)
index_path: /data/faiss/index

# After – versioned path
index_path: /data/faiss/index_v2_1_0   # corresponds to 1024‑dim model

2. Deploy a Rolling Upgrade with a Compatibility Layer

Introduce a shim that re‑projects vectors to the target dimension when a mismatch is detected.


# compatibility_shim.py
import numpy as np

def project(vector, target_dim):
    if len(vector) == target_dim:
        return vector
    # Simple zero‑padding or PCA projection (example)
    if len(vector) < target_dim:
        return np.pad(vector, (0, target_dim - len(vector)), mode='constant')
    else:
        # Truncate or apply learned linear projection
        return vector[:target_dim]

3. Add a Prometheus Recording Rule for Dimension Errors


# recording_rules.yml (before)
# No rule for dimension mismatches

# After – new rule
groups:
  - name: rag_dimension_mismatch
    rules:
      - record: rag_vector_dimension_mismatch_total
        expr: sum(increase(rag_embedding_dimension_mismatch_total[5m]))

Now the metric rag_vector_dimension_mismatch_total can be used in alerting.

4. Update Alerting Rules to Trigger on Missing Latency Spikes and Dimension Errors


# alerts.yml (before)
- alert: HighQueryLatency
  expr: histogram_quantile(0.99, sum(rate(rag_query_latency_seconds_bucket[5m])) by (le))
  for: 2m
  labels:
    severity: warning

# After – add dimension mismatch alert
- alert: RagEmbeddingDimensionMismatch
  expr: rag_vector_dimension_mismatch_total > 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Embedding dimension mismatch detected in RAG pipeline"
    description: "Ingestion and query services are using different embedding dimensions. Check model versions and index paths."

5. Deploy with a Controlled Service Discovery Pause

Temporarily freeze the Prometheus target list during the rollout to avoid partial scrapes:


# prometheus.yml snippet
scrape_configs:
  - job_name: 'rag-query-worker'
    static_configs:
      - targets: ['query-worker-1:9100', 'query-worker-2:9100']
    relabel_configs:
      - source_labels: [__address__]
        regex: '.*'
        action: keep
    # Add a pause via a custom exporter that returns 0 until rollout completes

Verify – Confirming the Fix

  1. Validate that both ingestion and query workers report the same model_version via the health endpoint.
  2. Run an end‑to‑end query and check that the documents array contains expected hits.
  3. Observe Prometheus metrics: 
    
rag_vector_dimension_mismatch_total == 0
rate(rag_query_latency_seconds_sum[5m]) returns to baseline (~35 ms)
  4. Check FAISS index metadata for the new dimension (e.g., 1024).
  5. Confirm that the new alert fires if a mismatch is artificially introduced (e.g., downgrade a single worker).

Prevent – Best Practices and Guardrails

  • Version‑locked index schemas: Store the embedding dimension in the index metadata and reject writes that do not match.
  • Metric guards: Export a custom rag_embedding_dimension_mismatch_total counter from each worker and record it via a Prometheus recording_rule.
  • Zero‑downtime rollout pattern: Use blue‑green deployment for the embedding model, switch traffic only after both ingestion and query services confirm the same version.
  • Scrape health checks: Configure scrape_timeout and honor_labels to ensure missing scrapes are visible (Prometheus Documentation – Remote Write & High‑Throughput Scraping).
  • Automated integration test: After each model bump, run a test that indexes a known document and queries it, asserting that the result count > 0.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

  • Why does the query latency drop instead of increase? The query worker discards the vector before performing the ANN search, resulting in an immediate return of zero hits. The latency measurement only captures the request handling time, not the index lookup.
  • Can I rely on HTTP status codes to detect dimension mismatches? No. The current client library treats a missing result set as a successful response. Adding explicit error codes in the API contract is recommended.
  • How do I expose the dimension mismatch metric from a custom exporter? Increment a counter named rag_embedding_dimension_mismatch_total each time the worker logs the mismatch. Prometheus will scrape it like any other metric.
  • What is the safest way to roll out a new embedding model in a high‑RPS environment? Deploy the new model in a separate process group, keep the old index immutable, and use a feature flag to switch ingestion and query services only after both groups report the same version.
  • Why didn’t the existing alert fire? The alert relied on latency thresholds. Because the mismatch caused the query path to exit early, latency decreased, staying below the alert condition. Missing scrape data during the rollout also prevented the alert from evaluating correctly (Prometheus Documentation – Alerting Rules).