Mistral AI RAG Retrieval Dominance After Rolling Update

Q: Problem Description – Retrieval Dominance After a Rolling Update

A production Mistral AI Retrieval‑Augmented Generation (RAG) pipeline uses a hybrid search that combines:

Q: Root Cause Analysis

The hybrid search configuration is stored in a HybridSearchConfig object that lives in a pod‑local file system. According to the Mistral AI RAG Guide, the weights must be synchronized to a shared storage (e.g., ConfigMap, S3, or the SDK persistence endpoint) before a pod is terminated.

Q: 1. Inspect the running configuration

# Query the hybrid endpoint for the active weights curl -s -X POST https://api.mistral.ai/v1/search/hybrid \ -H "Authorization: Bearer $TOKEN" \ -d '{"model":"rag-retriever","query":"sample"}' | jq '.weights' # Expected output (correctly persisted) { "retrieval_weight": 0.6, "rerank_weight": 0.4 } 2. Check pod logs for weight‑sync errors kubectl logs -l app=mistral-rag -c rag-worker --since=5m | grep -E "HybridSearchConfigError|WeightSyncFailed" Typical output during a failed rollout:

Q: 3. Verify persistence endpoint health

# Example: health check for the SDK persistence service curl -f http://weight-sync-service:8080/healthz && echo "OK" If this command fails, the rolling update will lose weight state.

Q: 4. Examine the rolling update strategy

# Show the Deployment rollout strategy kubectl get deployment mistral-rag -o yaml | grep -A5 strategy Ensure the strategy includes maxUnavailable: 0 and a preStop hook that runs the weight export script.

Q: 1. Add a preStop hook that exports the current weights

apiVersion: apps/v1 kind: Deployment metadata: name: mistral-rag spec: template: spec: containers: - name: rag-worker image: mistral/rag:{{ .Values.imageTag }} lifecycle: preStop: exec: command: ["/bin/sh", "-c", "/app/export_weights.sh"] env: - name: WEIGHT_SYNC_ENDPOINT value: "http://weight-sync-service:8080/weights" export_weights.sh:

Q: 2. Modify the startup script to pull persisted weights

# startup.sh #!/bin/sh set -e WEIGHT_ENDPOINT="http://weight-sync-service:8080/weights" if curl -sf "$WEIGHT_ENDPOINT" > /tmp/weights.json; then python -

In this article

Table of Contents

Problem Description – Retrieval Dominance After a Rolling Update

A production Mistral AI Retrieval‑Augmented Generation (RAG) pipeline uses a hybrid search that combines:

Vector‑based retrieval (weight = retrieval_weight)
Transformer‑based reranking (weight = rerank_weight)

During a rolling update that introduces a new reranker model version, operators observed:

Hybrid search results suddenly favor raw retrieval scores.
Relevance metrics dropped 30‑45 % (see fintech incident 2024‑07).

Logs contain errors such as:

HybridSearchConfigError: Rerank weight not found in configuration
WeightSyncFailed: Unable to persist rerank_weight to shared storage
Score imbalance detected: retrieval_score >> rerank_score

In extreme cases the rerank_weight resets to the default 0.2 or even 0, effectively disabling reranking.

Root Cause Analysis

The hybrid search configuration is stored in a HybridSearchConfig object that lives in a pod‑local file system. According to the Mistral AI RAG Guide, the weights must be synchronized to a shared storage (e.g., ConfigMap, S3, or the SDK persistence endpoint) before a pod is terminated.

During a rolling update the following sequence occurs:

Old pods receive a SIGTERM and begin shutdown.
The SDK attempts to write the current retrieval_weight and rerank_weight to the persistence endpoint.
If the endpoint is unreachable (network glitch, permission change) the SDK logs WeightSyncFailed and proceeds with termination.
New pods start with a fresh HybridSearchConfig instance, falling back to the hard‑coded defaults (retrieval_weight=0.8, rerank_weight=0.2) as described in the API reference.
Because the new reranker model may have a higher baseline score, the unchanged retrieval weight now dominates the combined score, producing the observed retrieval‑only results.

Community reports (GitHub #842, Stack Overflow 78543219) confirm that missing the --preserve-weights flag or not exporting the ConfigMap in a pre_stop hook leads to exactly this drift.

Investigation and Debugging

Follow these steps to reproduce and isolate the issue.

1. Inspect the running configuration

# Query the hybrid endpoint for the active weights
curl -s -X POST https://api.mistral.ai/v1/search/hybrid \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"model":"rag-retriever","query":"sample"}' | jq '.weights'

# Expected output (correctly persisted)
{
  "retrieval_weight": 0.6,
  "rerank_weight": 0.4
}

2. Check pod logs for weight‑sync errors

kubectl logs -l app=mistral-rag -c rag-worker --since=5m | grep -E "HybridSearchConfigError|WeightSyncFailed"

Typical output during a failed rollout:

2024-06-20T14:32:11Z WARN HybridSearchConfigError: Rerank weight not found in configuration
2024-06-20T14:32:12Z ERROR WeightSyncFailed: Unable to persist rerank_weight to shared storage (connection refused)

3. Verify persistence endpoint health

# Example: health check for the SDK persistence service
curl -f http://weight-sync-service:8080/healthz && echo "OK"

If this command fails, the rolling update will lose weight state.

4. Examine the rolling update strategy

# Show the Deployment rollout strategy
kubectl get deployment mistral-rag -o yaml | grep -A5 strategy

Ensure the strategy includes maxUnavailable: 0 and a preStop hook that runs the weight export script.

5. Reproduce the drift locally

Start a container with a stale config, trigger a shutdown, and watch the weight reset:

docker run -d --name rag-test mistral/rag:latest
docker exec rag-test python -c "
from mistral.sdk import HybridSearchConfig
cfg = HybridSearchConfig()
cfg.retrieval_weight = 0.5
cfg.rerank_weight = 0.5
cfg.save('/tmp/config.json')
"
docker stop rag-test
# New container starts without the saved file → defaults applied

Resolution – Preserve and Re‑synchronize Weights

The fix consists of two parts: guarantee weight persistence during termination and enforce weight loading on startup.

1. Add a `preStop` hook that exports the current weights

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mistral-rag
spec:
  template:
    spec:
      containers:
      - name: rag-worker
        image: mistral/rag:{{ .Values.imageTag }}
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "/app/export_weights.sh"]
        env:
        - name: WEIGHT_SYNC_ENDPOINT
          value: "http://weight-sync-service:8080/weights"

export_weights.sh:

#!/bin/sh
python - <<'PY'
from mistral.sdk import HybridSearchConfig
import requests, os, json

cfg = HybridSearchConfig.load()
payload = {
    "retrieval_weight": cfg.retrieval_weight,
    "rerank_weight": cfg.rerank_weight,
    "model_version": os.getenv("MODEL_VERSION")
}
resp = requests.post(
    os.getenv("WEIGHT_SYNC_ENDPOINT"),
    json=payload,
    timeout=5
)
resp.raise_for_status()
PY

2. Modify the startup script to pull persisted weights

# startup.sh
#!/bin/sh
set -e
WEIGHT_ENDPOINT="http://weight-sync-service:8080/weights"
if curl -sf "$WEIGHT_ENDPOINT" > /tmp/weights.json; then
  python - <<'PY'
import json, os
from mistral.sdk import HybridSearchConfig
with open("/tmp/weights.json") as f:
    data = json.load(f)
cfg = HybridSearchConfig()
cfg.retrieval_weight = data["retrieval_weight"]
cfg.rerank_weight = data["rerank_weight"]
cfg.model_version = data["model_version"]
cfg.save("/app/config/hybrid.json")
PY
else
  echo "Weight endpoint unavailable – using defaults"
fi
exec mistral-rag-worker

3. Enable the SDK flag to keep weights across restarts

# When launching the container
docker run -e MISTRAL_SDK_FLAGS="--preserve-weights" mistral/rag:latest

Before / After Comparison

Scenario	Retrieval Weight	Rerank Weight	Result
Before fix (default reset)	0.8	0.2	Retrieval dominates, relevance ↓ 40 %
After fix (persisted)	0.6	0.4	Balanced scores, relevance restored

Validation – Confirm the Fix Works

Deploy the updated manifest with kubectl apply -f deployment.yaml.
Trigger a rolling update (e.g., bump imageTag).
During the rollout, verify the export hook runs:

kubectl logs -l app=mistral-rag -c rag-worker | grep "export_weights.sh"

Expected log snippet:

2024-06-22T03:15:02Z INFO Exported weights: {"retrieval_weight":0.6,"rerank_weight":0.4}
2024-06-22T03:15:03Z INFO Weight sync succeeded

After the rollout, query the hybrid endpoint again:

curl -s -X POST https://api.mistral.ai/v1/search/hybrid \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"model":"rag-retriever","query":"sample"}' | jq '.weights'

Output should match the persisted values (e.g., 0.6 and 0.4).

Run an A/B relevance test against a known benchmark dataset. The metric (e.g., NDCG@10) should return to pre‑update levels.

Prevention and Best Practices

Stateful weight synchronization: Always configure a preStop hook or use the SDK --preserve-weights flag.
Health‑check the weight‑sync service and add an alert on WeightSyncFailed errors.
Version‑lock the weight schema: Include model_version in the persisted payload and reject mismatched versions at startup (see Rolling Update Strategy).
Zero‑downtime rollout: Set maxUnavailable: 0 and maxSurge: 1 to ensure at least one pod with correct weights is always serving.
Automated testing: Add a CI step that starts a container, changes weights, performs a rolling restart, and asserts the weights survive.

FAQ – Related Questions

Why does the rerank weight reset to 0 after a blue‑green deployment?
Because the new pod starts without loading the persisted HybridSearchConfig. Without a preStop export or the --preserve-weights flag, the SDK falls back to its default configuration.
How can I verify which weights are currently active in a running pod?
Execute curl -X POST …/search/hybrid with a dummy query and inspect the .weights field, or read the on‑disk config file (/app/config/hybrid.json).
Can I store the weights in a ConfigMap instead of a custom service?
Yes. The SDK accepts any reachable HTTP endpoint. Mount a ConfigMap as a volume, expose it via a sidecar that serves the JSON, and point WEIGHT_SYNC_ENDPOINT to that sidecar.
What alert thresholds should I set for weight drift?
Monitor the log pattern Score imbalance detected: retrieval_score >> rerank_score. Trigger an alert when it appears more than 3 times in a 5‑minute window.
Is it safe to change retrieval_weight and rerank_weight at runtime?
The API allows dynamic updates via PATCH /v1/search/hybrid/config, but the change must be persisted before the next pod termination; otherwise a rollback will overwrite it.

Related Topic Hub: LLM Systems Troubleshooting Hub