Mistral AI Hybrid Search Scoring Discrepancy Between Staging And Production

Q: Investigation and Debugging 1. Verify model and index versions

# Inspect the model version label on the running pod kubectl exec -n prod $(kubectl get pod -n prod -l app=mistral-ai -o jsonpath="{.items[0].metadata.name}") -- \ cat /models/version.txt # Expected output: v2.1.0 # Query the index version via the API curl -s -X GET "https://prod.api.mistral.ai/v1/index/version" -H "Authorization: Bearer $TOKEN" Production returned {"index_version":3} while staging returned {"index_version":4}, confirming a mismatch.

Q: 2. Compare Helm values

# Render effective values for both environments helm get values mistral-ai -n staging --all helm get values mistral-ai -n prod --all ParameterStagingProduction TOP_K1010 SCORE_WEIGHT.lexical0.40.7 SCORE_WEIGHT.vector0.60.3 deterministictruefalse 3. Check index refresh status # Examine the index refresh job logs kubectl logs -n prod -l job-name=mistral-index-refresh -c refresh Log excerpt:

Q: 4. Reproduce nondeterminism locally

# Run two identical queries without deterministic flag curl -s -X POST "https://prod.api.mistral.ai/v1/hybrid-search" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"query":"machine learning basics","top_k":5}' Repeated calls returned different ordering for the 3rd and 4th results, confirming nondeterministic tie‑breaking.

Q: 2. Align Helm configuration across environments

Extract common scoring parameters into a shared values-common.yaml and reference it in both staging and production releases.

Problem Description

Symptoms and Impact

After the nightly CI/CD pipeline retrains the Mistral AI embedding model and rolls out the new container image, the /v1/hybrid-search endpoint returns noticeably different relevance scores in the production Kubernetes cluster compared to the staging cluster.

Typical observations:

Top‑K results differ in ordering despite identical query text.
Score drift of up to 0.12 on a 0‑1 scale.

Production logs contain warnings such as:


[2026-06-22T14:32:10Z] WARN  HybridSearchScoringError: embedding dimension mismatch – expected 768, got 1024
[2026-06-22T14:32:12Z] WARN  ScoreDivergenceWarning: score variance exceeds 0.05 between consecutive runs

Staging runs are deterministic (identical scores on repeated queries), while production shows nondeterministic tie‑breaking.

Operational Impact

End‑users experience inconsistent search relevance, leading to reduced click‑through rates and a spike in support tickets. The discrepancy also violates the Service Level Objective (SLO) for search relevance stability.

Root Cause Analysis

Multiple independent evidence sources converge on three primary contributors:

Vector index version mismatch: The CI/CD pipeline refreshes the embedding model but does not trigger an index rebuild in production. Staging rebuilds the index on each deployment, while production continues querying the previous index version. This aligns with the incident described in the evidence package where “staging used the new index while production continued to query the previous index, leading to score drift.”
Configuration drift via Helm values: Production Helm values override TOP_K and SCORE_WEIGHT (lexical vs. vector weighting). The official Mistral AI Documentation – Hybrid Search Scoring and Parameters states that SCORE_WEIGHT defaults to {lexical: 0.4, vector: 0.6} unless explicitly set. A production values.yaml file set SCORE_WEIGHT: {lexical: 0.7, vector: 0.3}, causing different hybrid scores.
Non‑deterministic random seed: The /v1/hybrid-search API supports a deterministic flag. Staging deployments enable it (per the deployment guide), but production omitted the flag, resulting in nondeterministic tie‑breaking as reported in Stack Overflow question 82210158.

Investigation and Debugging

1. Verify model and index versions

# Inspect the model version label on the running pod
kubectl exec -n prod $(kubectl get pod -n prod -l app=mistral-ai -o jsonpath="{.items[0].metadata.name}") -- \
  cat /models/version.txt
# Expected output: v2.1.0

# Query the index version via the API
curl -s -X GET "https://prod.api.mistral.ai/v1/index/version" -H "Authorization: Bearer $TOKEN"

Production returned {"index_version":3} while staging returned {"index_version":4}, confirming a mismatch.

2. Compare Helm values

# Render effective values for both environments
helm get values mistral-ai -n staging --all
helm get values mistral-ai -n prod --all

Parameter	Staging	Production
TOP_K	10	10
SCORE_WEIGHT.lexical	0.4	0.7
SCORE_WEIGHT.vector	0.6	0.3
deterministic	true	false

3. Check index refresh status

# Examine the index refresh job logs
kubectl logs -n prod -l job-name=mistral-index-refresh -c refresh

Log excerpt:


2026-06-22T13:58:02Z INFO  Index refresh started for version 4
2026-06-22T13:58:05Z ERROR IndexVersionMismatchError: queried index version 3, but model expects version 4
2026-06-22T13:58:06Z INFO  Skipping index rebuild due to cached layer

The job aborted because the pod used a cached Docker layer that still referenced the old index artifact (see the incident about “cached Docker layer” in the evidence package).

4. Reproduce nondeterminism locally


# Run two identical queries without deterministic flag
curl -s -X POST "https://prod.api.mistral.ai/v1/hybrid-search" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query":"machine learning basics","top_k":5}'

Repeated calls returned different ordering for the 3rd and 4th results, confirming nondeterministic tie‑breaking.

Resolution

1. Enforce index rebuild on every model rollout

Update the CI/CD pipeline to include an explicit index refresh step that blocks deployment until the new index is verified.

# CI/CD stage (pseudo‑YAML)
- name: Refresh Vector Index
  run: |
    helm upgrade --install mistral-index-refresh charts/mistral-index \
      --namespace prod \
      --set modelVersion=${MODEL_VERSION} \
      --wait --timeout 10m

2. Align Helm configuration across environments

Extract common scoring parameters into a shared values-common.yaml and reference it in both staging and production releases.

# values-common.yaml
TOP_K: 10
SCORE_WEIGHT:
  lexical: 0.4
  vector: 0.6
deterministic: true

Then in each environment:


helm upgrade --install mistral-ai charts/mistral-ai \
  -n prod -f values-common.yaml -f values-prod.yaml

3. Enable deterministic mode in production

Add the flag to the API server startup arguments.

# Deployment manifest snippet (before)
args:
  - "--model-dir=/models/mistral-hybrid"
  - "--port=8080"

# After
args:
  - "--model-dir=/models/mistral-hybrid"
  - "--port=8080"
  - "--deterministic=true"

4. Prevent Docker layer caching issues

Force a clean build for production images.


# In the production Docker build step
docker build --no-cache -t registry.example.com/mistral-ai:${MODEL_VERSION} .

Validation

Score Consistency Check


# Query both environments with deterministic flag
curl -s -X POST "https://staging.api.mistral.ai/v1/hybrid-search" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query":"machine learning basics","top_k":5,"deterministic":true}' > staging.json

curl -s -X POST "https://prod.api.mistral.ai/v1/hybrid-search" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query":"machine learning basics","top_k":5,"deterministic":true}' > prod.json

diff -u staging.json prod.json

The diff should be empty, indicating identical scores and ordering.

Index Version Verification


curl -s "https://prod.api.mistral.ai/v1/index/version" | jq .
# Expected: {"index_version":4}

Health‑check Endpoint


curl -s "https://prod.api.mistral.ai/v1/healthz"
# Expected response: {"status":"ok","model_version":"v2.1.0","index_version":4}

Prevention and Best Practices

Versioned index artifacts: Store each vector index with a semantic version tag in the data lake and require the model version to match the index version during startup.
Deterministic flag as default: Include --deterministic=true in the base container image to avoid accidental omission.
Configuration as code: Keep scoring parameters in a single source of truth (e.g., values-common.yaml) and lint Helm values for drift during CI.
Post‑deployment verification job: Run a lightweight query against the new deployment and compare scores to a baseline stored in a ConfigMap; fail the rollout if variance exceeds 0.02.
Cache busting strategy: Use --no-cache or unique build arguments (e.g., BUILD_ID) to guarantee fresh images for production.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does the hybrid search score differ only after a CI/CD redeploy?
The redeploy updates the embedding model but, without an explicit index refresh, production continues using the old vector index. Scoring combines lexical and vector components; a stale index changes the vector contribution, leading to score drift.
How can I confirm which index version a pod is using?
Inspect the INDEX_VERSION environment variable inside the pod or query the /v1/index/version endpoint. The value should match the model’s model_version label.
Is the random seed the only source of nondeterminism?
No. Besides the seed, mismatched SCORE_WEIGHT and an outdated index also produce divergent results. All three must be aligned for truly deterministic scoring.
Can I safely disable deterministic mode in production?
Disabling it is acceptable only if your ranking algorithm tolerates tie‑breaking randomness and you have downstream processes that can handle score variance. For most relevance‑critical workloads, keep deterministic=true.
What monitoring alerts should I add to catch this early?
Set alerts on:
- Log pattern HybridSearchScoringError or ScoreDivergenceWarning.
- Metric mistral_hybrid_score_variance exceeding 0.05.
- Index version lag between model_version and index_version greater than 0.