MLflow RAG query decomposition fails during rolling update

Problem – RAG query decomposition fails during a rolling update

During a rolling update of an MLflow‑served Retrieval‑Augmented Generation (RAG) model, inference requests start returning errors such as:

Failed to decompose query: embedding dimension mismatch (expected 768, got 512)
CacheKeyCollisionError: Duplicate cache entry for key 'rag_decompose:customer_id=123' detected during rolling update
ModelVersionNotFoundError: Requested version '2' for model 'rag-retriever' not found in registry
InferenceError: Query decomposition failed due to version conflict – loaded tokenizer vocab differs from expected version

The failure only appears while two model versions are simultaneously running (old and new). Once the rollout finishes and only one version remains, the service returns to normal.

Root Cause – Interaction between model registry, shared cache, and rolling‑update semantics

MLflow’s model registry (see official docs) allows multiple versions of a model to coexist, each identified by model_name and model_version. The serving container loads the requested version on first request and caches:

• the tokenizer / pre‑processor (Python objects),
• the vector store client (often a Redis instance), and
• temporary files under /tmp/model_cache (used by the mlflow.pyfunc predict implementation).

In a Kubernetes rolling update (see MLflow on Kubernetes guide) the old pods are terminated only after new pods pass readiness probes. While both sets of pods are alive:

1. Shared filesystem cache: By default the Helm chart mounts a emptyDir volume that is shared across all pods on the same node. Old pods continue to use the cache populated with version 1 artifacts, while new pods write version 2 artifacts to the same directory.
2. Redis key namespace collision: The RAG decomposition step stores intermediate embeddings under a static key pattern (rag_decompose:{customer_id}). When two versions write to the same Redis database, keys from version 1 overwrite or collide with version 2, causing CacheKeyCollisionError and mismatched embedding dimensions.
3. In‑memory singleton vector store: Some implementations keep a singleton VectorStore object in module‑level globals. The singleton is not version‑aware; when a new pod imports the same module it re‑uses the old instance, leading to “embedding dimension mismatch” errors (see the fintech incident of March 2024).
4. ModelRegistryConcurrencyError: The registry client updates the stage field of a model version during the rollout. Simultaneous updates from old and new pods cause a race condition, leaving the registry in an inconsistent state and returning ModelVersionNotFoundError for some requests.

Collectively these issues cause the RAG pipeline’s query‑decomposition step to receive the wrong tokenizer, wrong vector store, or corrupted cached embeddings, which manifests as the observed errors.

Debug – Step‑by‑step investigation

1. Confirm multiple versions are serving

kubectl get pods -l app=mlflow-serve -o wide
NAME                                 READY   STATUS    RESTARTS   AGE   IP          NODE
mlflow-serve-7f9c9d5d5b-abcde        1/1     Running   0          2m    10.1.2.5    node-1
mlflow-serve-7f9c9d5d5b-fghij        1/1     Running   0          2m    10.1.2.6    node-2

Both pods report the same model_name=rag-retriever but different model_version in their environment variables.

2. Inspect logs for cache‑related errors

kubectl logs mlflow-serve-7f9c9d5d5b-abcde | grep -i "CacheKeyCollisionError"
2024-05-12 14:03:21,874 ERROR mlflow.pyfunc: Failed to decompose query: CacheKeyCollisionError: Duplicate cache entry for key 'rag_decompose:customer_id=123' detected during rolling update

3. Check Redis key namespaces

redis-cli -h redis.internal.svc.cluster.local KEYS "rag_decompose:*"
1) "rag_decompose:customer_id=123"
2) "rag_decompose:customer_id=124"

If the same key appears from different pods, the namespace is not version‑scoped.

4. Verify the tokenizer version used by each pod

kubectl exec -it mlflow-serve-7f9c9d5d5b-abcde -- python - <<'PY'
import mlflow.pyfunc, json, os
model = mlflow.pyfunc.load_model(os.getenv("MLFLOW_MODEL_URI"))
print("Tokenizer vocab size:", model.metadata.get_input_schema().metadata.get("vocab_size"))
PY
Tokenizer vocab size: 30522

Compare with the new pod; a mismatch indicates stale in‑memory objects.

5. Detect ModelRegistryConcurrencyError

kubectl logs mlflow-serve-7f9c9d5d5b-fghij | grep -i "ModelRegistryConcurrencyError"
2024-05-12 14:04:02,112 WARN mlflow.tracking: ModelRegistryConcurrencyError: Unable to update model stage because another process is modifying the same version

Solution – Making the rolling update cache‑safe and version‑aware

1. Isolate model cache per pod

Modify the Helm values to mount a emptyDir with a unique sub‑directory per pod (e.g., using the pod name as a suffix). This prevents cross‑pod file‑system collisions.

Before (default values.yaml)

volumeMounts:
  - name: model-cache
    mountPath: /tmp/model_cache
volumes:
  - name: model-cache
    emptyDir: {}

After (add pod‑specific sub‑path)

volumeMounts:
  - name: model-cache
    mountPath: /tmp/model_cache/$(POD_NAME)
    subPathExpr: $(POD_NAME)
volumes:
  - name: model-cache
    emptyDir: {}
env:
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name

2. Namespace Redis keys by model version

Update the RAG decomposition code to prepend the model version to every cache key.

Before

def cache_key(customer_id):
    return f"rag_decompose:{customer_id}"

After

def cache_key(customer_id, version):
    return f"rag_decompose:v{version}:{customer_id}"

Pass model_version obtained from the request context into the function.

3. Ensure tokenizer and vector store are instantiated per‑request or per‑model version

Replace module‑level singletons with a factory that caches objects keyed by version.

# utils.py
from functools import lru_cache

@lru_cache(maxsize=None)
def get_tokenizer(version: str):
    # Load tokenizer assets from the version‑specific artifact directory
    path = f"/tmp/model_cache/{version}/tokenizer"
    return Tokenizer.load(path)

@lru_cache(maxsize=None)
def get_vector_store(version: str):
    # Create a Redis‑backed store with a version‑scoped namespace
    return RedisVectorStore(namespace=f"rag_store:v{version}")

4. Graceful shutdown of old pods

Configure terminationGracePeriodSeconds and a preStop hook that flushes in‑memory caches and clears the per‑pod cache directory.

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "rm -rf /tmp/model_cache/$(POD_NAME)/* && sleep 5"]
terminationGracePeriodSeconds: 30
readinessProbe:
  httpGet:
    path: /health
    port: 5000
  initialDelaySeconds: 5
  periodSeconds: 10

5. Serialize stage transitions in the Model Registry

When promoting a new version to Production, use the mlflow.models.set_version flag with --force disabled, and serialize the operation via a CI/CD lock (e.g., a Kubernetes ConfigMap lock).

mlflow models set-version-tag \
  --model-name rag-retriever \
  --version 2 \
  --stage Production \
  --no-force

Verification – Confirm the fix works

1. Deploy the updated Helm chart

helm upgrade mlflow-serve ./mlflow-serve-chart \
  -f values.yaml \
  --set image.tag=v2.1.0

2. Run a health‑check that includes the version query parameter

curl -s "http://mlflow-serve.default.svc.cluster.local/invocations?model_name=rag-retriever&model_version=2" \
  -H "Content-Type: application/json" \
  -d '{"queries":["What is the policy for overseas transfers?"]}'

Expected response (truncated):

{
  "answers": ["Overseas transfers are processed within 2‑3 business days..."],
  "metadata": {"model_version":"2"}
}

3. Inspect Redis for version‑scoped keys

redis-cli KEYS "rag_decompose:v2:*"
1) "rag_decompose:v2:customer_id=123"

4. Check pod logs for the absence of cache‑collision messages

kubectl logs -l app=mlflow-serve | grep -i "CacheKeyCollisionError"
# (no output)

Prevention – Operational guardrails

• Per‑pod cache isolation: Always mount /tmp/model_cache with a pod‑specific sub‑path. This is the recommended default in the MLflow Helm chart after version 2.0.
• Version‑scoped external caches: When using Redis, Elasticsearch, or any external store for RAG preprocessing, include the model version (or a UUID derived from the model’s run_id) in the key namespace.
• Readiness & liveness probes: Ensure the readiness probe validates that the model version has been fully loaded before the pod receives traffic. A typical probe checks the /health endpoint with the target version as a query param.
• Graceful termination: Set terminationGracePeriodSeconds ≥ 30 s and implement a preStop hook that clears any local caches to avoid stale artifacts lingering after pod shutdown.
• Serialized registry updates: Use a lock (e.g., a ConfigMap or external lock service) around any mlflow models set-stage calls during a rollout to prevent ModelRegistryConcurrencyError.
• Monitoring alerts: Create alerts for the following patterns:
  • Log rate of CacheKeyCollisionError > 0 per minute.
  • Metric mlflow_inference_latency_seconds spikes > 2× baseline when model_version changes.
  • Redis key growth in the rag_decompose namespace exceeding expected cardinality.

FAQ – Common follow‑up questions

Q1: Why does the error only appear when two versions run simultaneously?
A: The shared cache (filesystem or Redis) is not version‑aware. When both versions write to the same location, the older artifacts overwrite the newer ones, causing mismatched tokenizers or embedding dimensions. Once the rollout finishes, only one version writes, eliminating the collision.

Q2: Can I keep a single Redis instance and still avoid key collisions?
A: Yes. Prefix every key with a version identifier (e.g., rag_decompose:v{version}:) or use Redis logical databases (SELECT 0, 1, …) per version. The prefix approach is simpler and works with existing connection pools.

Q3: What is the recommended way to flush the per‑pod cache during a rollout?
A: Configure a preStop hook that deletes the pod‑specific sub‑directory (/tmp/model_cache/${POD_NAME}) and ensure terminationGracePeriodSeconds allows the hook to complete before the pod is killed.

Q4: How do I make the readiness probe wait for the correct model version to load?
A: Expose an endpoint such as /health?model_version=2 that attempts a lightweight inference (e.g., a dummy query) using the requested version. Return HTTP 200 only when the model and its tokenizer are fully instantiated.

Q5: Is there a built‑in MLflow flag to isolate caches per version?
A: As of MLflow 2.2, there is no automatic version‑scoped cache. The recommended pattern is to control cache locations via environment variables (e.g., MLFLOW_MODEL_CACHE_DIR) and set them per pod using the pod name or version identifier.

Related Topic Hub: Model Serving Troubleshooting Hub