mlflow

MLflow RAG query decomposition fails during rolling update

by
cable network

Problem – RAG query decomposition fails during a rolling update During a rolling update of an MLflow‑served Retrieval‑Augmented Generation (RAG) model, inference requests start returning errors such as: Failed to decompose query: embedding dimension mismatch (expected 768, got 512) CacheKeyCollisionError: Duplicate cache entry for key ‘rag_decompose:customer_id=123’ detected during rolling update ModelVersionNotFoundError: Requested version ‘2’ for … Read more

MLflow training job rejected by Kubernetes admission controller

by
photo of computer cables

Problem Description When launching an MLflow training run that uses the Kubernetes backend, the pod creation fails with an admission‑controller error. The MLflow client reports a kubectl error similar to the following: Error from server (Forbidden): pods “mlflow-train-7f9c8d5b9-xyz” is forbidden: exceeded quota: compute-resources, request: cpu=8, memory=16Gi Other observed messages include: admission webhook “validation.gatekeeper.sh” denied the … Read more