mlflow

MLflow audio artifact logging fails in Docker: missing ffmpeg library

by
a rack of servers in a server room

Problem When running MLflow inside a Docker container, attempts to log or retrieve audio artifacts fail with an exception similar to: Traceback (most recent call last): File “/usr/local/lib/python3.10/site-packages/mlflow/tracking/client.py”, line 1245, in log_artifact _upload_artifact(local_path, artifact_path, run_id, client) File “/usr/local/lib/python3.10/site-packages/mlflow/tracking/client.py”, line 1103, in _upload_artifact _log_artifact(local_path, artifact_path, run_id, client) File “/usr/local/lib/python3.10/site-packages/mlflow/tracking/client.py”, line 1156, in _log_artifact _log_artifact_to_artifact_repo(local_path, artifact_path, run_id, … Read more

MLflow context window overflow during high‑frequency inference logging

by
a rack of servers in a server room

Problem During a production rollout of a real‑time inference service, the MLflow tracking server started rejecting mlflow.log_metric and mlflow.log_artifact calls originating from a high‑frequency streaming pipeline. The client observed the following error in the pipeline logs: 2026-09-28 14:12:03,721 ERROR mlflow.tracking.context – Context window overflow: attempted to write 128 KB into a 64 KB window (request_id=7f3c9a2e‑b1d4‑4e5a‑a2c1‑d9f1e2c3b6e7) Symptoms include: … Read more

MLflow tracking server run registration fails due to DNS lookup error

by
cable network

Problem: MLflow Tracking Server Fails to Register Runs Due to DNS Lookup Errors When a client attempts to start a new run, the tracking server raises an exception similar to: socket.gaierror: [Errno -2] Name or service not known mlflow.exceptions.MlflowException: Unable to register run Traceback (most recent call last): … urllib3.exceptions.NewConnectionError: Failed to establish a new … Read more

RAG pipeline context injection failure after MLflow model update

by
a rack of servers in a server room

Problem: RAG pipeline context injection failure after MLflow model update During a blue‑green deployment, traffic was shifted from the green version of the rag-service endpoint to the newly promoted blue version. After the switch, the Retrieval‑Augmented Generation (RAG) pipeline began sending prompts to the LLM that contained an empty or malformed context variable. Typical symptoms … Read more

MLflow model loading timeout during iterative training

by
cable network

Problem – Intermittent Model Loading Timeouts During Iterative Training When a training loop repeatedly saves checkpoints to a remote MLflow tracking server and then calls mlflow.pyfunc.load_model at the start of each epoch, the script occasionally hangs or crashes with a requests.exceptions.ReadTimeout. Typical error messages observed in the logs include: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host=’mlflow-tracking.mycompany.com’, port=443): Read timed out. … Read more

RAG retrieval degradation after MLflow model validation

by
img IX mining rig inside white and gray room

Problem – Retrieval‑Augmented Generation (RAG) Hybrid Search Degradation During MLflow Validation During automated pre‑deployment validation runs orchestrated by an MLflow Project in the CI/CD pipeline, the hybrid retriever’s relevance metrics collapse: Recall@10 drops by 40‑50 % compared with manual runs. Logs contain errors such as ScoreNormalizationError: Sparse scores exceed dense scores range and warnings like HybridSearchWeightOverflow. … Read more

MLflow tracking server write conflicts after etcd quorum loss

by
photo of computer cables

Problem Description During a high‑throughput inference deployment (≈10 k requests / s), the MLflow tracking server began returning errors when logging run parameters, metrics, and feature versions. The observed symptoms included: etcdserver: request timed out in the MLflow server logs. etcdserver: leader changed followed by retries. Duplicate run UUIDs and missing metric rows in the PostgreSQL backend. Intermittent … Read more

MLflow token refresh failure during long-running training jobs

by
photo of computer cables

Problem – Intermittent 401 Unauthorized Errors During Long‑Running Training In a multi‑node GPU cluster orchestrated by Kubernetes, a distributed PyTorch/TensorFlow training job pushes metrics, parameters, and artifacts to a remote MLflow tracking server that is protected by an OIDC/OAuth2 provider. After several hours of execution the training process receives repeated 401 Unauthorized responses: HTTPError 401 – Unauthorized while … Read more

MLflow tracking server not found in local Docker setup

by
a close-up of a server room

Problem: MLflow tracking server not found in a local Docker development setup During local development an engineer runs an MLflow tracking server inside a Docker container (or via docker‑compose) and then starts a separate client container (or a host‑side script) that attempts to log runs. The client immediately fails with errors such as: mlflow.exceptions.RestException: Unable … Read more

MLflow tracking server auth fails after secret rotation

by
a close-up of a server room

Problem – Inconsistent Authentication After Secret Rotation During a blue‑green rollout of an ML pipeline on Kubernetes, the MLflow tracking server stopped logging experiments. The server reported authentication failures against the backend database despite the new deployment using the updated secret ARN. Typical log excerpt: 2024-06-28 14:32:11,842 ERROR mlflow.store.sqlalchemy_store SQLAlchemyStore: Backend store connection failed: sqlalchemy.exc.OperationalError: … Read more