Model Serving

MLflow audio artifact logging fails in Docker: missing ffmpeg library

by
a rack of servers in a server room

Problem When running MLflow inside a Docker container, attempts to log or retrieve audio artifacts fail with an exception similar to: Traceback (most recent call last): File “/usr/local/lib/python3.10/site-packages/mlflow/tracking/client.py”, line 1245, in log_artifact _upload_artifact(local_path, artifact_path, run_id, client) File “/usr/local/lib/python3.10/site-packages/mlflow/tracking/client.py”, line 1103, in _upload_artifact _log_artifact(local_path, artifact_path, run_id, client) File “/usr/local/lib/python3.10/site-packages/mlflow/tracking/client.py”, line 1156, in _log_artifact _log_artifact_to_artifact_repo(local_path, artifact_path, run_id, … Read more

TGI node OOM during large batch inference with autoscaling disabled

by
The image shows an open bible book of genesis

Problem – TGI Node Out‑of‑Memory (OOM) During Large Batch Inference with Autoscaling Disabled In a production Kubernetes deployment of Text Generation Inference (TGI), a batch job that aggregates thousands of text‑generation requests into a single inference call caused the pod to be killed with an OOM error. Autoscaling was deliberately disabled, so the node could … Read more

ReplicaSet scaling failure after blue‑green rollout of Hugging Face model

by
white and black letter b

ReplicaSet Scaling Failure After Blue‑Green Rollout of a Hugging Face Model Problem During a blue‑green deployment of a Hugging Face Transformers API served by TorchServe, the new version’s ReplicaSet does not scale out. Increasing spec.replicas in the Deployment has no effect: no new Pods are created, or the Pods remain in Pending state indefinitely. Typical … Read more

TensorRT model download fails after OAuth2 token acquisition timeout

by
black and white car door

Problem In an event‑driven inference pipeline, each job pulls a TensorRT engine from a secured model registry. The registry requires an OAuth2 access token. After a recent update to the token‑service, the OAuth2 flow intermittently times out, causing the model download request to abort. The symptom appears as failed inference jobs that never receive a … Read more

ONNX Runtime authentication errors after secret rotation in multi‑region replication

by
the letter x on a black background

Problem After a scheduled secret rotation, ONNX Runtime instances deployed in several cloud regions start failing to load model artifacts. The runtime logs contain authentication errors, and the model loading step hangs or aborts. This outage appears only after the rotation completes, despite the secret manager entries being updated correctly. Root Cause ONNX Runtime uses … Read more

MLflow context window overflow during high‑frequency inference logging

by
a rack of servers in a server room

Problem During a production rollout of a real‑time inference service, the MLflow tracking server started rejecting mlflow.log_metric and mlflow.log_artifact calls originating from a high‑frequency streaming pipeline. The client observed the following error in the pipeline logs: 2026-09-28 14:12:03,721 ERROR mlflow.tracking.context – Context window overflow: attempted to write 128 KB into a 64 KB window (request_id=7f3c9a2e‑b1d4‑4e5a‑a2c1‑d9f1e2c3b6e7) Symptoms include: … Read more

TensorRT inference runtime error: tokenizer encoding fails on non-ASCII input

by
Modern airport terminal building with signage

Problem During live inference serving with TensorRT engines, requests that contain non‑ASCII characters (e.g., “café”, “naïve”, or emojis) trigger a runtime exception inside the tokenizer layer. The failure manifests as a UnicodeEncodeError or as silently corrupted token IDs, causing the entire inference request to be aborted. Typical symptoms observed in production logs: 2026-09-25 12:34:56,783 ERROR … Read more

ONNX Runtime model load timeout after S3 API throttling in Kubernetes

by
A large building with a sign on top of it

Problem In a production Kubernetes deployment the ONNX Runtime inference service intermittently fails to load models stored in cloud object storage (e.g., Amazon S3, Google Cloud Storage). The failure manifests as a model‑load timeout followed by inference request timeouts. The issue appears only after a period of sustained traffic and disappears after a pod restart. … Read more

RAG pipeline returns empty results during rolling update in PyTorch

by
a purple and black background with a purple and black logo

Problem Description During a Kubernetes rolling update of a TorchServe‑hosted Retrieval‑Augmented Generation (RAG) service, the RAGRetriever began returning an empty list of documents for every query. The symptom manifested as log entries such as: INFO 2026-09-21 12:34:56,789 RAGRetriever: No documents retrieved for query ERROR 2026-09-21 12:34:57,102 faiss index not found at path /model/index.faiss WARNING 2026-09-21 … Read more

vLLM node port conflict after adding second deployment on-prem

by
Black cables and wires connected to the back of modular LED display panels

vLLM NodePort Conflict After Adding a Second On‑Prem Deployment Problem Description After deploying a second vLLM instance on the same on‑premises server, both services fail to start. The startup logs contain errors such as: OSError: [Errno 98] Address already in use vLLM server failed to start: socket.error: [Errno 48] Address already in use FailedCreatePodSandBox: failed … Read more