Model Serving

TensorRT model serving fails after certificate expiration

by
black and white car door

Problem Description During a routine A/B test that routes production traffic across two GPU‑accelerated inference endpoints (Triton Server with TensorRT‑optimized models), the following symptoms appeared: gRPC clients started failing with UNAVAILABLE: SSL handshake failed. HTTP API gateway returned 502 Bad Gateway – TLS handshake timeout. Model loading logs showed Failed to load TLS certificate: certificate has … Read more

MLflow tracking server write conflicts after etcd quorum loss

by
photo of computer cables

Problem Description During a high‑throughput inference deployment (≈10 k requests / s), the MLflow tracking server began returning errors when logging run parameters, metrics, and feature versions. The observed symptoms included: etcdserver: request timed out in the MLflow server logs. etcdserver: leader changed followed by retries. Duplicate run UUIDs and missing metric rows in the PostgreSQL backend. Intermittent … Read more

CUDA OOM error during batched audio inference in PyTorch

by
a neon circle with a snake on it

Problem – Intermittent CUDA OOM and Tensor Shape Mismatches During Batched Audio Inference In a Kubernetes cluster that runs containerized PyTorch inference services, pods processing variable‑length audio streams occasionally crash with: CUDA out of memory. Tried to allocate 2.34 GiB (GPU 0; 8.00 GiB total capacity; 6.12 GiB already allocated; 1.23 GiB free; 6.12 GiB … Read more

vLLM citation formatting errors after RAG prompt augmentation

by
img IX mining rig inside white and gray room

Problem Description – Inconsistent Citation Formatting in vLLM RAG Pipelines In a production Kubernetes deployment that serves high‑throughput GPU‑accelerated inference, the vLLM engine is used to generate responses augmented with retrieved documents (RAG). The downstream reference validator expects a strict JSON payload that contains a citations array, each citation delimited by the marker syntax defined … Read more

Hugging Face Trainer FileNotFoundError on shared dataset directory

by
Robotic figure with blue and red details against sky

Problem Description The Hugging Face Trainer crashes during the initialization phase of a distributed training run on a Kubernetes cluster. The failure manifests as a FileNotFoundError or PermissionError when the trainer tries to read the shared dataset directory or create the checkpoint folder. Typical log excerpts from the failing pod (rank 2) are: 2026-07-31 10:12:45,123 … Read more

MLflow token refresh failure during long-running training jobs

by
photo of computer cables

Problem – Intermittent 401 Unauthorized Errors During Long‑Running Training In a multi‑node GPU cluster orchestrated by Kubernetes, a distributed PyTorch/TensorFlow training job pushes metrics, parameters, and artifacts to a remote MLflow tracking server that is protected by an OIDC/OAuth2 provider. After several hours of execution the training process receives repeated 401 Unauthorized responses: HTTPError 401 – Unauthorized while … Read more

Hugging Face RAG retrieval empty results on A100 GPU cluster

by
Robotic figure with blue and red details against sky

Problem Description – Empty Retrieval Results on A100/H100 GPU Cluster When executing a Retrieval‑Augmented Generation (RAG) pipeline on a multi‑node GPU cluster (A100/H100), the RAGRetriever consistently returns an empty document list despite: Valid natural‑language queries. A fully populated FAISS index built from the knowledge base. Successful generation of query embeddings on the GPU. Typical log … Read more

vLLM token refresh fails after cluster restoration from snapshot

by
A building that has a reflection of it in the water

Problem: vLLM token refresh fails after cluster restoration from snapshot During disaster‑recovery drills and production failovers, a vLLM cluster restored from a persistent snapshot repeatedly logs errors such as: TokenRefreshError: token expired Invalid token in session state Failed to refresh token: authentication failed Session token mismatch after restore Token cache not found or corrupted All … Read more

MLflow tracking server not found in local Docker setup

by
a close-up of a server room

Problem: MLflow tracking server not found in a local Docker development setup During local development an engineer runs an MLflow tracking server inside a Docker container (or via docker‑compose) and then starts a separate client container (or a host‑side script) that attempts to log runs. The client immediately fails with errors such as: mlflow.exceptions.RestException: Unable … Read more

ONNX Runtime RAG inference fails due to malformed prompt templates

by
img IX mining rig inside white and gray room

Problem – RAG inference fails because the prompt template is malformed In a distributed micro‑services deployment the Retrieval‑Augmented Generation (RAG) service invokes an ONNX Runtime sidecar via gRPC. The sidecar builds a prompt by substituting variables (e.g., {{question}}, {{context}}) into a JSON‑encoded template before feeding the concatenated string to a transformer model. Symptoms observed in … Read more