Model Serving

MLflow model loading timeout during iterative training

by
cable network

Problem – Intermittent Model Loading Timeouts During Iterative Training When a training loop repeatedly saves checkpoints to a remote MLflow tracking server and then calls mlflow.pyfunc.load_model at the start of each epoch, the script occasionally hangs or crashes with a requests.exceptions.ReadTimeout. Typical error messages observed in the logs include: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host=’mlflow-tracking.mycompany.com’, port=443): Read timed out. … Read more

Hugging Face Transformers vision token sequence exceeds max_position_embeddings

by
Transformer robot

Problem: Vision token sequence exceeds max_position_embeddings in Hugging Face Transformers When processing high‑resolution images with VisionEncoderDecoderModel, ViTModel or CLIPModel, the image is split into fixed‑size patches. Each patch becomes a token, and the total token count must be ≤ the model’s max_position_embeddings (e.g., 197 for ViT‑B/16, 512 for CLIP). In production pipelines that ingest variable‑resolution … Read more

vLLM JSON deserialization errors during high concurrency testing

by
img IX mining rig inside white and gray room

Problem: JSON Deserialization Errors in vLLM Function‑Calling under High Concurrency During sustained load testing of the /v1/chat/completions endpoint with function calling enabled, engineers observed intermittent failures such as: JSONDecodeError: Expecting value: line 1 column 1 (char 0) Tool call response schema validation failed: missing required field ‘name’ vLLM request dropped: JSON parsing error RuntimeError: Failed … Read more

vLLM inference failure after model weights update

by
photo of computer cables

vLLM Inference Failure After Model Weights Update Problem Description After a scheduled model checkpoint rollout, a fleet of vLLM workers began returning errors during request handling. Typical symptoms observed across the cluster were: Log excerpt: [2024-08-07 10:12:03] ERROR vllm.engine.engine: Failed to load model: weight shape mismatch Traceback (most recent call last): File “…/vllm/engine/engine.py”, line 312, … Read more

Hugging Face streamer connection timeout during token generation

by
Transformer robot

Problem – Intermittent Streamer Time‑outs During Token Generation When a FastAPI gateway proxies a request to a GPU‑hosted TextGenerationPipeline with streamer=True, the client receives a socket hang‑up or ReadTimeoutError after the first token (or sometimes before any token is emitted). The failure is reproducible under concurrent load and manifests as: ReadTimeoutError: Server timed out while … Read more

PyTorch tensor shape mismatch in RAG context injection

by
a neon circle with a snake on it

Problem – Tensor Shape Mismatch in RAG Context Injection When using a Retrieval‑Augmented Generation (RAG) pipeline inside a Docker‑based Jupyter sandbox, the model often drops or truncates tokens from the retrieved documents. The symptom manifests as missing context during generation and degraded answer quality. Typical log excerpt: context_input_ids.shape: torch.Size([3, 210]) context_attention_mask.shape: torch.Size([3, 128]) RuntimeError: Expected … Read more

PyTorch log_prob() returns -inf with large batch sizes and AMP

by
a purple and black background with a purple and black logo

Problem Statement When benchmarking high‑throughput models on NVIDIA A100 GPUs with torch.cuda.amp.autocast enabled, the log_prob() method of torch.distributions frequently returns -inf or NaN for otherwise valid probability tensors. The symptom appears only when the batch size crosses a hardware‑dependent threshold (e.g., 8 k, 16 k, or higher) and manifests as: loss = -inf (nan) Subsequent backward passes … Read more

RAG retrieval degradation after MLflow model validation

by
img IX mining rig inside white and gray room

Problem – Retrieval‑Augmented Generation (RAG) Hybrid Search Degradation During MLflow Validation During automated pre‑deployment validation runs orchestrated by an MLflow Project in the CI/CD pipeline, the hybrid retriever’s relevance metrics collapse: Recall@10 drops by 40‑50 % compared with manual runs. Logs contain errors such as ScoreNormalizationError: Sparse scores exceed dense scores range and warnings like HybridSearchWeightOverflow. … Read more

TensorRT model serving fails after certificate expiration

by
black and white car door

Problem Description During a routine A/B test that routes production traffic across two GPU‑accelerated inference endpoints (Triton Server with TensorRT‑optimized models), the following symptoms appeared: gRPC clients started failing with UNAVAILABLE: SSL handshake failed. HTTP API gateway returned 502 Bad Gateway – TLS handshake timeout. Model loading logs showed Failed to load TLS certificate: certificate has … Read more

MLflow tracking server write conflicts after etcd quorum loss

by
photo of computer cables

Problem Description During a high‑throughput inference deployment (≈10 k requests / s), the MLflow tracking server began returning errors when logging run parameters, metrics, and feature versions. The observed symptoms included: etcdserver: request timed out in the MLflow server logs. etcdserver: leader changed followed by retries. Duplicate run UUIDs and missing metric rows in the PostgreSQL backend. Intermittent … Read more