Model Serving

vLLM JSON output malformed after CUDA upgrade to 12.4

by
a close-up of a server room

Problem Description Symptoms and Impact After upgrading the Docker image to CUDA 12.4, vLLM v0.6.3 started emitting malformed JSON for function‑calling prompts. The downstream FastAPI gateway receives a truncated or syntactically invalid JSON payload, which triggers json.JSONDecodeError exceptions and causes request drops. Log snippet from a worker thread: INFO vllm.engine.output_parser – Received partial JSON: {“name”: “search”, “parameters”: … Read more

RAG pipeline context injection failure after MLflow model update

by
a rack of servers in a server room

Problem: RAG pipeline context injection failure after MLflow model update During a blue‑green deployment, traffic was shifted from the green version of the rag-service endpoint to the newly promoted blue version. After the switch, the Retrieval‑Augmented Generation (RAG) pipeline began sending prompts to the LLM that contained an empty or malformed context variable. Typical symptoms … Read more

MLflow model loading timeout during iterative training

by
cable network

Problem – Intermittent Model Loading Timeouts During Iterative Training When a training loop repeatedly saves checkpoints to a remote MLflow tracking server and then calls mlflow.pyfunc.load_model at the start of each epoch, the script occasionally hangs or crashes with a requests.exceptions.ReadTimeout. Typical error messages observed in the logs include: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host=’mlflow-tracking.mycompany.com’, port=443): Read timed out. … Read more

Hugging Face Transformers vision token sequence exceeds max_position_embeddings

by
Transformer robot

Problem: Vision token sequence exceeds max_position_embeddings in Hugging Face Transformers When processing high‑resolution images with VisionEncoderDecoderModel, ViTModel or CLIPModel, the image is split into fixed‑size patches. Each patch becomes a token, and the total token count must be ≤ the model’s max_position_embeddings (e.g., 197 for ViT‑B/16, 512 for CLIP). In production pipelines that ingest variable‑resolution … Read more

vLLM JSON deserialization errors during high concurrency testing

by
img IX mining rig inside white and gray room

Problem: JSON Deserialization Errors in vLLM Function‑Calling under High Concurrency During sustained load testing of the /v1/chat/completions endpoint with function calling enabled, engineers observed intermittent failures such as: JSONDecodeError: Expecting value: line 1 column 1 (char 0) Tool call response schema validation failed: missing required field ‘name’ vLLM request dropped: JSON parsing error RuntimeError: Failed … Read more

vLLM inference failure after model weights update

by
photo of computer cables

vLLM Inference Failure After Model Weights Update Problem Description After a scheduled model checkpoint rollout, a fleet of vLLM workers began returning errors during request handling. Typical symptoms observed across the cluster were: Log excerpt: [2024-08-07 10:12:03] ERROR vllm.engine.engine: Failed to load model: weight shape mismatch Traceback (most recent call last): File “…/vllm/engine/engine.py”, line 312, … Read more

Hugging Face streamer connection timeout during token generation

by
Transformer robot

Problem – Intermittent Streamer Time‑outs During Token Generation When a FastAPI gateway proxies a request to a GPU‑hosted TextGenerationPipeline with streamer=True, the client receives a socket hang‑up or ReadTimeoutError after the first token (or sometimes before any token is emitted). The failure is reproducible under concurrent load and manifests as: ReadTimeoutError: Server timed out while … Read more

PyTorch tensor shape mismatch in RAG context injection

by
a neon circle with a snake on it

Problem – Tensor Shape Mismatch in RAG Context Injection When using a Retrieval‑Augmented Generation (RAG) pipeline inside a Docker‑based Jupyter sandbox, the model often drops or truncates tokens from the retrieved documents. The symptom manifests as missing context during generation and degraded answer quality. Typical log excerpt: context_input_ids.shape: torch.Size([3, 210]) context_attention_mask.shape: torch.Size([3, 128]) RuntimeError: Expected … Read more

PyTorch log_prob() returns -inf with large batch sizes and AMP

by
a purple and black background with a purple and black logo

Problem Statement When benchmarking high‑throughput models on NVIDIA A100 GPUs with torch.cuda.amp.autocast enabled, the log_prob() method of torch.distributions frequently returns -inf or NaN for otherwise valid probability tensors. The symptom appears only when the batch size crosses a hardware‑dependent threshold (e.g., 8 k, 16 k, or higher) and manifests as: loss = -inf (nan) Subsequent backward passes … Read more

RAG retrieval degradation after MLflow model validation

by
img IX mining rig inside white and gray room

Problem – Retrieval‑Augmented Generation (RAG) Hybrid Search Degradation During MLflow Validation During automated pre‑deployment validation runs orchestrated by an MLflow Project in the CI/CD pipeline, the hybrid retriever’s relevance metrics collapse: Recall@10 drops by 40‑50 % compared with manual runs. Logs contain errors such as ScoreNormalizationError: Sparse scores exceed dense scores range and warnings like HybridSearchWeightOverflow. … Read more