Model Serving

ONNX Runtime inference fails after CI/CD deployment due to incorrect routing

by
img IX mining rig inside white and gray room

Problem – ONNX Runtime Inference Fails After CI/CD Deployment During a routine CI/CD rollout, newly built ONNX model containers are deployed to a Kubernetes cluster. After the deployment completes, API calls that should hit the onnx-runtime service return HTTP 502/504 errors and the ONNX Runtime logs show Failed to load model: file not found. The … Read more

PyTorch CRD validation failure due to tensor dimension mismatch

by
a neon circle with a snake on it

Problem – CRD validation failure due to tensor dimension mismatch During a multi‑node distributed training run, the pipeline aborts with a validation error similar to: ValueError: Expected tensor of shape (N, C, H, W) but got (N, C, H, W, 1) – CRD validation failed Other observed symptoms include: Inconsistent batch shapes reported by the … Read more

vLLM text image modality mismatch after GPU passthrough

by
A ribbon with gradient colors floats in the air.

Problem – Text‑Image Modality Mismatch after GPU Passthrough When running a multimodal model (e.g., llava‑1.5‑7b) inside a Docker container that uses the NVIDIA Container Toolkit, the generated responses contain text that is unrelated to the supplied image. The model appears to process the textual prompt correctly but ignores the visual content, resulting in incoherent or … Read more