Model Serving

TensorRT engine build fails with embedding dimension mismatch after model change

by
a view of a restaurant through a window at night

Problem: TensorRT Engine Build Fails with Embedding Dimension Mismatch After Model Change A real‑time video streaming inference pipeline converts a PyTorch model containing an nn.Embedding layer to a TensorRT engine for sub‑millisecond latency. After a recent model revision that altered the embedding size, the engine build aborts with a validation error: [2026-10-21 14:03:12] [ERROR] [TensorRT] … Read more

TGI service unreachable during benchmark due to DNS resolution failure

by
white printer paper on white textile

Problem – TGI Service Unreachable During Benchmark During automated performance benchmarking of a Text Generation Inference (TGI) deployment, the benchmark script repeatedly fails with a DNS resolution error: 2026-10-18 14:32:07,123 ERROR benchmark.py: Failed to resolve host “tgi-model-prod.example.com”: Name or service not known Traceback (most recent call last): File “/opt/benchmark/benchmark.py”, line 87, in run ip = … Read more

TorchServe pods pending after custom scheduler can’t match GPU resource requests

by
a purple and black background with a purple and black logo

Problem – TorchServe Pods Remain Pending When Using a Custom Scheduler In a Kubernetes cluster that hosts GPU‑enabled node pools, a real‑time inference service is deployed with TorchServe. The deployment succeeds, but the resulting pods stay in Pending state indefinitely. The custom scheduler (configured via a scheduler profile or extender) reports that it cannot find … Read more

vLLM API server connection timeout after rolling deployment in Kubernetes

by
The text JEV VS LLM* written in black dots on a glass window

Problem After a rolling deployment of the vLLM inference service in a Kubernetes cluster, client applications began receiving connection timeout, connection refused, and HTTP 502 Bad Gateway responses when calling the vLLM API endpoint. The issue appears only after the deployment finishes; the service was reachable before the rollout. Root Cause The timeout originated from … Read more

Intermittent JSON schema validation error during DDP multi‑GPU training

by
Robotic figure with blue and red details against sky

Problem: Intermittent JSON Schema Validation Errors in DDP Multi‑GPU Training During distributed training of a large language model with torch.nn.parallel.DistributedDataParallel (DDP) on a 4‑GPU A100 node, the model is configured to emit structured JSON responses via the transformers StructuredOutput utilities. Occasionally, a RuntimeError is raised: RuntimeError: Structured output validation failed: generated JSON does not conform … Read more

Intermittent streaming response interruption in PyTorch microservice under high concurrency

by
a neon circle with a snake on it

Problem – Intermittent Streaming Response Interruption In a Kubernetes‑deployed microservice that serves a PyTorch model via a streaming HTTP endpoint, clients occasionally receive truncated inference results. The HTTP connection is closed before the model finishes generating the full token sequence, leading to partial outputs and downstream processing failures. The issue appears only under high concurrency … Read more

TensorRT inference API server unreachable in staging after firewall update

by
a view of a restaurant through a window at night

Problem After a scheduled firewall rule update, the TensorRT inference API server in the staging environment stopped accepting TCP connections on port 8000. All client services that invoke /v1/predict receive a connection timed out error, while the same request works fine in the production environment. Typical client error: requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host=’staging-infer.example.com’, port=8000): Max retries exceeded with … Read more

Kubernetes storage class not found error deploying Hugging Face model

by
A detailed red and blue robot with glowing eyes

Problem: “StorageClass not found” when deploying a Hugging Face Transformers model During the CI/CD pipeline a Deployment that mounts a PersistentVolumeClaim for the model artifacts fails with the following error: error: failed to create pod sandbox: rpc error: code = Unknown desc = failed to provision volume with StorageClass “fast-ssd”: storageclass.storage.k8s.io “fast-ssd” not found The … Read more

vLLM RAG hybrid search weight imbalance on edge node after config change

by
A man with styled hair wearing a white t-shirt with red lettering

Problem After a recent configuration change on an edge‑computing node running vLLM, the Retrieval‑Augmented Generation (RAG) hybrid search started to favor one modality almost exclusively. In practice: Keyword‑based retrieval (bm25) returns no results even when the query contains exact terms. Dense vector retrieval dominates the relevance score, causing unrelated passages to be selected. Latency spikes … Read more

Admission controller rejecting PyTorch pod creation on kind cluster

by
a purple and black background with a purple and black logo

Problem When attempting to launch a PyTorch training job on a local kind cluster, the pod creation fails with an admission controller error. The user sees the following output from kubectl apply and kubectl describe pod: $ kubectl apply -f pytorch-job.yaml pod/pytorch-trainer created error: error when creating “pytorch-job.yaml”: admission webhook “podsecuritypolicy.k8s.io” denied the request: spec.securityContext.runAsUser: … Read more