pod-management

Kubernetes pod crash loop after canary deploying Hugging Face Transformers model

by
A white robot car toy with blue and red stripes

Problem Description A canary rollout of a Hugging Face transformers model on a Kubernetes cluster results in the newly created pods entering CrashLoopBackOff. The failure manifests during the model initialization phase, before the inference server becomes ready. Typical log excerpts include: 2024-08-30T12:15:42.123Z ERROR [model_loader] RuntimeError: CUDA out of memory. Tried to allocate 12.34 GiB. This … Read more

Canary Elasticsearch pods timing out after Calico network policy change

by
photo of computer cables

Problem – Canary Elasticsearch Pods Time Out After Calico NetworkPolicy Change During a canary rollout of a new Elasticsearch version, pods labeled app=es-canary were unable to reach the primary elasticsearch-master service. All HTTP requests to port 9200 and transport requests to port 9300 resulted in connection‑timeout errors, causing indexing and query failures and halting the … Read more

Weaviate scheduler pods pending due to node selector mismatch in GKE

by
a rack of servers in a server room

Problem In a Google Kubernetes Engine (GKE) cluster running a Weaviate deployment, the weaviate-scheduler pods remain in the Pending state indefinitely. The kubectl describe pod output shows a scheduling failure related to node selector constraints, despite apparent node capacity. Root Cause The Weaviate Helm chart defines a nodeSelector for the scheduler component that targets a … Read more

Docker container audio/video drift after scaling inference pods in Kubernetes

by
blue and red cargo ship on sea during daytime

Problem Description The multimodal inference service runs inside Docker containers on a Kubernetes cluster. Each pod receives a live video feed with an accompanying audio track, processes them with an AI model, and re‑encodes the synchronized stream for downstream consumption. After enabling the Horizontal Pod Autoscaler (HPA) and scaling from 2 to 8 inference pods, … Read more

ReplicaSet fails to scale GPU pods after benchmark load

by
black and silver sony cassette player

Problem: ReplicaSet Fails to Scale GPU Pods During Benchmark Load During a performance benchmark of a deep‑learning workload, a ReplicaSet that requests nvidia.com/gpu resources stalls after a subset of pods become running. The remaining pods stay in Pending with scheduler events such as: 0/5 nodes are available: 5 Insufficient nvidia.com/gpu Typical impact includes: Benchmark run … Read more

GPT-4 function call JSON schema errors after scaling to 10k pods

by
a laptop computer sitting on top of a wooden table

Problem – Malformed Function Call JSON at 10k‑Pod Scale During a batch job that dispatches 10,000+ concurrent requests to the OpenAI chat/completions endpoint, the downstream workers began receiving function‑call payloads that failed JSON‑schema validation. Typical symptoms observed in the logs were: JSONDecodeError: Expecting value: line 1 column 1 (char 0) – empty or truncated response. … Read more

TGI container OOM crash loading 30B model on 16GiB pod

by
A book page with printed text and handwritten notes

Problem Description In a staging environment a tgi:latest container is deployed in a Kubernetes pod with 16 GiB of RAM. During startup the container crashes with an out‑of‑memory (OOM) termination: kubectl describe pod tgi-staging-abc123 … State: Waiting Reason: OOMKilled … Events: Type Reason Age From Message —- —— —- —- ——- Warning OOMKill 2m kubelet, node-01 … Read more

Redis pod stuck pending PVC on edge node after node reboot

by
Rows of black server racks with white logos in a data center

Problem – Redis pod stuck pending PVC on an edge node after node reboot A Redis StatefulSet deployed to an edge‑computing node fails to start. The pod remains in Pending because its PersistentVolumeClaim (redis-data) never becomes Bound. The symptom blocks data persistence and makes the Redis service unavailable. Typical pod description: kubectl describe pod redis-0 … Read more

Elasticsearch pod placement failure during high concurrency testing

by
photo of computer cables

Problem: Elasticsearch pod placement failures under high‑concurrency load testing During automated benchmark runs that combine Elasticsearch Rally suites with custom ML vector‑search latency tests, the Kubernetes scheduler repeatedly emits FailedScheduling events and refuses to bind new Elasticsearch pods. The observable symptoms include: Pod events such as “FailedScheduling: 0/5 nodes are available: 5 Insufficient cpu” and … Read more

LlamaIndex StatefulSet update fails with pod termination delay

by
photo of computer cables

Problem Description A production LlamaIndex service deployed as a StatefulSet began failing during rolling updates. Pods entered the Terminating phase for more than ten minutes, PVC unmounts stalled, and the vector‑store index became corrupted. The symptoms manifested as: Ingress returned 503 Service Unavailable – LlamaIndex API not ready. Ingestion pipelines timed out with Failed to … Read more