Engineering Logs for Troubleshooting & Debugging
Problem Description A production LlamaIndex service deployed as a StatefulSet began failing during rolling updates. Pods entered the Terminating phase for more than ten minutes, PVC unmounts stalled, and the vector‑store index became corrupted. The symptoms manifested as: Ingress returned 503 Service Unavailable – LlamaIndex API not ready. Ingestion pipelines timed out with Failed to … Read more
Problem Description In a Haystack deployment that runs multi‑GPU training jobs, the scheduler pod remains Pending with events such as: 0/5 nodes are available: 5 Insufficient gpu or FailedScheduling: pod has unschedulable: no nodes match node selector These failures prevent the haystack-scheduler from launching training workers, effectively blocking any multi‑GPU training pipeline. Root Cause Analysis … Read more
Problem Description A Kubernetes Deployment that runs tensorflow_model_server was updated to serve a new model version. After the rollout the pods entered a Running state but never became Ready. The following symptoms were observed: Readiness probe failures: Readiness probe failed: Get http://localhost:8501/v1/models/my_model/metadata: dial tcp 127.0.0.1:8501: connect: connection refused Container logs show the server exiting immediately: … Read more
Problem – Pods OOMKilled After Increasing AI Model Size An AI inference service deployed on a multi‑node Kubernetes cluster started failing shortly after the model binary grew from 2 GB to 4 GB. The symptoms observed across the cluster were: State: Terminated Reason: OOMKilled in kubectl describe pod output. Node‑level memory pressure events and occasional pod eviction … Read more
Problem Description The Horizontal Pod Autoscaler (HPA) attached to a Deployment that serves Hugging Face Transformers inference never creates additional replicas, even when the pods report CPU utilization above the configured target (e.g., 80%). The symptom manifests as: CPU usage in kubectl top pod consistently >90%. Inference latency spikes and occasional request time‑outs. HPA status shows … Read more
Problem Description The Horizontal Pod Autoscaler (HPA) in a Kubernetes‑based CI/CD pipeline is expected to add or remove training pods based on GPU utilization. In practice the HPA never scales out: the replica count stays at 1 even when GPU usage spikes to 95 % during model training. The symptom manifests as a growing job queue, … Read more
Problem – Nginx Serves Stale Data After a Kubernetes Pod Update In an on‑premises data‑center, Nginx (or the Nginx Ingress Controller) is used as a reverse proxy for a micro‑services architecture. Service endpoints are discovered through the Kubernetes EndpointSlice API. After a rolling update or a pod crash, Nginx continues to forward traffic to the … Read more
Kubernetes Scheduler Failing to Place AI Pods on AMD GPU Edge Nodes Problem Description In a remote edge site the AI inference workload never leaves the Pending state. Typical scheduler messages include: 0/5 nodes are available: 5 node(s) didn’t match node selector. Insufficient amd.com/gpu: 0/5 nodes are available: 5 node(s) have insufficient resources. scheduler: no … Read more
Problem: Kubernetes Secret Rotation Failure for Multi‑GPU Training Pods In a production AI training platform, each training job runs in a pod that requests multiple GPUs. Access to the GPU resource provider (e.g., NVIDIA GPU Cloud, a proprietary licensing server, or a cloud‑native GPU quota service) is gated by short‑lived credentials stored in a Secret. … Read more
Problem – Frequent Pod Evictions on AMD GPU Nodes During Disaster Recovery During disaster‑recovery (DR) drills the AI training platform experiences a surge of concurrent training jobs. On clusters that use AMD MI250X GPUs the kubelet repeatedly evicts pods with the reason=OutOfmemory condition. Typical symptoms observed: Pods transition to Evicted status within minutes of the … Read more