Engineering Logs for Troubleshooting & Debugging
Problem: Pods Evicted Under Memory Pressure During Large‑Scale Model Evaluation on Azure During batch evaluation of transformer‑based models (12‑20 GB checkpoint size) on Azure virtual machines managed by AKS, operators observed repeated pod evictions: Events: Type Reason Age From Message —- —— —- —- ——- Normal Killing 5m kubelet Killing container with id docker://evaluator:Need to kill … Read more
Problem – Grafana Dashboard Rendering Fails Under Load In a GPU‑focused LLM training cluster (64 × A100, 48 × H100 for inference) operators observed the following symptoms after adding per‑GPU DCGM metrics and reducing the dashboard refresh interval to 5s: Grafana pod repeatedly OOMKilled: panic: runtime: out of memory HTTP 504 Gateway Timeout responses from the Grafana UI. CPU … Read more
Docker Container Memory Allocation Failure During Multi‑Region Replication Problem Description During a scheduled multi‑region data replication job, Docker containers that host the replication agents repeatedly terminate with out‑of‑memory (OOM) errors. The failure manifests as: 2024-07-15T03:12:47.321Z containerd[1234]: failed to create shim task: failed to start container: OCI runtime create failed: container_linux.go:380: starting container process caused “process_linux.go:449: … Read more
Problem – “GPT‑3.5 collection creation fails during streaming with memory allocation error” In a distributed inference service that streams completions to many simultaneous users, the OpenAI client throws errors such as: Error: Memory allocation failed RuntimeError: CUDA out of memory OpenAIError: Server error – 500 – Failed to create collection OSError: [Errno 12] Cannot allocate … Read more