Engineering Logs for Troubleshooting & Debugging
Problem – Token Refresh Failure During Disaster Recovery on AMD GPU Systems During a planned disaster‑recovery (DR) drill, AI inference workloads that rely on AMD Instinct GPUs stopped abruptly after the failover to the backup cluster. The ROCm runtime emitted the following error: TokenRefreshError: failed to refresh access token – ROCm runtime aborts with exit … Read more
Problem Description The Horizontal Pod Autoscaler (HPA) in a Kubernetes‑based CI/CD pipeline is expected to add or remove training pods based on GPU utilization. In practice the HPA never scales out: the replica count stays at 1 even when GPU usage spikes to 95 % during model training. The symptom manifests as a growing job queue, … Read more
Problem Description In a Kubernetes‑based microservices architecture that uses AMD GPUs for Retrieval‑Augmented Generation (RAG), the generated citations are malformed. Observed symptoms include: Missing source fields in the JSON payload returned by the generation service. Inconsistent or empty metadata.title and metadata.author entries. Intermittent duplicate citation entries when pods restart under memory pressure. Log excerpts such … Read more
Kubernetes Scheduler Failing to Place AI Pods on AMD GPU Edge Nodes Problem Description In a remote edge site the AI inference workload never leaves the Pending state. Typical scheduler messages include: 0/5 nodes are available: 5 node(s) didn’t match node selector. Insufficient amd.com/gpu: 0/5 nodes are available: 5 node(s) have insufficient resources. scheduler: no … Read more
AMD GPU CLI Argument Parsing Failure During Disaster Recovery Problem Description (Symptoms and Impact) During automated disaster‑recovery runs on AMD GPU clusters, the recovery scripts invoke ROCm utilities such as rocm-smi, rocminfo, and rocm-bandwidth-test. The scripts abort with parsing errors, causing the scheduler (e.g., Slurm) to mark all GPUs as unavailable. Typical log excerpts are: … Read more
Problem – Collection Creation Failure During Canary Deployment on AMD MI210 Nodes During a rolling canary deployment of a new AI model on a Kubernetes cluster that contains AMD Instinct MI210 accelerators, pods repeatedly terminate with the following error: [2026-06-27 10:12:34] ERROR: Failed to create collection: ROCm runtime version 5.5 does not match driver version 5.6 … Read more
Problem – Frequent Pod Evictions on AMD GPU Nodes During Disaster Recovery During disaster‑recovery (DR) drills the AI training platform experiences a surge of concurrent training jobs. On clusters that use AMD MI250X GPUs the kubelet repeatedly evicts pods with the reason=OutOfmemory condition. Typical symptoms observed: Pods transition to Evicted status within minutes of the … Read more
Problem – PVC Stuck in Pending on an AMD GPU‑Enabled Cluster In a Kubernetes cluster where nodes are provisioned with AMD GPUs (ROCm), AI workloads that require GPU‑accelerated storage often create a PersistentVolumeClaim (PVC). Operators observe that the PVC never transitions to Bound and remains in the Pending phase despite the presence of storage back‑ends … Read more
Problem: Batch Ingestion Fails on AMD GPU in an Air‑Gapped Environment In a secure, air‑gapped compute cluster the nightly AI batch‑ingestion pipeline aborts with errors indicating missing ROCm runtime components. Typical log excerpts look like: Traceback (most recent call last): File “/opt/pipeline/ingest.py”, line 42, in import torch ImportError: libamdhip64.so: cannot open shared object file: No … Read more
Problem – NVIDIA GPU Webhook Timeouts Under High API Traffic During peak loads on an API gateway that forwards real‑time inference requests to a Triton Inference Server (or similar GPU‑accelerated service), clients begin receiving HTTP 504 or custom webhook timeout errors. Typical log excerpts look like: 2024-05-28T14:12:03.421Z [ERROR] Triton Inference Server: Request timed out after … Read more