Engineering Logs for Troubleshooting & Debugging
Problem Description In an AMD‑GPU‑enabled Kubernetes cluster, AI training jobs launched by an event‑driven pipeline (e.g., Kubeflow + Argo Events) fail to start because the associated PersistentVolumeClaim (PVC) stays in the Pending state. The pod is scheduled onto a GPU node, but the volume never binds, leading to errors such as: persistentvolumeclaim “my-pvc” is pending … Read more
Problem – etcd Cluster State Divergence Triggering AMD GPU Node Scheduling Errors In a production AI training platform that relies on Kubernetes to schedule AMD GPU‑accelerated workloads, operators observed a sudden increase in pod pending states and job failures. The symptoms were traced back to GPU worker nodes being marked NotReady by the kubelet, despite … Read more
Problem – Webhook POST Times Out After AMD GPU Inference in Docker In a CI/CD pipeline a Docker container runs an AI model on an AMD Instinct GPU using ROCm. The inference step finishes successfully, but the subsequent HTTP POST to an external webhook hangs and eventually fails with a timeout, causing the job to … Read more
AMD GPU Nodes TLS Handshake Failure to Artifact Storage After Driver Update Problem Description After upgrading the ROCm driver suite to 6.0.3 on a mixed‑GPU Kubernetes cluster (NVIDIA A100/H100 and AMD Instinct MI250), all pods scheduled on AMD GPU nodes began failing to download model artifacts from the central artifact-storage service (Azure Blob, S3, or … Read more
Problem Description During long‑running AI training jobs inside a Docker container, the AMD GPU controller manager (amdgpu) crashes repeatedly. The symptom manifests as the container exiting with code 137 (SIGKILL) and loss of GPU access. Typical log excerpts are: dmesg: amdgpu: controller manager crashed, resetting GPU Docker log: error while creating device /dev/kfd: No such … Read more
Problem – CRD Validation Rejects AMD GPU Specification in Deployment Manifest When deploying an AI inference service that requires an AMD GPU, the kubectl apply -f deployment.yaml command fails with a validation error from the AMD GPU admission webhook. Typical error output looks like: error: admission webhook “validate.amd.com” denied the request: spec.template.spec.containers[0].resources.limits.amd.com/gpu: Invalid value: “1” … Read more
Problem Description In an air‑gapped deployment of large language models (LLMs) on AMD GPUs, inference jobs abort once the prompt length exceeds a certain token count. The failure manifests as memory allocation errors that appear unrelated to the reported free VRAM. Typical log excerpts: 2026-07-12 14:03:21 [INFO] Starting inference on model LLaMA-13B 2026-07-12 14:03:22 [ERROR] … Read more
Problem – Token Refresh Failure During Disaster Recovery on AMD GPU Systems During a planned disaster‑recovery (DR) drill, AI inference workloads that rely on AMD Instinct GPUs stopped abruptly after the failover to the backup cluster. The ROCm runtime emitted the following error: TokenRefreshError: failed to refresh access token – ROCm runtime aborts with exit … Read more
Problem Description In a Kubernetes‑based microservices architecture that uses AMD GPUs for Retrieval‑Augmented Generation (RAG), the generated citations are malformed. Observed symptoms include: Missing source fields in the JSON payload returned by the generation service. Inconsistent or empty metadata.title and metadata.author entries. Intermittent duplicate citation entries when pods restart under memory pressure. Log excerpts such … Read more
Kubernetes Scheduler Failing to Place AI Pods on AMD GPU Edge Nodes Problem Description In a remote edge site the AI inference workload never leaves the Pending state. Typical scheduler messages include: 0/5 nodes are available: 5 node(s) didn’t match node selector. Insufficient amd.com/gpu: 0/5 nodes are available: 5 node(s) have insufficient resources. scheduler: no … Read more