GPU Infrastructure

CRD validation error rejecting AMD GPU spec in deployment manifest

by
a black and silver electronic device

Problem – CRD Validation Rejects AMD GPU Specification in Deployment Manifest When deploying an AI inference service that requires an AMD GPU, the kubectl apply -f deployment.yaml command fails with a validation error from the AMD GPU admission webhook. Typical error output looks like: error: admission webhook “validate.amd.com” denied the request: spec.template.spec.containers[0].resources.limits.amd.com/gpu: Invalid value: “1” … Read more

CUDA OOM during long sequence inference on NVIDIA GPU

by
the nvidia logo is displayed on a table

Problem Description On‑premise LLM inference nodes equipped with NVIDIA GPUs (e.g., A100 40 GiB, RTX 3090 24 GiB) crash when processing prompts that exceed the pre‑allocated key‑value (KV) cache size. The failure manifests as a CUDA out‑of‑memory (OOM) exception during the attention kernel launch. Typical log excerpts: [2026-08-10 14:32:07] ERROR – torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 … Read more

AMD GPU context window overflow in air-gapped environment

by
a black and silver electronic device

Problem Description In an air‑gapped deployment of large language models (LLMs) on AMD GPUs, inference jobs abort once the prompt length exceeds a certain token count. The failure manifests as memory allocation errors that appear unrelated to the reported free VRAM. Typical log excerpts: 2026-07-12 14:03:21 [INFO] Starting inference on model LLaMA-13B 2026-07-12 14:03:22 [ERROR] … Read more

AMD GPU token refresh failure during disaster recovery

by
Amd ryzen 6000 series processor on circuit board background

Problem – Token Refresh Failure During Disaster Recovery on AMD GPU Systems During a planned disaster‑recovery (DR) drill, AI inference workloads that rely on AMD Instinct GPUs stopped abruptly after the failover to the backup cluster. The ROCm runtime emitted the following error: TokenRefreshError: failed to refresh access token – ROCm runtime aborts with exit … Read more

Kubernetes HPA not scaling pods with NVIDIA GPU metrics

by
black and silver sony cassette player

Problem Description The Horizontal Pod Autoscaler (HPA) in a Kubernetes‑based CI/CD pipeline is expected to add or remove training pods based on GPU utilization. In practice the HPA never scales out: the replica count stays at 1 even when GPU usage spikes to 95 % during model training. The symptom manifests as a growing job queue, … Read more

RAG citation formatting issues with AMD GPU in Kubernetes

by
Amd ryzen 6000 series processor on circuit board background

Problem Description In a Kubernetes‑based microservices architecture that uses AMD GPUs for Retrieval‑Augmented Generation (RAG), the generated citations are malformed. Observed symptoms include: Missing source fields in the JSON payload returned by the generation service. Inconsistent or empty metadata.title and metadata.author entries. Intermittent duplicate citation entries when pods restart under memory pressure. Log excerpts such … Read more

Kubernetes scheduler failing to place AI pods on AMD GPU edge nodes

by
Amd ryzen 6000 series processor on circuit board background

Kubernetes Scheduler Failing to Place AI Pods on AMD GPU Edge Nodes Problem Description In a remote edge site the AI inference workload never leaves the Pending state. Typical scheduler messages include: 0/5 nodes are available: 5 node(s) didn’t match node selector. Insufficient amd.com/gpu: 0/5 nodes are available: 5 node(s) have insufficient resources. scheduler: no … Read more

AMD GPU CLI argument parsing failure during disaster recovery

by
a computer case with a neon sign on the side of it

AMD GPU CLI Argument Parsing Failure During Disaster Recovery Problem Description (Symptoms and Impact) During automated disaster‑recovery runs on AMD GPU clusters, the recovery scripts invoke ROCm utilities such as rocm-smi, rocminfo, and rocm-bandwidth-test. The scripts abort with parsing errors, causing the scheduler (e.g., Slurm) to mark all GPUs as unavailable. Typical log excerpts are: … Read more

ROCm version mismatch during canary deployment on AMD GPU

by
lighted black and gray graphics card

Problem – Collection Creation Failure During Canary Deployment on AMD MI210 Nodes During a rolling canary deployment of a new AI model on a Kubernetes cluster that contains AMD Instinct MI210 accelerators, pods repeatedly terminate with the following error: [2026-06-27 10:12:34] ERROR: Failed to create collection: ROCm runtime version 5.5 does not match driver version 5.6 … Read more

Kubernetes pod eviction due to OOM on AMD GPU nodes

by
lighted black and gray graphics card

Problem – Frequent Pod Evictions on AMD GPU Nodes During Disaster Recovery During disaster‑recovery (DR) drills the AI training platform experiences a surge of concurrent training jobs. On clusters that use AMD MI250X GPUs the kubelet repeatedly evicts pods with the reason=OutOfmemory condition. Typical symptoms observed: Pods transition to Evicted status within minutes of the … Read more