pod-eviction

Kubernetes pod eviction due to OOM on AMD GPU nodes

by
lighted black and gray graphics card

Problem – Frequent Pod Evictions on AMD GPU Nodes During Disaster Recovery During disaster‑recovery (DR) drills the AI training platform experiences a surge of concurrent training jobs. On clusters that use AMD MI250X GPUs the kubelet repeatedly evicts pods with the reason=OutOfmemory condition. Typical symptoms observed: Pods transition to Evicted status within minutes of the … Read more

RabbitMQ pod eviction during CI/CD pipeline load testing

by
photo of computer cables

RabbitMQ pod eviction during CI/CD pipeline load testing Problem During nightly CI/CD pipeline runs the RabbitMQ StatefulSet is repeatedly evicted by the kube‑scheduler. The eviction manifests as a FailedScheduling event and the pod enters Terminating followed by a recreation of the pod with a new name. The pipeline then fails with connection timeouts and lost … Read more