storage

AMD GPU nodes TLS handshake failure to artifact storage after driver update

by
Amd ryzen 6000 series processor on circuit board background

AMD GPU Nodes TLS Handshake Failure to Artifact Storage After Driver Update Problem Description After upgrading the ROCm driver suite to 6.0.3 on a mixed‑GPU Kubernetes cluster (NVIDIA A100/H100 and AMD Instinct MI250), all pods scheduled on AMD GPU nodes began failing to download model artifacts from the central artifact-storage service (Azure Blob, S3, or … Read more

TGI checkpoint storage class not found after A/B deployment

by
A book page with printed text and blank lined paper

TGI checkpoint storage class not found after A/B deployment Problem During an A/B testing rollout of Text Generation Inference (TGI) instances, the newly created replica fails to start and logs an error similar to: Failed to load checkpoint: storage class “fast-ssd” not found Other observed symptoms include: Pod tgi-a-12345 stays in Pending state. kubectl describe … Read more