deployment

IAM permission denied when attaching role during blue‑green EC2 deployment

by
Race car with cartoon graphics and aws logo

Problem – IAM Permission Denied When Attaching Role During Blue‑Green EC2 Deployment A fully automated blue‑green deployment creates a fresh EC2 fleet, assigns a pre‑defined IAM role to each instance, switches the load balancer target group, and finally terminates the old fleet. During the AssociateIamInstanceProfile step the deployment script aborts with: ClientError: An error occurred (UnauthorizedOperation) when … Read more

vLLM API server connection timeout after rolling deployment in Kubernetes

by
The text JEV VS LLM* written in black dots on a glass window

Problem After a rolling deployment of the vLLM inference service in a Kubernetes cluster, client applications began receiving connection timeout, connection refused, and HTTP 502 Bad Gateway responses when calling the vLLM API endpoint. The issue appears only after the deployment finishes; the service was reachable before the rollout. Root Cause The timeout originated from … Read more

nginx pods PVC pending after multi-region Kubernetes rollout

by
Rows of black server racks with white logos in a data center

Problem During a staged rollout of nginx across three AWS regions (us-east-1, eu-west-1, ap-southeast-2) the newly created pods repeatedly entered the Pending phase because their PersistentVolumeClaims (PVCs) could not be bound. The issue manifested only after the rollout controller applied the StatefulSet manifest that references a StorageClass with volumeBindingMode: WaitForFirstConsumer. Operators observed the following symptoms: … Read more

Multimodal alignment error on GCP edge node after model rollout

by
photo of computer cables

Multimodal alignment error on GCP edge node after model rollout Problem The vision‑language model deployed on a GCP Compute Engine edge instance (e2-standard‑4 with an attached NVIDIA T4 GPU) began producing mismatched embeddings during real‑time inference. Video frames from the drone’s camera and IoT sensor streams (IMU, GPS, barometer) that should be jointly encoded were … Read more

ReplicaSet scaling failure after blue‑green rollout of Hugging Face model

by
white and black letter b

ReplicaSet Scaling Failure After Blue‑Green Rollout of a Hugging Face Model Problem During a blue‑green deployment of a Hugging Face Transformers API served by TorchServe, the new version’s ReplicaSet does not scale out. Increasing spec.replicas in the Deployment has no effect: no new Pods are created, or the Pods remain in Pending state indefinitely. Typical … Read more

Milvus index rebuild failure in air‑gapped deployment missing binary dependencies

by
A red kite soars against a green forest

Problem In an air‑gapped deployment of Milvus (v2.3.x), attempts to rebuild a vector index on an existing collection fail with errors indicating missing binary dependencies and permission denials. The failure occurs during the milvusctl index rebuild operation and leaves the collection without a usable index. Typical Symptoms CLI returns non‑zero exit code: milvusctl index rebuild … Read more

GPU OOM in inference service after blue‑green deployment on Kubernetes

by
white concrete building under blue sky during daytime

Problem The AI inference microservice runs on GPU‑enabled nodes in a Kubernetes cluster and consumes a high‑throughput Kafka topic. After a blue‑green deployment rollout, the new (green) pods start reporting CUDA out of memory (OOM) errors, causing request failures and a cascade of consumer lag in Kafka. Typical log excerpt from a failing pod: 2026-09-28 … Read more

vLLM node port conflict after adding second deployment on-prem

by
Black cables and wires connected to the back of modular LED display panels

vLLM NodePort Conflict After Adding a Second On‑Prem Deployment Problem Description After deploying a second vLLM instance on the same on‑premises server, both services fail to start. The startup logs contain errors such as: OSError: [Errno 98] Address already in use vLLM server failed to start: socket.error: [Errno 48] Address already in use FailedCreatePodSandBox: failed … Read more

Kafka vector index corruption after blue‑green deployment traffic switch

by
a couple of statues sitting on top of a stone ground

Problem – Vector Index Corruption After Blue‑Green Traffic Switch During a blue‑green rollout of a streaming analytics pipeline, the new version of the service began writing vector index files (Avro‑encoded embeddings) to a compacted Kafka topic while the previous version was still running. After the traffic cut‑over, downstream consumers started failing with deserialization errors such … Read more

Prometheus scrape 401 token limit exceeded during canary rollout

by
a golden statue of a man and a woman on a boat

Problem – 401 Token Limit Exceeded During Canary Scrape During a staged rollout of a new microservice, Prometheus began reporting scrape failures with the following error: error scraping target “http://service-canary:9090/metrics”: unexpected status code 401 Unauthorized (token limit exceeded) Additional log entries from the Prometheus server process show the token acquisition step failing: 2024-08-12T14:23:07Z level=error msg=”failed to fetch bearer … Read more