Cloud Infrastructure

IAM permission denied when attaching role during blue‑green EC2 deployment

by
Race car with cartoon graphics and aws logo

Problem – IAM Permission Denied When Attaching Role During Blue‑Green EC2 Deployment A fully automated blue‑green deployment creates a fresh EC2 fleet, assigns a pre‑defined IAM role to each instance, switches the load balancer target group, and finally terminates the old fleet. During the AssociateIamInstanceProfile step the deployment script aborts with: ClientError: An error occurred (UnauthorizedOperation) when … Read more

Compute Engine index rebuild failure during nightly batch on AI service

by
cable network

Problem During the nightly batch window a search index rebuild executed by the AI model‑serving workload on Google Compute Engine (GCE) VMs fails repeatedly. The failure prevents the latest documents from being searchable until the next successful run, causing stale results for downstream applications. Typical symptoms observed on the affected instances: Batch job exits with … Read more

Azure VM model loading timeout after DR failover to secondary region

by
a white computer mouse

Problem – Model Loading Timeout After DR Failover to Secondary Region During a planned disaster‑recovery (DR) failover, traffic is redirected from the primary Azure region to a secondary region where a standby Azure Virtual Machine (VM) hosts the AI inference service. The service fails to start because the model loading step exceeds the configured startup … Read more

RAG query decomposition error after adding new microservice on GCP Compute

by
photo of computer cables

RAG Query Decomposition Error After Adding a New Microservice on GCP Compute Problem After deploying a new stateless microservice (order‑processor) on a GCP Compute Engine instance, the Retrieval‑Augmented Generation (RAG) service began returning DecompositionError: unable to split query into sub‑queries. The error appears only for multi‑step queries that require data from both the existing inventory … Read more

Multimodal alignment error on GCP edge node after model rollout

by
photo of computer cables

Multimodal alignment error on GCP edge node after model rollout Problem The vision‑language model deployed on a GCP Compute Engine edge instance (e2-standard‑4 with an attached NVIDIA T4 GPU) began producing mismatched embeddings during real‑time inference. Video frames from the drone’s camera and IoT sensor streams (IMU, GPS, barometer) that should be jointly encoded were … Read more

Context window overflow on EC2 inference after sending oversized prompt

by
Female speaker presenting in front of a projector screen

Problem: Context Window Overflow on EC2‑Hosted LLM Inference When a batch job on an on‑premises server sends a prompt that exceeds the token limit of the deployed large language model (LLM), the inference API on the AWS EC2 instance returns a TokenLimitExceeded error. The failure manifests as: 2026-09-30T14:12:03.421Z ERROR inference_service: TokenLimitExceeded: Prompt length 8192 tokens … Read more

RAG citation JSON‑LD missing fields after model version upgrade

by
photo of computer cables

Problem: Malformed JSON‑LD Citations in RAG Responses after Model Version Upgrade The AI service running on Google Compute Engine (GCE) VMs generates Retrieval‑Augmented Generation (RAG) answers that include citations. After upgrading the underlying LLM from v1.2.0 to v1.3.0, the citation strings no longer conform to the required JSON‑LD schema. Missing fields (@type, url), stray delimiters, … Read more

Azure VM OOM error when launching AI inference after DR failover

by
Security, privacy, and performance status with fix options

Problem – Azure VM OOM error when launching AI inference after DR failover During a planned disaster‑recovery (DR) failover from the primary Azure region to a secondary region, the AI inference service fails to start. The container runtime reports a memory allocation failure and the process is terminated by the Linux OOM killer: 2024-09-15T08:12:34.567Z ERROR … Read more

Upsert timeout writing checkpoints from multi‑GPU EC2 training job

by
Female speaker presenting in front of a projector screen

Problem – Upsert Timeout When Writing Checkpoints from a Multi‑GPU EC2 Training Job During a distributed PyTorch training run on an m5.24xlarge instance with 8 NVIDIA A100 GPUs, the checkpointing routine attempts to write model metadata (epoch, loss, S3 URI) into a PostgreSQL database using an UPSERT statement. After the 3rd epoch the job aborts … Read more

Inference queue backlog on EC2 during CI/CD pipeline execution

by
Race car with cartoon graphics and aws logo

Problem During CI/CD pipeline runs that automatically deploy new model versions, the inference service running on a fleet of Amazon EC2 instances experiences a rapid increase in request latency. The inference_queue length grows to several thousand entries, causing time‑outs and failed inference jobs. Typical symptoms observed in the pipeline logs are: [2026-09-22T14:03:12Z] INFO inference_worker: Queue … Read more