rabbitmq

RabbitMQ scheduler pod unschedulable during disaster recovery failover

by
a rack of servers in a server room

Problem – RabbitMQ Scheduler Pod Unschedulable During Disaster Recovery Failover During a planned disaster‑recovery (DR) failover, the RabbitMQ rabbitmq-scheduler pod repeatedly enters the Pending state with the following event: Warning FailedScheduling 2m30s (x5 over 5m) default-scheduler 0/3 nodes are available: 3 node(s) had insufficient memory, 3 node(s) had taint {dr/secondary=true:NoSchedule}, and 3 node(s) had pod … Read more

RabbitMQ replica node stays offline after auto‑scaling in us‑east‑1

by
Rows of black server racks with white logos in a data center

RabbitMQ Replica Node Stays Offline After Auto‑Scaling in us-east-1 Problem In a multi‑region deployment that uses RabbitMQ federation and mirrored queues, the auto‑scaler (AWS Auto Scaling Group) successfully launches new EC2 instances in us-east-1. The instances boot, the RabbitMQ service starts, but the node never joins the existing cluster. The node remains in offline state … Read more

RabbitMQ RAG metadata filter dropping relevant docs after filter tweak

by
Black cables and wires connected to the back of modular LED display panels

Problem The Retrieval‑Augmented Generation (RAG) pipeline uses RabbitMQ to stream document metadata to a GPU‑accelerated inference service. After a recent change to the metadata filter, the pipeline began discarding documents that should have been included in the context set. The LLM responses became fragmented, missing critical facts, and the generation_quality_score metric dropped from 0.87 to … Read more

RAG decomposition service throws DecompositionFailedException on complex queries

by
Black cables and wires connected to the back of modular LED display panels

Problem: DecompositionFailedException in RAG Query Decomposition Service The RAG decomposition microservice consumes user queries from the rag.requests exchange, splits them into sub‑queries, and republishes each sub‑query to topic queues such as rag.subqueries.*. When a complex query (e.g., nested JSON, multi‑entity request, or very long prompt) is processed, the service throws DecompositionFailedException. The exception propagates, the … Read more

RabbitMQ SAML assertion invalid after failover to secondary cloud region

by
a close-up of a server room

Problem Description After an automatic disaster‑recovery (DR) failover to the secondary cloud region, AI training workers were unable to publish messages to RabbitMQ. The broker logged repeated authentication failures with the SAML plugin: 2024-07-31 12:04:18.732 [error] {rabbit_auth_backend_saml, handle_authentication_failure, [{error, “SAML assertion is expired”}, {error, “SAML assertion not yet valid”}, {error, “Invalid SAML audience”}, {error, “Signature … Read more

RabbitMQ queue saturation after increasing batch size in ML training

by
cable network

Problem: RabbitMQ Queue Saturation After Increasing Batch Size in ML Training In a Kubernetes‑deployed distributed training pipeline, the model workers publish gradient‑update messages to a single RabbitMQ queue. After the batch size was raised from 32 to 256, the training loop began to fail with: basic.publish returned: RESOURCE_LOCKED channel flow control activated Message dropped due … Read more

RabbitMQ node resource exhaustion in Docker container

by
img IX mining rig inside white and gray room

Problem Description In a development sandbox the RabbitMQ node runs inside a Docker container with the default image configuration (1 vCPU, 512 MiB RAM). After a burst of simulated AI‑model‑training orchestration messages the broker becomes unresponsive: Memory usage climbs to >95 % of the container limit. Docker daemon emits OOMKill and restarts the container. CPU spikes to 80‑100 % … Read more

RabbitMQ pod eviction during CI/CD pipeline load testing

by
photo of computer cables

RabbitMQ pod eviction during CI/CD pipeline load testing Problem During nightly CI/CD pipeline runs the RabbitMQ StatefulSet is repeatedly evicted by the kube‑scheduler. The eviction manifests as a FailedScheduling event and the pod enters Terminating followed by a recreation of the pod with a new name. The pipeline then fails with connection timeouts and lost … Read more