rabbitmq

Problem Description After an automatic disaster‑recovery (DR) failover to the secondary cloud region, AI training workers were unable to publish messages to RabbitMQ. The broker logged repeated authentication failures with the SAML plugin: 2024-07-31 12:04:18.732 [error] {rabbit_auth_backend_saml, handle_authentication_failure, [{error, “SAML assertion is expired”}, {error, “SAML assertion not yet valid”}, {error, “Invalid SAML audience”}, {error, “Signature … Read more

Problem: RabbitMQ Queue Saturation After Increasing Batch Size in ML Training In a Kubernetes‑deployed distributed training pipeline, the model workers publish gradient‑update messages to a single RabbitMQ queue. After the batch size was raised from 32 to 256, the training loop began to fail with: basic.publish returned: RESOURCE_LOCKED channel flow control activated Message dropped due … Read more

Problem Description In a development sandbox the RabbitMQ node runs inside a Docker container with the default image configuration (1 vCPU, 512 MiB RAM). After a burst of simulated AI‑model‑training orchestration messages the broker becomes unresponsive: Memory usage climbs to >95 % of the container limit. Docker daemon emits OOMKill and restarts the container. CPU spikes to 80‑100 % … Read more