photo of computer cables

RabbitMQ pod eviction during CI/CD pipeline load testing

by

RabbitMQ pod eviction during CI/CD pipeline load testing

Problem

During nightly CI/CD pipeline runs the RabbitMQ StatefulSet is repeatedly evicted by the kube‑scheduler. The eviction manifests as a FailedScheduling event and the pod enters Terminating followed by a recreation of the pod with a new name. The pipeline then fails with connection timeouts and lost messages.

Typical symptoms observed in the pipeline logs:


[INFO] Starting load test: publishing 10k messages per second
[ERROR] Connection to rabbitmq-0.rabbitmq.default.svc:5672 failed: EOF
[ERROR] pod rabbitmq-0 evicted: memory pressure

Cluster‑wide alerts:


KubePodEvicted{pod="rabbitmq-0"} 1
node_memory_pressure{node="worker-1"} 1

Root Cause

The StatefulSet runs with default resource requests (CPU = 100m, memory = 128Mi) and no limits. During load testing the broker processes a surge of publish‑confirm traffic that spikes resident set size (RSS) well beyond the node’s allocatable memory. The kubelet’s eviction_manager detects node‑level memory pressure and evicts the highest‑priority pod that does not have a PodDisruptionBudget protecting it – in this case the RabbitMQ pod.

Two contributing factors:

  • Insufficient resource requests/limits: The scheduler assumes the pod only needs 128 MiB, so it places it on a node already near capacity.
  • Absence of QoS guarantees: Without a memory limit, the container can consume all available memory, triggering node‑level pressure.

Because the pod is stateful, eviction forces a restart, causing the cluster to re‑elect a leader and lose in‑flight messages.

Debug

Step‑by‑step investigation performed during the failure window:

  1. Inspect the eviction event:

kubectl describe pod rabbitmq-0 | grep -A3 "Events"


Events:
  Type     Reason                 Age   From               Message
  ----     ------                 ----  ----               -------
  Warning  EvictionThresholdMet   2m    kubelet,worker-1  Pod rabbitmq-0 evicted due to memory pressure
  1. Check node memory pressure metrics:

kubectl top node worker-1


NAME        CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
worker-1    1800m        90%    30Gi / 32Gi     93%
  1. Examine RabbitMQ process memory usage during the test:

kubectl exec -it rabbitmq-0 -- bash -c "ps -o pid,rss,cmd -C beam.smp"


  PID   RSS CMD
  12   12456 /usr/lib/rabbitmq/bin/beam.smp -kernel inet_dist_listen_min 25672 ...
  1. Review the StatefulSet manifest for resource specifications:

kubectl get statefulset rabbitmq -o yaml | grep -A5 resources


resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  1. Capture a short packet trace to confirm message burst size:

kubectl exec -n default rabbitmq-0 -- tcpdump -i any -s 0 -w /tmp/trace.pcap port 5672 &
sleep 30
kill %1
kubectl cp rabbitmq-0:/tmp/trace.pcap ./trace.pcap

Analysis of the pcap showed a sustained 10k msgs/s publish rate during the test.

Solution

Apply proper resource guarantees and protect the StatefulSet with a PodDisruptionBudget. The following changes were made:

Before


apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
spec:
  serviceName: rabbitmq
  replicas: 3
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      containers:
      - name: rabbitmq
        image: rabbitmq:3.11-management
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"

After – add limits, higher requests, and a PDB


apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: rabbitmq-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: rabbitmq
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
spec:
  serviceName: rabbitmq
  replicas: 3
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      containers:
      - name: rabbitmq
        image: rabbitmq:3.11-management
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
          limits:
            cpu: "1"
            memory: "2Gi"
        env:
        - name: RABBITMQ_VM_MEMORY_HIGH_WATERMARK
          value: "0.75"   # 75 % of the container memory

Key changes:

  • Requests increased to 500 m CPU and 1 GiB memory, ensuring the scheduler places the pod on a node with sufficient headroom.
  • Limits added to cap memory at 2 GiB, preventing a single broker from exhausting node memory.
  • VM memory high watermark set to 75 % of the container limit, causing RabbitMQ to apply flow control before the OS OOM killer intervenes.
  • PodDisruptionBudget guarantees at least two replicas remain available, so an eviction would be blocked unless the entire StatefulSet is scaled down.

Verify

After applying the updated manifest:


kubectl apply -f rabbitmq-statefulset.yaml
kubectl rollout status statefulset/rabbitmq

Run the same load test and observe:


kubectl logs -f rabbitmq-0 | grep "memory"


2026-06-13 12:34:10.123 [info] Memory usage: 1.4GiB / 2GiB (70%)

Node pressure metrics return to normal:


kubectl top node worker-1


NAME        CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
worker-1    1200m        60%    20Gi / 32Gi     62%

No further EvictionThresholdMet events appear in kubectl get events during the test run.

Prevent

  • Capacity planning: Benchmark expected message rates and size, then size requests accordingly.
  • Memory high watermark tuning: Set RABBITMQ_VM_MEMORY_HIGH_WATERMARK to 0.6‑0.8 of the container limit to trigger broker‑level flow control before node pressure.
  • Horizontal scaling: Deploy additional RabbitMQ replicas or a separate queue per test suite to distribute load.
  • Monitoring & alerts: Add Prometheus alerts on rabbitmq_mem_used and node memory_pressure to catch early spikes.
  • CI/CD throttling: Limit concurrent load‑test jobs that target the same RabbitMQ cluster, or run them against an isolated test namespace.

FAQ

  1. Why does the pod get evicted even though I set requests? The original manifest only set a very low request (128 MiB). The scheduler placed the pod on a node already near capacity, so when the broker’s memory grew, node pressure triggered eviction.
  2. Can I rely on RabbitMQ’s internal flow control without setting a memory limit? No. Flow control only activates relative to the container’s memory limit. Without a limit, the broker cannot calculate a high‑watermark, and the OS may kill the process before RabbitMQ throttles.
  3. Do I need to adjust the Kubernetes QoS class? By providing both requests and limits, the pod receives a Guaranteed QoS class, which protects it from being the first victim of node‑level eviction.
  4. How can I test the new configuration before committing to CI? Deploy the updated StatefulSet to a staging namespace and run a synthetic load generator (e.g., rabbitmq-perf-test) at the expected peak rate, monitoring rabbitmq_mem_used and node memory.
  5. Is a PodDisruptionBudget enough to stop evictions? A PDB prevents voluntary disruptions (e.g., rolling updates) but does not block forced evictions due to node pressure. Proper resource sizing and limits are still required.

Related Topic Hub: Data Infrastructure Troubleshooting Hub