RabbitMQ Scheduler Pod Unschedulable During Disaster Recovery Failover

Q: Debug – Step‑by‑Step Investigation 1. Inspect pod events and node conditions

kubectl describe pod rabbitmq-scheduler-0 -n dr-namespace Key output excerpt:

Q: 2. Verify node taints and available resources

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}' kubectl top nodes Sample output:

Q: 3. Examine the scheduler pod spec generated by the RabbitMQ operator

kubectl get statefulset rabbitmq -n dr-namespace -o yaml | grep -A5 "scheduler" Output shows the scheduler inherits the same resource requests as the main RabbitMQ containers but lacks a toleration entry.

Q: 4. Check anti‑affinity rules on existing RabbitMQ pods

kubectl get pod -l app=rabbitmq -n dr-namespace -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].topologyKey}{"\n"}{end}' Result confirms each pod is bound to a distinct hostname, consuming all three nodes.

Q: 2. Add toleration for the DR taint to the scheduler pod template

scheduler: tolerations: - key: "dr/secondary" operator: "Equal" value: "true" effect: "NoSchedule" 3. Relax anti‑affinity for the scheduler only Scheduler pods do not need to avoid other RabbitMQ pods. Override the affinity for the scheduler container:

Q: 4. Apply the updated RabbitMQ custom resource (CR)

kubectl apply -f rabbitmq-dr-cr.yaml -n dr-namespace The operator will roll out the changes, recreating the scheduler pod with the new spec.

Q: Verify – Confirm Scheduler Pod is Running and Functional 1. Check pod status

kubectl get pod -l app=rabbitmq-scheduler -n dr-namespace Expected output:

In this article

Table of Contents

Problem – RabbitMQ Scheduler Pod Unschedulable During Disaster Recovery Failover

During a planned disaster‑recovery (DR) failover, the RabbitMQ rabbitmq-scheduler pod repeatedly enters the Pending state with the following event:


Warning  FailedScheduling  2m30s (x5 over 5m)  default-scheduler
  0/3 nodes are available: 3 node(s) had insufficient memory, 3 node(s) had taint {dr/secondary=true:NoSchedule}, 
  and 3 node(s) had pod affinity conflicts.

The failure prevents the scheduler from creating delayed messages, causing downstream services that rely on TTL‑based dead‑letter routing to stall.

Root Cause – Resource Constraints Coupled With Taint‑Based Node Isolation

The DR cluster is a minimal footprint environment consisting of three worker nodes, each provisioned with 8 GiB RAM and a dr/secondary=true:NoSchedule taint. The RabbitMQ StatefulSet for the DR site is configured with the following affinity and resource requests:


apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
spec:
  serviceName: rabbitmq
  replicas: 3
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: rabbitmq
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: rabbitmq
        image: rabbitmq:3.11-management
        resources:
          requests:
            cpu: "2000m"
            memory: "4Gi"
          limits:
            cpu: "4000m"
            memory: "8Gi"

Key points:

Memory request (4 GiB) per pod exceeds the available free memory on each node after the three primary RabbitMQ pods are scheduled (3 × 4 GiB = 12 GiB > 8 GiB per node).
The dr/secondary=true:NoSchedule taint isolates the DR nodes from regular workloads, but the scheduler pod does not have a toleration, so it cannot be placed on any node that does not carry the taint.
Pod anti‑affinity forces each RabbitMQ pod onto a distinct node, further fragmenting the limited resources.

Consequently, when the DR failover initiates, the scheduler pod cannot find a node that simultaneously satisfies:

Memory request.
Toleration for the DR taint.
Anti‑affinity constraints.

Debug – Step‑by‑Step Investigation

1. Inspect pod events and node conditions


kubectl describe pod rabbitmq-scheduler-0 -n dr-namespace

Key output excerpt:


Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  3m    default-scheduler  0/3 nodes are available: 3 node(s) had insufficient memory, 3 node(s) had taint {dr/secondary=true:NoSchedule}, and 3 node(s) had pod affinity conflicts.

2. Verify node taints and available resources


kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'
kubectl top nodes

Sample output:


node-a    [{"key":"dr/secondary","value":"true","effect":"NoSchedule"}]
node-b    [{"key":"dr/secondary","value":"true","effect":"NoSchedule"}]
node-c    [{"key":"dr/secondary","value":"true","effect":"NoSchedule"}]

NAME      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node-a    2500m        62%    10240Mi          80%
node-b    2300m        57%    11000Mi          85%
node-c    2100m        52%    9500Mi           73%

3. Examine the scheduler pod spec generated by the RabbitMQ operator


kubectl get statefulset rabbitmq -n dr-namespace -o yaml | grep -A5 "scheduler"

Output shows the scheduler inherits the same resource requests as the main RabbitMQ containers but lacks a toleration entry.

4. Check anti‑affinity rules on existing RabbitMQ pods


kubectl get pod -l app=rabbitmq -n dr-namespace -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].topologyKey}{"\n"}{end}'

Result confirms each pod is bound to a distinct hostname, consuming all three nodes.

Solution – Adjust Resource Requests, Add Tolerations, and Relax Anti‑Affinity

1. Reduce memory request for the scheduler

The scheduler does not need the full 4 GiB. Lower it to 256 MiB, which is sufficient for delayed‑message handling.


# Before (scheduler inherits full request)
resources:
  requests:
    cpu: "2000m"
    memory: "4Gi"
  limits:
    cpu: "4000m"
    memory: "8Gi"

# After (scheduler specific overrides)
scheduler:
  resources:
    requests:
      cpu: "250m"
      memory: "256Mi"
    limits:
      cpu: "500m"
      memory: "512Mi"

2. Add toleration for the DR taint to the scheduler pod template


scheduler:
  tolerations:
  - key: "dr/secondary"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

3. Relax anti‑affinity for the scheduler only

Scheduler pods do not need to avoid other RabbitMQ pods. Override the affinity for the scheduler container:


scheduler:
  affinity: {}

4. Apply the updated RabbitMQ custom resource (CR)


kubectl apply -f rabbitmq-dr-cr.yaml -n dr-namespace

The operator will roll out the changes, recreating the scheduler pod with the new spec.

Verify – Confirm Scheduler Pod is Running and Functional

1. Check pod status


kubectl get pod -l app=rabbitmq-scheduler -n dr-namespace

Expected output:


NAME                     READY   STATUS    RESTARTS   AGE
rabbitmq-scheduler-0     1/1     Running   0          2m

2. Validate delayed message handling

Publish a message with a TTL of 30 seconds and confirm it appears in the dead‑letter queue after the timeout.


# Publish with TTL
rabbitmqadmin publish exchange=amq.direct routing_key=test payload="test" properties='{"expiration":"30000"}'

# Consume from DLX after 35 seconds
rabbitmqadmin get queue=dlx_queue requeue=false count=1

Successful receipt indicates the scheduler is operational.

3. Monitor resource usage


kubectl top pod rabbitmq-scheduler-0 -n dr-namespace

Typical output:


NAME                 CPU(cores)   MEMORY(bytes)
rabbitmq-scheduler-0  120m         180Mi

Prevent – Recommendations for Future DR Deployments

Separate resource profiles: Define lightweight resource requests for auxiliary pods (scheduler, metrics exporter) that differ from the main RabbitMQ containers.
Explicit tolerations on all DR‑only workloads to avoid accidental scheduling on primary clusters.
Use node selectors or affinity instead of blanket anti‑affinity when node count is limited.
Capacity planning: Reserve at least 20 % of node memory for system daemons and burst workloads during failover.
Automated pre‑flight check: Script that validates total requested memory against node capacity before initiating a DR failover.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does the scheduler pod inherit the same resource requests as the main RabbitMQ pods?
The RabbitMQ operator uses a single resources block for all containers unless an explicit override is provided. Without a scheduler.resources section, the scheduler defaults to the primary pod’s values.
Can I keep the anti‑affinity rule for the scheduler?
Yes, but you must also increase node capacity or add additional nodes. The simplest fix is to remove the rule for the scheduler, as it does not benefit from spreading across hosts.
What happens if I forget to add the toleration?
The scheduler will continue to be rejected by the NoSchedule taint, resulting in the same “insufficient memory” message because the scheduler is never considered for placement on the tainted nodes.
Is it safe to lower the scheduler’s memory request to 128 MiB?
In most deployments 128 MiB is sufficient, but monitor rabbitmq_scheduled_messages metrics after the change. If you see spikes, increase to 256 MiB.
How can I automate the verification of scheduler health after a DR failover?
Create a Kubernetes Job that publishes a TTL message and checks the dead‑letter queue, then expose the result via a Prometheus metric or a simple exit code for CI pipelines.