RabbitMQ Node Resource Exhaustion In Docker Container

Q: 1. Inspect container resource usage

docker stats rabbitmq-sandbox # Expected output (example) CONTAINER ID NAME CPU % MEM USAGE / LIMIT NET I/O a1b2c3d4e5f6 rabbitmq-sandbox 92.5% 498MiB / 512MiB 12.3MB / 8.7MB 2. Query RabbitMQ memory alarm state docker exec rabbitmq-sandbox rabbitmqctl status | grep memory {memory,alarm} = true 3. Identify queues with the highest message count docker exec rabbitmq-sandbox rabbitmqctl list_queues name messages messages_ready messages_unacknowledged # Sample output training.tasks 215432 215432 0 model.status 10234 10234 0 4. Capture a short packet trace to confirm producer burst docker exec rabbitmq-sandbox tcpdump -i any -nn -s 0 port 5672 -c 20 -w /tmp/trace.pcap Analyze the pcap for a high rate of basic.publish frames.

Q: 5. Review Erlang VM memory statistics

docker exec rabbitmq-sandbox rabbitmqctl eval 'erlang:memory().' # Example output [{total,471859200}, {processes,12345678}, {processes_used,11223344}, {system,459513522}, {atom,123456}, {binary,9876543}, {code,21000000}, {ets,3456789}] Compare total against the container limit (512 MiB ≈ 536 870 912 bytes).

Q: 1. Apply back‑pressure at the producer

Modify the producer client to respect the basic.publish channel.flow signal or to use publisher confirms with a bounded in‑flight window.

Q: 2. Reduce the memory high watermark for containers

Set vm_memory_high_watermark to a lower fraction (e.g., 0.2) and optionally cap absolute memory with vm_memory_limit.

Q: 3. Configure Docker container limits and ulimits

Launch the container with explicit memory and ulimit values that match the broker configuration.

In this article

Table of Contents

Problem Description

In a development sandbox the RabbitMQ node runs inside a Docker container with the default image configuration (1 vCPU, 512 MiB RAM). After a burst of simulated AI‑model‑training orchestration messages the broker becomes unresponsive:

Memory usage climbs to >95 % of the container limit.
Docker daemon emits OOMKill and restarts the container.
CPU spikes to 80‑100 % and the management UI stops responding.

Log entries such as:


2023-10-12 14:23:45.123 [error] memory high watermark reached: 460 MiB / 512 MiB
2023-10-12 14:23:45.124 [error] {memory,alarm} = true
2023-10-12 14:23:46.001 [warning] disk free alarm: free space 1.2 GiB < 1 GiB
2023-10-12 14:23:46.005 [error] rabbitmq_server: memory alarm triggered – shutting down non‑essential processes

Running rabbitmqctl status shows:


{memory,alarm} = true
{disk_free,alarm} = true

The node repeatedly logs “memory high watermark reached” and eventually the Docker daemon kills the container, mirroring the incidents described in the official Memory and Flow Control guide and community reports (e.g. GitHub issue #3382, Stack Overflow 58692312).

Root Cause Analysis

RabbitMQ tracks memory consumption per Erlang process and per queue. When the total memory exceeds the configured vm_memory_high_watermark (default 0.4 of the system memory) it raises a memory alarm. In a containerized sandbox the effective system memory is the cgroup limit (512 MiB). The following factors combined to trigger the alarm:

Unbounded queue growth – Producers publish status updates for each training job without back‑pressure. Durable queues accumulated >200 k messages, each message header + payload consuming ~2 KB, quickly exhausting RAM.
Insufficient flow control – The default memory alarm throttles publishers but only after the high watermark is breached. By that time the container is already near OOM.
CPU overhead from queue indexing – Large unprocessed queues force frequent index rebuilds (see RabbitMQ Resource Usage documentation), driving CPU to 100 %.
Docker default limits – The official RabbitMQ image does not set --memory or ulimit values; the process can allocate up to the host’s limit, but the cgroup limit remains low, causing the Erlang VM to mis‑estimate available memory.

In short, the sandbox’s limited resources plus an unchecked message backlog caused the memory alarm, which cascaded into CPU saturation and an OOM kill.

Investigation and Debugging

Follow these steps to reproduce the symptom and isolate the offending component.

1. Inspect container resource usage

docker stats rabbitmq-sandbox
# Expected output (example)
CONTAINER ID   NAME               CPU %   MEM USAGE / LIMIT   NET I/O
a1b2c3d4e5f6   rabbitmq-sandbox   92.5%   498MiB / 512MiB     12.3MB / 8.7MB

2. Query RabbitMQ memory alarm state

docker exec rabbitmq-sandbox rabbitmqctl status | grep memory
{memory,alarm} = true

3. Identify queues with the highest message count

docker exec rabbitmq-sandbox rabbitmqctl list_queues name messages messages_ready messages_unacknowledged
# Sample output
training.tasks           215432   215432   0
model.status             10234    10234    0

4. Capture a short packet trace to confirm producer burst

docker exec rabbitmq-sandbox tcpdump -i any -nn -s 0 port 5672 -c 20 -w /tmp/trace.pcap

Analyze the pcap for a high rate of basic.publish frames.

5. Review Erlang VM memory statistics

docker exec rabbitmq-sandbox rabbitmqctl eval 'erlang:memory().' 
# Example output
[{total,471859200},
 {processes,12345678},
 {processes_used,11223344},
 {system,459513522},
 {atom,123456},
 {binary,9876543},
 {code,21000000},
 {ets,3456789}]

Compare total against the container limit (512 MiB ≈ 536 870 912 bytes).

Resolution

The fix consists of three parts: limit queue growth, tune RabbitMQ memory watermarks for containers, and configure Docker resource limits with proper ulimit settings.

1. Apply back‑pressure at the producer

Modify the producer client to respect the basic.publish channel.flow signal or to use publisher confirms with a bounded in‑flight window.

# Python pika example
channel.confirm_delivery()
while True:
    try:
        channel.basic_publish(exchange='',
                              routing_key='training.tasks',
                              body=payload,
                              mandatory=True)
    except pika.exceptions.NackError:
        # Back‑off when broker signals flow control
        time.sleep(0.5)

2. Reduce the memory high watermark for containers

Set vm_memory_high_watermark to a lower fraction (e.g., 0.2) and optionally cap absolute memory with vm_memory_limit.

Before (default rabbitmq.conf):

# /etc/rabbitmq/rabbitmq.conf
# No explicit memory settings – uses defaults

After (sandbox‑optimized rabbitmq.conf):

# /etc/rabbitmq/rabbitmq.conf
vm_memory_high_watermark.relative = 0.2
# Optional absolute limit (in bytes)
# vm_memory_limit = 400MiB

3. Configure Docker container limits and ulimits

Launch the container with explicit memory and ulimit values that match the broker configuration.

docker run -d --name rabbitmq-sandbox \
  --memory=512m --memory-swap=512m \
  --ulimit nofile=65536:65536 \
  -e RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+M 1024" \
  -p 5672:5672 -p 15672:15672 \
  rabbitmq:3.11-management

The +M 1024 flag raises the Erlang VM’s internal process limit, preventing premature alarm triggering due to low processes memory.

4. Enable automatic queue TTL for transient data

If the status messages are only needed for a short window, add a per‑queue message-ttl to let RabbitMQ discard stale messages.

docker exec rabbitmq-sandbox rabbitmqctl set_policy \
  ttl-policy "^training\." '{"message-ttl":60000}' --apply-to queues

Why the fix works

Lowering vm_memory_high_watermark causes the broker to raise the alarm earlier, giving the container’s OOM killer more headroom.
Back‑pressure stops producers from overwhelming the broker, keeping queue length bounded.
Explicit Docker limits ensure the Erlang VM sees the correct memory ceiling, aligning its internal calculations with cgroup constraints.
TTL policies automatically prune unconsumed messages, preventing indefinite growth.

Validation

After applying the changes, verify that the node no longer enters an alarm state under the same workload.

Restart the container with the new configuration.
Re‑run the producer burst.
Check that docker stats shows memory staying below 80 %:

CONTAINER ID   NAME               CPU %   MEM USAGE / LIMIT
b7c8d9e0f1a2   rabbitmq-sandbox   45.2%   380MiB / 512MiB

Confirm the alarm is cleared:

docker exec rabbitmq-sandbox rabbitmqctl status | grep memory
{memory,alarm} = false

Inspect queue lengths; they should remain within expected bounds (e.g., <2000 messages for a 60‑second TTL).

Prevention and Best Practices

Set container‑aware memory watermarks – always override the default vm_memory_high_watermark when running in cgroups.
Implement producer back‑pressure – use publisher confirms or channel.flow callbacks.
Use queue TTL or max‑length policies for transient data streams.
Monitor the management plugin metrics – watch memory_used, memory_limit, and disk_free alerts.
Allocate sufficient CPU – large queues trigger index rebuilds; a minimum of 2 vCPU is recommended for dev sandboxes handling bursts.
Enable RabbitMQ’s built‑in flow control – ensure vm_memory_high_watermark is lower than 0.5 to give the VM time to react before the container hits OOM.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does the memory alarm trigger only after a burst? The alarm is based on absolute memory usage. A sudden influx of messages pushes the broker past the vm_memory_high_watermark, which is only evaluated periodically.
Can I disable the memory alarm? Disabling it is unsafe; it removes RabbitMQ’s protection against OOM. Instead, tune the watermark and enforce back‑pressure.
How do I know which queue is consuming most memory? Use rabbitmqctl list_queues name memory (requires the rabbitmq_management plugin) or query the management UI’s “Memory” tab.
Is increasing --memory the only solution? It mitigates the symptom but does not address unbounded queue growth. Proper flow control and TTL policies are essential for long‑term stability.
What impact does vm_memory_limit have inside Docker? It caps the Erlang VM’s view of available memory, aligning it with the container’s cgroup limit and preventing the VM from assuming more RAM than is actually allocated.