RabbitMQ replica node stays offline after auto‑scaling in us‑east‑1

RabbitMQ Replica Node Stays Offline After Auto‑Scaling in us-east-1

Problem

In a multi‑region deployment that uses RabbitMQ federation and mirrored queues, the auto‑scaler (AWS Auto Scaling Group) successfully launches new EC2 instances in us-east-1. The instances boot, the RabbitMQ service starts, but the node never joins the existing cluster. The node remains in offline state in the management UI and rabbitmqctl cluster_status shows it as a disc node that is not part of the cluster.

Typical symptoms:

• Auto‑scaling event logs show instance i-0abcd1234efgh5678 launched.
• Systemd logs: systemd[1]: Started RabbitMQ broker.
• RabbitMQ logs contain:

2024-10-07 12:03:45.123 [error] <0.1234.0> failed to join cluster: {cannot_connect, {node_not_found, 'rabbit@ip-10-0-0-45.us-east-1.compute.internal'}}

• rabbitmqctl status on the new node reports {running_applications,[]} and {cluster_status,[]}.
• Federation links continue to work for existing nodes, but the new replica never receives mirrored queue data.

Root Cause

The failure originates from a mismatch between the cluster name and the node name resolution performed during the join process. The auto‑scaled instances are launched with the default hostname generated by the EC2 DNS resolver (ip-10-0-0-45). RabbitMQ, when started with the default RABBITMQ_NODENAME=rabbit@$(hostname -s), registers itself as rabbit@ip-10-0-0-45. However, the existing cluster nodes were started with a fully qualified domain name (FQDN) derived from the aws_route53_record that resolves to the private IP, e.g. rabbit@rabbit-us-east-1.internal.

Because the new node advertises a different name, the existing nodes cannot resolve it, and the Erlang distribution layer rejects the join request with node_not_found. The problem is amplified by the fact that the cluster_formation.classic_config section in rabbitmq.conf lists static node names (FQDNs) and does not include the auto‑generated hostname.

Additionally, the security groups allow traffic only on the private subnet CIDR. When the new node attempts to resolve the FQDN of the existing nodes, the DNS query falls back to the public resolver, returning a public IP that is blocked by the security group, causing a second failure mode: connection refused.

Investigation and Debugging

Follow these steps to reproduce the failure and collect evidence.

1. Confirm node name mismatch:

   # On the new node
   hostname -s
   # Expected: rabbit-us-east-1-01
   # Actual: ip-10-0-0-45
   
   # Show RabbitMQ node name
   sudo rabbitmqctl status | grep -i nodename
   # Output: {nodename,<<"rabbit@ip-10-0-0-45">>}
   

2. Check DNS resolution from the new node:

   # Resolve an existing cluster node
   nslookup rabbit-us-east-1-01.internal
   # Expected private IP (10.0.x.x)
   # Actual: NXDOMAIN or public IP
   

3. Inspect cluster formation config:

   # /etc/rabbitmq/rabbitmq.conf
   cluster_formation.classic_config.nodes.1 = rabbit@rabbit-us-east-1-01.internal
   cluster_formation.classic_config.nodes.2 = rabbit@rabbit-us-east-1-02.internal
   

4. Review security group rules:

   # Verify inbound Erlang port (25672) allows traffic from the new node's private IP
   aws ec2 describe-security-groups --group-ids sg-0abc1234def56789
   

5. Collect Erlang distribution logs:

   journalctl -u rabbitmq-server -n 50 | grep -i distribution
   2024-10-07 12:03:45.123 rabbit@ip-10-0-0-45 ... failed to join cluster: {cannot_connect, {node_not_found, 'rabbit@rabbit-us-east-1-01.internal'}}
   

Solution

Standardize node naming and ensure DNS resolution to private IPs before the RabbitMQ service starts. The fix consists of three parts:

1. Set a deterministic node name using the instance tag

Update the launch template/user data to export RABBITMQ_NODENAME based on the Name tag that follows the rabbit-- pattern.

# User data (cloud‑init)
#!/bin/bash
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
NAME=$(aws ec2 describe-tags --filters "Name=resource-id,Values=$INSTANCE_ID" "Name=key,Values=Name" --region us-east-1 --output text | cut -f5)
export RABBITMQ_NODENAME="rabbit@$NAME.internal"
echo "RABBITMQ_NODENAME=$RABBITMQ_NODENAME" >> /etc/rabbitmq/rabbitmq-env.conf

2. Ensure private DNS resolution

Enable enableDnsHostnames and enableDnsSupport on the VPC, and create a private Route 53 hosted zone that maps *.internal to the private IPs of each node. Add an A record for each node using the same Name tag.

3. Adjust rabbitmq.conf to use a dynamic node list

Replace the static classic_config.nodes list with the cluster_formation.peer_discovery_backend = rabbit_peer_discovery_aws plugin, which discovers peers via AWS tags.

# /etc/rabbitmq/rabbitmq.conf (before)
cluster_formation.classic_config.nodes.1 = rabbit@rabbit-us-east-1-01.internal
cluster_formation.classic_config.nodes.2 = rabbit@rabbit-us-east-1-02.internal

# /etc/rabbitmq/rabbitmq.conf (after)
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_aws
cluster_formation.aws.region = us-east-1
cluster_formation.aws.tag_key = Name
cluster_formation.aws.tag_value = rabbit-us-east-1
cluster_formation.aws.use_private_ip = true

After updating the launch template, perform a rolling replacement of the Auto Scaling Group so that new instances start with the corrected environment variables and configuration.

Verification

Validate that newly provisioned nodes join the cluster automatically.

1. Trigger a scale‑out event (e.g., increase desired capacity by 1).
2. On the new instance, confirm the node name:

   sudo rabbitmqctl status | grep -i nodename
   # Expected: {nodename,<<"rabbit@rabbit-us-east-1-03.internal">>}
   

3. Check cluster status from any existing node:

   sudo rabbitmqctl cluster_status
   # Output should list the new node under {running_nodes, [...]}
   

4. Verify federation links are healthy:

   curl -s http://localhost:15672/api/federation-links -u guest:guest | jq '.[] | select(.status=="running")'
   

5. Inspect RabbitMQ logs for successful join messages:

   grep -i "joined cluster" /var/log/rabbitmq/rabbit@*.log
   2024-10-07 12:15:02.789 [info] <0.5678.0> node 'rabbit@rabbit-us-east-1-03.internal' joined the cluster
   

Prevention and Best Practices

• Use peer discovery plugins (AWS, Kubernetes, Consul) instead of hard‑coded node lists for dynamic environments.
• Enforce a naming convention via EC2 tags and propagate it to RABBITMQ_NODENAME at boot time.
• Validate DNS resolution in a pre‑start health check script; abort startup if the node cannot resolve any peer.
• Monitor Erlang distribution metrics (rabbitmq_erlang_dist_node_up, rabbitmq_erlang_dist_node_down) and set alerts for sudden spikes.
• Restrict security groups to allow Erlang port (25672) and inter‑region federation ports (5671/5672) only from the VPC CIDR.
• Automated smoke test in the ASG lifecycle hook: after instance becomes InService, run rabbitmqctl cluster_status and report success/failure to CloudWatch.

FAQ

1. Why does the node appear “offline” even though the service is running?
   Because RabbitMQ registers itself with a hostname that existing cluster members cannot resolve, the Erlang distribution layer rejects the join request, leaving the node in an isolated state.
2. Can I keep the static classic_config.nodes list and still auto‑scale?
   It is possible by adding a script that updates the list on every scale‑out, but using a peer‑discovery plugin eliminates manual list maintenance and reduces configuration drift.
3. What ports must be opened between regions for federation to work?
   Erlang distribution (TCP 25672) for cluster formation, and the AMQP ports used by federation (5671 for TLS, 5672 for plain) must be reachable over the private VPC peering or Transit Gateway.
4. How do I troubleshoot a “node_not_found” error after fixing DNS?
   Check that the RABBITMQ_NODENAME environment variable matches the DNS record, and verify that the node can resolve its own FQDN (run hostname -f). Also confirm that the security group permits inbound traffic from the peer’s private IP on port 25672.
5. Is there a way to see which cipher suite is used for federation TLS connections?
   Enable ssl_options` in the federation upstream definition and inspect the connection logs:

   2024-10-07 12:20:11.345 [info] <0.1234.0> TLS handshake succeeded, cipher: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

Related Topic Hub: Data Infrastructure Troubleshooting Hub