Problem – RabbitMQ connection timeout during AI inference failover to DR site
During a planned failover of the AI inference pipeline from the primary data center to the disaster‑recovery (DR) site, the inference workers repeatedly log Connection timeout errors when trying to publish or consume messages from RabbitMQ. The failure manifests as:
- Inference requests stall after the first retry.
- Worker logs contain
pika.exceptions.AMQPConnectionError: Connection timed out. - Metrics show a sudden drop in
rabbitmq_queue_messages_readyand a spike inworker_request_latency_seconds.
The primary site operates normally; only the DR environment exhibits the timeout.
Root Cause – Network path, DNS, and client‑side reconnection handling
Three intertwined factors cause the timeout:
- Asymmetric DNS resolution. The inference service resolves the RabbitMQ virtual IP (VIP) to the primary address before the failover. After the VIP is moved to the DR cluster, the DNS cache on the worker nodes still points to the old IP until the TTL expires (default 300 s). The client attempts to connect to an IP that no longer routes to a RabbitMQ node, leading to a TCP SYN timeout.
- Load‑balancer health‑check delay. The external load balancer in front of the DR RabbitMQ nodes takes ~45 seconds to mark the new backend as healthy. During this window, connections are accepted by the LB but immediately reset, which the client interprets as a timeout.
- Missing exponential back‑off and connection‑reset handling in the client. The existing
pika.BlockingConnectionwrapper retries only once with a fixed 5‑second timeout. When the first attempt fails, the worker aborts the request instead of persisting until the DR cluster becomes reachable.
Because the primary site’s configuration mirrors the DR site, the issue is not a RabbitMQ version mismatch but a combination of DNS caching, LB health‑check timing, and insufficient reconnection logic.
Debug – Step‑by‑step investigation
1. Verify DNS resolution before and after failover
# On a worker node
$ dig +short rabbitmq.dr.example.com
10.0.1.45 # Primary VIP before failover
10.0.2.45 # Expected DR VIP after failover
After failover, the command still returned 10.0.1.45 for ~2 minutes, confirming stale cache.
2. Capture TCP handshake with tcpdump
# Capture SYN packets to the RabbitMQ port (5672)
$ sudo tcpdump -i eth0 -n port 5672 and '(tcp[tcpflags] & tcp-syn != 0)'
Sample output during the failure window:
12:34:56.789012 IP 10.1.5.22.54321 > 10.0.1.45.5672: Flags [S], seq 123456789, win 29200, length 0
12:34:56.789045 IP 10.0.1.45.5672 > 10.1.5.22.54321: Flags [R], seq 0, win 0, length 0
The reset (R) packet indicates the load balancer rejected the connection because the backend was not yet healthy.
3. Inspect RabbitMQ logs on the DR nodes
2024-03-15 02:01:12.345 [info] <0.1234.0> accepted TCP connection from 10.1.5.22:54321
2024-03-15 02:01:12.347 [error] <0.1234.0> connection attempt failed: connection reset by peer
The log confirms that the broker received the TCP SYN but the LB closed the socket before RabbitMQ could complete the handshake.
4. Review the client reconnection code
# current implementation (simplified)
def get_connection():
parameters = pika.ConnectionParameters(host='rabbitmq.dr.example.com',
port=5672,
socket_timeout=5)
try:
return pika.BlockingConnection(parameters)
except pika.exceptions.AMQPConnectionError as e:
logger.error("Connection timeout: %s", e)
raise
Only a single attempt is made; no back‑off or DNS refresh.
Solution – Robust reconnection strategy and infrastructure tweaks
1. Force DNS refresh on each connection attempt
# Updated helper
import socket
import time
import pika
import logging
logger = logging.getLogger(__name__)
def resolve_host(hostname):
# Resolve each call to avoid stale cache
return socket.gethostbyname(hostname)
def exponential_backoff(attempt, base=2, cap=30):
return min(cap, base ** attempt)
def get_connection(hostname='rabbitmq.dr.example.com', port=5672, max_retries=5):
for attempt in range(max_retries):
ip = resolve_host(hostname)
parameters = pika.ConnectionParameters(host=ip,
port=port,
socket_timeout=10,
heartbeat=30)
try:
conn = pika.BlockingConnection(parameters)
logger.info("Connected to RabbitMQ at %s:%s", ip, port)
return conn
except pika.exceptions.AMQPConnectionError as e:
wait = exponential_backoff(attempt)
logger.warning("Attempt %d/%d failed (%s). Retrying in %ds",
attempt + 1, max_retries, e, wait)
time.sleep(wait)
raise RuntimeError("Unable to connect to RabbitMQ after %d attempts" % max_retries)
Why it works: Each retry resolves the hostname anew, bypassing stale DNS entries. The exponential back‑off gives the load balancer time to mark the DR backend healthy, while the increased socket_timeout prevents premature failure.
2. Reduce DNS TTL for the RabbitMQ VIP
| Current TTL | Recommended TTL |
|---|---|
| 300 seconds | 30 seconds |
Update the authoritative DNS zone:
$TTL 30
rabbitmq.dr.example.com. IN A 10.0.2.45
3. Adjust load‑balancer health‑check interval
- Set
health_check_intervalto 5 seconds. - Set
healthy_thresholdto 2. - Set
unhealthy_thresholdto 2.
This reduces the window where connections are accepted but the backend is not ready.
4. Deploy a sidecar “connection watchdog” in the inference pod
# Dockerfile snippet for sidecar
FROM python:3.11-slim
RUN pip install pika
COPY watchdog.py /app/watchdog.py
CMD ["python", "/app/watchdog.py"]
The watchdog continuously attempts to open a short‑lived connection and publishes a heartbeat metric. If the metric drops, the main inference container can trigger a graceful restart.
Verify – Confirm that the failover now succeeds without timeouts
- Simulate failover. Move the RabbitMQ VIP to the DR site and observe DNS propagation with
dig. Verify the TTL is 30 seconds. - Run a connection test script.
# test_connection.py
from connection_helper import get_connection
if __name__ == "__main__":
conn = get_connection()
print("Connection established:", conn.is_open)
Expected output after failover:
Attempt 1/5 failed (Connection timed out). Retrying in 2s
Attempt 2/5 failed (Connection timed out). Retrying in 4s
Attempt 3/5 succeeded
Connection established: True
connection reset by peer entries after the LB health‑check becomes healthy.worker_request_latency_seconds returns to baseline and rabbitmq_queue_messages_ready resumes normal growth.Prevent – Operational guardrails to avoid recurrence
- Monitoring: Alert on
rabbitmq_connection_failed_totalspikes > 5 minutes. - DNS health check: Verify that the VIP resolves to a single IP within the expected TTL after any topology change.
- Load‑balancer readiness probe: Ensure the probe checks the actual RabbitMQ port (5672) and not just TCP reachability.
- Client library upgrade: Use
pika1.3+ which includes built‑in DNS resolution per connection. - Chaos testing: Periodically simulate a VIP move in a staging environment to validate the reconnection logic.
Related Topic Hub: Data Infrastructure Troubleshooting Hub
FAQ
- Why does the connection succeed after several retries instead of failing immediately? Each retry forces a fresh DNS lookup, allowing the stale entry to expire. The exponential back‑off also aligns with the load balancer’s health‑check cycle, giving the DR backend time to become healthy.
- Can I keep the original single‑attempt code and just increase the timeout? Increasing the timeout masks the underlying DNS and LB readiness issues; the client would still block for the full timeout period on each request, degrading latency. A proper retry strategy is required.
- Do I need to change RabbitMQ clustering settings? No. The problem is external to the broker. However, ensure that the DR cluster mirrors the primary’s policies (mirrored queues, HA policy) so that once the connection is established, message durability is preserved.
- What if my inference service uses a different client library (e.g.,
amqporaio-pika)? The same principles apply: resolve the hostname per connection, implement exponential back‑off, and respect the load‑balancer health‑check timing. Most libraries expose asocket_timeoutand a way to catch connection errors. - Is reducing the DNS TTL to 30 seconds safe for production? Yes, provided the DNS zone is managed centrally and the TTL reduction does not increase load on authoritative servers. The benefit of rapid failover outweighs the minor increase in query volume.