a close-up of a server room

RabbitMQ SAML assertion invalid after failover to secondary cloud region

by

Problem Description

After an automatic disaster‑recovery (DR) failover to the secondary cloud region, AI training workers were unable to publish messages to RabbitMQ. The broker logged repeated authentication failures with the SAML plugin:


2024-07-31 12:04:18.732 [error] {rabbit_auth_backend_saml, handle_authentication_failure,
  [{error, "SAML assertion is expired"},
   {error, "SAML assertion not yet valid"},
   {error, "Invalid SAML audience"},
   {error, "Signature validation failed"}]}
authentication handshake failed: SAML assertion invalid

Consequences included:

  • Training jobs stalled because they could not enqueue work items.
  • Back‑pressure cascaded to downstream inference services.
  • Metrics showed a sudden drop in rabbitmq_queue_messages_publish_total and an increase in rabbitmq_authentication_failure_total.

Root Cause Analysis

The failure stemmed from three intertwined issues that became visible only after the secondary region took over:

  1. System‑clock drift: The secondary cluster’s VMs were 45 seconds ahead of the Identity Provider (IdP) clock. The SAML plugin validates the NotBefore and NotOnOrAfter timestamps in the assertion. When the broker’s time is ahead, the plugin emits “SAML assertion not yet valid”. This matches the GitHub issue #312 and the AWS DR case study (2022).
  2. TLS certificate rotation mismatch: During the failover the secondary region performed an automated TLS certificate rollover. The IdP metadata still referenced the previous signing certificate, causing the plugin to reject the signature with “Signature validation failed” (see GitHub issue #2745 and the Azure SR incident 2023).
  3. SAML audience URI change: RabbitMQ’s service entity ID is built from the broker’s public hostname. After promotion the hostname changed from mq-primary.example.com to mq-dr.example.net. The IdP metadata contained the old audience URI, leading to “Invalid SAML audience” errors (mailing‑list thread 2023‑09‑15).

Each symptom appears in the official RabbitMQ SAML plugin documentation under “Clock skew handling” and “IdP metadata requirements”. The combination of these failures caused the generic “SAML assertion invalid” message that surfaced in the logs.

Investigation and Debugging

Below is a reproducible step‑by‑step debugging workflow that was used in the incident.

1. Capture the authentication error details


# Tail the auth logs with the rabbitmq_auth_backend_saml tag
journalctl -u rabbitmq-server -f | grep rabbit_auth_backend_saml

2. Verify system time on each node


# Check current time and NTP status
date -u
timedatectl status
ntpq -p

Expected output on a healthy node shows offset < 100 ms. In the failed region the offset was +45 s.

3. Inspect the SAML assertion returned by the IdP


# Capture a raw SAML response using curl and decode the Base64 payload
curl -s -L -u user:pwd "https://idp.example.com/saml2/idp/SSOService.php?SAMLRequest=..." -o saml_response.xml
openssl base64 -d -in saml_response.xml | xmllint --format -

Look for <NotBefore> and <NotOnOrAfter> timestamps and the <Audience> element.

4. Compare IdP metadata with broker configuration


# Show the metadata file used by RabbitMQ
cat /etc/rabbitmq/saml/idp_metadata.xml | grep -i certificate

If the certificate fingerprint does not match the one advertised by the IdP, signature validation will fail.

5. Review TLS certificate chain for the broker


# Show the current broker certificate and its expiry
openssl x509 -in /etc/rabbitmq/ssl/cert.pem -noout -text | grep -E "Subject:|Not After"
# Verify that the certificate chain is trusted by the IdP
openssl verify -CAfile /etc/rabbitmq/ssl/ca.pem /etc/rabbitmq/ssl/cert.pem

6. Check RabbitMQ SAML plugin configuration


# rabbitmq.conf excerpt
saml.idp_metadata = /etc/rabbitmq/saml/idp_metadata.xml
saml.service_entity_id = https://mq-dr.example.net/
saml.clock_skew = 120

Note the saml.clock_skew default (60 s) and the current value.

Resolution

All three root causes were addressed. The steps below include before/after configuration snippets.

1. Synchronize clocks across the secondary cluster


# Before (drift of +45 s)
timedatectl status
   Local time: Thu 2024-08-01 12:05:13 UTC
   NTP synchronized: yes
   NTP service: active
   Offset: 45.123s

# After (NTP correction)
timedatectl set-ntp true
systemctl restart systemd-timesyncd
date -u
   Thu 2024-08-01 12:05:58 UTC

Ensured all nodes run the same NTP pool and set maxpoll to 16 to reduce drift.

2. Refresh IdP metadata with the new signing certificate

Downloaded the updated metadata from the IdP and replaced the local file.


# Before (old certificate fingerprint)

  
    
      MIID...oldfingerprint...
    
  


# After (new certificate fingerprint)
curl -s https://idp.example.com/metadata > /etc/rabbitmq/saml/idp_metadata.xml

3. Update the service entity ID to match the promoted hostname


# rabbitmq.conf – before
saml.service_entity_id = https://mq-primary.example.com/

# rabbitmq.conf – after
saml.service_entity_id = https://mq-dr.example.net/

Restarted the RabbitMQ application to load the new value:


rabbitmqctl stop_app
rabbitmqctl start_app

4. Adjust clock‑skew tolerance as a safety net

While the primary fix is NTP synchronization, the plugin can be configured to tolerate up to 180 seconds of skew during DR events.


# rabbitmq.conf addition
saml.clock_skew = 180

5. Rotate TLS certificates in lock‑step with the IdP

Implemented a CI/CD pipeline that:

  • Generates a new broker certificate.
  • Uploads the public certificate to the IdP metadata endpoint.
  • Waits for IdP acknowledgement before swapping the broker’s cert.pem.

Validation

After applying the fixes, the following checks confirmed successful recovery.

Functional test


# Publish a test message from a training worker
python - <<'PY'
import pika, os
creds = pika.PlainCredentials('saml_user', os.getenv('SAML_TOKEN'))
params = pika.ConnectionParameters(host='mq-dr.example.net', port=5671,
                                   ssl=True, credentials=creds)
conn = pika.BlockingConnection(params)
ch = conn.channel()
ch.queue_declare(queue='test')
ch.basic_publish(exchange='', routing_key='test', body='hello')
print('published')
PY

Output:


published

Log verification


journalctl -u rabbitmq-server | grep "SAML assertion"
# No lines containing "invalid", "expired", or "not yet valid"

Metrics

  • rabbitmq_authentication_failure_total returned to baseline (< 0.01 % of requests).
  • Publish rate on training.jobs queue restored to 120 msg/s.

Prevention and Best Practices

Area Recommended Guardrail
Clock synchronization Deploy a dedicated NTP server per region; enforce maxpoll <= 8 on all RabbitMQ VMs; monitor system.time.offset via Prometheus.
SAML metadata freshness Automate metadata pull every 12 h; validate certificate fingerprint against IdP; alert on parsing errors.
TLS certificate lifecycle Coordinate certificate rotation with IdP via a pre‑deployment webhook; keep both old and new certs in IdP metadata for a 48‑hour overlap.
Audience URI consistency Parameterize saml.service_entity_id from a region‑specific environment variable; store the variable in a central config store (e.g., Consul).
Plugin configuration drift Store rabbitmq.conf in version control; enforce CI linting that checks saml.clock_skew and saml.service_entity_id values per region.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

  1. Why does the error say “SAML assertion is expired” even though the token was just issued?
    Because the broker’s clock is ahead of the IdP. The NotOnOrAfter timestamp appears in the past from the broker’s perspective, triggering the “expired” check.
  2. Can I increase saml.clock_skew to avoid clock‑skew errors?
    Yes, but it only masks the symptom. Large skews weaken security and can cause replay attacks. The proper fix is to synchronize clocks via NTP.
  3. Do TLS certificate rotations always affect SAML validation?
    Only if the IdP signs assertions with the broker’s public certificate (common in mutual‑TLS setups). The IdP must be updated with the new signing certificate before the broker starts using it.
  4. What is the correct audience URI format for RabbitMQ?
    It must match the saml.service_entity_id setting, typically the HTTPS URL that clients use to reach the broker (e.g., https://mq-dr.example.net/). Changing the hostname without updating the IdP metadata causes “Invalid SAML audience”.
  5. How can I test SAML authentication without affecting production workers?
    Use the rabbitmqctl eval command to invoke the plugin’s validate_assertion/1 function with a captured assertion, or run a temporary worker that authenticates via the SAML token and publishes a single test message.