NGINX SAML assertion invalid after canary deployment of AI inference service

Problem

During a canary rollout of an AI inference service, NGINX began returning “SAML assertion invalid” errors for requests that were successfully authenticated in the stable environment. The failure manifested only for traffic routed to the new canary pods, causing intermittent authentication failures for end‑users attempting to invoke the model endpoint.

Root Cause

The NGINX auth_saml module validates the SAML <Assertion> signature against the public key of the Identity Provider (IdP). The canary deployment introduced a new container image that changed the TLS termination point:

  • The stable service terminated TLS at the upstream AI service, forwarding the original SAML POST body unchanged.
  • The canary pods added an nginx sidecar that performed SSL off‑loading before the request reached the AI inference container.

Off‑loading caused NGINX to read the request body for TLS decryption, which consumed the POST payload. When the auth_saml module later attempted to verify the SAML assertion, the request body was empty, resulting in the “SAML assertion invalid” error.

Debug

Step‑by‑step investigation that isolated the issue:

1. Observe the error in NGINX logs


2024/10/12 14:23:07 [error] 12#12: *3456 auth_saml: SAML assertion invalid: missing or empty assertion (client: 10.2.3.45, server: inference-proxy, request: "POST /predict HTTP/1.1")

2. Compare request bodies for stable vs. canary


# Capture request body on the stable path
curl -s -X POST https://stable.example.com/predict \
  -d '@saml-request.xml' -H "Content-Type: text/xml" -o /dev/null -w "%{size_upload}\n"
# Output: 1245

# Capture request body on the canary path (via tcpdump)
sudo tcpdump -i any -s 0 -w /tmp/canary.pcap host 10.2.3.45 and port 443
# After decoding the pcap:
# No HTTP payload present – only TLS handshake frames

3. Verify NGINX configuration differences


# Stable upstream definition
upstream ai_backend {
    server 10.1.0.10:8443;
}

# Canary upstream with sidecar TLS termination
upstream ai_backend_canary {
    server 10.1.0.20:8443;   # AI container
    server 10.1.0.21:8443;   # sidecar TLS terminator
}

4. Check module behavior

The auth_saml module reads the request body using ngx_http_request_body. If the body has already been read (e.g., by proxy_pass with proxy_request_buffering off), the module receives an empty buffer.

5. Confirm sidecar reads the body


# Inside the sidecar container
docker exec -it sidecar-canary /bin/sh
# The sidecar uses nginx with:
proxy_set_body $request_body;
# This directive forces NGINX to read $request_body, consuming it.

Solution

Prevent the sidecar from consuming the request body before the auth_saml module validates it. Two viable approaches:

Approach A – Move SAML validation upstream (before TLS off‑load)

Place the auth_saml directive in the front‑facing NGINX instance that terminates TLS for both stable and canary traffic.


# Front‑facing NGINX (unchanged for stable, updated for canary)
server {
    listen 443 ssl;
    ssl_certificate /etc/nginx/certs/proxy.crt;
    ssl_certificate_key /etc/nginx/certs/proxy.key;

    # SAML authentication applied here
    auth_saml on;
    auth_saml_idp_metadata /etc/nginx/saml/idp-metadata.xml;
    auth_saml_sp_entity_id https://proxy.example.com;
    auth_saml_sp_private_key /etc/nginx/saml/sp.key;
    auth_saml_sp_certificate /etc/nginx/saml/sp.crt;

    location /predict {
        proxy_pass http://ai_backend_canary;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Approach B – Preserve request body for downstream modules

If the sidecar must remain, configure it to buffer the request body without consuming it:


# Sidecar NGINX (canary only)
server {
    listen 8443 ssl;
    ssl_certificate /etc/nginx/certs/sidecar.crt;
    ssl_certificate_key /etc/nginx/certs/sidecar.key;

    # Do NOT read $request_body here
    proxy_pass http://ai_inference_container:8080;
    proxy_request_buffering on;   # default, ensures body is buffered
    proxy_set_header Content-Length $content_length;
    proxy_set_header Content-Type $content_type;
}

Additionally, add proxy_set_body "" to explicitly avoid body consumption.

Verify

After applying the chosen fix, repeat the validation steps:

  1. Send a SAML‑authenticated request to the canary endpoint.
  2. Confirm the NGINX log no longer shows “SAML assertion invalid”.
  3. Inspect the response – the AI inference service should return a 200 with the model prediction.

2024/10/12 14:45:12 [info] 12#12: *3890 auth_saml: assertion validated successfully for user "alice@example.com"

Run a health‑check script:


#!/usr/bin/env bash
curl -s -o /dev/null -w "%{http_code}" https://canary.example.com/predict \
  -d '@saml-request.xml' -H "Content-Type: text/xml"
# Expected output: 200

Prevent

Implement safeguards to avoid similar regressions in future canary rollouts:

  • Configuration linting: Enforce that any upstream proxy that terminates TLS must have proxy_request_buffering on and must not reference $request_body before authentication modules.
  • Automated tests: Add an integration test that sends a SAML POST through the canary path and asserts a 200 response.
  • Canary monitoring: Track auth_saml: SAML assertion invalid log frequency with a Prometheus counter; alert if the rate exceeds a threshold within the first 5 minutes of deployment.
  • Documentation: Record the required ordering of NGINX modules – authentication must precede any body‑consuming proxy.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

  1. Why does the error appear only for the canary and not for the stable version?
    The canary introduced an SSL‑terminating sidecar that read the request body before the auth_saml module could validate it. The stable path forwards the raw TLS stream directly to the AI container, preserving the body for validation.
  2. Can I keep the sidecar and still use auth_saml in the same NGINX instance?
    Yes, but you must ensure the sidecar does not consume the body. Set proxy_request_buffering on and avoid any directives that reference $request_body before the SAML module runs.
  3. Is there a way to debug missing request bodies in NGINX?
    Enable error_log debug; and look for messages like “request body already read”. You can also dump the body to a temporary file with proxy_body_in_file on; and inspect it.
  4. What monitoring metric should I use to detect this issue early?
    Create a Prometheus counter that increments on the log pattern “auth_saml: SAML assertion invalid”. Alert when the rate > 0 for a 1‑minute window during a deployment.
  5. Will enabling proxy_request_buffering off ever be appropriate with SAML?
    Disabling buffering streams the request directly to the upstream, which always consumes the body before the SAML module runs. Therefore it should never be used when the same NGINX instance performs SAML validation.