401 Error From AI Inference Pod After SAML Assertion Validation Fails

Q: Verify

Run the same curl command used in the debug step:

In this article

Table of Contents

401 error from AI inference pod after SAML assertion validation fails

Problem

In a Kubernetes‑hosted microservices platform, the AI inference service (ai‑model‑svc) began returning HTTP 401 responses for internal calls that should be authenticated via SAML‑based SSO. The failure is logged as “SAML assertion invalid” by the authentication middleware, causing downstream clients to see a hard 401 without any payload.

Typical request flow:

Client microservice obtains a SAML assertion from the IdP.
Assertion is attached as a bearer token in the Authorization: Bearer header.
ai‑model‑svc receives the request, passes the token to the saml‑auth‑middleware for validation.
Middleware rejects the token and returns 401.

Root Cause

The middleware validates the SAML assertion against the IdP’s signing certificate and checks the NotOnOrAfter condition. Two regressions introduced simultaneously caused the validation to fail:

Certificate rotation mismatch – The IdP rotated its signing certificate, but the ConfigMap saml‑cert‑config mounted into the pod still contained the previous certificate. The middleware therefore could not verify the signature.
Clock skew – The node’s system clock drifted by >5 minutes after a recent kernel update. The NotOnOrAfter timestamp in the assertion fell outside the acceptable window, triggering the “invalid assertion” error.

Both issues manifested as the same log message, making the initial diagnosis ambiguous.

Debug

Step‑by‑step investigation that reproduced the failure and isolated the cause.

1. Capture failing request logs

kubectl logs -n ml-prod -l app=ai-model-svc -c auth-middleware --tail=100

Sample output:

2026-10-13T08:12:45Z ERROR middleware: SAML assertion invalid: signature verification failed
2026-10-13T08:12:45Z INFO  request-id=abc123 path=/v1/predict user=service-b
2026-10-13T08:12:45Z WARN  middleware: Assertion NotOnOrAfter condition failed (now=2026-10-13T08:12:45Z, expiry=2026-10-13T08:07:30Z)

2. Verify the signing certificate used by the IdP

# Export current IdP signing cert
openssl s_client -connect idp.example.com:443 -showcerts

Compare the fingerprint with the one stored in the ConfigMap:

kubectl get configmap saml-cert-config -n ml-prod -o yaml

3. Check node clock synchronization

# On one of the worker nodes
timedatectl status
# Example output
               Local time: Mon 2026-10-13 08:13:02 UTC
           Universal time: Mon 2026-10-13 08:13:02 UTC
                 RTC time: Mon 2026-10-13 08:13:02
                Time zone: UTC (UTC, +0000)
System clock synchronized: no
NTP service: active
RTC in local TZ: no

The “System clock synchronized: no” flag indicated NTP was not running.

4. Reproduce with a known‑good assertion

curl -v -H "Authorization: Bearer $(cat valid_assertion.b64)" \
  https://ai-model-svc.ml-prod.svc.cluster.local/v1/predict

Result: 401 still returned, confirming the middleware rejection is not request‑specific.

Solution

Two corrective actions were applied.

1. Update the SAML signing certificate ConfigMap

Before:

<configmap name="saml-cert-config">
  <data>
    <signing.crt>-----BEGIN CERTIFICATE-----
MIID...old...AB
-----END CERTIFICATE-----</signing.crt>
  </data>
</configmap>

After pulling the new cert from the IdP and applying:

# Extract new cert
openssl s_client -connect idp.example.com:443 -showcerts

<configmap name="saml-cert-config">
  <data>
    <signing.crt>-----BEGIN CERTIFICATE-----
MIID...new...XY
-----END CERTIFICATE-----</signing.crt>
  </data>
</configmap>

# Apply the updated ConfigMap
kubectl apply -f saml-cert-config.yaml

Restart the pods to mount the new ConfigMap:

kubectl rollout restart deployment/ai-model-svc -n ml-prod

2. Restore NTP synchronization on all worker nodes

# Enable and start chrony (or systemd-timesyncd)
sudo systemctl enable --now chronyd
# Verify sync
timedatectl status | grep "System clock synchronized"

After the service started, the clock drift reduced to <0.5 seconds.

Verify

Run the same curl command used in the debug step:

curl -s -o /dev/null -w "%{http_code}\n" \
  -H "Authorization: Bearer $(cat valid_assertion.b64)" \
  https://ai-model-svc.ml-prod.svc.cluster.local/v1/predict

Expected output:

Additional verification:

Check middleware logs for successful validation messages.
Confirm the ConfigMap now contains the new fingerprint:

kubectl exec -n ml-prod $(kubectl get pod -l app=ai-model-svc -o name | head -n1) \
  -- cat /etc/saml/signing.crt | openssl x509 -noout -fingerprint

Result should match the IdP’s current signing certificate fingerprint.

Prevent

Automated certificate rotation – Use a sidecar that periodically fetches the IdP metadata URL and updates the saml‑cert‑config ConfigMap via a Kubernetes Job. Trigger a rolling restart only when the fingerprint changes.
Clock health monitoring – Deploy a DaemonSet that runs chronyc tracking and exports a Prometheus metric node_clock_offset_seconds. Alert if offset > 2 seconds.
Graceful fallback – Extend the middleware to accept a secondary trusted certificate bundle during a rotation window (e.g., keep the previous cert for 48 hours).
Integration test – Add a CI pipeline stage that generates a fresh SAML assertion against a test IdP and validates the end‑to‑end flow, catching mismatched certificates before promotion.

FAQ

Why does the 401 appear only after a certificate rotation? The middleware validates the signature against the locally stored certificate. When the IdP rotates its signing key, the old certificate no longer verifies the signature, causing the middleware to reject every assertion even though the token payload is otherwise correct.
Can I temporarily bypass SAML validation to keep the service running? It is possible to enable a “maintenance mode” flag that skips signature verification, but this opens a security hole. A safer temporary measure is to load the previous certificate alongside the new one, allowing both signatures to be accepted during the overlap period.
How do I know if clock skew is the culprit? The middleware logs include a specific message about the NotOnOrAfter condition failing. Correlate that with the node’s timedatectl output; a drift of >5 minutes typically triggers the failure.
What metric should I monitor to detect future SAML validation failures early? Export a counter saml_assertion_validation_failure_total with labels reason (e.g., signature, clock_skew). Alert on a sudden spike or on any non‑zero value over a short window.
Is there a way to debug the exact part of the assertion that failed? Enable debug logging in the middleware (e.g., set LOG_LEVEL=debug). The middleware will then log the parsed assertion fields, the certificate fingerprint used, and the exact timestamp comparison, which helps pinpoint whether the failure is due to signature, audience, issuer, or time conditions.

Related Topic Hub: Distributed Systems Troubleshooting Hub