Google Gemini SAML Assertion Invalid Error During Inter‑service Call

Q: 1. Capture the failing request

curl -v -H "Authorization: Bearer $(gcloud auth print-access-token)" \ https://gemini.googleapis.com/v1beta/models/my-model:generateText Typical Gemini log entry (from Cloud Logging schema):

Q: 3. Check system clocks

# On a GKE node date -u # Compare with IdP server time ssh idp-admin@idp.example.com "date -u" If the difference exceeds 5 minutes, the assertion will be rejected as expired.

Q: 4. Validate IdP metadata against Gemini SP configuration

# Download IdP metadata curl -s https://idp.example.com/metadata.xml -o idp-metadata.xml # Extract Audience URI grep -i "entityID" idp-metadata.xml Cross‑reference the extracted entityID with the SP Entity ID configured in Gemini (see the Workload Identity Federation guide).

Q: 5. Inspect the trust store for signing certificates

# List certificates trusted by Gemini (via gcloud) gcloud beta iam workload-identity-pools describe my-pool \ --location=global --format="json(trustConfig)" Confirm the certificate fingerprint matches the current IdP signing certificate.

Q: Fix A – Clock Skew

Synchronize node clocks with NTP and enable Google Cloud’s chrony service.

In this article

Table of Contents

Problem Description

Microservices that run on GKE, Cloud Run, or Cloud Functions invoke the Google Gemini API using SAML‑based workload identity federation. During inter‑service calls the Gemini client returns:

SAML Assertion is invalid: Expired

SAML Assertion is invalid: Audience mismatch

These errors prevent the services from authenticating to Gemini, causing request failures such as HTTP 403 responses and cascading timeouts across the mesh.

Root Cause Analysis

According to the Gemini authentication guide and the Identity Platform SAML configuration reference, Gemini validates the following elements of a SAML assertion:

Temporal validity: NotBefore and NotOnOrAfter must be within the allowed clock skew (default ±5 minutes).
Audience (SP Entity ID): Must exactly match the Gemini Service Provider (SP) entity ID configured in the IdP metadata.
Signature: Must be signed with a certificate that Gemini trusts (present in the SP trust store).
Subject: A NameID element is required for workload identity federation.

Real incidents from the evidence package show three recurring failure modes:

Symptom	Underlying cause
Expired assertion	IdP clock drift or insufficient `NotOnOrAfter` window.
Audience mismatch	SP Entity ID changed after a Gemini endpoint migration.
Signature validation failed	Signing certificate rotated without updating Gemini’s trust store.

In the observed production outage, the error “SAML Assertion is invalid: Expired” was traced to a 5‑minute clock skew on the GKE node pool, while a separate “Audience mismatch” error originated from a stale AudienceURI after a migration to gemini.googleapis.com/v1beta.

Investigation and Debugging

Follow these steps to isolate the exact failure mode.

1. Capture the failing request

curl -v -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  https://gemini.googleapis.com/v1beta/models/my-model:generateText

Typical Gemini log entry (from Cloud Logging schema):

{
  "timestamp": "2026-09-10T14:23:41.123Z",
  "severity": "ERROR",
  "jsonPayload": {
    "error": "SAML Assertion is invalid: Audience mismatch",
    "service": "gemini",
    "requestId": "abcd-1234"
  }
}

2. Verify the SAML assertion content

Extract the Base64‑encoded assertion from the token exchange request (Python example):

import base64, xml.etree.ElementTree as ET, json, requests

resp = requests.post(
    "https://sts.googleapis.com/v1/token",
    data={
        "grant_type": "urn:ietf:params:oauth:grant-type:token-exchange",
        "subject_token_type": "urn:ietf:params:oauth:token-type:saml2",
        "subject_token": ""
    })
assertion_xml = base64.b64decode(resp.json()["subject_token"])
print(assertion_xml.decode())

Inspect the Audience, NotBefore, NotOnOrAfter, and Signature elements.

3. Check system clocks

# On a GKE node
date -u
# Compare with IdP server time
ssh idp-admin@idp.example.com "date -u"

If the difference exceeds 5 minutes, the assertion will be rejected as expired.

4. Validate IdP metadata against Gemini SP configuration

# Download IdP metadata
curl -s https://idp.example.com/metadata.xml -o idp-metadata.xml

# Extract Audience URI
grep -i "entityID" idp-metadata.xml

Cross‑reference the extracted entityID with the SP Entity ID configured in Gemini (see the Workload Identity Federation guide).

5. Inspect the trust store for signing certificates

# List certificates trusted by Gemini (via gcloud)
gcloud beta iam workload-identity-pools describe my-pool \
  --location=global --format="json(trustConfig)"

Confirm the certificate fingerprint matches the current IdP signing certificate.

Resolution

Apply the fix that corresponds to the identified root cause.

Fix A – Clock Skew

Synchronize node clocks with NTP and enable Google Cloud’s chrony service.

# On each node (or via DaemonSet)
sudo systemctl enable --now chronyd
sudo chronyc tracking

After enabling NTP, the “Expired” error disappears.

Fix B – Audience Mismatch

Update the IdP metadata to use the correct Gemini SP Entity ID (https://gemini.googleapis.com/ for v1, or https://gemini.googleapis.com/v1beta after migration).

# Before (incorrect)


# After (correct)

Reload the metadata on the IdP and restart the SAML service.

Fix C – Certificate Rotation

Upload the new signing certificate to Gemini’s trust store.

# Export new certificate fingerprint
openssl x509 -in new-signing.crt -noout -fingerprint -sha256

# Update trust config
gcloud beta iam workload-identity-pools update my-pool \
  --location=global \
  --add-trust-config="certificate_fingerprint=SHA256:AB:CD:EF:..."

Verify the update with:

gcloud beta iam workload-identity-pools describe my-pool \
  --location=global --format="json(trustConfig)"

Validation

After applying the fix, perform the following checks:

Re‑run the Gemini request and confirm a 200 OK response.
Search Cloud Logging for any remaining “SAML Assertion is invalid” entries.
Inspect the token exchange response for a valid access_token field.
Run a health‑check endpoint in each microservice that reports the current SAML assertion status.

# Expected log entry
{
  "severity": "INFO",
  "jsonPayload": {
    "message": "Authenticated to Gemini successfully",
    "service": "order-service"
  }
}

Prevention and Best Practices

Clock synchronization: Enforce NTP on all compute resources; consider --enable-chrony flag in GKE node pools.
Metadata version control: Store IdP metadata in a version‑controlled repository and automate validation against Gemini’s SP Entity ID during CI/CD.
Certificate rotation policy: Rotate signing certificates no more frequently than every 90 days and automate trust store updates using gcloud beta iam workload-identity-pools update.
Audience verification test: Include a post‑deployment integration test that parses the generated SAML assertion and asserts the Audience matches the expected SP URI.
Monitoring: Create a Cloud Monitoring alert on log entries with severity=ERROR and "SAML Assertion is invalid" to catch regressions within 5 minutes.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does the “Expired” error appear only after a rolling update?
During a rolling update, new pods may inherit the node’s time offset before the NTP daemon starts, causing a temporary clock skew that expires the generated assertions.
How can I programmatically verify the Audience URI in a generated assertion?
Decode the Base64 assertion and parse the Audience element with an XML library (e.g., Python’s xml.etree.ElementTree) and compare it to the expected SP Entity ID.
Can I use a shorter token lifetime to mitigate clock skew?
Gemini enforces a minimum NotOnOrAfter of 5 minutes. Reducing the lifetime below this threshold will cause immediate rejection.
What should I do if the IdP certificate rotation is coordinated across multiple regions?
Stage the new certificate in Gemini’s trust store first, verify successful authentication from a subset of services, then retire the old certificate after confirming no failures.
Is there a way to bypass SAML and use service‑account keys for Gemini?
Yes, you can switch to workload identity federation with OIDC or use native Google service‑account keys, but this changes the security model and may require code changes.