Google Gemini OAuth2 Token Acquisition Failure After Service Account Rotation

Q: Debug 1. Examine pipeline logs

2024-10-12 03:14:27,842 ERROR token_fetcher.py:45 - Failed to obtain access token: google.auth.exceptions.DefaultCredentialsError: Failed to retrieve the default credentials. Check that the GOOGLE_APPLICATION_CREDENTIALS environment variable is set correctly. 2. Verify the credential file path $ echo $GOOGLE_APPLICATION_CREDENTIALS /home/etl_user/keys/gemini-service-account.json $ ls -l /home/etl_user/keys/gemini-service-account.json -rw-r----- 1 etl_user etl_user 2.1K Oct 11 22:13 /home/etl_user/keys/gemini-service-account.json The file exists but contains the previous key. Attempting to parse it yields:

Q: 3. Inspect IAM permissions

$ gcloud iam service-accounts get-iam-policy \ new-gemini-sa@my-data-pipeline.iam.gserviceaccount.com \ --format=json { "bindings": [ { "role": "roles/aiplatform.viewer", "members": [ "serviceAccount:new-gemini-sa@my-data-pipeline.iam.gserviceaccount.com" ] } ] } Notice the missing roles/aiplatform.user binding.

Q: 4. Capture the token request

$ curl -v -X POST https://oauth2.googleapis.com/token \ -d "grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer" \ -d "assertion=eyJhbGciOiJSUzI1NiIsImtpZCI6Ij..." 2>&1 | grep HTTP > POST /token HTTP/1.1 < HTTP/1.1 400 Bad Request { "error": "invalid_grant", "error_description": "Invalid JWT: Invalid issuer." } The JWT is signed with the old private key, confirming the credential mismatch.

Q: 2. Update IAM policy to include required role

# Grant Gemini API access $ gcloud projects add-iam-policy-binding my-data-pipeline \ --member="serviceAccount:new-gemini-sa@my-data-pipeline.iam.gserviceaccount.com" \ --role="roles/aiplatform.user" 3. Refresh environment variable (if path changed) # In the pipeline’s startup script export GOOGLE_APPLICATION_CREDENTIALS=/home/etl_user/keys/gemini-service-account.json 4. Restart the pipeline workers Workers cache the credential loader; a restart forces re‑initialisation.

Q: 2. Gemini API call sanity check

$ curl -X POST https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent \ -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \ -H "Content-Type: application/json" \ -d '{"contents":[{"role":"user","parts":[{"text":"Hello"}]}]}' { "candidates": [ { "content": { "role": "model", "parts": [{"text":"Hello! How can I help you today?"}] } } ] } Successful response confirms the pipeline can now invoke Gemini.

In this article

Table of Contents

Problem

An automated ETL pipeline invokes the Google Gemini API for model inference. After rotating the Google Cloud service account used for authentication, the pipeline repeatedly fails to acquire an OAuth2 access token. All subsequent Gemini requests return 401 Unauthorized errors, causing the data processing job to abort.

Root Cause

The failure stems from a mismatch between the newly created service account key and the cached credentials used by the pipeline’s google-auth-library client. During rotation, the old key file was removed, but the environment variable GOOGLE_APPLICATION_CREDENTIALS still pointed to the stale path. Additionally, the IAM policy on the new service account omitted the roles/aiplatform.user role required for Gemini API access. The combination of an invalid key file and insufficient permissions prevents the OAuth2 token endpoint (https://oauth2.googleapis.com/token) from issuing a token.

Debug

1. Examine pipeline logs


2024-10-12 03:14:27,842 ERROR token_fetcher.py:45 - Failed to obtain access token: 
google.auth.exceptions.DefaultCredentialsError: 
Failed to retrieve the default credentials. 
Check that the GOOGLE_APPLICATION_CREDENTIALS environment variable is set 
correctly.

2. Verify the credential file path


$ echo $GOOGLE_APPLICATION_CREDENTIALS
/home/etl_user/keys/gemini-service-account.json
$ ls -l /home/etl_user/keys/gemini-service-account.json
-rw-r----- 1 etl_user etl_user 2.1K Oct 11 22:13 /home/etl_user/keys/gemini-service-account.json

The file exists but contains the previous key. Attempting to parse it yields:


$ cat /home/etl_user/keys/gemini-service-account.json | head -n 5
{
  "type": "service_account",
  "project_id": "my-data-pipeline",
  "private_key_id": "abcdef1234567890",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvAIBADANB...

3. Inspect IAM permissions


$ gcloud iam service-accounts get-iam-policy \
    new-gemini-sa@my-data-pipeline.iam.gserviceaccount.com \
    --format=json
{
  "bindings": [
    {
      "role": "roles/aiplatform.viewer",
      "members": [
        "serviceAccount:new-gemini-sa@my-data-pipeline.iam.gserviceaccount.com"
      ]
    }
  ]
}

Notice the missing roles/aiplatform.user binding.

4. Capture the token request


$ curl -v -X POST https://oauth2.googleapis.com/token \
  -d "grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer" \
  -d "assertion=eyJhbGciOiJSUzI1NiIsImtpZCI6Ij..." 2>&1 | grep HTTP
> POST /token HTTP/1.1
< HTTP/1.1 400 Bad Request
{
  "error": "invalid_grant",
  "error_description": "Invalid JWT: Invalid issuer."
}

The JWT is signed with the old private key, confirming the credential mismatch.

Solution

1. Deploy the new service account key

Generate a fresh key for the rotated service account and replace the stale file.


# Generate key
$ gcloud iam service-accounts keys create \
    /home/etl_user/keys/gemini-service-account.json \
    --iam-account=new-gemini-sa@my-data-pipeline.iam.gserviceaccount.com

# Secure permissions
$ chmod 640 /home/etl_user/keys/gemini-service-account.json
$ chown etl_user:etl_user /home/etl_user/keys/gemini-service-account.json

2. Update IAM policy to include required role


# Grant Gemini API access
$ gcloud projects add-iam-policy-binding my-data-pipeline \
    --member="serviceAccount:new-gemini-sa@my-data-pipeline.iam.gserviceaccount.com" \
    --role="roles/aiplatform.user"

3. Refresh environment variable (if path changed)


# In the pipeline’s startup script
export GOOGLE_APPLICATION_CREDENTIALS=/home/etl_user/keys/gemini-service-account.json

4. Restart the pipeline workers

Workers cache the credential loader; a restart forces re‑initialisation.


$ systemctl restart etl-pipeline.service

Before / After comparison of the IAM policy

Before (rotated key only) After (added role)

Before (rotated key only)	After (added role)
`{ "bindings": [ { "role": "roles/aiplatform.viewer", "members": [ "serviceAccount:new-gemini-sa@my-data-pipeline.iam.gserviceaccount.com" ] } ] }`	`{ "bindings": [ { "role": "roles/aiplatform.viewer", "members": [ "serviceAccount:new-gemini-sa@my-data-pipeline.iam.gserviceaccount.com" ] }, { "role": "roles/aiplatform.user", "members": [ "serviceAccount:new-gemini-sa@my-data-pipeline.iam.gserviceaccount.com" ] } ] }`

{
  "bindings": [
    {
      "role": "roles/aiplatform.viewer",
      "members": [
        "serviceAccount:new-gemini-sa@my-data-pipeline.iam.gserviceaccount.com"
      ]
    }
  ]
}

{
  "bindings": [
    {
      "role": "roles/aiplatform.viewer",
      "members": [
        "serviceAccount:new-gemini-sa@my-data-pipeline.iam.gserviceaccount.com"
      ]
    },
    {
      "role": "roles/aiplatform.user",
      "members": [
        "serviceAccount:new-gemini-sa@my-data-pipeline.iam.gserviceaccount.com"
      ]
    }
  ]
}

Verify

1. Token acquisition test


$ python - <<'PY'
from google.auth import default
creds, _ = default()
creds.refresh()
print("Access token:", creds.token[:20] + "...")
PY
Access token: ya29.a0AfH6SM...

The token is successfully refreshed without errors.

2. Gemini API call sanity check


$ curl -X POST https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent \
  -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
  -H "Content-Type: application/json" \
  -d '{"contents":[{"role":"user","parts":[{"text":"Hello"}]}]}'
{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [{"text":"Hello! How can I help you today?"}]
      }
    }
  ]
}

Successful response confirms the pipeline can now invoke Gemini.

Prevent

Automated credential rotation: Use gcloud iam service-accounts keys rotate in a CI/CD step that atomically updates the file and restarts dependent services.
IAM role validation: Include a post‑rotation script that checks for required roles (e.g., roles/aiplatform.user) via gcloud projects get-iam-policy and fails the deployment if missing.
Health check for token fetch: Add a lightweight cron job that calls gcloud auth application-default print-access-token and alerts on failures.
Configuration drift detection: Store the expected checksum of the credential file in a ConfigMap or secret and monitor for unexpected changes.
Logging guardrails: Ensure logs capture both the path of GOOGLE_APPLICATION_CREDENTIALS and the result of the token request (status code, error description).

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does the pipeline still use the old key after I replaced the JSON file?
The google-auth-library caches the credential object for the lifetime of the process. A process restart (or explicit reload) is required for the new key to be read.
Can I avoid setting GOOGLE_APPLICATION_CREDENTIALS manually?
Yes. When running on GKE or Cloud Run, attach the service account directly to the workload. The metadata server will provide short‑lived tokens, eliminating file‑based keys.
What IAM role is the minimum required for Gemini inference?
roles/aiplatform.user (or a custom role that includes aiplatform.models.predict) is required. Viewer or read‑only roles are insufficient for token acquisition.

How can I detect a missing role before deployment?
Use a pre‑deployment script that runs:


gcloud projects get-iam-policy $PROJECT_ID \
  --flatten="bindings[].members" \
  --filter="bindings.role:roles/aiplatform.user AND bindings.members:serviceAccount:$SA_EMAIL"

If the output is empty, abort the deployment.

Is there a way to programmatically verify token validity?
Call the token introspection endpoint (https://oauth2.googleapis.com/tokeninfo?access_token=TOKEN) and check that audience matches https://generativelanguage.googleapis.com/ and expires_in is positive.