TGI model load fails with IAM permission denied during DR failover

For Secret Manager, grant the accessor role:

gcloud secrets add-iam-policy-binding tgi-config-prod \
  --member=serviceAccount:tgi-dr-sa@project-id.iam.gserviceaccount.com \
  --role=roles/secretmanager.secretAccessor \
  --project=project-id

Redeploy the TGI pods so they pick up the updated service account permissions.

Verify

After applying the changes, confirm that the model loads without IAM errors:

1. Restart the deployment:

kubectl rollout restart deployment tgi -n tgi-dr

2. Watch the pod logs for successful download messages:

kubectl logs -f -l app=tgi -n tgi-dr
2026-10-25T14:45:12.001Z INFO model_loader.go:115] downloaded model.tar.gz (2.3 GB) from gs://tgi-models-prod/model_v3.tar.gz
2026-10-25T14:45:13.210Z INFO config_loader.go:45] loaded secret tgi-config-prod
2026-10-25T14:45:14.000Z INFO server.go:78] TGI service ready on :8080

3. Run a health‑check request:

curl -s http://tgi-dr-service.tgi-dr.svc.cluster.local:8080/healthz | jq .
{
  "status": "ok",
  "model": "model_v3",
  "region": "europe-west1"
}

If the health endpoint returns "status":"ok", the DR failover is functional.

Prevent

• Policy as code: Store bucket IAM bindings and Secret Manager policies in Terraform or Config Connector. Include the allowed regions list as a variable so DR regions are automatically added during environment provisioning.
• Automated permission validation: Add a CI step that runs gcloud iam test-permissions for the DR service account against all required resources. Fail the pipeline if any permission is missing.
• Region‑agnostic bucket conditions: Prefer using resource.location in ["us-central1","europe-west1"] instead of equality checks, reducing the chance of forgetting a DR region.
• Service account inheritance: Create a single “tgi‑svc‑sa” with the superset of permissions and reference it from both primary and DR deployments, reducing drift.
• Monitoring alerts: Set up a Cloud Monitoring alert on container/logs for the pattern “IAM permission denied” with a threshold of 5 occurrences in 5 minutes.

FAQ

1. Why does the primary region work but the DR region fails with the same service account?
   The bucket IAM policy includes a conditional expression that restricts access to resource.location == "us-central1". When the request originates from europe-west1, the condition evaluates to false, causing the denial even though the role is present.
2. Can I avoid modifying bucket conditions by using a signed URL?
   Yes, generating a signed URL for the model artifact bypasses IAM checks, but it adds operational complexity and expires. Granting the correct IAM role is the preferred, auditable solution.
3. Do I need to update the IAM policy for every new DR region?
   If you use a static equality condition, yes. Instead, define the condition using an in list or a regular expression that matches all authorized regions, which only requires updating when adding a completely new region.
4. What if the DR service account already has the roles but still sees “permission denied”?
   Verify that the bucket’s uniformBucketLevelAccess is enabled. If disabled, object‑level ACLs may override IAM permissions. Ensure the objects themselves grant READER access to the service account or enable uniform bucket-level access.
5. How can I test the IAM configuration before a real failover?
   Use gcloud iam test-permissions for each required permission and perform a dry‑run download with gsutil cp -n (no‑clobber) to confirm read access without affecting the production bucket.

Related Topic Hub: Model Serving Troubleshooting Hub