ChromaDB IAM Permission Denied On Collections During Rolling Update

Q: Problem Description

During a Kubernetes rolling update of a production ChromaDB deployment, operators observed a burst of PermissionDenied errors when the newly started pods attempted to read existing collections. Typical log entries look like:

Q: 1. Pre‑stop hook to force token refresh

Add a preStop lifecycle hook that deletes the cached token, forcing the kubelet to request a new one before the pod terminates.

In this article

Table of Contents

Problem Description

During a Kubernetes rolling update of a production ChromaDB deployment, operators observed a burst of PermissionDenied errors when the newly started pods attempted to read existing collections. Typical log entries look like:

2024-07-12T14:23:07Z ERROR  PermissionDenied: User does not have permission to access collection 7f9c2a3b
2024-07-12T14:23:07Z ERROR  403 Forbidden: IAM role arn:aws:iam::123456789012:role/chroma-db-read lacks chroma.collections.read permission
2024-07-12T14:23:08Z ERROR  AccessDenied: token expired or invalid while accessing collection

The errors are transient—once the rollout completes, the pods regain normal access. However, the window of failure can cause request time‑outs for downstream services and trigger alert fatigue.

Root Cause Analysis

ChromaDB authenticates each request against an IAM‑backed policy (ChromaDB Authentication & Authorization). In AWS‑based clusters the pod’s service account is bound to an IAM role via IRSA. The rolling update sequence introduces two timing gaps:

IAM token expiration: The terminating pod’s projected service‑account token (a JWT) expires after the default token-expiration of 1 hour. When the pod is killed, the token may already be near expiry, causing the new pod’s first request to be signed with an expired token.
IAM role propagation delay: When a new pod is scheduled, the kubelet fetches a fresh token from the metadata service. AWS may take up to 30 seconds to propagate the role’s permissions to the token (GitHub Issue #1582). During this window the token lacks the chroma.collections.read permission, resulting in 403 responses.

Both symptoms match the real‑world incident on an EKS cluster where “the terminating pod’s IAM token expires before the new pod acquires a fresh token” (GitHub Issue #1459).

Investigation and Debugging

Follow these steps to reproduce the failure and collect evidence:

Watch the rollout and capture pod events:

kubectl rollout status deployment/chroma-db -n production
kubectl get pods -n production -w

Inspect the service‑account token inside a newly created pod:

# exec into the pod
kubectl exec -it chroma-db-7f9c2a3b-xyz -n production -- cat /var/run/secrets/kubernetes.io/serviceaccount/token | head -c 200
# decode JWT header (optional)
echo <token> | cut -d. -f1 | base64 -d | jq .

Check the IAM role attached to the service account:

kubectl get serviceaccount chroma-sa -n production -o yaml
aws eks describe-addon-versions --cluster-name prod-cluster
# Verify role binding
aws iam get-role --role-name chroma-db-read

Query the STS token to see its expiration:

aws sts assume-role-with-web-identity \
  --role-arn arn:aws:iam::123456789012:role/chroma-db-read \
  --role-session-name test \
  --web-identity-token file:///var/run/secrets/kubernetes.io/serviceaccount/token \
  --duration-seconds 3600 | jq .Credentials.Expiration

Enable ChromaDB request tracing (if enabled) to capture the exact error payload:

curl -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  https://chroma.internal/api/collections/7f9c2a3b/metadata -v

Typical output during the failure window:

< HTTP/1.1 403 Forbidden
< Content-Type: application/json
< X-Request-ID: a1b2c3d4
{
  "error": "PermissionDenied: User does not have permission to access collection 7f9c2a3b"
}

Resolution

The fix consists of two complementary changes: ensure the token is fresh on pod start and give the IAM role enough time to propagate.

1. Pre‑stop hook to force token refresh

Add a preStop lifecycle hook that deletes the cached token, forcing the kubelet to request a new one before the pod terminates.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: chroma-db
spec:
  replicas: 3
  selector:
    matchLabels:
      app: chroma-db
  template:
    metadata:
      labels:
        app: chroma-db
    spec:
      serviceAccountName: chroma-sa
      containers:
      - name: chroma
        image: trychroma/chroma:latest
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "rm -f /var/run/secrets/kubernetes.io/serviceaccount/token"]
        env:
        - name: AWS_ROLE_ARN
          value: arn:aws:iam::123456789012:role/chroma-db-read

2. Increase token TTL and add a warm‑up init container

Configure the service‑account token projection to a longer TTL (e.g., 6 hours) and use an init container that performs a harmless ChromaDB request, guaranteeing a valid token before the main container starts.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: chroma-sa
automountServiceAccountToken: true
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chroma-db
spec:
  template:
    spec:
      serviceAccountName: chroma-sa
      automountServiceAccountToken: true
      # Token projection with extended TTL
      projected:
        sources:
        - serviceAccountToken:
            path: token
            expirationSeconds: 21600   # 6 hours
      initContainers:
      - name: token-warmup
        image: curlimages/curl:7.85.0
        command: ["sh", "-c"]
        args:
          - |
            TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
            curl -s -o /dev/null -w "%{http_code}" -H "Authorization: Bearer $TOKEN" https://chroma.internal/healthz
        volumeMounts:
        - name: sa-token
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      containers:
      - name: chroma
        # ... existing spec ...

Why the fix works

The preStop hook removes the stale token, preventing the terminating pod from re‑using an almost‑expired JWT.
Extending expirationSeconds gives the new pod a larger window to obtain a fresh token.
The init container forces the kubelet to fetch a token and validates it against ChromaDB before the main container begins handling traffic, effectively “warming up” the IAM credentials.

Validation

After applying the changes, perform a controlled rolling update and verify that no PermissionDenied errors appear:

# Trigger rollout
kubectl rollout restart deployment/chroma-db -n production

# Watch logs for a full minute
kubectl logs -f deployment/chroma-db -n production | grep PermissionDenied || echo "No errors found"

Expected outcome:

No errors found

Additional checks:

Confirm the token TTL:

kubectl exec -it $(kubectl get pod -l app=chroma-db -n production -o jsonpath="{.items[0].metadata.name}") -n production -- cat /var/run/secrets/kubernetes.io/serviceaccount/token | cut -d. -f2 | base64 -d | jq .exp

Validate IAM permissions via AWS CLI:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/chroma-db-read \
  --action-names chroma:ReadCollection \
  --resource-arns arn:aws:chroma:us-east-1:123456789012:collection/7f9c2a3b

Prevention and Best Practices

Use IAM role propagation buffers: Add a sleep 30 or a health‑check loop in the init container to give AWS up to 30 seconds for role propagation.
Monitor token expiration metrics: Export kubelet_serviceaccount_token_expiration_seconds to Prometheus and alert when the remaining TTL drops below 300 seconds during a rollout.
Pin exact IAM permissions: Follow the ChromaDB Deployment Guide and grant only chroma.collections.read and chroma.collections.write as needed.
Enable graceful termination: Set terminationGracePeriodSeconds to at least 60 seconds so the pre‑stop hook can run reliably.
Test rolling updates in a staging namespace before applying to production, using the same service‑account and IAM role bindings.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Why does the error only appear during a rolling update?
Because the new pod starts with a freshly projected service‑account token that may be expired or not yet propagated, while the terminating pod still holds a near‑expiry token. The short overlap window exposes the missing permission.
Can I disable IAM role propagation delay?
No. Propagation is an eventual‑consistency property of AWS IAM. The recommended approach is to add a warm‑up step or a small sleep before the pod begins serving traffic.
Is the pre‑stop hook sufficient for GKE Workload Identity?
Yes. Workload Identity also uses projected tokens; deleting the token forces the node to fetch a new one with the correct bindings. Ensure the service account has the iam.serviceAccounts.actAs permission.
What if I cannot modify the deployment (e.g., third‑party helm chart)?
Add a sidecar container that performs the token warm‑up and shares the token volume with the main container, or use a postStart hook to achieve the same effect.
How do I differentiate between token expiration and missing IAM permissions?
Expired tokens return AccessDenied: token expired or invalid, whereas missing permissions return 403 Forbidden: IAM role ... lacks chroma.collections.read permission. Checking the JWT exp claim helps isolate the cause.