ChromaDB secret rotation failure after Vault token update in Kubernetes

ChromaDB Secret Rotation Failure After Vault Token Update in Kubernetes

Problem

The production ChromaDB service is intermittently failing to authenticate against its backend store. Errors manifest as connection refusals and query timeouts. The failure correlates with the scheduled secret rotation that pulls new credentials from HashiCorp Vault. After a recent Vault token renewal, the rotation job no longer updates the CHROMADB_PASSWORD secret in the pod’s environment, causing the application to continue using expired credentials.

Root Cause

Vault token renewal changed the token’s entity_id and associated policies. The ExternalSecret custom resource (or secret-store) that fetches ChromaDB credentials was still referencing the old token path (auth/kubernetes/login) without the updated service_account binding. Consequently:

• The Vault client inside the secret-rotation Job authenticates successfully but receives a 403 Forbidden when reading the chroma-db/creds secret because the token lacks the read capability.
• The rotation Job exits with a non‑zero status, leaving the Secret object unchanged.
• Pods continue to mount the stale Secret, leading to authentication errors in ChromaDB.

Debug

Step‑by‑step investigation performed in the affected cluster:

1. Inspect pod logs for authentication errors

kubectl logs -n prod -l app=chroma-db -c app | grep -i "authentication"

2026-10-30T14:12:07Z ERROR auth failed for user “chroma_app”: invalid credentials (expired password)

2. Check the secret‑rotation Job status

kubectl get job -n prod chromadb-secret-rotate -o yaml

status:
failed: 1
conditions:
– type: Failed
status: “True”
reason: BackoffLimitExceeded

3. View the Job’s container logs

kubectl logs -n prod job/chromadb-secret-rotate -c rotate

2026-10-30T14:11:55Z INFO authenticating to Vault using Kubernetes auth

2026-10-30T14:11:57Z ERROR failed to read secret “chroma-db/creds”: permission denied (HTTP 403)

4. Verify Vault token capabilities

vault token capabilities -token=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token) chroma-db/creds

capabilities = []

5. Inspect the ServiceAccount and RoleBinding used by the rotation Job

kubectl get serviceaccount -n prod chromadb-rotator -o yaml

kubectl get rolebinding -n prod chromadb-rotator-binding -o yaml

The binding references a ClusterRole that was recently modified to remove the read permission on secret/data/chroma-db/*.

Solution

Restore the correct Vault policy and update the Kubernetes authentication configuration so the rotation Job can read the secret.

1. Re‑apply the Vault policy with read capability

# policies/chroma-db.hcl
path "secret/data/chroma-db/*" {
  capabilities = ["read", "list"]
}

vault policy write chromadb-read policies/chroma-db.hcl

2. Update the Kubernetes auth role to map the ServiceAccount to the new policy

# vault/k8s-role.hcl
path "auth/kubernetes/role/chromadb-rotator" {
  capabilities = ["create", "update", "read"]
}

vault write auth/kubernetes/role/chromadb-rotator \
    bound_service_account_names=chromadb-rotator \
    bound_service_account_namespaces=prod \
    policies=chromadb-read \
    ttl=1h

3. Amend the ExternalSecret (or SecretProviderClass) to reference the correct Vault role

# external-secret.yaml (before)
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: chromadb-credentials
  namespace: prod
spec:
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: chromadb-secret
  dataFrom:
  - extract:
      key: secret/data/chroma-db/creds
  # missing role reference

# external-secret.yaml (after)
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: chromadb-credentials
  namespace: prod
spec:
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: chromadb-secret
  dataFrom:
  - extract:
      key: secret/data/chroma-db/creds
  refreshInterval: 1h
  authentication:
    secretRef:
      name: vault-token
      key: token
    role: chromadb-rotator   # <-- added role binding

4. Redeploy the rotation Job to verify it can now fetch the secret

kubectl delete job -n prod chromadb-secret-rotate
kubectl apply -f job/rotate-secret.yaml

Verify

Confirm that the secret is refreshed and the application resumes normal operation.

1. Check the newly created Secret data:

kubectl get secret chromadb-secret -n prod -o yaml

data:
CHROMADB_USERNAME: Y2hyb21hX3VzZXI=
CHROMADB_PASSWORD: c2VjcmV0X3Bhc3N3b3Jk

Inspect the rotation Job logs for a successful read:

kubectl logs -n prod job/chromadb-secret-rotate -c rotate

2026-10-30T14:15:02Z INFO successfully retrieved secret "chroma-db/creds"

2026-10-30T14:15:03Z INFO updated Kubernetes Secret "chromadb-secret"

Validate ChromaDB connectivity:

kubectl exec -n prod -c app $(kubectl get pod -n prod -l app=chroma-db -o jsonpath="{.items[0].metadata.name}") -- \
    curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/healthz

200

Prevent

Implement safeguards to avoid recurrence:

• Policy drift detection: Use vault policy read in CI to compare against a stored baseline; fail the pipeline on mismatches.
• Automated token health checks: Deploy a CronJob that periodically calls vault token lookup and verifies required capabilities for the rotation role.
• ExternalSecret health monitoring: Enable the external-secrets.io/refresh metric and set an alert if a secret has not been refreshed within its refreshInterval.
• RBAC review process: Require pull‑request approval for any change to the ClusterRole that grants Vault access.
• Graceful secret rollout: Configure the Deployment to use envFrom with secretKeyRef and enable imagePullPolicy: IfNotPresent so pods restart only after the secret is successfully updated.

FAQ

1. Why did the rotation job succeed after the token renewal but still fail to fetch the secret?

   The token renewal generated a new token without the chromadb-read policy attached. Authentication succeeded, but the token lacked read permission on the secret path, resulting in a 403 error.

2. Can I use a static Vault token for secret rotation?

   Static tokens bypass token renewal issues but introduce security risks. The recommended approach is to keep Kubernetes auth with a ServiceAccount and ensure the associated Vault role always includes the required policies.

3. How do I know if a secret rotation actually updated the Kubernetes Secret?

   Check the metadata.annotations["external-secrets.io/refresh"] timestamp or query the secret’s resourceVersion before and after rotation. A change indicates a successful update.

4. What if the rotation job crashes before completing?

   Configure the Job with backoffLimit and restartPolicy: OnFailure. Additionally, enable external-secrets.io/refreshInterval so the controller retries independently of the Job.

5. Is there a way to test Vault policy changes without affecting production?

   Deploy a separate dev namespace with a copy of the ExternalSecret and a test ServiceAccount. Run the rotation Job there; if it succeeds, promote the policy to production.

Related Topic Hub: Vector Databases Troubleshooting Hub