a close-up of a server room

ChromaDB token refresh failure during live data ingestion

by

Problem: ChromaDB Token Refresh Failure During Live Data Ingestion

In a production streaming pipeline that continuously upserts embedded documents into a hosted ChromaDB instance, the following symptoms were observed after roughly one hour of operation:

  • WebSocket connections dropped with connection closed with code 4001 (authentication_failed).
  • HTTP 401 Unauthorized responses on /upsert and /add endpoints, e.g.: 
    
HTTP/1.1 401 Unauthorized
Content-Type: application/json
{
  "detail": "Token expired"
}
  • gRPC calls returned UNAUTHENTICATED errors: 
    
grpc.UNAUTHENTICATED: token_refresh_error: failed to obtain new token
  • Backlog accumulation – a pipeline ingesting 5 k embeddings / sec built a queue of ~360 k records during a 12‑minute outage (AWS Lambda incident).

These failures interrupt the low‑latency guarantee required for real‑time event processing and cause downstream SLA breaches.

Root Cause Analysis

Token Lifecycle in ChromaDB

According to the ChromaDB Authentication Guide, a client obtains an access token via the /auth/token endpoint. The token has a configurable TTL (default 1 hour) and must be refreshed using the /auth/refresh endpoint before expiry. The SDK injects the refreshed token into the Authorization: Bearer <token> header for subsequent HTTP, gRPC, and WebSocket requests.

Observed Failure Path

  1. At T+60 min the access token reaches its TTL.
  2. The SDK invokes the refresh endpoint. In many production environments the refresh request is throttled or returns 403 Forbidden (“Token refresh rejected”) – see GitHub Issue #1389 and the Kubernetes incident where the auth provider throttled the request.
  3. The SDK retries the refresh according to an exponential back‑off policy, eventually exhausting its max_retry count and raising TokenRefreshError: Maximum retry attempts exceeded while refreshing access token (GitHub Issue #1245).
  4. Because the SDK does not receive a new token, it continues to use the stale token for existing WebSocket connections. The server validates the token on each frame and closes the socket with code 4001 (authentication_failed).
  5. All subsequent HTTP/gRPC upsert calls carry the expired token, resulting in 401 Unauthorized or UNAUTHENTICATED errors.

The core reasons are:

  • Refresh endpoint failure due to throttling, mis‑configured OIDC client, or server‑side 500 errors (on‑premise incident).
  • SDK bug where the refreshed token is not propagated to an already‑opened WebSocket (GitHub Issue #1389).
  • Insufficient retry/back‑off configuration causing the client to give up before the auth provider recovers.

Investigation and Debugging Steps

1. Capture Authentication‑Related Logs


2024-07-15T12:00:58.123Z INFO  chromadb.client: Refresh token request sent
2024-07-15T12:00:58.456Z WARN  chromadb.client: Refresh token response 403 Forbidden
2024-07-15T12:01:00.001Z ERROR chromadb.client: TokenRefreshError: Maximum retry attempts exceeded
2024-07-15T12:01:02.789Z WARN  websocket: connection closed with code 4001 (authentication_failed)
2024-07-15T12:01:03.112Z ERROR http: POST /upsert 401 Unauthorized

2. Verify Token TTL and Refresh Endpoint Directly


# Inspect current token expiry (JWT payload)
jwt decode $(cat /var/run/chroma/access_token.jwt) | jq .exp

# Manually invoke refresh endpoint
curl -X POST https://chroma.example.com/auth/refresh \
     -H "Authorization: Bearer $(cat /var/run/chroma/refresh_token.jwt)" \
     -d '{"grant_type":"refresh_token"}' -i

Expected successful response:


HTTP/1.1 200 OK
Content-Type: application/json
{
  "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...",
  "expires_in": 3600
}

3. Inspect WebSocket Metadata Propagation


# Enable SDK debug mode
export CHROMADB_DEBUG=1

# Observe the token used for each frame (client logs)
2024-07-15T12:01:01.234Z DEBUG websocket: sending frame with Authorization: Bearer

4. Check Rate‑Limit Headers from Auth Provider


curl -I -X POST https://auth.example.com/token \
     -d 'client_id=xyz&client_secret=abc&grant_type=client_credentials'

# Look for:
# X-RateLimit-Remaining: 0
# Retry-After: 30

5. Validate SDK Version

GitHub Issue #1245 identifies the bug in chromadb==0.3.7. Verify the version in use:


pip show chromadb
Name: chromadb
Version: 0.3.7

Resolution

1. Upgrade the SDK to a Version with Fixed Refresh Logic

Version 0.3.9 includes:

  • Propagation of refreshed tokens to active WebSocket connections.
  • Configurable refresh_retry_max and refresh_backoff_base parameters.

# Before
pip install "chromadb==0.3.7"

# After
pip install "chromadb==0.3.9"

2. Extend Token TTL or Use Long‑Lived Service Accounts

Modify the deployment configuration (see Security Best Practices) to set access_token_ttl = 86400 (24 h) for service‑to‑service ingestion.


# chroma_config.yaml
auth:
  token_ttl_seconds: 86400
  refresh_token_ttl_seconds: 2592000

3. Adjust Refresh Retry Policy

Provide explicit retry settings when constructing the client:


from chromadb import Client

client = Client(
    api_url="https://chroma.example.com",
    auth_token="initial-access-token",
    refresh_token="initial-refresh-token",
    refresh_retry_max=10,
    refresh_backoff_base=2,   # exponential back‑off: 2,4,8,...
)

4. Implement a Guarded Refresh Callback (Workaround for OIDC Throttling)

If the auth provider imposes strict rate limits, introduce a local cache that reuses a successfully refreshed token for up to retry_window_seconds before attempting another refresh.


import time
from threading import Lock

class CachedRefresher:
    def __init__(self, client, cache_ttl=300):
        self.client = client
        self.cache_ttl = cache_ttl
        self._lock = Lock()
        self._cached_token = None
        self._expires_at = 0

    def get_token(self):
        with self._lock:
            now = time.time()
            if self._cached_token and now < self._expires_at:
                return self._cached_token
            # perform refresh
            resp = self.client.refresh_token()
            self._cached_token = resp["access_token"]
            self._expires_at = now + resp["expires_in"] - 30  # safety margin
            return self._cached_token

# usage
refresher = CachedRefresher(client)
client.set_auth_callback(refresher.get_token)

Verification

  1. Token Refresh Success 
    
2024-07-15T13:00:01.112Z INFO  chromadb.client: Refresh token request sent
2024-07-15T13:00:01.345Z INFO  chromadb.client: Refresh token succeeded, new expires_in=3600
  2. WebSocket Remains Open 
    
2024-07-15T13:00:02.001Z DEBUG websocket: connection authenticated with new token
2024-07-15T13:00:02.050Z INFO  websocket: ping/pong healthy
  3. Upsert Calls Return 200 
    
2024-07-15T13:00:03.210Z INFO  http: POST /upsert 200 OK
  4. Metrics – In Grafana, the chroma_ingest_success_total counter should show no drops, and the chroma_token_refresh_failure_total metric should be zero.

Prevention and Best Practices

  • Monitor Token Health – Export chroma_token_expiry_seconds and set an alert when remaining TTL < 300 s.
  • Centralize Secret Management – Store refresh tokens in AWS Secrets Manager or HashiCorp Vault and rotate them with a cadence shorter than the access token TTL.
  • Use Service Accounts with Long‑Lived Tokens when possible; avoid per‑request user tokens for high‑throughput ingestion.
  • Configure Auth Provider Rate Limits to accommodate bursty refresh traffic (e.g., allow 5 requests per minute per client ID).
  • Enable SDK Debug Logging in Staging to catch token propagation bugs before production rollout.
  • Graceful Shutdown Hook – On process termination, close WebSocket connections after sending a final close frame with a fresh token to avoid “authentication_failed” logs.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

  1. Why does the token refresh succeed when I call the endpoint manually but fail inside the SDK?

    The SDK in versions prior to 0.3.9 does not correctly handle Retry-After headers and aborts after the default max_retry count. Upgrading the SDK or configuring refresh_retry_max resolves the issue.

  2. Can I rely on the refresh token indefinitely?

    Refresh tokens also have a TTL (default 30 days). Monitor refresh_token_expiry_seconds and rotate the underlying OIDC client secret before it expires.

  3. My WebSocket stays open after the access token expires – is that expected?

    WebSocket servers validate the token on each frame. If the server is mis‑configured to skip re‑validation, the connection may appear stable but will eventually be closed when the client sends a frame with an expired token. Ensure the server follows the spec described in the Streaming Client Documentation.

  4. How do I differentiate between a 401 caused by an expired token versus an insufficient scope?

    Check the error body. An expired token returns "Token expired", while scope issues return "Insufficient scopes for operation". The WWW-Authenticate header also includes error="invalid_token" for expiry and error="insufficient_scope" for scope problems.

  5. Is exponential back‑off safe for token refresh, or can it cause longer outages?

    Exponential back‑off reduces load on the auth provider but can increase the window of failure. Pair it with a max_retry that is high enough to survive transient throttling (e.g., 10 attempts) and a safety margin in the access token TTL (e.g., refresh 5 minutes before expiry).