ChromaDB connection timeout during RAG inference serving

Problem – ChromaDB Connection Timeout During RAG Inference Serving

When deploying a Retrieval‑Augmented Generation (RAG) service, the inference container repeatedly fails to retrieve documents from the ChromaDB vector store. The request handler logs a ConnectionTimeoutError after the client’s requests call exceeds the configured 5 second deadline, and the service returns a 504 response to the downstream API consumer.

Typical symptom stack trace:

2026-10-30 14:22:07,842 ERROR inference_worker.py:112
Traceback (most recent call last):
  File "/app/inference_worker.py", line 98, in retrieve_documents
    results = chroma_client.query(
  File "/usr/local/lib/python3.11/site-packages/chromadb/api/client.py", line 210, in query
    resp = self._http_client.post(url, json=payload, timeout=self.timeout)
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1234, in post
    response = self.request("POST", url, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1192, in request
    raise ConnectTimeout("Connection timed out.")
httpx.ConnectTimeout: Connection timed out.

In addition, occasional log entries show authentication failures from the ChromaDB server:

2026-10-30 14:22:09,113 WARN chroma_server.py:45
Authentication failed for token: ****

The failure prevents the RAG pipeline from augmenting LLM responses with relevant context, effectively breaking the inference service.

Root Cause – Mismatched Network & Authentication Configuration

Investigation revealed two intertwined root causes:

• Network reachability issue: The inference container runs in a Kubernetes pod behind a NetworkPolicy that only permits egress to ports 80/443. ChromaDB is exposed on port 8000 (HTTP) and 8443 (HTTPS). The policy blocked traffic, causing the client’s TCP SYN to be dropped, which the HTTP client interprets as a timeout.
• Token‑based authentication mismatch: The ChromaDB server was upgraded to v0.4.0, which introduced mandatory JWT validation. The inference service still used the legacy static API key stored in a ConfigMap. The server rejected the key, emitting the “Authentication failed” warning. Because the request never completed the TLS handshake, the client timed out before receiving the 401 response.

Both problems manifested as a timeout because the client’s retry logic masks the underlying 401 error when the TCP connection cannot be established.

Debug – Systematic Investigation Steps

1. Verify pod‑to‑service connectivity

# From inside the inference pod
kubectl exec -it $(kubectl get pod -l app=inference -o jsonpath='{.items[0].metadata.name}') -- /bin/sh

# Test TCP connectivity to ChromaDB service
nc -zvw5 chromadb-service.default.svc.cluster.local 8000
# Expected output:
# Connection to chromadb-service.default.svc.cluster.local 8000 port [tcp/http] succeeded!

If the command hangs or reports “Connection timed out”, a network policy or Service/Endpoint misconfiguration is present.

2. Inspect NetworkPolicy rules

kubectl get networkpolicy -n default -o yaml | grep -A5 -B5 "podSelector:
  matchLabels:
    app: inference"

Confirm that egress rules include the ChromaDB service ports.

3. Capture traffic with tcpdump

kubectl exec -it $(kubectl get pod -l app=inference -o jsonpath='{.items[0].metadata.name}') -- \
  tcpdump -i any -nn -s 0 port 8000 or port 8443 -c 5 -w /tmp/capture.pcap

# Retrieve the pcap for analysis
kubectl cp $(kubectl get pod -l app=inference -o jsonpath='{.items[0].metadata.name}'):/tmp/capture.pcap ./capture.pcap

Examine the pcap in Wireshark; a missing SYN‑ACK indicates egress blockage.

4. Validate authentication token

# Decode the JWT used by the inference service (if any)
jwt decode $(cat /etc/secrets/chroma_jwt) --no-verify

Check the exp claim and the iss field against the server’s expected issuer.

5. Review ChromaDB server logs

kubectl logs -l app=chromadb -c server --tail=100
2026-10-30 14:20:55,012 INFO  server.py:112 Starting ChromaDB on 0.0.0.0:8000
2026-10-30 14:22:09,113 WARN  auth.py:45 Authentication failed for token: ****
2026-10-30 14:22:09,115 INFO  server.py:210 Request from 10.2.3.45:54321 rejected (401)

Solution – Align Network Policy and Authentication Mechanisms

1. Update NetworkPolicy to allow ChromaDB egress

Before:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: inference-egress
spec:
  podSelector:
    matchLabels:
      app: inference
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
    ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443

After: add the ChromaDB ports.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: inference-egress
spec:
  podSelector:
    matchLabels:
      app: inference
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
    ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443
    - protocol: TCP
      port: 8000   # HTTP endpoint of ChromaDB
    - protocol: TCP
      port: 8443   # HTTPS endpoint of ChromaDB

Apply the updated policy:

kubectl apply -f inference-egress.yaml

2. Switch to JWT authentication compatible with ChromaDB v0.4.0

Generate a new token using the server’s secret key (or configure the server to accept the legacy API key).

Before (legacy config):

# config.yaml
chroma:
  api_key: "legacy-static-key"
  endpoint: "http://chromadb-service:8000"

After (JWT config):

# config.yaml
chroma:
  jwt_token: "{{ env.CHROMA_JWT }}"   # injected via secret
  endpoint: "https://chromadb-service:8443"
  verify_ssl: true

Update the Kubernetes secret:

# Generate JWT (example using python-jose)
python - <<'PY'
import jwt, datetime, os
payload = {
    "sub": "inference-service",
    "iss": "my-org",
    "exp": datetime.datetime.utcnow() + datetime.timedelta(hours=2)
}
secret = os.getenv("CHROMA_JWT_SECRET")
token = jwt.encode(payload, secret, algorithm="HS256")
print(token)
PY

# Store token in a secret
kubectl create secret generic chroma-auth \
  --from-literal=CHROMA_JWT=$(cat token.txt) \
  --dry-run=client -o yaml | kubectl apply -f -

Mount the secret as an environment variable in the inference deployment:

env:
- name: CHROMA_JWT
  valueFrom:
    secretKeyRef:
      name: chroma-auth
      key: CHROMA_JWT

3. Adjust client timeout (optional)

If network latency is expected to be higher, increase the HTTP client timeout to avoid premature failures.

# inference_worker.py
import httpx

chroma_client = httpx.Client(
    base_url=os.getenv("CHROMA_ENDPOINT"),
    timeout=30.0,               # increased from default 5s
    headers={"Authorization": f"Bearer {os.getenv('CHROMA_JWT')}"}
)

Verify – Confirm the Fix Works in Production

1. Redeploy the inference service and the updated NetworkPolicy.
2. Run a health‑check endpoint that forces a ChromaDB query:

curl -s http://inference-service.default.svc.cluster.local/healthz?test=chroma | jq .
{
  "chroma_connection": "ok",
  "retrieved_documents": 3
}

Expected log entry:

2026-10-30 14:45:12,021 INFO inference_worker.py:112 Document retrieval succeeded (3 hits)

Run a load test (e.g., hey or locust) and verify that latency remains < 200 ms for the retrieval step.

Prevent – Operational Guardrails and Monitoring

• NetworkPolicy audit: Include ChromaDB ports in the egress whitelist for any pod that accesses the vector store. Use a CI lint rule that validates NetworkPolicy objects against a whitelist of required service ports.
• Authentication drift detection: Deploy a periodic job that attempts a lightweight ping query to ChromaDB using the current token. Alert on 401/403 responses.
• Metrics: Export httpx.request.duration and httpx.request.errors to Prometheus. Create alerts for error rate > 1% or latency > 5 s.
• Secret rotation policy: Rotate JWTs every 24 h and enforce a exp claim no longer than 2 h. Automate rotation with a Kubernetes CronJob.
• Version pinning: Keep the ChromaDB client library version aligned with the server major version. Use requirements.txt with an explicit version range (e.g., chromadb>=0.4.0,<0.5.0).

FAQ – Common Follow‑Up Questions

1. Why does the inference service work in dev but time out in prod?
   Dev clusters often have permissive egress rules and run ChromaDB on the default HTTP port (80/443). Production enforces strict NetworkPolicy and the server was upgraded to require JWTs, causing both connectivity and auth failures.
2. Can I keep using the legacy API key?
   Only if you downgrade the ChromaDB server to a version prior to the JWT enforcement (v0.3.x) or explicitly disable the require_auth flag in the server config. Maintaining the legacy key is discouraged because it bypasses token expiration.
3. What if I need to run ChromaDB behind a load balancer?
   Expose the service via a Service of type LoadBalancer or Ingress that terminates TLS, then configure the inference client to use the external URL and set verify_ssl=False only for internal testing. Ensure the load balancer forwards the Authorization header.
4. How do I debug intermittent timeouts that appear only under load?
   Collect tcpdump traces during peak traffic, look for SYN retransmissions or TCP window stalls. Correlate with pod CPU/Memory metrics; a saturated CPU can delay the client’s request processing, causing the server to close idle connections.
5. Is there a way to test JWT validity without hitting the full query endpoint?
   ChromaDB exposes a /auth/validate endpoint. A simple curl -H "Authorization: Bearer $TOKEN" https://chromadb:8443/auth/validate returns 200 if the token is accepted, 401 otherwise.

Related Topic Hub: Vector Databases Troubleshooting Hub