Intermittent ChromaDB similarity query latency spikes after rolling deployment

Because the pool size is defined per‑process, the issue only appears during deployments that increase pod count; under steady state the 2‑pod configuration stays within the DB limit.

Debugging Steps

1. Confirm connection‑pool exhaustion

   
   kubectl exec -it $(kubectl get pod -l app=chromadb -o name | head -n1) -- \
     bash -c "psql -U chroma -d chroma -c 'SELECT count(*) FROM pg_stat_activity WHERE state = ''active'';'"
   

   Typical output during a spike:

   
    count 
   -------
       30
   (1 row)
   

2. Inspect PostgreSQL configuration

   
   kubectl exec -it postgres-0 -- \
     psql -U postgres -c "SHOW max_connections;"
   

   Output:

   
    max_connections 
   -----------------
    30
   (1 row)
   

3. Correlate pod replica count with connection count

   
   kubectl get deployment chromadb -o jsonpath='{.spec.replicas}'
   kubectl get pods -l app=chromadb -o jsonpath='{range .items[*]}{.metadata.name}:{.status.containerStatuses[0].ready}{" "}{end}'
   

   During rollout the replica count jumps from 2 to 4, while the old pods are still terminating, confirming overlapping capacity.

4. Check driver timeout settings

   
   # Inside a running container
   cat /app/chromadb/config.yaml | grep connection_timeout
   

   Result:

   
   connection_timeout: 5s
   

Solution

The fix consists of two coordinated changes:

1. Reduce per‑pod connection pool size

Update the ChromaDB deployment to set CHROMA_DB_MAX_POOL_SIZE=5, ensuring that even with four replicas the total connections stay ≤ 20.

Before (environment variables snippet):

env:
  - name: CHROMA_DB_MAX_POOL_SIZE
    value: "10"
  - name: CHROMA_DB_CONNECTION_TIMEOUT
    value: "5s"

After:

env:
  - name: CHROMA_DB_MAX_POOL_SIZE
    value: "5"
  - name: CHROMA_DB_CONNECTION_TIMEOUT
    value: "5s"

2. Increase PostgreSQL max_connections and enable a connection pooler

Modify the PostgreSQL StatefulSet to raise max_connections to 100 and deploy PgBouncer as a sidecar to multiplex connections.

Before (postgresql.conf snippet):

max_connections = 30

After:

max_connections = 100

Deploy PgBouncer with a pool_mode = transaction and default_pool_size = 20. Update ChromaDB’s DSN to point to the PgBouncer service (e.g., postgres://chroma:p@gbouncer:6432/chroma).

3. Add a readiness probe that waits for the pool to be initialized

This prevents the service from receiving traffic before it can acquire the minimum number of connections.

readinessProbe:
  exec:
    command: ["sh", "-c", "pg_isready -U chroma -d chroma && curl -sf http://localhost:8000/healthz"]
  initialDelaySeconds: 15
  periodSeconds: 5

Verification

1. Deploy the updated manifests

   
   kubectl apply -f chromadb-deployment.yaml
   kubectl apply -f postgres-config.yaml
   kubectl apply -f pgbouncer.yaml
   

2. Run a load test against the /query endpoint

   
   hey -c 20 -n 2000 http://staging-chromadb:8000/query
   

   Expected result: 95‑th percentile latency <200 ms, no >5 s outliers.

3. Check connection metrics

   
   kubectl exec -it postgres-0 -- \
     psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;"
   

   Should stay below 20 even during peak load.

4. Validate health endpoint

   
   curl -s http://staging-chromadb:8000/healthz | jq .
   

   Response contains "status":"ok" and "db_pool_ready":true.

Prevention

• Define CHROMA_DB_MAX_POOL_SIZE based on the known max_connections of the backing database and the maximum replica count.
• Deploy a dedicated connection pooler (PgBouncer, Pgpool) for any service that performs high‑concurrency vector searches.
• Include a readiness probe that verifies DB connectivity before marking the pod ready.
• Monitor db_connection_acquire_seconds and set an alert when the 95‑th percentile exceeds 1 s.
• During rolling updates, use maxSurge: 0 and maxUnavailable: 1 to avoid temporary over‑provisioning of pods.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

1. Why did the latency spikes only appear during rolling deployments?
   Because the deployment temporarily increased the number of pods, each requesting its full connection pool. The sum exceeded PostgreSQL’s max_connections, causing connection queuing only while the old pods were terminating.

2. Can I keep the default pool size and just raise max_connections?
   Raising max_connections alone works, but it masks the underlying scaling risk. Properly sizing the pool per pod and using a pooler provides a safety margin and reduces DB resource consumption.

3. Is PgBouncer mandatory?
   Not mandatory, but it decouples the number of application connections from the database limit, simplifies scaling, and adds features like transaction pooling that are beneficial for short‑lived similarity queries.

4. What metric should I watch to detect a similar issue early?
   Watch db_connection_acquire_seconds_bucket{le="5"} and the PostgreSQL pg_stat_activity count. A rising tail in acquisition latency indicates pool exhaustion.

5. How does the readiness probe prevent the problem?
   Kubernetes will not route traffic to a pod until the probe succeeds. By checking that the DB pool can acquire at least one connection, we ensure the pod is ready to serve queries, avoiding early traffic that would otherwise block on connection acquisition.