photo of computer cables

ChromaDB batch ingestion failure with 504 Gateway Timeout in Kubernetes

by

Problem – Intermittent Batch Ingestion Failures in ChromaDB

When a dedicated ingestion service streams large Kafka batches to the ChromaDB /api/v1/ingest endpoint, the operation fails with two distinct symptoms:

  • ERROR - client: Connection reset by peer (errno 104) while sending batch upsert request.
  • 504 Gateway Timeout - upstream request timed out (nginx ingress) while processing /api/v1/ingest endpoint.

The failures are intermittent, appear only with payloads larger than ~10 MB, and cause downstream processing pipelines to stall.

Root Cause Analysis

1. HTTP server limits in ChromaDB

The ChromaDB HTTP server enforces three configurable timeouts (ChromaDB Configuration – HTTP server settings):

Parameter Default Effect
max_request_size 5 MB Requests larger than this are rejected with HTTP 413.
read_timeout 30s Maximum time the server will wait for the request body.
write_timeout 30s Maximum time to write a response.

Batch payloads of 10–20 MB exceed max_request_size, causing the server to close the connection. The client sees “Connection reset by peer”. The ingress controller then reports a 504 because the upstream connection was terminated.

2. Ingress timeout configuration

NGINX ingress defaults to proxy_read_timeout 60s (GitHub issue #342). Bulk upserts can take several minutes, especially when the vector dimension is high. When the upstream processing exceeds 60 seconds, the ingress drops the connection and returns 504.

3. Pod resource exhaustion

Insufficient memory leads to OOM kills (Kubernetes Slack #vector-db thread). An OOM‑killed worker exits abruptly, producing “Connection reset by peer” on the client side and “Worker failed to start – RuntimeError: Event loop is closed” in the pod logs.

4. Istio sidecar timeout (if present)

When Istio is injected, the default HTTP timeout is 15 seconds. Bulk upserts exceed this, triggering a 504 from the sidecar (GitHub issue #389).

Investigation and Debugging Steps

Log Inspection


# ChromaDB pod logs
2024-05-21T12:34:56Z uvicorn.error: Worker failed to start – RuntimeError: Event loop is closed
2024-05-21T12:35:02Z app.error: request body too large (HTTP 413)

# Ingress controller logs
2024-05-21T12:35:07Z nginx: *12345 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.2.3.4, server: chromadb.example.com, request: "POST /api/v1/ingest HTTP/1.1", upstream: "http://10.8.0.12:8000/api/v1/ingest", host: "chroma.example.com"

Metric Examination

  • Check container_memory_usage_bytes – spikes to 95 % before OOM events.
  • Observe nginx_ingress_controller_upstream_response_time_seconds – many requests > 60 s.

Network Capture (optional)


tcpdump -i eth0 -s 0 -w ingest.pcap host chromadb.example.com and port 80

Look for TCP RST packets coinciding with the client timestamps.

Configuration Validation


# Verify ChromaDB config map
kubectl -n vectordb get configmap chromadb-config -o yaml

# Verify ingress annotations
kubectl -n vectordb get ingress chromadb-ingress -o yaml

Resolution – Step‑by‑Step Fixes

1. Increase HTTP server limits in ChromaDB

Update the ConfigMap to raise max_request_size and timeouts as per the ingestion API reference (ChromaDB Documentation – Ingestion API).

# Before (configmap.yaml)
data:
  max_request_size: "5MB"
  read_timeout: "30s"
  write_timeout: "30s"

# After (configmap.yaml)
data:
  max_request_size: "50MB"
  read_timeout: "300s"
  write_timeout: "300s"

Apply the change and restart the pods:


kubectl -n vectordb apply -f configmap.yaml
kubectl -n vectordb rollout restart deployment/chromadb

2. Tune NGINX ingress timeouts

Add or update the following annotations on the ingress resource (GitHub issue #342):

# Before
metadata:
  name: chromadb-ingress
  annotations:
    kubernetes.io/ingress.class: nginx

# After
metadata:
  name: chromadb-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"

Redeploy the ingress:


kubectl -n vectordb apply -f ingress.yaml

3. Allocate sufficient memory and enable autoscaling

Adjust the deployment resources and add a HorizontalPodAutoscaler (HPA) to handle bursty loads:

# Before (deployment.yaml)
resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "1"
    memory: "1Gi"

# After (deployment.yaml)
resources:
  requests:
    cpu: "1000m"
    memory: "2Gi"
  limits:
    cpu: "2000m"
    memory: "4Gi"
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: chromadb-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: chromadb
  minReplicas: 2
  maxReplicas: 6
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

4. Adjust Istio timeout (if Istio is used)

Patch the VirtualService to extend timeout:


apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: chromadb
spec:
  hosts:
  - chromadb.example.com
  http:
  - route:
    - destination:
        host: chromadb
    timeout: 300s

Verification – Confirming the Fix

Functional Test


# Simulate a 20 MB batch upsert
curl -X POST https://chroma.example.com/api/v1/ingest \
  -H "Content-Type: application/json" \
  --data-binary @large_batch.json \
  -w "\nHTTP %{http_code}\n"

Expected output:


{"status":"success","upserted":10000}
HTTP 200

Log and Metric Check

  • Ensure no “request body too large” or “upstream timed out” entries appear.
  • Confirm nginx_ingress_controller_upstream_response_time_seconds shows values < 300 s.
  • Validate pod memory usage stays < 80 % during peak ingestion.

Readiness Probe

After the changes, the readiness probe should succeed:


kubectl -n vectordb get pod -l app=chromadb -o jsonpath="{.items[*].status.conditions[?(@.type=='Ready')].status}"

Expected output: True

Prevention – Operational Guardrails

  • Monitoring: Alert on nginx_ingress_controller_upstream_response_time_seconds > 250 s and on pod OOMKill events.
  • Ingress defaults: Set cluster‑wide NGINX default annotations for proxy-read-timeout and proxy-body-size to accommodate vector workloads.
  • Batch sizing: Follow the ChromaDB Performance Tuning – Batch size guidelines. Keep individual upsert batches under 5 MB when possible, and use client‑side chunking.
  • Resource planning: Define requests and limits that reflect the maximum expected payload size and processing time.
  • Autoscaling policies: Enable both CPU‑based HPA and a custom metric‑based scaler that watches ingestion queue depth.

FAQ – Common Follow‑Up Questions

  1. Why does the ingestion work locally but fail in the cluster?
    Local runs bypass the ingress and typically have higher OS limits. In Kubernetes the request passes through NGINX (or Istio) which enforces stricter timeouts and body‑size limits.
  2. Can I keep the default max_request_size and still ingest large batches?
    Yes, by splitting the payload into smaller chunks (e.g., 4 MB each) and sending them sequentially. This respects the server limit while avoiding timeouts.
  3. Do I need to increase uvicorn workers?
    Increasing workers (e.g., from 1 to 4) helps when CPU is the bottleneck, but it does not solve request‑size or timeout issues. Use it in conjunction with the other fixes if CPU saturation is observed.
  4. How do I know if Istio is the source of the 504?
    Check the Envoy access logs (kubectl logs -l app=istio-proxy) for entries with upstream_request_timeout. If present, adjust the VirtualService timeout as shown above.
  5. Is there a way to surface the exact timeout value that caused the 504?
    Enable debug level logging on the NGINX ingress controller (controller.config.log-level: debug) and look for messages like “upstream timed out (110: Connection timed out)”. The log includes the configured timeout.

Related Topic Hub: Vector Databases Troubleshooting Hub