ChromaDB Container Crash Loop On-premises OOM Error

Q: Problem – ChromaDB Container Crash Loop on‑Premises OOM Error

Deployments of ChromaDB on shared on‑premises servers often encounter a crash loop where the Docker container is repeatedly killed and restarted. Typical symptoms include:

Q: Investigation – Debugging the Crash Loop 1. Inspect container resource limits

docker inspect --format='{{json .HostConfig.Memory}}' chromadb_container docker inspect --format='{{json .HostConfig.MemorySwap}}' chromadb_container Typical output for a mis‑configured container:

Q: 2. Monitor memory usage in real time

docker stats chromadb_container --no-stream Sample output during ingestion:

Q: 3. Review Docker daemon and kernel logs

journalctl -u docker.service | grep -i "OOMKilled" grep -i "Out of memory" /var/log/kern.log Typical kernel message:

Q: 4. Check health‑check logs

docker logs --tail 20 chromadb_container Relevant excerpt:

Q: Solution – Stabilizing the Deployment 1. Increase container memory limits and enable swap

Before (Docker‑Compose snippet):

In this article

Table of Contents

Problem – ChromaDB Container Crash Loop on‑Premises OOM Error

Deployments of ChromaDB on shared on‑premises servers often encounter a crash loop where the Docker container is repeatedly killed and restarted. Typical symptoms include:

Docker daemon reports OOMKilled for the container.
Health‑check failures such as HTTP 500 Internal Server Error from the /healthz endpoint.
Docker logs show Killed process 1 (python) total-vm:123456kB, anon-rss:98765kB, file-rss:0kB, uid:0.
Docker CLI prints docker: Error response from daemon: container <id> is restarting.

These events prevent ChromaDB from serving queries or persisting new vectors, effectively taking the vector store offline.

Root Cause – Why the Container Exhausts Memory

ChromaDB stores vectors in‑memory while indexing and serves them via a Python FastAPI process. The memory consumption grows with:

Number of vectors ingested (each vector can be 256‑1024 dimensions).
Batch indexing jobs that allocate temporary NumPy arrays.
Concurrent query load that triggers copy‑on‑write pages.

On a host with limited RAM (e.g., 8 GB) and a Docker memory limit of 2 GB (set via mem_limit), the process quickly exceeds its cgroup quota during large ingestions. Docker’s OOM handling (Docker Engine Documentation – OOM handling) kills the process, and the container’s restart policy restarts it, causing a crash loop.

Two additional factors amplify the problem:

Missing swap configuration – Without memswap_limit, the container cannot use host swap, so any spike beyond the hard limit triggers immediate OOM kill (GitHub issue #112).
Health‑check timing – ChromaDB’s built‑in health endpoint returns 500 once the process runs out of memory (GitHub issue #158). Docker interprets the failure as unhealthy and restarts the container, adding to the loop.

Investigation – Debugging the Crash Loop

1. Inspect container resource limits

docker inspect --format='{{json .HostConfig.Memory}}' chromadb_container
docker inspect --format='{{json .HostConfig.MemorySwap}}' chromadb_container

Typical output for a mis‑configured container:

2048MiB   // Memory limit
0         // MemorySwap (no swap allowed)

2. Monitor memory usage in real time

docker stats chromadb_container --no-stream

Sample output during ingestion:

CONTAINER ID   NAME               CPU %   MEM USAGE / LIMIT   MEM %   NET I/O
abcd1234       chromadb_container 12.5%   2.1GiB / 2GiB       105%    12.3kB / 9.8kB

The MEM % > 100% indicates the container is over its limit and will be OOM‑killed.

3. Review Docker daemon and kernel logs

journalctl -u docker.service | grep -i "OOMKilled"
grep -i "Out of memory" /var/log/kern.log

Typical kernel message:

kernel: Out of memory: Kill process 1234 (chroma) score 950 or sacrifice child
kernel: Killed process 1234 (chroma) total-vm:123456kB, anon-rss:98765kB, file-rss:0kB, uid:0

4. Check health‑check logs

docker logs --tail 20 chromadb_container

Relevant excerpt:

2024-05-28T14:32:11.123Z [chroma] MemoryError: Unable to allocate 256000000 bytes
2024-05-28T14:32:12.001Z Health check failed: HTTP 500 Internal Server Error

5. Verify ChromaDB configuration variables

Environment variables controlling memory are documented in the ChromaDB Configuration Reference. Common ones:

CHROMA_MEMORY_LIMIT – maximum bytes the process may allocate.
CHROMA_HEALTH_CHECK_TIMEOUT – seconds before the health endpoint is considered failed.

Solution – Stabilizing the Deployment

1. Increase container memory limits and enable swap

Before (Docker‑Compose snippet):

services:
  chromadb:
    image: chromadb/chroma:latest
    mem_limit: 2g
    # memswap_limit omitted – defaults to 0 (no swap)
    restart: always

After – allocate sufficient RAM and allow swap (adjust values based on host capacity; for an 8 GB host, 4 GB + 2 GB swap is a safe starting point):

services:
  chromadb:
    image: chromadb/chroma:latest
    mem_limit: 4g
    memswap_limit: 6g   # 4 GB RAM + 2 GB swap
    environment:
      - CHROMA_MEMORY_LIMIT=3500000000   # ~3.5 GB in bytes
    restart: always

Why it works: Docker now permits the container to exceed the physical limit by using swap, preventing immediate OOM kill. The CHROMA_MEMORY_LIMIT env var caps the process below the cgroup limit, giving the runtime a graceful failure path instead of a hard kill.

2. Tune ChromaDB health‑check parameters

Add a longer timeout and a more tolerant start period:

services:
  chromadb:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/healthz"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s

This prevents Docker from restarting the container while the process is still loading large batches.

3. Isolate memory‑intensive workloads

If the host runs PostgreSQL or other services, consider:

Moving ChromaDB to a dedicated node or VM.
Setting --oom-score-adj to a lower value (e.g., -500) so the kernel prefers killing less critical containers first.

Command example:

docker run -d --name chromadb \
  --memory=4g --memory-swap=6g \
  --oom-score-adj=-500 \
  -e CHROMA_MEMORY_LIMIT=3500000000 \
  chromadb/chroma:latest

4. Apply resource requests/limits in Kubernetes (if applicable)

For a pod definition, increase requests and limits to match the expected peak usage:

apiVersion: v1
kind: Pod
metadata:
  name: chromadb
spec:
  containers:
  - name: chromadb
    image: chromadb/chroma:latest
    resources:
      requests:
        memory: "4Gi"
      limits:
        memory: "6Gi"
    env:
    - name: CHROMA_MEMORY_LIMIT
      value: "5000000000"
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8000
      initialDelaySeconds: 60
      periodSeconds: 30

Verification – Confirming the Fix

Redeploy the container/pod with the updated configuration.
Watch the container status:

docker ps -f name=chromadb

Expected output shows STATUS as Up … without the (health: starting) or (unhealthy) flag.

Trigger a moderate ingestion (e.g., 300k vectors) and monitor memory:

docker stats chromadb_container

MEM % should stay below 90 % and never exceed the MEM LIMIT line.

Check health endpoint after ingestion:

curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8000/healthz

Should return 200. No “HTTP 500” lines in docker logs.

Prevention – Guardrails for Future Deployments

Capacity planning: Estimate peak memory based on vector dimensions and batch size. The ChromaDB docs recommend at least 1.5 × the expected resident set size.
Monitoring: Set alerts on container_memory_working_set_bytes (Prometheus) and on Docker daemon OOM events.
Health‑check tuning: Align start_period with the longest expected indexing job.
Swap awareness: Enable memswap_limit only when host swap is provisioned; otherwise allocate more RAM.
Isolation: Run memory‑heavy services on separate cgroups or VMs to avoid cross‑service OOM cascades.

FAQ – Common Follow‑Up Questions

Why does the container crash only after a large ingestion and not on startup?
During ingestion ChromaDB allocates temporary NumPy buffers proportional to the batch size. Startup memory usage stays well below the limit, so OOM only appears when the batch exceeds the cgroup quota.
Can I rely on Docker’s default OOM‑killer without setting memswap_limit?
No. Without swap the kernel kills the process as soon as physical memory is exhausted, leading to immediate restarts. Enabling memswap_limit or raising the RAM limit gives the process room to complete the allocation.
How do I determine a safe value for CHROMA_MEMORY_LIMIT?
Measure the resident set size (RSS) during a representative ingestion run (e.g., using docker stats or ps -o rss). Set CHROMA_MEMORY_LIMIT to 80‑90 % of the container’s mem_limit to leave headroom for GC and OS overhead.
What if other containers on the same host also trigger OOM?
Docker’s OOM score defaults to 0 for all containers. Adjust --oom-score-adj for less critical services (e.g., -500) so the kernel prefers killing them before ChromaDB.
Is there a way to make the health check ignore temporary memory spikes?
Increase CHROMA_HEALTH_CHECK_TIMEOUT and the Docker healthcheck.start_period. This gives the process extra time to recover before Docker marks the container unhealthy.

Related Topic Hub: Vector Databases Troubleshooting Hub