ChromaDB OOM Error During High-throughput Vector Insertion Benchmarking

Q: Problem: OOM Errors During High‑Throughput Vector Insertion Benchmarking

When benchmarking ChromaDB on a single‑node VM (16 GB RAM, 8 vCPU) with 10 M+ 768‑dimensional embeddings, the Python process is terminated by the kernel after a few gigabytes of RSS growth. Typical failure messages observed include:

Q: 1. Reduce max_batch_size and limit concurrency

# before client = chromadb.Client() client.add(vectors, ids, embeddings, batch_size=1000) # default # after client = chromadb.Client() client.add(vectors, ids, embeddings, batch_size=250) # smaller batches # limit workers to 2 for a 16 GB node 2. Tune HNSW parameters Lower M and ef_construction to values that fit the memory budget (see the Indexing Parameters guide).

Q: 4. Adjust SQLite cache and journal mode

# before (implicit defaults) # SQLite uses default page cache (~2000 pages) and DELETE journal mode # after sqlite3 /path/to/db.sqlite "PRAGMA cache_size = -512000;" # 512 MiB cache sqlite3 /path/to/db.sqlite "PRAGMA journal_mode = WAL;" # Write‑ahead logging reduces journal size 5. Switch to Parquet with streaming writer (optional) If disk I/O is not the bottleneck, the Parquet backend can be used with a reduced buffer size.

Q: 6. Enforce a hard memory limit via cgroups (production safeguard)

# create a cgroup with 14 GB limit sudo cgcreate -g memory:/chroma echo $((14*1024*1024*1024)) | sudo tee /sys/fs/cgroup/memory/chroma/memory.limit_in_bytes sudo cgexec -g memory:chroma python benchmark.py Validation: Confirming the Fix After applying the configuration, re‑run the benchmark and verify:

In this article

Table of Contents

Problem: OOM Errors During High‑Throughput Vector Insertion Benchmarking

When benchmarking ChromaDB on a single‑node VM (16 GB RAM, 8 vCPU) with 10 M+ 768‑dimensional embeddings, the Python process is terminated by the kernel after a few gigabytes of RSS growth. Typical failure messages observed include:

MemoryError: Unable to allocate 1.2 GiB
sqlite3.OperationalError: database or disk is full
OSError: [Errno 12] Cannot allocate memory

Two concrete runs illustrate the symptom:

Insertion via client.add() in batches of 10 k caused RSS to climb to ~2.3 GB, after which SQLite’s journal file expanded beyond 8 GB and the process was killed.
Four concurrent workers with an HNSW index triggered MemoryError: Unable to allocate 1.2 GiB during hnsw_index.build(), with each worker retaining a full copy of the vector matrix.

Root Cause Analysis

The OOM condition originates from a combination of:

Batch buffer allocation: ChromaDB’s ingestion loop creates a NumPy array sized (max_batch_size, dim). With the default max_batch_size=1000 and 768‑dim vectors, each batch consumes roughly 1000 × 768 × 4 ≈ 2.9 MiB. Under concurrent workers this multiplies, quickly exhausting RAM.
HNSW index materialisation: The HNSW construction keeps the entire vector matrix in memory while building the graph. As documented in the Memory Management and Index Settings section, the ef_construction parameter controls the temporary buffer size; high values (e.g., 400) inflate memory usage dramatically.
SQLite page cache pressure: Bulk inserts write a large journal file. Community logs (Discord thread 2024‑03‑12) show the page cache growing unchecked, eventually swapping to disk and causing the kernel OOM killer.
Parquet writer buffering: When using the Parquet backend, the writer buffers the full batch before flushing. In the “Parquet backend runs out of memory” issue (#917), this caused peak memory >14 GB during persist_index.

In short, the default configuration assumes ample memory for a single‑threaded load. High‑throughput, multi‑worker benchmarks exceed those assumptions, leading to uncontrolled allocations.

Investigation and Debugging Steps

Follow this reproducible checklist to isolate the memory hotspot:

Capture process memory growth:
```
watch -n 1 "ps -o pid,rss,cmd -p $(pgrep -f chromadb)"
```
Observe RSS spikes when a new batch starts.

Enable ChromaDB debug logging (environment variable CHROMA_LOG_LEVEL=debug) and look for messages like:

WARN  chroma.index.hnsw - 'EF construction value too high for current memory budget, reducing to 120'

Inspect SQLite page cache size:
```
sqlite3 /path/to/db.sqlite "PRAGMA cache_size;"
```
A negative value indicates size in KiB; values > 4 GB are a red flag.

Profile NumPy allocations with tracemalloc:

import tracemalloc, sys
tracemalloc.start()
# run a single batch insertion
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:5]:
    print(stat)

Look for large (max_batch_size, dim) allocations.

Check HNSW construction memory:

grep -i "hnsw_index.build" -R $(python -c "import site, sys; print(site.getsitepackages()[0])")

Identify the temporary np.ndarray that holds the full vector set.

Monitor SQLite journal growth:
```
ls -lh /path/to/db.sqlite-journal
```
If the file exceeds a few GB, SQLite is the bottleneck.

Resolution: Memory‑Optimised Configuration

Apply the following changes. They target each identified hotspot.

1. Reduce `max_batch_size` and limit concurrency

# before
client = chromadb.Client()
client.add(vectors, ids, embeddings, batch_size=1000)  # default

# after
client = chromadb.Client()
client.add(vectors, ids, embeddings, batch_size=250)  # smaller batches
# limit workers to 2 for a 16 GB node

2. Tune HNSW parameters

Lower M and ef_construction to values that fit the memory budget (see the Indexing Parameters guide).

# before (default)
hnsw_params = {"M": 32, "ef_construction": 400}

# after
hnsw_params = {"M": 16, "ef_construction": 200}
client.create_collection(name="vectors", metadata={"hnsw": hnsw_params})

3. Disable persistent in‑memory index during bulk load

Persisting the index after every batch forces a full copy of the graph into memory.

# before
client.persist()  # called inside the ingestion loop

# after
# defer persistence until after all batches complete
...
for batch in batches:
    client.add(..., batch_size=250)
client.persist()  # single call at the end

4. Adjust SQLite cache and journal mode

# before (implicit defaults)
# SQLite uses default page cache (~2000 pages) and DELETE journal mode

# after
sqlite3 /path/to/db.sqlite "PRAGMA cache_size = -512000;"   # 512 MiB cache
sqlite3 /path/to/db.sqlite "PRAGMA journal_mode = WAL;"    # Write‑ahead logging reduces journal size

5. Switch to Parquet with streaming writer (optional)

If disk I/O is not the bottleneck, the Parquet backend can be used with a reduced buffer size.

# before
client = chromadb.Client(settings={"persist_directory": "/data/chroma", "storage": "parquet"})

# after
client = chromadb.Client(settings={
    "persist_directory": "/data/chroma",
    "storage": "parquet",
    "parquet_write_buffer": 64 * 1024 * 1024  # 64 MiB per write
})

6. Enforce a hard memory limit via cgroups (production safeguard)

# create a cgroup with 14 GB limit
sudo cgcreate -g memory:/chroma
echo $((14*1024*1024*1024)) | sudo tee /sys/fs/cgroup/memory/chroma/memory.limit_in_bytes
sudo cgexec -g memory:chroma python benchmark.py

Validation: Confirming the Fix

After applying the configuration, re‑run the benchmark and verify:

RSS never exceeds ~10 GB (allowing headroom for OS).
SQLite journal file remains < 500 MiB.
No MemoryError or OOM kill messages in dmesg.
All 10 M vectors are persisted and searchable.

Sample validation commands:

# Verify vector count
client = chromadb.Client()
col = client.get_collection("vectors")
print(col.count())  # should output 10000000

# Spot‑check a random vector
result = col.query(embedding=rand_vec, n_results=5)
print(result)

Metrics from prometheus (if enabled) should show stable memory usage and no spikes during batch ingestion.

Prevention and Best Practices

Practice	Why it matters	Implementation
Cap `max_batch_size` relative to RAM	Prevents large NumPy buffers per worker	Use `max_batch_size = floor(RAM / (dim * 4 * concurrency * safety_factor))`
Tune HNSW parameters for workload	Reduces temporary graph buffers	Set `M ≤ 16`, `ef_construction ≤ 200` for 16 GB nodes
Persist index once after bulk load	Avoids repeated full‑graph copies	Call `client.persist()` only at the end
Configure SQLite WAL and cache size	Limits journal growth and page‑cache RAM pressure	Run `PRAGMA journal_mode=WAL;` and `PRAGMA cache_size=-512000;`
Monitor memory via cgroup or container limits	Provides early OOM detection before kernel kill	Set `memory.limit_in_bytes` or Docker `--memory` flag

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Why does increasing ef_construction cause OOM even though it only affects search quality?
ef_construction determines the size of the temporary candidate list during graph building. The list is stored in a NumPy array proportional to ef_construction × dim. On a 768‑dim vector, ef_construction=400 allocates ~1.2 GiB per worker, which quickly exhausts RAM.
Can I keep the default max_batch_size=1000 if I use a larger VM?
Yes, but you must ensure that RAM ≥ (max_batch_size × dim × 4 × concurrency × safety_factor). For 768‑dim vectors and 4 workers, a 64 GB instance is a safe baseline.
Is the SQLite journal file the only source of memory pressure?
No. The journal is a symptom of excessive write buffering. The primary pressure comes from the in‑memory HNSW matrix and the NumPy batch buffers. Reducing those buffers also shrinks the journal.
Should I switch to the Parquet backend for large ingestion?
Parquet reduces disk I/O but buffers whole batches in memory. If you keep batch sizes small (< 500) and set parquet_write_buffer to ≤ 64 MiB, Parquet can be used safely.
How can I detect that the process will OOM before it happens?
Enable Prometheus metrics for process_resident_memory_bytes and set an alert at 80 % of the node’s RAM. Additionally, run the benchmark inside a cgroup with a hard limit; the kernel will emit Memory cgroup out of memory messages before killing the process.