ChromaDB RAG Query Decomposition Error During Multi‑GPU Training

Q: Root Cause – Concurrency limits and non‑thread‑safe components

ChromaDB’s query API supports decomposition parameters that split a prompt into sub_query_ids (see the Query API reference). The decomposition step relies on:

Q: Investigation and Debugging 1. Log inspection

Enable ChromaDB debug logging (CHROMA_LOG_LEVEL=DEBUG) and capture a failing request:

Q: Resolution – Making the RAG pipeline concurrency‑safe 1. Use a process‑isolated client per GPU worker

Instantiate a fresh ChromaClient inside each worker process instead of sharing a global instance.

Q: 4. Enable connection pooling for the HTTP layer

If using the REST server, configure uvicorn workers and a keep‑alive pool:

In this article

Table of Contents

Problem – RAG query decomposition failures in a multi‑GPU training pipeline

When scaling a Retrieval‑Augmented Generation (RAG) workflow to multiple GPUs, engineers observed that chroma_client.query calls intermittently returned malformed or empty sub‑queries. The downstream LLM received no context, leading to generation failures or hallucinations. Typical error messages included:

ValueError: Query decomposition returned an empty list of sub‑queries.
RuntimeError: SQLite database is locked
ChromaError: Retrieval returned 0 results

The issue manifested only under concurrent load from several GPU workers, while single‑GPU or local runs behaved correctly.

Root Cause – Concurrency limits and non‑thread‑safe components

ChromaDB’s query API supports decomposition parameters that split a prompt into sub_query_ids (see the Query API reference). The decomposition step relies on:

A shared tokeniser instance (not thread‑safe).
The persistence backend (SQLite by default) which enforces a single writer lock.

In a multi‑GPU setting, each rank (or worker) creates its own Python thread or process but often reuses the same ChromaClient instance. Under high concurrency the following interactions break:

Simultaneous tokenisation attempts corrupt the internal buffer, producing sub_query_ids that are null or duplicated (see the open‑source demo repo issue where a shared global tokeniser caused dropped sub‑queries).
The default SQLite backend serialises writes; concurrent reads/writes trigger “database is locked” errors (GitHub Issue #913). When SQLite returns a partial JSON payload, the client deserialises an empty sub_query_ids list.
Without a connection pool, HTTP requests to the ChromaDB server are queued, causing timeouts (TimeoutError: waiting for query response …) and incomplete responses.

Collectively these race conditions lead to the observed malformed sub‑queries and empty retrieval results.

Investigation and Debugging

1. Log inspection

Enable ChromaDB debug logging (CHROMA_LOG_LEVEL=DEBUG) and capture a failing request:


2024-09-13 10:12:45,321 DEBUG chroma.server.query Decomposing prompt: "Explain the risk of X..."
2024-09-13 10:12:45,322 ERROR chroma.server.db SQLite locked: database is locked
2024-09-13 10:12:45,323 WARN  chroma.server.query Returned empty sub_query_ids

The pattern of “SQLite locked” followed by an empty sub_query_ids list matches the incidents reported by the fintech startup and the research lab case study.

2. Reproducing the race condition locally

Run two parallel processes that share a client instance:


import multiprocessing as mp
from chromadb import Client

def worker():
    client = chroma_client  # shared global
    resp = client.query(
        collection_name="docs",
        query_text="What is the impact of Y?",
        decomposition={"max_subqueries": 5}
    )
    print(resp)

if __name__ == "__main__":
    chroma_client = Client()
    processes = [mp.Process(target=worker) for _ in range(2)]
    for p in processes: p.start()
    for p in processes: p.join()

Typical output shows one process receiving a full response, the other getting:


{'ids': [], 'embeddings': [], 'documents': [], 'metadatas': [], 'sub_query_ids': None}

3. Backend verification

Check which persistence backend is active (default is SQLite). In the container logs:


INFO  chroma.persistence Using SQLite backend at /data/chroma.db

If the workload requires high concurrency, the official Performance & Scaling Guide recommends PostgreSQL or DuckDB with connection pooling.

Resolution – Making the RAG pipeline concurrency‑safe

1. Use a process‑isolated client per GPU worker

Instantiate a fresh ChromaClient inside each worker process instead of sharing a global instance.

# Before (shared client)
chroma_client = Client()
def train_worker(rank):
    # reused client
    ...

# After (per‑process client)
def train_worker(rank):
    client = Client()  # new instance per rank
    # use client for RAG queries
    ...

2. Switch to a concurrency‑friendly persistence backend

Configure ChromaDB to use PostgreSQL (or DuckDB with a pool) in chroma_settings.yaml:

# Before (SQLite)
persistence:
  type: sqlite
  path: /data/chroma.db

# After (PostgreSQL)
persistence:
  type: postgresql
  host: db.internal
  port: 5432
  database: chroma
  user: chroma_user
  password: ${CHROMA_DB_PASSWORD}
  pool_size: 20

PostgreSQL supports multiple concurrent readers/writers, eliminating the “database is locked” error.

3. Isolate the tokeniser per worker

Chroma’s default tokeniser is a singleton. Override it with a thread‑local instance:

from chromadb.tokenizer import Tokenizer
import threading

thread_local = threading.local()

def get_tokenizer():
    if not hasattr(thread_local, "tokenizer"):
        thread_local.tokenizer = Tokenizer()
    return thread_local.tokenizer

# Pass the tokenizer to the client (pseudo‑API)
client = Client(tokenizer_factory=get_tokenizer)

This prevents buffer corruption that leads to null sub_query_ids.

4. Enable connection pooling for the HTTP layer

If using the REST server, configure uvicorn workers and a keep‑alive pool:


uvicorn chroma.server:app --host 0.0.0.0 --port 8000 --workers 4 --keep-alive 30

5. Adjust query decomposition parameters

Reduce the number of sub‑queries per request to lower per‑request payload size, mitigating timeouts:


resp = client.query(
    collection_name="docs",
    query_text=prompt,
    decomposition={"max_subqueries": 3, "min_chunk_size": 50}
)

Verification – Confirming that the fix works

Functional test

Run a multi‑GPU training run with the updated configuration and assert that every query returns at least one retrieval chunk:


for rank in range(num_gpus):
    resp = client.query(... )
    assert resp["documents"], f"Rank {rank} returned empty results"
print("All ranks returned non‑empty retrievals")

Log validation

Search logs for the absence of the previous error patterns:


grep -i "SQLite locked" logs/*.log   # should return no lines
grep -i "empty sub_query_ids" logs/*.log   # should return no lines

Performance metrics

Monitor query latency and DB connection pool usage (Prometheus metrics chroma_db_query_duration_seconds, chroma_db_pool_active_connections). Latency should stabilise below 200 ms per query, and active connections should stay within the configured pool size.

Prevention – Operational guardrails for future scaling

Per‑process client instances: Enforce client creation inside the training worker entry point.
Choose a scalable backend: Default to PostgreSQL for any workload with >4 concurrent workers.
Connection pooling: Configure both DB and HTTP pools with headroom (e.g., 2× expected concurrency).
Tokeniser isolation: Use thread‑local or process‑local tokenisers; avoid global singletons.
Monitoring: Alert on database is locked errors, query timeouts, and empty sub_query_ids counts.
Testing: Include a stress test that spawns N workers (where N = number of GPUs × 2) and validates non‑empty retrievals before any production rollout.

FAQ – Common follow‑up questions

Why does the error appear only when using multiple GPUs? Each GPU worker runs in its own process but previously shared a global client and tokeniser. Concurrency bugs surface only when those shared resources are accessed simultaneously.
Can I keep using SQLite if I limit the number of workers? SQLite can handle read‑only concurrency, but any write (e.g., inserting new embeddings) will still serialize. For >2 concurrent writers, switch to PostgreSQL or DuckDB with pooling.
How do I verify which sub‑queries were generated? Enable CHROMA_LOG_LEVEL=DEBUG and inspect the sub_query_ids field in the JSON response. The debug log prints the full decomposition payload.
Is there a built‑in way to make the tokeniser thread‑safe? As of the current release, the tokeniser is not thread‑safe. The recommended pattern is to instantiate a separate tokeniser per thread or process, as shown in the resolution section.
What timeout should I set for large batches? The default 30 seconds may be insufficient under heavy load. Increase to 60 seconds and monitor latency; adjust based on observed query duration histograms.

Related Topic Hub: Vector Databases Troubleshooting Hub