a rack of servers in a server room

ChromaDB slice endpoint returning stale data on A100 GPU cluster

by

ChromaDB Slice Endpoint Returning Stale Data on an A100/H100 GPU Cluster

Problem Description (Symptoms and Impact)

When querying the /slice endpoint on a multi‑node GPU cluster (A100 or H100), the response contains embeddings that were present before a recent bulk upsert. The stale data persists for several minutes, causing downstream model inference to drift and leading to missed SLA targets.

Typical observations:

  • API response payload includes an error field such as:
{
  "error": "StaleDataError",
  "detail": "Slice data may be outdated, consider calling /refresh"
}
  • Log entry in the ChromaDB pod:
WARN  chroma::index::cache - Cache miss for slice request, serving from stale snapshot
  • GPU node logs show NCCL synchronization warnings:
NCCL WARN: Timeout while synchronizing GPU buffers, possible data race in slice operation

Impact includes:

  • Incorrect similarity search results for up to ~15 minutes (as seen in the biotech startup incident, Q2 2024).
  • Increased error rates in downstream pipelines that rely on fresh embeddings.
  • Unnecessary alert noise from health checks that compare recent upserts against slice results.

Root Cause Analysis

The stale data originates from a combination of two mechanisms:

  1. Write‑through cache invalidation delay: ChromaDB persists vectors to disk according to the Persistence and Caching documentation. On GPU‑accelerated deployments, writes are first staged in GPU memory and flushed asynchronously. The cache invalidation hook that marks the in‑memory index as dirty is triggered only after a successful NCCL barrier. In the incident at the biotech startup, the NCCL barrier timed out, leaving the index in a stale snapshot.
  2. Missing explicit refresh: The /slice API supports an optional refresh=true query parameter (see the API reference). When omitted, the endpoint may serve from the last consistent snapshot. Community issue #842 and Stack Overflow question 81927412 both point out that callers often forget to invoke chroma.flush() or the /refresh endpoint after bulk upserts on GPU workers.

In practice, the following sequence caused the problem:

  • Bulk upsert of 1 M embeddings on node 1 (A100).
  • GPU memory buffers are written, but NCCL synchronization across the 4‑node ring fails due to a transient network jitter.
  • Write‑ahead log (WAL) is persisted, but the in‑memory index on nodes 2‑4 never receives the invalidation signal.
  • Subsequent /slice calls on any node read from the stale in‑memory snapshot, triggering the warning shown above.

Investigation and Debugging Steps

Follow these steps to reproduce and isolate the issue in a controlled environment:

  1. Confirm stale slice behavior:
curl -X POST https://chroma.example.com/api/slice \
  -H "Content-Type: application/json" \
  -d '{"ids": ["vec_12345"], "include_vectors": true}'

Expected: vector matches the most recent upsert.
Observed: vector matches the pre‑upsert value.

  1. Check cache and sync logs on each GPU node:
journalctl -u chroma.service | grep -i "cache\|NCCL"

Look for messages such as:

WARN  chroma::index::cache - Cache miss for slice request, serving from stale snapshot
NCCL WARN: Timeout while synchronizing GPU buffers, possible data race in slice operation
  1. Validate write‑through status using the internal health endpoint (if enabled):
curl http://localhost:8000/_internal/health | jq .write_sync

Possible values: "ok" or "pending". A "pending" state indicates that the WAL has not been flushed to the shared index.

  1. Force a manual refresh and observe the change:
curl -X POST https://chroma.example.com/api/refresh

After the call, repeat the slice request. The vector should now be up‑to‑date.

  1. Inspect NCCL barrier statistics to confirm timeouts:
nccl-tests/build/all_reduce_perf -b 8 -e 64M -f 2 -g 4

High latency or timeout errors correlate with the stale‑data windows observed.

Resolution (Fix Implementation)

The fix consists of two complementary actions: ensuring synchronous flush after bulk writes and configuring the deployment to enforce stronger NCCL barrier timeouts.

1. Add explicit chroma.flush() after bulk upserts

Update the ingestion script to call the flush API before returning control to the client.

Before:

# bulk_upsert.py
client.upsert(vectors)
print("Upsert complete")

After:

# bulk_upsert.py
client.upsert(vectors)
client.flush()  # forces write‑through and cache invalidation
print("Upsert and flush complete")

2. Adjust GPU deployment environment variables

According to the GPU deployment guide, set the following variables to tighten synchronization:

export CHROMA_GPU_SYNC_TIMEOUT=30   # seconds, default is 10
export CHROMA_CACHE_INVALIDATE=true
export NCCL_DEBUG=INFO

Redeploy the service with the updated environment. The longer timeout gives NCCL enough time to complete the barrier even under transient network congestion.

3. Enable automatic refresh on slice requests (optional)

If the client cannot guarantee a flush, configure the slice endpoint to request a refresh automatically:

curl -X POST "https://chroma.example.com/api/slice?refresh=true" \
  -H "Content-Type: application/json" \
  -d '{"ids": ["vec_12345"], "include_vectors": true}'

This adds a lightweight barrier before serving the slice, ensuring consistency at the cost of a few milliseconds of latency.

Verification (Validation Steps)

After applying the fixes, perform the following checks:

  1. Functional test: Run a bulk upsert followed by an immediate slice request with refresh=true. Verify that the returned vector matches the newly upserted data.
  2. Log inspection: Ensure no Cache miss for slice request, serving from stale snapshot warnings appear in the logs for at least 30 minutes after the upsert.
  3. Metrics validation: If you expose Prometheus metrics, watch chroma_slice_stale_seconds (or similar) drop to zero after the flush.
  4. Stress test: Simulate concurrent upserts on all GPU nodes and issue slice requests at 1‑second intervals. Confirm that the latency increase due to refresh=true stays below the SLA threshold (e.g., <5 ms).

Operational Experience (Lessons Learned)

  • Relying on the default asynchronous flush is risky on GPU clusters where NCCL barriers can be delayed. Explicitly flushing after bulk operations eliminates the race condition.
  • Misleading symptom: the API returns a successful HTTP 200 with a vector payload, leading engineers to assume the data is fresh. The warning log entry is the only indicator of staleness.
  • In mixed‑precision indexing (H100 scenario), the in‑memory index can diverge from persisted storage even without NCCL timeouts. A manual cache purge (chroma.refresh()) is required after any change to the indexing precision.
  • Production alerts should monitor both chroma.flush_latency_seconds and NCCL timeout counters to catch synchronization regressions early.

Best Practices and Prevention

  • Always invoke client.flush() (or call /refresh) after bulk upserts, especially when using GPU workers.
  • Set CHROMA_GPU_SYNC_TIMEOUT to a value that exceeds the worst‑case network latency in your environment.
  • Enable CHROMA_CACHE_INVALIDATE=true to force cache invalidation on every successful write.
  • Instrument your deployment with Prometheus metrics for write_sync status and NCCL barrier health.
  • Include a health check that performs a lightweight slice with refresh=true and verifies the result against a known recent upsert.

Related Questions (FAQ)

  1. Why does the slice endpoint return stale data only after bulk upserts?
    Because bulk upserts are staged in GPU memory and flushed asynchronously. Without an explicit flush or successful NCCL barrier, the in‑memory cache remains unchanged.
  2. Do I need to call /refresh for every slice request?
    No. Call it only when you cannot guarantee that a prior write has been flushed. Enabling CHROMA_CACHE_INVALIDATE=true reduces the need for manual refreshes.
  3. How can I monitor NCCL synchronization health?
    Set NCCL_DEBUG=INFO and export the nccl_sync_timeout_seconds metric via the ChromaDB exporter. Alert if the metric exceeds the configured CHROMA_GPU_SYNC_TIMEOUT.
  4. Is the issue specific to A100 GPUs?
    The root cause is tied to the asynchronous write‑through path, which exists on both A100 and H100. However, H100 mixed‑precision indexing adds an extra divergence risk, as seen in the July 2024 research lab outage.
  5. Can I disable caching altogether to avoid staleness?
    Setting CHROMA_CACHE_INVALIDATE=false disables automatic invalidation but forces every slice to read from disk, dramatically increasing latency. It is not recommended for production workloads.

Related Topic Hub: Vector Databases Troubleshooting Hub