ChromaDB secret rotation failure after Vault token update in Kubernetes

photo of computer cables

ChromaDB Secret Rotation Failure After Vault Token Update in Kubernetes Problem The production ChromaDB service is intermittently failing to authenticate against its backend store. Errors manifest as connection refusals and query timeouts. The failure correlates with the scheduled secret rotation that pulls new credentials from HashiCorp Vault. After a recent Vault token renewal, the rotation … Read more

ChromaDB connection timeout during RAG inference serving

Black cables and wires connected to the back of modular LED display panels

Problem – ChromaDB Connection Timeout During RAG Inference Serving When deploying a Retrieval‑Augmented Generation (RAG) service, the inference container repeatedly fails to retrieve documents from the ChromaDB vector store. The request handler logs a ConnectionTimeoutError after the client’s requests call exceeds the configured 5 second deadline, and the service returns a 504 response to the downstream … Read more

Intermittent ChromaDB similarity query latency spikes after rolling deployment

a rack of servers in a server room

Problem During a rolling deployment of the ChromaDB service (v0.4.2) in a staging Kubernetes cluster, similarity‑search queries that normally return in ~100 ms began exhibiting intermittent latency spikes exceeding 5 s. The spikes were not correlated with CPU or memory pressure on the pod. Typical log excerpt during a spike: 2024-10-28T14:12:03.721Z INFO chromadb.query: Executing similarity search for … Read more

ChromaDB query API 400 Bad Request after client library version mismatch

photo of computer cables

Problem: 400 Bad Request from ChromaDB Query API in an Air‑Gapped Deployment In an isolated internal network the client library attempts to invoke /api/v1/query on a ChromaDB server. The server returns HTTP 400 with a body similar to: { “detail”: “Invalid request payload: expected object with key ‘queries’, got null” } Symptoms observed: Python client raises requests.exceptions.HTTPError: 400 … Read more

ChromaDB multi-region replication causing RAG prompt placeholder missing

photo of computer cables

Problem When ChromaDB is deployed with multi‑region replication, Retrieval‑Augmented Generation (RAG) prompts occasionally render with the placeholder {{retrieved_text}} left untouched. The generated output contains the literal placeholder instead of the expected document snippets, causing downstream language model calls to fail or produce nonsensical answers. Typical symptom in the application logs: 2024-10-08 14:22:31,842 ERROR rag_engine – … Read more

Sandbox cannot resolve ChromaDB host DNS during vector store init

photo of computer cables

Problem During development in a sandbox environment the application fails to initialise the ChromaDB vector store. The client library throws a DNS resolution error and aborts before any HTTP request is sent. Observed log excerpt: 2026-10-07 14:22:31,842 ERROR chromadb.client – Failed to resolve host ‘chroma-db.internal.svc.cluster.local’: Name or service not known Traceback (most recent call last): … Read more

ChromaDB streaming response drops under high concurrency in production

a rack of servers in a server room

Problem – Intermittent Streaming Response Drops in ChromaDB Under High Concurrency In a production environment that serves dozens of simultaneous query calls with stream=True, operators observed that the response stream terminates prematurely. Clients receive only a subset of the expected result chunks, leading to incomplete search results and downstream processing errors. Typical symptoms: HTTP/2 or … Read more

ChromaDB slice endpoint returning stale data on A100 GPU cluster

a rack of servers in a server room

ChromaDB Slice Endpoint Returning Stale Data on an A100/H100 GPU Cluster Problem Description (Symptoms and Impact) When querying the /slice endpoint on a multi‑node GPU cluster (A100 or H100), the response contains embeddings that were present before a recent bulk upsert. The stale data persists for several minutes, causing downstream model inference to drift and … Read more

ChromaDB RAG query decomposition error during multi‑GPU training

a rack of servers in a server room

Problem – RAG query decomposition failures in a multi‑GPU training pipeline When scaling a Retrieval‑Augmented Generation (RAG) workflow to multiple GPUs, engineers observed that chroma_client.query calls intermittently returned malformed or empty sub‑queries. The downstream LLM received no context, leading to generation failures or hallucinations. Typical error messages included: ValueError: Query decomposition returned an empty list … Read more

ChromaDB IAM permission denied on collections during rolling update

a rack of servers in a server room

Problem Description During a Kubernetes rolling update of a production ChromaDB deployment, operators observed a burst of PermissionDenied errors when the newly started pods attempted to read existing collections. Typical log entries look like: 2024-07-12T14:23:07Z ERROR PermissionDenied: User does not have permission to access collection 7f9c2a3b 2024-07-12T14:23:07Z ERROR 403 Forbidden: IAM role arn:aws:iam::123456789012:role/chroma-db-read lacks chroma.collections.read … Read more