Vector Databases

Weaviate controller manager crashes with segfault during high load inference

by
a close-up of a server room

Problem – Controller‑manager crashes with segmentation fault under high‑load inference In a multi‑node Weaviate cluster running on Kubernetes with GPU‑accelerated inference, the controller‑manager pod repeatedly terminates with a SIGSEGV during spikes of concurrent /v1/objects or /v1/graphql queries. The failure manifests as: Segmentation fault (core dumped) [signal SIGSEGV: segmentation fault] at 0x7f… in libtorch.so panic: runtime … Read more

Pinecone index schema mismatch during blue-green deployment

by
brown pine cone in close up photography

Problem — Structured Output Validation Errors During Blue‑Green Deployment During a blue‑green rollout of a new Pinecone index version, the green environment began returning validation errors such as: Schema validation failed: expected field ‘metadata.category’ of type string, got integer or PineconeException: Index schema mismatch – query dimensions (1536) do not match index dimensions (1024) These errors manifested … Read more

Weaviate container crash loop in Kubernetes deployment

by
photo of computer cables

Problem – Weaviate Container CrashLoopBackOff in Kubernetes A production deployment of Weaviate on a multi‑node Kubernetes cluster repeatedly enters CrashLoopBackOff. The pod terminates shortly after start, causing the service to be unavailable for queries. Typical symptoms observed in the cluster: Pod status: CrashLoopBackOff after a few seconds. Readiness probe errors such as: Readiness probe failed: … Read more

ChromaDB batch ingestion failure with 504 Gateway Timeout in Kubernetes

by
photo of computer cables

Problem – Intermittent Batch Ingestion Failures in ChromaDB When a dedicated ingestion service streams large Kafka batches to the ChromaDB /api/v1/ingest endpoint, the operation fails with two distinct symptoms: ERROR – client: Connection reset by peer (errno 104) while sending batch upsert request. 504 Gateway Timeout – upstream request timed out (nginx ingress) while processing … Read more

Weaviate CronJob scheduling conflict in Kubernetes

by
a close-up of a server room

Problem Description In a production Weaviate deployment several CronJob resources are used: Nightly backup (02:00) Index compaction (02:30) Data synchronization (every 15 minutes) Operators observed the following symptoms: Backup logs contain “Failed to acquire lock for backup”. Kubernetes controller logs show “CronJob is already running” for the index job. Occasional “Backoff limit exceeded” events after the … Read more

Weaviate streaming API incomplete results under high traffic

by
photo of computer cables

Weaviate Streaming API Incomplete Results Under High Traffic Problem – Symptoms and Impact During peak load the /v1/objects/{className}/{id}/stream endpoint returns truncated JSON payloads or the client receives a connection reset by peer error. Typical observations include: Logs in Weaviate: 2024-05-12T14:23:07Z WARN streaming response incomplete: expected 12456 bytes, got 8421 API gateway (Kong) reports: 504 Gateway … Read more

Pinecone audio embeddings fail in air-gapped environment

by
tilt-shift photography of tree

Problem Description In an air‑gapped deployment the audio‑embedding service that feeds vectors to Pinecone returns either an empty vector ([]) or a vector of length 0. Subsequent query calls to the Pinecone index fail with errors such as: Invalid vector length: expected 512, got 0 Empty query vector Vector dimension mismatch Symptoms observed in production … Read more

ChromaDB stop sequence ignored during disaster recovery

by
cable network

Problem – Stop Sequence Ignored During Disaster Recovery During a disaster‑recovery (DR) operation on a multi‑node ChromaDB cluster, the configured stop_sequence was not honoured. The restoration process continued past the intended termination point, resulting in: Partial overwriting of existing embedding files. Duplicate document IDs across shards. Truncated vectors – up to 12 % of dimensions lost … Read more

Milvus ReplicaSet scaling failure during high concurrency benchmarking

by
A red kite soars against a green forest.

Problem – Milvus ReplicaSet Scaling Failure Under High‑Concurrency Benchmark During a distributed performance benchmark of Milvus 2.x, the team attempted to increase the replicaCount of the QueryNode (and optionally the Proxy) to meet a target query throughput of ~10 k QPS. The scaling operation consistently failed: New QueryNode pods remained in Pending or entered CrashLoopBackOff. Existing … Read more