Engineering Logs for Troubleshooting & Debugging
Problem – Weaviate Container CrashLoopBackOff in Kubernetes A production deployment of Weaviate on a multi‑node Kubernetes cluster repeatedly enters CrashLoopBackOff. The pod terminates shortly after start, causing the service to be unavailable for queries. Typical symptoms observed in the cluster: Pod status: CrashLoopBackOff after a few seconds. Readiness probe errors such as: Readiness probe failed: … Read more
Problem – Intermittent Batch Ingestion Failures in ChromaDB When a dedicated ingestion service streams large Kafka batches to the ChromaDB /api/v1/ingest endpoint, the operation fails with two distinct symptoms: ERROR – client: Connection reset by peer (errno 104) while sending batch upsert request. 504 Gateway Timeout – upstream request timed out (nginx ingress) while processing … Read more
Problem Description In a production Weaviate deployment several CronJob resources are used: Nightly backup (02:00) Index compaction (02:30) Data synchronization (every 15 minutes) Operators observed the following symptoms: Backup logs contain “Failed to acquire lock for backup”. Kubernetes controller logs show “CronJob is already running” for the index job. Occasional “Backoff limit exceeded” events after the … Read more
Weaviate Streaming API Incomplete Results Under High Traffic Problem – Symptoms and Impact During peak load the /v1/objects/{className}/{id}/stream endpoint returns truncated JSON payloads or the client receives a connection reset by peer error. Typical observations include: Logs in Weaviate: 2024-05-12T14:23:07Z WARN streaming response incomplete: expected 12456 bytes, got 8421 API gateway (Kong) reports: 504 Gateway … Read more
Problem Description In an air‑gapped deployment the audio‑embedding service that feeds vectors to Pinecone returns either an empty vector ([]) or a vector of length 0. Subsequent query calls to the Pinecone index fail with errors such as: Invalid vector length: expected 512, got 0 Empty query vector Vector dimension mismatch Symptoms observed in production … Read more
Problem – Stop Sequence Ignored During Disaster Recovery During a disaster‑recovery (DR) operation on a multi‑node ChromaDB cluster, the configured stop_sequence was not honoured. The restoration process continued past the intended termination point, resulting in: Partial overwriting of existing embedding files. Duplicate document IDs across shards. Truncated vectors – up to 12 % of dimensions lost … Read more
Problem – FAISS index access denied after EC2 instance reboot During a disaster‑recovery (DR) drill an EC2 instance was launched to replace a failed node. The instance runs a Python service that restores a FAISS index from an S3 bucket using boto3 and then loads the index with faiss.read_index. After the reboot the service crashes … Read more
Problem – Milvus ReplicaSet Scaling Failure Under High‑Concurrency Benchmark During a distributed performance benchmark of Milvus 2.x, the team attempted to increase the replicaCount of the QueryNode (and optionally the Proxy) to meet a target query throughput of ~10 k QPS. The scaling operation consistently failed: New QueryNode pods remained in Pending or entered CrashLoopBackOff. Existing … Read more
Problem – ChromaDB Container Crash Loop on‑Premises OOM Error Deployments of ChromaDB on shared on‑premises servers often encounter a crash loop where the Docker container is repeatedly killed and restarted. Typical symptoms include: Docker daemon reports OOMKilled for the container. Health‑check failures such as HTTP 500 Internal Server Error from the /healthz endpoint. Docker logs … Read more
Problem Description The controller manager pods in a multi‑region Pinecone deployment are crashing during synchronous index replication. Symptoms observed across us-east-1, eu-west-1 and ap-southeast-2 include: Pod restarts with exit code 137 (OOM kill) in the Kubernetes events. Log entries such as: 2024-05-28T14:12:03Z controller_manager: out of memory 2024-05-28T14:12:04Z replication sync failed: connection reset by peer 2024-05-28T14:12:05Z … Read more