Engineering Logs for Troubleshooting & Debugging
Problem – Stop Sequence Ignored During Disaster Recovery During a disaster‑recovery (DR) operation on a multi‑node ChromaDB cluster, the configured stop_sequence was not honoured. The restoration process continued past the intended termination point, resulting in: Partial overwriting of existing embedding files. Duplicate document IDs across shards. Truncated vectors – up to 12 % of dimensions lost … Read more
Problem – FAISS index access denied after EC2 instance reboot During a disaster‑recovery (DR) drill an EC2 instance was launched to replace a failed node. The instance runs a Python service that restores a FAISS index from an S3 bucket using boto3 and then loads the index with faiss.read_index. After the reboot the service crashes … Read more
Problem – Milvus ReplicaSet Scaling Failure Under High‑Concurrency Benchmark During a distributed performance benchmark of Milvus 2.x, the team attempted to increase the replicaCount of the QueryNode (and optionally the Proxy) to meet a target query throughput of ~10 k QPS. The scaling operation consistently failed: New QueryNode pods remained in Pending or entered CrashLoopBackOff. Existing … Read more
Problem – ChromaDB Container Crash Loop on‑Premises OOM Error Deployments of ChromaDB on shared on‑premises servers often encounter a crash loop where the Docker container is repeatedly killed and restarted. Typical symptoms include: Docker daemon reports OOMKilled for the container. Health‑check failures such as HTTP 500 Internal Server Error from the /healthz endpoint. Docker logs … Read more
Problem Description The controller manager pods in a multi‑region Pinecone deployment are crashing during synchronous index replication. Symptoms observed across us-east-1, eu-west-1 and ap-southeast-2 include: Pod restarts with exit code 137 (OOM kill) in the Kubernetes events. Log entries such as: 2024-05-28T14:12:03Z controller_manager: out of memory 2024-05-28T14:12:04Z replication sync failed: connection reset by peer 2024-05-28T14:12:05Z … Read more