Engineering Logs for Troubleshooting & Debugging
Problem After a rolling deployment of the vLLM inference service in a Kubernetes cluster, client applications began receiving connection timeout, connection refused, and HTTP 502 Bad Gateway responses when calling the vLLM API endpoint. The issue appears only after the deployment finishes; the service was reachable before the rollout. Root Cause The timeout originated from … Read more
Problem Description During a Kubernetes rolling update of a TorchServe‑hosted Retrieval‑Augmented Generation (RAG) service, the RAGRetriever began returning an empty list of documents for every query. The symptom manifested as log entries such as: INFO 2026-09-21 12:34:56,789 RAGRetriever: No documents retrieved for query ERROR 2026-09-21 12:34:57,102 faiss index not found at path /model/index.faiss WARNING 2026-09-21 … Read more
Problem Description During a rolling upgrade of an on‑premises Kubernetes‑managed etcd cluster (v3.5.0 → v3.5.7) the AI model‑serving stack began loading outdated model parameters. The model‑config microservice reads configuration from etcd, writes the current config_version to a Redis key (model:config), and caches the full parameter payload with a 12‑hour TTL. After the upgrade the following … Read more
Problem Description During a rolling deployment of DeepSeek on a 5‑node GKE cluster, API calls to /v1/collections started failing with a runtime exception. The failure manifested as: java.lang.RuntimeException: Failed to write collection to PostgreSQL at com.deepseek.service.CollectionService.create(CollectionService.java:112) Caused by: org.postgresql.util.PSQLException: Connection is closed at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:247) Additional logs showed HikariCP pool timeouts: HikariPool-1 – Connection is not … Read more
Problem Description During a Kubernetes rolling update of a production ChromaDB deployment, operators observed a burst of PermissionDenied errors when the newly started pods attempted to read existing collections. Typical log entries look like: 2024-07-12T14:23:07Z ERROR PermissionDenied: User does not have permission to access collection 7f9c2a3b 2024-07-12T14:23:07Z ERROR 403 Forbidden: IAM role arn:aws:iam::123456789012:role/chroma-db-read lacks chroma.collections.read … Read more
Problem Description During a rolling update of a production Meta LLaMA RAG service, the retrieval‑augmented generation pipeline intermittently fails to retrieve embeddings. The failure manifests as: grpc deadline exceeded: context deadline exceeded while attempting to query vector store VectorStoreError: connection refused – unable to establish TCP connection to Faiss endpoint TimeoutError: failed to retrieve embeddings … Read more
OpenAI GPT‑4o API Rate Limit Exceeded During Kubernetes Rolling Update Problem Description (Symptoms and Impact) During a rolling update of the text‑generation microservice on GKE, a burst of 429 Too Many Requests responses was observed. The symptoms included: Log entries such as: [WARN] request_id=abc123 – OpenAI API returned 429 – Rate limit exceeded – retry_after=30s Failed … Read more
Problem: NodePort conflicts during rolling updates During a rolling update of a Deployment that exposes a Service of type NodePort, new Pods are scheduled while the old Pods are still running. If the Service definition specifies a static nodePort (or relies on the default allocation) and the update creates a second Service object (e.g., during … Read more
Problem – RAG query decomposition fails during a rolling update During a rolling update of an MLflow‑served Retrieval‑Augmented Generation (RAG) model, inference requests start returning errors such as: Failed to decompose query: embedding dimension mismatch (expected 768, got 512) CacheKeyCollisionError: Duplicate cache entry for key ‘rag_decompose:customer_id=123’ detected during rolling update ModelVersionNotFoundError: Requested version ‘2’ for … Read more
Problem Description – Retrieval Dominance After a Rolling Update A production Mistral AI Retrieval‑Augmented Generation (RAG) pipeline uses a hybrid search that combines: Vector‑based retrieval (weight = retrieval_weight) Transformer‑based reranking (weight = rerank_weight) During a rolling update that introduces a new reranker model version, operators observed: Hybrid search results suddenly favor raw retrieval scores. Relevance metrics dropped 30‑45 % (see … Read more