Jordan Lee

LlamaIndex OOM error during Kafka batch processing in production

by
Rows of black server racks with white logos in a data center

LlamaIndex OOM error during Kafka batch processing in production Problem In a high‑throughput event‑driven pipeline, a LlamaIndex worker consumes document batches from Kafka, creates embeddings using a transformer model, and stores the indexed vectors in a vector store. Under load the worker crashes with an out‑of‑memory (OOM) termination: 2024-10-12 03:45:21,874 ERROR worker.py:124 — Process terminated … Read more

Kafka StatefulSet rolling update stalls after PVC conflict in Kubernetes

by
white concrete building under blue sky during daytime

Problem A Kubernetes StatefulSet that runs a Kafka cluster is stuck during a rolling update. The update was triggered to apply a new container image and a small configuration change. After the first pod (kafka-0) is updated successfully, the next pod (kafka-1) never becomes Ready. The rollout remains in Progressing state indefinitely, causing the API … Read more

GPT-3.5 checkpoint backup fails with storage class not found during DR

by
photo of computer cables

Problem During a disaster‑recovery (DR) run‑book, the automated backup job that persists GPT‑3.5 checkpoint files to the configured backup bucket fails with the following error: 2026-11-05T14:32:07Z ERROR backup_job: storage class not found: “fast-ssd-backup” 2026-11-05T14:32:07Z INFO backup_job: aborting checkpoint upload for model “gpt-3.5” The failure occurs only in the DR region; the same backup pipeline works … Read more

ONNX Runtime RAG vector store connection timeout during batch inference

by
A large building with a sign on top of it

Problem – ONNX Runtime RAG Vector Store Connection Timeout During Batch Inference Symptoms observed in production batch runs: Batch inference jobs exit with RuntimeError: Retrieval failed – connection timeout. Log entries from the ONNX Runtime host show repeated authentication failures: 2026-11-05 02:13:41,872 [ERROR] onnxruntime.capi._pybind_state: RetrievalError: Failed to query vector store at https://vectordb.internal:443/v1/query – error: ConnectTimeoutError: … Read more

Milvus memory allocation failure during batch insert after multi‑GPU training

by
a bird of prey is flying in the air

Problem: Milvus Memory Allocation Failure During Batch Insert After Multi‑GPU Training In a production pipeline, a distributed multi‑GPU training job streams high‑dimensional embeddings (e.g., 1536‑dim float vectors) to a Milvus vector database running on a single node with 64 GiB RAM. When the training job attempts to insert a batch of ~200 k vectors, Milvus logs show … Read more

Google Gemini OAuth2 token acquisition failure after service account rotation

by
Google sign

Problem An automated ETL pipeline invokes the Google Gemini API for model inference. After rotating the Google Cloud service account used for authentication, the pipeline repeatedly fails to acquire an OAuth2 access token. All subsequent Gemini requests return 401 Unauthorized errors, causing the data processing job to abort. Root Cause The failure stems from a … Read more

Grafana 10 RAG prompt rendering error due to undefined variables

by
cable network

Problem Grafana 10’s new LLM integration allows panels to generate answers using Retrieval‑Augmented Generation (RAG). In our on‑premises deployment the RAG prompt template fails to compile, producing errors such as: [2024-10-12 03:14:27] error plugin.rag Error rendering RAG prompt: variable “question” is undefined [2024-10-12 03:14:27] error plugin.rag Template compilation failed: unexpected token “}}” at line 4 The … Read more

Pinecone hybrid search vector dominance after scaling multi‑GPU RAG

by
brown pinecones

Pinecone Hybrid Search Vector Dominance After Scaling Multi‑GPU RAG Problem When running a Retrieval‑Augmented Generation (RAG) pipeline on a multi‑GPU training cluster, the hybrid search query in Pinecone returns results that are overwhelmingly ranked by dense vector similarity, while the sparse BM25 component contributes almost nothing. The symptom is a noticeable drop in relevance for … Read more

Prometheus scrape timeout from inference pods after network policy applied

by
angel statue on brown wooden floor

Problem After applying a restrictive NetworkPolicy to the AI inference serving namespace, Prometheus started reporting scrape timeouts for all /metrics endpoints exposed by the inference pods. The failure manifests as: level=error ts=2026-11-03T12:45:27.123Z caller=scrape.go:842 component=prometheus msg=”Scrape failed” target=”inference-0.inference.svc.cluster.local:8080″ err=”context deadline exceeded (client timeout exceeded while awaiting headers)” Metrics collection stops, causing dashboards to show stale data … Read more

Kubernetes StatefulSet rollout error after ONNX Runtime multi-GPU update

by
System information displayed in a terminal window with red text on a black background

Problem A Kubernetes StatefulSet that runs ONNX Runtime for distributed multi‑GPU training fails during a rollout after upgrading the ONNX Runtime container to a version that adds multi‑GPU support. The controller logs contain: controller-runtime: failed to update StatefulSet “onnx-trainer”: the object has been modified; please try again later New pods remain in Pending state with … Read more