Jordan Lee

RAG chunk overlap duplicate embeddings after increasing chunk size on GCP Compute

by
Rows of black server racks with white logos in a data center

Problem – Overlapping Chunk Configuration Produces Duplicate Embeddings in a RAG Pipeline on GCP Compute In a production Retrieval‑Augmented Generation (RAG) service running on Google Cloud Compute Engine VMs, engineers observed that increasing the chunk_size parameter caused a sudden rise in duplicate vector embeddings. The vector store returned multiple identical entries for the same passage, … Read more

Weaviate 429 rate limit after batch vector inserts during traffic spike

by
a rack of servers in a server room

Weaviate 429 Rate‑limit after batch vector inserts during traffic spike Problem An event‑driven pipeline receives bursts of data events, transforms each payload into a vector embedding, and writes the embeddings to a Weaviate instance using the /v1/batch/objects endpoint. During traffic spikes the pipeline starts returning HTTP 429 “Rate limit exceeded” responses, causing: Back‑pressure on upstream microservices Ingestion … Read more

Staging AMD GPU node volume mount failure during container init

by
black and green digital device

Problem – Staging AMD GPU Node Volume Mount Failure During Container Init In the staging environment, AI training pipelines are orchestrated on AMD GPU compute nodes via Kubernetes. Each job expects a host‑backed dataset directory to be mounted into the container at /data. During pod initialization the mount operation fails, the container starts without the … Read more

Pinecone RAG chunk overlap misconfiguration causing duplicate vectors

by
brown pine cone in close up photography

Problem During development of a Retrieval‑Augmented Generation (RAG) pipeline that stores chunk embeddings in Pinecone, the index shows: Duplicate vectors for many text fragments. Gaps where expected chunks are missing. Retrieval results returning overlapping passages, causing hallucinations in downstream LLM responses. Typical log excerpt from the ingestion script: 2024-10-26 14:02:13,842 INFO ingest.py:124 – Processing document: … Read more

Hugging Face token limit exceeded during disaster log batch inference

by
Robotic figure with blue and red details against sky

Problem: Token‑limit Exceeded During Batch Inference of Disaster‑Recovery Logs During a recent disaster‑recovery drill, a pipeline that feeds multi‑megabyte incident logs, situational reports, and recovery plans into a Hugging Face Transformer model began failing with the following error: RuntimeError: Token indices sequence length is longer than the model’s maximum context length (4096). Truncate or split the … Read more

GPT-4o inference scheduler pod placement failure after node affinity change

by
Openai logo with green and white cylindrical letters

Problem – Scheduler Pod Fails to Schedule After Node‑Affinity Change The GPT‑4o inference scheduler pod remains in Pending state despite the cluster having free capacity. The issue appeared after a change to the pod’s nodeAffinity rules intended to isolate inference workloads on GPU‑enabled nodes. Observed symptoms Pod status: Pending Events from kubectl describe pod gpt4o-scheduler-xxxx … Read more

ChromaDB query API 400 Bad Request after client library version mismatch

by
photo of computer cables

Problem: 400 Bad Request from ChromaDB Query API in an Air‑Gapped Deployment In an isolated internal network the client library attempts to invoke /api/v1/query on a ChromaDB server. The server returns HTTP 400 with a body similar to: { “detail”: “Invalid request payload: expected object with key ‘queries’, got null” } Symptoms observed: Python client raises requests.exceptions.HTTPError: 400 … Read more

TGI model load fails with IAM permission denied during DR failover

by
a close up of a container with words on it

TGI model load fails with IAM permission denied during DR failover Problem During a disaster‑recovery (DR) failover to the secondary region, the Text Generation Inference (TGI) service repeatedly aborts model loading. The pod logs contain IAM permission denied errors when accessing the model artifacts stored in Cloud Storage and when reading the configuration stored in … Read more

Claude rolling update cron overlap causing duplicate model refresh tasks

by
Glossy purple abstract shapes with reflections

Problem – Duplicate Model Refresh Tasks During Rolling Updates During a rolling update of the Anthropic Claude service, operators observed that the model-refresh CronJob executed twice for a single refresh window. The duplicate executions caused: Increased CPU and memory pressure on the refreshed pods. Conflicting writes to the shared model cache, resulting in occasional cache … Read more

LLM API function call format error after HAProxy header rewrite

by
Rows of black server racks with white logos in a data center

Problem – Function‑Calling Format Error When LLM API Is Behind HAProxy During model evaluation the inference service returns a 400 Bad Request with a body similar to: { “error”: { “code”: “invalid_request_error”, “message”: “Function call JSON is malformed” } } The error only appears when the request traverses HAProxy, which is used to load‑balance multiple … Read more