LLM Systems

DeepSeek collection creation fails with runtime exception after rolling deployment

by
A person holding a cell phone in their hand

Problem Description During a rolling deployment of DeepSeek on a 5‑node GKE cluster, API calls to /v1/collections started failing with a runtime exception. The failure manifested as: java.lang.RuntimeException: Failed to write collection to PostgreSQL at com.deepseek.service.CollectionService.create(CollectionService.java:112) Caused by: org.postgresql.util.PSQLException: Connection is closed at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:247) Additional logs showed HikariCP pool timeouts: HikariPool-1 – Connection is not … Read more

Google Gemini pods pending despite free A100/H100 GPUs in Kubernetes

by
a close up of a cell phone with buttons

Problem: Gemini Training Pods Remain Pending Despite Free A100/H100 GPUs In a production Vertex AI cluster that mixes NVIDIA A100 and H100 nodes, engineers observed that new gemini training pods never transition to Running. The scheduler reports: FailedScheduling: 0/6 nodes are available: 6 Insufficient nvidia.com/gpu. Other observed symptoms include: Pod events showing pod didn’t match … Read more

Google Gemini CRD validation failure on kind after operator install

by
computer screen showing google search

Problem – Gemini CRD Validation Failure on Kind After Operator Install When applying a GeminiModel or GeminiEndpoint custom resource to a local kind cluster, kubectl apply aborts with validation errors such as: error: error validating “gemini-model.yaml”: error validating data: ValidationError(GeminiModel.spec): missing required field “model” in io.k8s.api.core.v1.ConfigMap error: unable to recognize “gemini-deployment.yaml”: no matches for kind … Read more

DeepSeek API server unreachable during high‑concurrency model evaluation

by
A person holding a cell phone in their hand

Problem Description During large‑scale model evaluation runs, scripts that invoke the https://api.deepseek.com/v1/chat/completions endpoint start failing after a few seconds of sustained traffic. The most common error messages observed are: Error: connect ETIMEDOUT https://api.deepseek.com/v1/chat/completions HTTP 502 Bad Gateway – received from DeepSeek load balancer during peak load HTTP 504 Gateway Timeout – request exceeded server timeout … Read more

Hybrid search score mismatch after dense model update in CI

by
A close up of a cell phone with a keyboard

Problem Description Nightly integration tests for the DeepSeek search service started failing after the dense encoder was upgraded to v2.1. The regression suite validates that the top‑k ranking produced by the hybrid (dense + sparse) scorer is deterministic across builds. After the model update the following error appeared in the CI logs: HybridScoreMismatchError: dense_score=0.8423, sparse_score=0.2311, combined_score variance … Read more

LLaMA tokenizer returns missing token IDs for Unicode emojis during evaluation

by
Rows of black server racks with white logos in a data center

Problem Description Symptoms and Impact During validation of a multilingual dataset that contains user‑generated comments, the LLaMA evaluation script crashes or produces malformed loss values. Typical log excerpts are: 2026-09-03 10:12:45,231 – INFO – Processing line 8427 2026-09-03 10:12:45,232 – WARNING – Tokenizer output length mismatch: expected 128 tokens, got 124 2026-09-03 10:12:45,233 – ERROR … Read more

GPU container crash loop for GPT-4o after driver upgrade

by
a cell phone sitting on top of a laptop computer

Problem Description The GPT‑4o inference service runs in a Kubernetes pod that launches a Docker container with GPU acceleration (A100/H100). After upgrading the NVIDIA driver from 525.xx to 550.xx, the pod entered a crash loop: Container exit code 1 with log line: Failed to initialize NVML: Driver/library version mismatch Subsequent restarts showed Segmentation fault (core … Read more

etcd configuration drift after adding AWS node breaks GPT-3.5 routing

by
Rows of black server racks with white logos in a data center

Problem Description After provisioning a new EC2 instance and adding it as an etcd member to a hybrid on‑premises/AWS cluster, the OpenAI GPT‑3.5 routing layer began serving stale model versions. Requests that should have been directed to gpt-3.5-turbo-0613 were occasionally routed to the older gpt-3.5-turbo-0301 endpoint, causing latency spikes and inconsistent billing. Observed symptoms Intermittent … Read more

Google Gemini query latency spikes in staging during peak CI load

by
a person holding a cell phone in their hand

Problem: Google Gemini Query Latency Spikes in Staging During Peak CI Load During nightly integration runs the staging environment experiences Gemini model response times up to 5× the SLA threshold (e.g., 300 ms → 1.5 s). The spikes are correlated with the start of a large CI batch (≈200 concurrent test jobs). Symptoms include: Latency histogram in … Read more

Mistral AI RAG citation JSON‑LD schema mismatch on edge node

by
A person holding a smart phone in their hand

Problem Description – Mistral AI RAG citation JSON‑LD schema mismatch on edge node Edge deployments of Mistral AI’s Retrieval‑Augmented Generation (RAG) pipeline are failing to produce citations that conform to the required JSON‑LD schema. Downstream parsers – e.g., knowledge‑graph ingest services – reject the payload, leading to: Runtime crashes in Kubernetes edge pods (e.g., jsonschema.exceptions.ValidationError: … Read more