LLM Systems

DeepSeek collection creation fails with 500 error under high traffic

by
A person holding a cell phone in their hand

Problem Description During peak traffic periods the DeepSeek POST /v1/collections endpoint returns intermittent HTTP 500 responses. The failures manifest as timeouts and generic error bodies such as: HTTP/1.1 500 Internal Server Error Content-Type: application/json { “error”: “Internal Server Error: collection creation failed” } Observed symptoms include: Spike in ERROR – CreateCollection – Timeout after 30s log … Read more

GPT-4 response truncation after token limit exceeded

by
Smartphone screen displays ai assistant options.

Problem: GPT‑4 Responses Truncate After Token Limit Is Exceeded During development in a sandbox environment, engineers observed that GPT‑4 completions stop mid‑sentence, omit closing JSON braces, or otherwise cut off content. The API returns HTTP 200, but the choices[0].text field ends exactly at the configured max_tokens value, and the finish_reason is set to length. Downstream parsers … Read more

Anthropic Claude multi-GPU training stale endpoint slice data issue

by
Anthropic Claude

Problem – Stale Endpoint Slice Data in Anthropic Claude Multi‑GPU Training When training Claude models across multiple GPU nodes, operators have observed inconsistent gradients, divergent loss curves, and occasional training crashes. Typical symptoms include error messages such as: StaleSliceError: endpoint slice data is out of sync (expected version 4, got version 3) DataSliceOutOfSync – Slice … Read more

GPT-4 API service failed to bind port in Kubernetes cluster

by
a computer screen with a web page on it

Problem – GPT‑4 API Service Fails to Bind Port in a Hybrid‑Cloud Kubernetes Cluster The GPT‑4 API proxy is deployed as a Deployment with a Service of type NodePort. On several nodes the pod enters CrashLoopBackOff with logs such as: Error: listen tcp 0.0.0.0:5000: bind: address already in use 2024-06-20T14:32:01.123Z WARN openai-proxy: Failed to start … Read more

GPT-4o model loading timeout during production deployment behind load balancer

by
Chatgpt atlas app icon on abstract background

Problem Description The production service that wraps the OpenAI GPT‑4o model is failing to become ready during deployment. The symptom set observed across multiple environments includes: Container logs ending with Error: model loading timed out after 10000ms. HTTP 504 Gateway Timeout responses from the load balancer. Kubernetes readiness probe failures: Health check failed: GPT‑4o not ready … Read more

Mistral AI hybrid search scoring discrepancy between staging and production

by
a person holding a cell phone in their hand

Problem Description Symptoms and Impact After the nightly CI/CD pipeline retrains the Mistral AI embedding model and rolls out the new container image, the /v1/hybrid-search endpoint returns noticeably different relevance scores in the production Kubernetes cluster compared to the staging cluster. Typical observations: Top‑K results differ in ordering despite identical query text. Score drift of … Read more

OpenAI GPT-4o token refresh failure during real-time streaming

by
a computer screen with a web page on it

Problem Description During high‑throughput real‑time streaming with OpenAI GPT‑4o, the inference pipeline intermittently drops the connection. The client receives HTTP 401 responses such as: Error: Invalid API Key HTTP/1.1 401 Unauthorized Content-Type: application/json { “error”: { “message”: “Invalid request: token expired”, “type”: “invalid_request_error”, “code”: “invalid_api_key” } } Log excerpts from the Python SDK show the same … Read more

Mistral AI RAG retrieval dominance after rolling update

by
A person holding a smart phone in their hand

Problem Description – Retrieval Dominance After a Rolling Update A production Mistral AI Retrieval‑Augmented Generation (RAG) pipeline uses a hybrid search that combines: Vector‑based retrieval (weight = retrieval_weight) Transformer‑based reranking (weight = rerank_weight) During a rolling update that introduces a new reranker model version, operators observed: Hybrid search results suddenly favor raw retrieval scores. Relevance metrics dropped 30‑45 % (see … Read more

Mistral AI tokenizer encoding errors in multi-GPU training

by
A close up of a cell phone on a table

Problem Description When training a Mistral‑based model on a multi‑GPU cluster using torch.distributed data‑parallelism, the tokenizer occasionally returns different token ID sequences for the same input text on different ranks. The symptom manifests as: Sudden spikes in loss or RuntimeError: CUDA illegal memory access during the forward pass. Batch failures with errors such as ValueError: … Read more

Qwen rolling update external tool parsing errors

by
yellow and white square box

Problem Description During a rolling update of a Qwen cluster, the model fails to parse the output of an external tool that it invokes as part of its processing pipeline. The failure manifests as runtime exceptions such as: JSONDecodeError: Expecting value: line 1 column 1 (char 0) protobuf parsing error: mismatched wire type ValueError: Unexpected … Read more