LLM Systems

Qwen model hallucinated descriptions during concurrent image-text inference

by
A wooden table topped with scrabble tiles spelling queen ali

Problem Description During a high‑traffic period the Qwen multimodal endpoint started returning unrelated or fabricated image captions. The issue manifested only when the API gateway routed dozens to thousands of concurrent image‑text requests through a load‑balanced pool of inference pods. Typical symptoms observed in logs and client responses: JSON response field caption contains text unrelated … Read more

Mistral AI model invoking external API with malformed JSON payload

by
a person holding a cell phone in their hand

Problem: Mistral AI model returns malformed JSON payloads when invoking external APIs during concurrent A/B test traffic splits In a production A/B testing setup, traffic is split between two Mistral model variants: Variant Temperature top_p Observed failure rate A 0.6 0.8 ~2 % B 0.9 0.95 34 % ↑ (SchemaValidationError) Under high concurrency, Variant B frequently produces JSON that: … Read more

DeepSeek token refresh failure in isolated Docker container

by
A person holding a cell phone in their hand

Problem: DeepSeek token refresh failure in an isolated Docker container When a DeepSeek client runs inside a headless Docker container that uses an immutable filesystem, restricted network egress, and non‑root user permissions, the automatic OAuth2 token refresh cycle stops working after the first access token expires. Subsequent inference requests receive HTTP 401 Unauthorized responses, and the … Read more

OpenAI GPT-3.5 API rate limit exceeded during parallel test runs

by
a close-up of a server room

Problem Description During automated model evaluation runs in a CI/CD pipeline, intermittent HTTP 429 (RateLimitError) responses are observed. The failures abort the test stage and block downstream deployment steps. Typical console output from a Python test worker: openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details. Response payload: { “error”: { “message”: … Read more

Inconsistent prompt templates across API gateway nodes OpenAI GPT-4

by
Openai logo with green and white cylindrical letters.

Inconsistent Prompt Templates Across API Gateway Nodes (OpenAI GPT‑4o) Problem During an A/B test that splits traffic between control and variant groups, the chat completion service began returning divergent outputs for the same user cohort. The symptoms observed were: Control requests sometimes received the temperature=0.7 value that belongs to the variant, and vice‑versa. System prompts … Read more

Kubernetes pod stuck after TensorFlow model update

by
red and black no smoking sign

Problem Description A Kubernetes Deployment that runs tensorflow_model_server was updated to serve a new model version. After the rollout the pods entered a Running state but never became Ready. The following symptoms were observed: Readiness probe failures: Readiness probe failed: Get http://localhost:8501/v1/models/my_model/metadata: dial tcp 127.0.0.1:8501: connect: connection refused Container logs show the server exiting immediately: … Read more

Anthropic Claude container crashing during large dataset evaluation

by
Abstract purple and gold elements within a reflective swirling structure.

Problem – Claude Container CrashLoop During Large‑Dataset Evaluation In a Kubernetes‑based evaluation pipeline the anthropic/claude container repeatedly enters CrashLoopBackOff while processing a benchmark dataset that exceeds 10 k prompts (≈5 GB JSON). The pod never reaches the Running state long enough to complete the evaluation workflow. Typical symptom log excerpt: kubectl logs claude-eval-7f9c9d8c5b-abcde -c claude … 2024-07-14T10:22:31.412Z … Read more

Meta LLaMA query latency spike after model update

by
cable network

Problem Description In a development sandbox that runs Meta LLaMA (v1) on a single NVIDIA A100 GPU, query latency jumped from an average of 150 ms to > 800 ms** within a two‑hour window. The workload is low‑concurrency (1‑5 queries per minute) and uses the default inference script from the Meta LLaMA 2 Documentation. The spike broke … Read more

OpenAI GPT-4 OAuth2 flow interrupted during high volume API benchmarking

by
a computer screen with a web page on it

Problem Description During an automated performance benchmark of the OpenAI GPT‑4 endpoint, a custom Python script launches hundreds of concurrent requests to measure throughput and latency. After a short ramp‑up period (≈30 seconds) the benchmark is interrupted by a surge of HTTP 401 Unauthorized responses. The script retries the failed calls, but each retry also receives 401, … Read more

GPT-3.5 collection creation fails during streaming with memory allocation error

by
img IX mining rig inside white and gray room

Problem – “GPT‑3.5 collection creation fails during streaming with memory allocation error” In a distributed inference service that streams completions to many simultaneous users, the OpenAI client throws errors such as: Error: Memory allocation failed RuntimeError: CUDA out of memory OpenAIError: Server error – 500 – Failed to create collection OSError: [Errno 12] Cannot allocate … Read more