logprob

Google Gemini logprob NaN after network latency spike in hybrid cloud

by
Google logo neon light signage

Problem Description During inference with the Google Gemini model in a hybrid‑cloud deployment, the logprob field in the response payload intermittently contains NaN or null values. The symptom manifests as: Inconsistent token scoring – generated text appears correct, but downstream ranking logic that relies on log probabilities fails. API errors such as HTTP 400 with … Read more

PyTorch model logprob inconsistencies under high gRPC traffic

by
blue and red cargo ship on sea during daytime

Problem Description During a sustained load test of a Docker‑containerized language model serving endpoint, the logprob values returned by the PyTorch model became inconsistent. Under low traffic the model produced deterministic log‑softmax outputs, but at peak rates (≈200 RPS) the following symptoms were observed: Log probabilities drifted by up to 0.12 nats between identical inputs. Intermittent errors … Read more