Distributed Systems

HAProxy config reload fails due to invalid function call format

by
photo of computer cables

Problem – HAProxy reload fails with “invalid function call format” During automated deployments in a GitLab CI pipeline the HAProxy service aborts its reload step. The CI job reports a validation error such as: ERROR: configuration file contains an invalid function call format at line 42 parse error: unexpected token ‘(‘ in fetch expression Invalid … Read more

HAProxy context window overflow during ML training loop

by
img IX mining rig inside white and gray room

Problem – HAProxy Context Window Overflow During ML Training Loop In a Kubernetes‑based distributed training pipeline, dozens of GPU worker pods issue HTTP GET/POST requests to a shared model storage service through an HAProxy ingress. During hyper‑parameter sweeps and large batch fetches the following symptoms were observed: HAProxy logs repeatedly contain tune.bufsize exceeded, context window … Read more

PyTorch model logprob inconsistencies under high gRPC traffic

by
blue and red cargo ship on sea during daytime

Problem Description During a sustained load test of a Docker‑containerized language model serving endpoint, the logprob values returned by the PyTorch model became inconsistent. Under low traffic the model produced deterministic log‑softmax outputs, but at peak rates (≈200 RPS) the following symptoms were observed: Log probabilities drifted by up to 0.12 nats between identical inputs. Intermittent errors … Read more

Kubernetes CRD validation errors after schema update

by
a group of blue boxes

Problem – Validation Errors After Updating a CRD Schema After a recent schema change to a shared Custom Resource Definition (CRD) used by multiple AI/ML services, kubectl apply and CI/CD pipelines began failing with errors such as: error: admission webhook “v1.crdvalidation.k8s.io” denied the request: spec: Required value Other observed messages include: validation failed: spec.replicas: Invalid … Read more

Nginx reverse proxy causing tokenizer encoding errors

by
a rack of servers in a server room

Problem – Tokenizer Encoding Errors Behind an Nginx Reverse Proxy During automated model deployment in a CI/CD pipeline (GitLab CI, Jenkins, Azure DevOps, or GitHub Actions), the inference service receives POST requests containing raw text that must be tokenized. In several incidents the service raised exceptions such as: UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x80 … Read more

Docker API server refuses connection from data pipeline

by
red and blue cargo ship on sea during daytime

Problem Description The data‑pipeline service, defined in a docker‑compose.yml file, attempts to call the API server at http://api:5000/api during ingestion. The call consistently fails with either a timeout or a connection‑refused error, causing the pipeline to abort. Typical log excerpts: pipeline | 2026-06-12T08:14:32Z ERROR: dial tcp 172.18.0.3:5000: connect: connection refused pipeline | 2026-06-12T08:14:32Z ERROR: Get … Read more

HAProxy traffic imbalance during blue-green deployment

by
a close-up of a server room

Problem – HAProxy traffic imbalance during a blue‑green deployment During a scheduled blue‑green rollout of a new service version, the production HAProxy 2.8 front‑end observed a ~40 % increase in query latency. The spike appeared only after the first batch of green servers was added to the blue_green backend and persisted until the rollout completed. Typical … Read more

Nginx webhook timeout during long GPU training jobs

by
cable network

Problem Description During multi‑GPU training jobs running in a Kubernetes cluster, webhook callbacks that report progress to an external observability platform (e.g., Datadog, Prometheus remote‑write) start failing after a few minutes of computation. The Nginx ingress controller logs errors such as: [error] 12345#0: *6789 upstream timed out (110: Connection timed out) while reading response header … Read more