Jordan Lee

Qwen shard rebalancing error during model weight update

by
black sedan parked beside tree during daytime

Problem Description – ShardRebalanceError During Model Weight Update During the CI/CD integration of Qwen‑7B, the distributed training job aborts after a weight checkpoint is pushed. The failure manifests as a ShardRebalanceError indicating that the data partitioning across training nodes is inconsistent. Typical log excerpts look like: [rank 0] ERROR: ShardRebalanceError: inconsistent partitioning across nodes [rank … Read more

Nginx reverse proxy causing tokenizer encoding errors

by
a rack of servers in a server room

Problem – Tokenizer Encoding Errors Behind an Nginx Reverse Proxy During automated model deployment in a CI/CD pipeline (GitLab CI, Jenkins, Azure DevOps, or GitHub Actions), the inference service receives POST requests containing raw text that must be tokenized. In several incidents the service raised exceptions such as: UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x80 … Read more

Kubernetes StatefulSet update fails during rolling upgrade TGI deployment

by
Open bible page with handwritten notes on the side.

Problem Description A high‑traffic Text Generation Inference (TGI) service is deployed as a StatefulSet on a Kubernetes cluster. During a rolling upgrade the controller stalls after the first pod is updated. Subsequent pods never reach the Running/Ready state, causing a partial service outage and a noticeable drop in request capacity. Typical symptoms observed in the … Read more

PostgreSQL vision token embedding overflow during ML training

by
a close-up of a server room

Problem – Vision Token Embedding Overflow During ML Training A machine‑learning pipeline extracts image embeddings (e.g., CLIP vision tokens) and stores each intermediate token vector in a PostgreSQL table while training. During a nightly training run the following symptoms were observed: Rapid disk consumption: storage grew from 200 GB to 1.2 TB within a few hours. PostgreSQL … Read more

MLflow training job rejected by Kubernetes admission controller

by
photo of computer cables

Problem Description When launching an MLflow training run that uses the Kubernetes backend, the pod creation fails with an admission‑controller error. The MLflow client reports a kubectl error similar to the following: Error from server (Forbidden): pods “mlflow-train-7f9c8d5b9-xyz” is forbidden: exceeded quota: compute-resources, request: cpu=8, memory=16Gi Other observed messages include: admission webhook “validation.gatekeeper.sh” denied the … Read more

RAG retrieval empty results with vLLM in hybrid cloud

by
A ribbon with gradient colors floats in the air.

Problem Description In a hybrid‑cloud deployment of a Retrieval‑Augmented Generation (RAG) pipeline that uses vLLM for inference, the retrieval step consistently returns an empty list or null despite: Valid user queries arriving at the API endpoint. Document embeddings successfully indexed in the vector store (e.g., Milvus or Pinecone). No errors reported by the vLLM server … Read more

Docker API server refuses connection from data pipeline

by
red and blue cargo ship on sea during daytime

Problem Description The data‑pipeline service, defined in a docker‑compose.yml file, attempts to call the API server at http://api:5000/api during ingestion. The call consistently fails with either a timeout or a connection‑refused error, causing the pipeline to abort. Typical log excerpts: pipeline | 2026-06-12T08:14:32Z ERROR: dial tcp 172.18.0.3:5000: connect: connection refused pipeline | 2026-06-12T08:14:32Z ERROR: Get … Read more

HAProxy traffic imbalance during blue-green deployment

by
a close-up of a server room

Problem – HAProxy traffic imbalance during a blue‑green deployment During a scheduled blue‑green rollout of a new service version, the production HAProxy 2.8 front‑end observed a ~40 % increase in query latency. The spike appeared only after the first batch of green servers was added to the blue_green backend and persisted until the rollout completed. Typical … Read more

ONNX Runtime inference fails after CI/CD deployment due to incorrect routing

by
img IX mining rig inside white and gray room

Problem – ONNX Runtime Inference Fails After CI/CD Deployment During a routine CI/CD rollout, newly built ONNX model containers are deployed to a Kubernetes cluster. After the deployment completes, API calls that should hit the onnx-runtime service return HTTP 502/504 errors and the ONNX Runtime logs show Failed to load model: file not found. The … Read more

PostgreSQL HPA not scaling under high CPU usage

by
img IX mining rig inside white and gray room

Problem – PostgreSQL HPA Not Scaling Under High CPU Usage The PostgreSQL deployment runs in a Kubernetes cluster that is part of an AI‑driven orchestration platform. During traffic spikes the API gateway generates thousands of concurrent requests per second, causing PostgreSQL CPU utilization to rise sharply. Expected behavior is that the Horizontal Pod Autoscaler (HPA) … Read more