networking

Kubernetes NetworkPolicy blocking API gateway inbound traffic after policy update

by
a group of blue squares

Problem The API gateway service (gateway-prod) suddenly stopped receiving inbound HTTP requests after a routine NetworkPolicy update. The symptom manifested as 5xx errors from downstream services and connection timeouts observed by clients. Observed logs 2026-10-05T14:32:10.123Z kubelet[1234]: E0905 14:32:10.123456 1234 network_plugin.go:215] Failed to program iptables rules: iptables-restore v1.8.4: line 23: Invalid argument 2026-10-05T14:32:10.124Z kube-proxy[5678]: I0905 14:32:10.124567 … Read more

GPT-3.5 webhook timeout after moving to air‑gapped network

by
photo of computer cables

GPT‑3.5 Webhook Timeout After Moving to an Air‑Gapped Network Problem After migrating an OpenAI GPT‑3.5 integration to an air‑gapped data center, the application began receiving webhook timeout errors when the model attempted to send its response back to the caller. The failure manifests as HTTP 504 responses from the webhook endpoint and corresponding error entries in … Read more

Kubernetes NetworkPolicy blocking model-serving pods from Redis cache

by
a group of blue boxes

Problem – Model‑Serving Pods Cannot Reach Redis Cache In a production Kubernetes cluster the AI inference workloads run in the model-serving namespace. The Redis cache that stores model artifacts lives in the data-services namespace and is exposed via the Service redis-data-services (port 6379). After a recent deployment of a default‑deny NetworkPolicy in model-serving, inference requests … Read more

Kafka video/LiDAR timestamp drift after intermittent network on edge node

by
a group of people walking down a street next to tall buildings

Problem – Timestamp Drift Between Video and LiDAR Streams on an Edge Kafka Node In a low‑power edge deployment a local Kafka broker aggregates two high‑rate topics: video‑frames – 30 fps H.264 payloads, timestamps generated by the camera firmware. lidar‑points – 10 Hz point‑cloud packets, timestamps generated by the LiDAR driver. During intermittent Wi‑Fi outages the following … Read more

Google Gemini logprob NaN after network latency spike in hybrid cloud

by
Google logo neon light signage

Problem Description During inference with the Google Gemini model in a hybrid‑cloud deployment, the logprob field in the response payload intermittently contains NaN or null values. The symptom manifests as: Inconsistent token scoring – generated text appears correct, but downstream ranking logic that relies on log probabilities fails. API errors such as HTTP 400 with … Read more

Canary Elasticsearch pods timing out after Calico network policy change

by
photo of computer cables

Problem – Canary Elasticsearch Pods Time Out After Calico NetworkPolicy Change During a canary rollout of a new Elasticsearch version, pods labeled app=es-canary were unable to reach the primary elasticsearch-master service. All HTTP requests to port 9200 and transport requests to port 9300 resulted in connection‑timeout errors, causing indexing and query failures and halting the … Read more

LlamaIndex training loop failed to fetch data due to network policy

by
photo of computer cables

Problem Description During a nightly training run, the LlamaIndex training loop crashes when it attempts to load external documents or call remote APIs. The pod logs contain errors such as: urllib3.exceptions.MaxRetryError: Failed to establish a new connection: [Errno 111] Connection refused requests.exceptions.ConnectionError: HTTPConnectionPool(host=’example.com’, port=80): Max retries exceeded with url: /data.json (Caused by NewConnectionError) EgressPolicyDenied: traffic … Read more

Qwen local development blocked by network policy

by
yellow and white square box

Problem – Qwen Local Development Blocked by Network Policy During local development of Qwen‑based applications, the SDK fails to download model weights or reach the Qwen API. Typical symptoms observed on a developer workstation behind a corporate firewall include: ConnectionError: Failed to establish a new connection: [Errno 111] Connection refused HTTPError 403: Forbidden – https://huggingface.co/…/resolve/main/pytorch_model.bin … Read more

PostgreSQL OAuth2 token refresh fails due to network latency

by
photo of computer cables

Problem – OAuth2 Token Refresh Fails on Edge Nodes In a remote industrial monitoring deployment an edge‑computing node runs PostgreSQL 15 with OAuth2‑based client authentication. When the network latency to the central authentication service exceeds a few hundred milliseconds, connections are abruptly terminated and the following errors appear in the PostgreSQL log: 2024-05-12 10:23:45.123 UTC … Read more