Engineering Logs for Troubleshooting & Debugging
Problem: NodePort conflicts during rolling updates During a rolling update of a Deployment that exposes a Service of type NodePort, new Pods are scheduled while the old Pods are still running. If the Service definition specifies a static nodePort (or relies on the default allocation) and the update creates a second Service object (e.g., during … Read more
Problem – Latency Spike in Kubernetes‑Hosted AI Inference Service During peak traffic windows the inference endpoint that serves ~100 ms predictions suddenly starts responding in 1 s +. The spike is repeatable, lasts for the duration of the load burst, and then returns to baseline once traffic subsides. Key observations: CPU and memory usage on GPU‑accelerated pods stay … Read more
Problem – Validation Errors After Updating a CRD Schema After a recent schema change to a shared Custom Resource Definition (CRD) used by multiple AI/ML services, kubectl apply and CI/CD pipelines began failing with errors such as: error: admission webhook “v1.crdvalidation.k8s.io” denied the request: spec: Required value Other observed messages include: validation failed: spec.replicas: Invalid … Read more