VLLM API Server Connection Timeout After Rolling Deployment In Kubernetes

Q: Problem

After a rolling deployment of the vLLM inference service in a Kubernetes cluster, client applications began receiving connection timeout, connection refused, and HTTP 502 Bad Gateway responses when calling the vLLM API endpoint. The issue appears only after the deployment finishes; the service was reachable before the rollout.

Q: Root Cause

The timeout originated from a mismatch between the ClusterIP Service and the pod IPs that were replaced during the rollout. Two independent misconfigurations combined to break connectivity:

Q: Debug 1. Verify Service and Endpoints

kubectl get svc vllm-api -n inference -o yaml kubectl get endpoints vllm-api -n inference -o yaml Typical output before the rollout:

Q: 2. Inspect Pod Logs

kubectl logs -l app=vllm -n inference --tail=50 Sample log snippet showing the model loading delay:

Q: 3. Check Readiness Probe Definition

kubectl get deployment vllm -n inference -o yaml | grep -A5 readinessProbe readinessProbe: httpGet: path: /healthz port: 80 initialDelaySeconds: 5 periodSeconds: 10 failureThreshold: 3 The /healthz endpoint returns 200 as soon as the HTTP server starts, not when the model is ready.

Q: 4. Review NetworkPolicy

kubectl get networkpolicy vllm-allow -n inference -o yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: vllm-allow spec: podSelector: matchLabels: app: vllm policyTypes: - Ingress ingress: - from: - podSelector: {} ports: - protocol: TCP port: 80 The selector matches app=vllm, but the new pods were deployed with an additional label version=v2 without updating the policy, causing them to be excluded.

Q: 5. Capture Network Traffic (optional)

kubectl exec -n ingress $(kubectl get pod -l app=ingress -n ingress -o name | head -n1) -- \ tcpdump -i any -nn port 80 and host 10.12.3.45 -c 5 -vv Result showed SYN packets sent to the new pod IPs with no ACK response, confirming the NetworkPolicy block.

Q: 3. Apply a RollingUpdate strategy that respects readiness

# deployment.yaml snippet strategy: type: RollingUpdate rollingUpdate: maxSurge: 25% maxUnavailable: 0 # do not terminate a pod until a new one is Ready 4. Redeploy kubectl apply -f deployment.yaml -n inference kubectl apply -f networkpolicy.yaml -n inference kubectl rollout status deployment/vllm -n inference Verify 1. Confirm Pods Reach Ready State kubectl get pods -l app=vllm -n inference -o wide All pods should show READY 1/1 and Status: Running after the /ready endpoint succeeds.

Problem

After a rolling deployment of the vLLM inference service in a Kubernetes cluster, client applications began receiving connection timeout, connection refused, and HTTP 502 Bad Gateway responses when calling the vLLM API endpoint. The issue appears only after the deployment finishes; the service was reachable before the rollout.

Root Cause

The timeout originated from a mismatch between the ClusterIP Service and the pod IPs that were replaced during the rollout. Two independent misconfigurations combined to break connectivity:

Readiness probe misconfiguration: The probe returned success before the vLLM process finished loading the model weights. New pods were marked ready while still initializing, causing the Service to route traffic to pods that could not accept connections.
NetworkPolicy allowing only established connections: The policy used port: 80 and protocol: TCP but did not include podSelector: {} for the new pod labels introduced in the rollout. When the old pods terminated, the remaining pods were not covered by the policy, leading to dropped SYN packets and a 502 from the Ingress controller.

Because the Service continued to advertise the old endpoints until the Endpoints object was reconciled, clients experienced intermittent connection failures during the transition window.

Debug

1. Verify Service and Endpoints

kubectl get svc vllm-api -n inference -o yaml

kubectl get endpoints vllm-api -n inference -o yaml

Typical output before the rollout:

apiVersion: v1
kind: Endpoints
metadata:
  name: vllm-api
subsets:
- addresses:
  - ip: 10.12.3.45
  ports:
  - port: 80
    protocol: TCP

During the rollout the addresses list contained IPs of pods that were still initializing, which the readiness probe did not guard against.

2. Inspect Pod Logs

kubectl logs -l app=vllm -n inference --tail=50

Sample log snippet showing the model loading delay:

[2024-10-12 08:15:02] INFO Loading model weights (size=12GB)...
[2024-10-12 08:15:02] INFO Server listening on 0.0.0.0:80
[2024-10-12 08:15:02] INFO Ready to accept requests

The Ready to accept requests line is emitted before the loading completes, which is why the readiness probe succeeded too early.

3. Check Readiness Probe Definition

kubectl get deployment vllm -n inference -o yaml | grep -A5 readinessProbe

readinessProbe:
  httpGet:
    path: /healthz
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3

The /healthz endpoint returns 200 as soon as the HTTP server starts, not when the model is ready.

4. Review NetworkPolicy

kubectl get networkpolicy vllm-allow -n inference -o yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vllm-allow
spec:
  podSelector:
    matchLabels:
      app: vllm
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector: {}
    ports:
    - protocol: TCP
      port: 80

The selector matches app=vllm, but the new pods were deployed with an additional label version=v2 without updating the policy, causing them to be excluded.

5. Capture Network Traffic (optional)

kubectl exec -n ingress $(kubectl get pod -l app=ingress -n ingress -o name | head -n1) -- \
  tcpdump -i any -nn port 80 and host 10.12.3.45 -c 5 -vv

Result showed SYN packets sent to the new pod IPs with no ACK response, confirming the NetworkPolicy block.

Solution

1. Fix Readiness Probe

Modify the probe to query an endpoint that reports model‑load completion. Add a /ready route that returns 200 only after the model is fully loaded.

# before (deployment.yaml)
readinessProbe:
  httpGet:
    path: /healthz
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3

# after (deployment.yaml)
readinessProbe:
  httpGet:
    path: /ready
    port: 80
  initialDelaySeconds: 30   # give model loading time
  periodSeconds: 10
  failureThreshold: 6

2. Update NetworkPolicy to Include New Labels

Either broaden the selector or add a separate policy for the new version.

# before (networkpolicy.yaml)
podSelector:
  matchLabels:
    app: vllm

# after (networkpolicy.yaml)
podSelector:
  matchExpressions:
  - key: app
    operator: In
    values:
    - vllm
  - key: version
    operator: In
    values:
    - v1
    - v2

3. Apply a RollingUpdate strategy that respects readiness

# deployment.yaml snippet
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 0   # do not terminate a pod until a new one is Ready

4. Redeploy

kubectl apply -f deployment.yaml -n inference
kubectl apply -f networkpolicy.yaml -n inference
kubectl rollout status deployment/vllm -n inference

Verify

1. Confirm Pods Reach Ready State

kubectl get pods -l app=vllm -n inference -o wide

All pods should show READY 1/1 and Status: Running after the /ready endpoint succeeds.

2. Check Service Endpoints

kubectl get endpoints vllm-api -n inference -o jsonpath='{.subsets[*].addresses[*].ip}'

All listed IPs must belong to pods that have passed the updated readiness probe.

3. Test API Connectivity

curl -s -o /dev/null -w "%{http_code}" http://vllm-api.inference.svc.cluster.local/v1/models

Expected output: 200

4. Verify No 502 from Ingress

kubectl logs -l app=ingress -n ingress --tail=20 | grep "502"

No lines containing 502 should appear after the rollout.

Prevent

Readiness probes tied to business logic: Expose a health endpoint that reflects actual service readiness (e.g., model load completion, GPU warm‑up).
NetworkPolicy version‑agnostic selectors: Use matchExpressions or label inheritance to avoid accidental exclusion during versioned rollouts.
Zero‑downtime RollingUpdate: Set maxUnavailable: 0 and ensure initialDelaySeconds accommodates cold‑start time.
Automated smoke test in CI/CD: After each rollout, run a lightweight inference request against the Service to verify end‑to‑end connectivity before promoting to production.
Monitoring alerts: Create alerts on http_request_duration_seconds timeouts and on kube_endpoint_changes_total spikes to catch similar regressions early.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why did the Service still route traffic to pods that were not ready? The readiness probe returned success as soon as the HTTP server started, not when the model was fully loaded. Kubernetes therefore considered the pod ready and added its IP to the Service endpoints.
Can I keep the old /healthz endpoint and add a new one? Yes. Keep /healthz for liveness checks, but configure the readiness probe to use /ready (or another endpoint) that reflects true service availability.
Do I need to restart the Ingress controller after fixing the NetworkPolicy? No. Updating the NetworkPolicy is sufficient; the controller watches the policy and immediately applies the new rules.
How can I detect similar label mismatches in future deployments? Include a lint step in your CI pipeline that validates that all pods selected by a NetworkPolicy have matching labels, using tools like kube-score or custom scripts.
What timeout values are safe for vLLM model loading? It depends on model size; for 12 GB models a initialDelaySeconds of 30–45 seconds is typical, but monitor actual load times in staging and adjust accordingly.