Kubernetes StatefulSet Update Fails During Rolling Upgrade TGI Deployment

Q: 1. Inspect StatefulSet status and events

kubectl get sts tgi -o wide kubectl describe sts tgi Typical output excerpt:

Q: 2. Check pod lifecycle and logs

# Pod stuck in Terminating kubectl get pod tgi-0 -o jsonpath='{.status.phase}' kubectl logs tgi-0 -c tgi --tail=50 Sample log snippet:

Q: 3. Verify readiness probe behavior under load

kubectl get pod tgi-1 -o yaml | grep readinessProbe -A5 Typical probe definition from the official TGI quick‑start:

Q: 4. Capture network activity to confirm open connections

# Run on the node hosting the terminating pod sudo ss -tapn | grep 8080 # 8080 = TGI container port # Or a short tcpdump sudo tcpdump -i eth0 -n port 8080 and host <load‑balancer‑IP> -c 10 5. Review Helm chart values that affect rollout # values.yaml excerpt service: updateStrategy: RollingUpdate terminationGracePeriodSeconds: 30 readinessProbe: periodSeconds: 10 failureThreshold: 3 Solution The fix consists of three coordinated changes:

In this article

Table of Contents

Problem Description

A high‑traffic Text Generation Inference (TGI) service is deployed as a StatefulSet on a Kubernetes cluster. During a rolling upgrade the controller stalls after the first pod is updated. Subsequent pods never reach the Running/Ready state, causing a partial service outage and a noticeable drop in request capacity.

Typical symptoms observed in the cluster:

Only the first replica transitions to the new image; the rest remain in Pending or Terminating.
kubectl describe sts tgi shows events such as "FailedCreate pod/... timed out waiting for the condition".
Readiness probe failures: "Readiness probe failed: HTTP request failed with status code 503".
Pod logs contain errors like RuntimeError: CUDA out of memory or SIGTERM received, but process did not exit within terminationGracePeriodSeconds.
Cluster‑wide latency spikes as the load balancer retries against the few remaining healthy replicas.

Root Cause Analysis

The failure stems from the interaction of three StatefulSet mechanisms with TGI’s runtime characteristics:

Ordered pod termination (default podManagementPolicy: OrderedReady) forces the controller to wait for the first pod to become Terminated before creating the second. TGI keeps long‑lived HTTP connections open for in‑flight inference batches. When a SIGTERM is delivered, the container does not close those connections promptly, so the pod stays in Terminating for minutes.
Readiness probe timeouts under load. The default probe configuration (e.g., periodSeconds: 10, failureThreshold: 3) is too aggressive for a model warm‑up that can take >30 s on GPU. The probe repeatedly reports 503, causing the controller to consider the pod “not ready” and abort the rollout.
Insufficient termination grace period. The Helm chart ships with terminationGracePeriodSeconds: 30. TGI needs more time to flush pending inference batches and release CUDA resources. When the grace period expires, the pod is killed (exit code 137) and the StatefulSet controller retries, leading to the “timed out waiting for the condition” error.

These factors combine to create a deadlock: the first pod never finishes termination, preventing the next pod from being created, and the rollout controller eventually times out.

Investigation and Debugging Steps

Below is a reproducible debugging workflow that isolates each contributing factor.

1. Inspect StatefulSet status and events

kubectl get sts tgi -o wide
kubectl describe sts tgi

Typical output excerpt:

Events:
  Type    Reason            Age   From                Message
  ----    ------            ----  ----                -------
  Normal  SuccessfulCreate  2m    statefulset-controller  create Pod tgi-0
  Normal  SuccessfulDelete  1m    statefulset-controller  delete Pod tgi-0
  Warning FailedCreate      30s   statefulset-controller  timed out waiting for the condition

2. Check pod lifecycle and logs

# Pod stuck in Terminating
kubectl get pod tgi-0 -o jsonpath='{.status.phase}'
kubectl logs tgi-0 -c tgi --tail=50

Sample log snippet:

2024-05-28T12:14:02.123Z INFO  Received SIGTERM, beginning graceful shutdown...
2024-05-28T12:14:02.124Z WARN  12 active inference streams still open
2024-05-28T12:14:32.001Z ERROR SIGTERM received, but process did not exit within terminationGracePeriodSeconds

3. Verify readiness probe behavior under load

kubectl get pod tgi-1 -o yaml | grep readinessProbe -A5

Typical probe definition from the official TGI quick‑start:

readinessProbe:
  httpGet:
    path: /health
    port: 80
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

During a rolling upgrade, the /health endpoint returns 503 until the model finishes loading, triggering the failure threshold after ~30 s.

4. Capture network activity to confirm open connections

# Run on the node hosting the terminating pod
sudo ss -tapn | grep 8080   # 8080 = TGI container port
# Or a short tcpdump
sudo tcpdump -i eth0 -n port 8080 and host <load‑balancer‑IP> -c 10

5. Review Helm chart values that affect rollout

# values.yaml excerpt
service:
  updateStrategy: RollingUpdate
terminationGracePeriodSeconds: 30
readinessProbe:
  periodSeconds: 10
  failureThreshold: 3

Solution

The fix consists of three coordinated changes:

1. Switch to parallel pod management

Allow the StatefulSet controller to create the next replica without waiting for the previous one to terminate.

# before (ordered rollout)
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: tgi
spec:
  podManagementPolicy: OrderedReady   # default
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0

# after (parallel rollout)
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: tgi
spec:
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0

2. Extend termination grace period and add a `preStop` hook

The hook explicitly closes HTTP connections and signals the TGI process to stop accepting new requests.

# before
terminationGracePeriodSeconds: 30
# after
terminationGracePeriodSeconds: 120
preStop:
  exec:
    command: ["/bin/sh", "-c", "curl -X POST http://localhost:80/shutdown && sleep 5"]

The /shutdown endpoint is provided by the TGI container (see TGI docs) and forces a clean exit of all inference threads.

3. Tune readiness probe for model warm‑up

Increase the interval and failure threshold so the probe tolerates the longer load time.

# before
readinessProbe:
  periodSeconds: 10
  failureThreshold: 3
# after
readinessProbe:
  periodSeconds: 15
  failureThreshold: 8
  initialDelaySeconds: 60

4. Optional: Deploy a PodDisruptionBudget (PDB)

Guarantee that at least n-1 replicas stay available during the upgrade.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: tgi-pdb
spec:
  minAvailable: 2   # for a 3‑replica StatefulSet
  selector:
    matchLabels:
      app: tgi

Verification

Roll out the new manifest and watch the controller:

kubectl apply -f statefulset.yaml
kubectl rollout status sts/tgi --watch

All pods should transition to Ready without the “timed out waiting for the condition” event.

Confirm graceful shutdown by checking pod termination logs:

kubectl logs tgi-0 -c tgi --tail=20 | grep "graceful shutdown"

Expected line: INFO Graceful shutdown completed, exiting.

Validate health endpoint after the rollout:

curl -s http://tgi-0.default.svc.cluster.local/health
# Should return 200 OK

Monitor request latency on the load balancer (e.g., Prometheus query histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))) to ensure no spike.

Prevention and Best Practices

Use podManagementPolicy: Parallel for inference workloads that keep long‑lived connections.
Set terminationGracePeriodSeconds based on model size and batch flush time (typically 90‑120 s for GPU‑accelerated models).
Implement a preStop hook that calls TGI’s /shutdown endpoint or sends SIGTERM to the server process.
Adjust readiness probes to accommodate the longest expected warm‑up time; keep initialDelaySeconds generous.
Deploy a PodDisruptionBudget to prevent accidental full‑service outages during future upgrades.
Monitor GPU memory pressure (e.g., nvidia-smi metrics) because OOM during warm‑up can cause readiness probe failures.
Enable structured logging (JSON) for the TGI container so that health‑check failures are easily searchable.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why does the rollout hang after the first replica?
Because the default OrderedReady policy waits for the first pod to finish termination. TGI holds open HTTP connections and does not exit within the default 30 s grace period, blocking subsequent pod creation.
Can I keep OrderedReady and still avoid the deadlock?
Only if you guarantee that the pod terminates quickly, e.g., by adding a preStop hook that forces connection closure and by increasing terminationGracePeriodSeconds. Parallel management is simpler for high‑throughput inference services.
What readiness probe values work for a 13 GB GPT‑NeoX model?
A practical baseline is initialDelaySeconds: 60, periodSeconds: 15, and failureThreshold: 8. Adjust upward if model loading exceeds 2 minutes.
How do I verify that the preStop hook actually closed connections?
After issuing a rollout, run kubectl exec tgi-0 -- ss -tapn | grep 8080 during termination. No ESTABLISHED sockets should remain once the hook completes.
Is a PodDisruptionBudget required for StatefulSets?
Not required, but highly recommended for services that must maintain a minimum number of ready replicas during upgrades, especially when using podManagementPolicy: Parallel.