StatefulSet Update Fails With Incompatible Node Configs In Kubernetes

Q: Problem – StatefulSet update fails with incompatible node configurations in a vLLM deployment

A hybrid‑cloud environment runs a vllm StatefulSet that stores model files on Persistent Volume Claims (PVCs) and is front‑ended by an Ingress controller. After introducing a new node pool (AWS EKS) and applying a Helm upgrade that adds a nodeSelector for GPU accelerators, the rollout stalls. Pods remain in Pending or NotReady and the following errors appear in the controller logs:

Q: 1. Inspect the StatefulSet definition

kubectl get statefulset vllm -n vllm -o yaml Key sections to examine:

Q: 2. Verify node labels and taints

kubectl get nodes -L accelerator -o wide Typical output:

Q: 3. Check for the “not‑compatible” taint

kubectl describe node ip-10-0-4-78 | grep Taint Example output:

Q: 4. Review PVC events

kubectl describe pvc model-pvc-0 -n vllm Relevant event excerpt:

Q: 5. Correlate readiness probe failures

kubectl logs vllm-0 -n vllm --previous | tail -n 20 Sample log tail:

Q: 6. Validate Helm values diff

helm diff upgrade vllm ./charts/vllm \ --values values-prod.yaml \ --values values-aws.yaml \ --namespace vllm Look for changes to nodeSelector and storageClassName.

In this article

Table of Contents

Problem – StatefulSet update fails with incompatible node configurations in a vLLM deployment

A hybrid‑cloud environment runs a vllm StatefulSet that stores model files on Persistent Volume Claims (PVCs) and is front‑ended by an Ingress controller. After introducing a new node pool (AWS EKS) and applying a Helm upgrade that adds a nodeSelector for GPU accelerators, the rollout stalls. Pods remain in Pending or NotReady and the following errors appear in the controller logs:

Error: statefulset.apps “vllm” is invalid: spec.template.spec.nodeSelector: Invalid value: map[string]string{“accelerator”:”nvidia”}: node selector conflict with existing pods

Readiness probe failed: Get http://10.0.0.12:8080/healthz: dial tcp 10.0.0.12:8080: connect: connection refused

FailedMount: mounting volume “model-pvc” failed: permission denied while mounting “fs-12345678.efs.us-east-1.amazonaws.com:/”

The StatefulSet never reaches the desired replica count, causing downstream request failures and a breach of the Service Level Objective (SLO) for latency.

Root Cause Analysis

Three intertwined factors cause the update deadlock:

Node selector conflict – The Helm chart added nodeSelector: {"accelerator":"nvidia"} to the pod template (see vLLM Helm chart documentation). Existing pods were already scheduled on on‑prem nodes that lack the accelerator=nvidia label. Kubernetes rejects the update because a StatefulSet cannot change the node affinity of already‑running pods without violating the StatefulSet immutability rules.
Kernel / container runtime ABI mismatch – On‑premises nodes run Linux kernel 4.15 while the new AWS nodes run 5.10. The vLLM container image expects the newer containerd ABI; on older kernels the runtime applies the node.kubernetes.io/not-compatible taint, causing new pods to stay Pending (see the “Hybrid‑cloud incident” evidence).
PVC class divergence – The PVCs were originally provisioned with the aws-efs storage class. When the new node pool was added, a different storage class (efs-gp2) was referenced in the Helm values, leading to FailedMount errors because the underlying EFS file system permissions differ between clusters (see “Production outage” evidence).

Because the StatefulSet controller enforces ordered pod updates, the first pod that cannot be scheduled blocks the entire rollout, producing the “node selector conflict” error and the “Readiness probe constantly failing” symptom.

Investigation and Debugging

Follow these steps to reproduce the failure and isolate each component.

1. Inspect the StatefulSet definition

kubectl get statefulset vllm -n vllm -o yaml

Key sections to examine:

spec:
  serviceName: vllm-headless
  replicas: 3
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      nodeSelector:
        accelerator: nvidia   # <-- newly added
      containers:
      - name: vllm
        image: vllmproject/vllm:latest
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
        volumeMounts:
        - name: model-pvc
          mountPath: /models
  volumeClaimTemplates:
  - metadata:
      name: model-pvc
    spec:
      storageClassName: efs-gp2   # <-- changed from aws-efs
      accessModes: ["ReadWriteMany"]
      resources:
        requests:
          storage: 500Gi

2. Verify node labels and taints

kubectl get nodes -L accelerator -o wide

Typical output:

NAME                STATUS   ROLES    AGE   VERSION   LABELS
ip-10-0-1-12        Ready       45d   v1.23.6   accelerator=nvidia
ip-10-0-2-34        Ready       45d   v1.23.6   <none>
ip-10-0-3-56        Ready       45d   v1.23.6   accelerator=nvidia
ip-10-0-4-78        Ready       12d   v1.24.2   <none>

Notice that the on‑prem nodes lack the accelerator=nvidia label, confirming the selector conflict.

3. Check for the “not‑compatible” taint

kubectl describe node ip-10-0-4-78 | grep Taint

Example output:

Taints: node.kubernetes.io/not-compatible:NoSchedule

4. Review PVC events

kubectl describe pvc model-pvc-0 -n vllm

Relevant event excerpt:

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 3m kubelet, ip-10-0-4-78 mounting volume "model-pvc" failed: permission denied while mounting "fs-12345678.efs.us-east-1.amazonaws.com:/"

5. Correlate readiness probe failures

kubectl logs vllm-0 -n vllm --previous | tail -n 20

Sample log tail:

2024-06-28T12:34:56.789Z INFO Starting vLLM server on port 8080

2024-06-28T12:34:57.001Z ERROR Health check failed: model directory not mounted

6. Validate Helm values diff

helm diff upgrade vllm ./charts/vllm \
  --values values-prod.yaml \
  --values values-aws.yaml \
  --namespace vllm

Look for changes to nodeSelector and storageClassName.

Resolution – Align node selectors, kernel compatibility, and PVC storage classes

1. Make node selector compatible with existing pods

Option A – Add the missing label to on‑prem nodes (if they have GPUs):

kubectl label node ip-10-0-2-34 accelerator=nvidia --overwrite
kubectl label node ip-10-0-4-78 accelerator=nvidia --overwrite

Option B – Remove the selector from the StatefulSet if GPU is not required on all nodes. Edit the Helm values:

# values-prod.yaml
nodeSelector: {}

Then perform a rolling upgrade:

helm upgrade vllm ./charts/vllm \
  -f values-prod.yaml \
  -n vllm

2. Resolve kernel / runtime mismatch

Upgrade the on‑prem kernel to at least 5.4 or migrate the workload to a node pool that matches the AWS kernel version. Example using a custom node image:

# Example: create a new on‑prem node pool with updated kernel
kops create instancegroup --name=cluster.example.com \
  --role=Node --subnet=us-west-2a \
  --node-labels="accelerator=nvidia" \
  --node-taints="" \
  --image=ubuntu-20.04-5.10

After the new nodes are ready, cordon and drain the old nodes:

kubectl cordon ip-10-0-2-34
kubectl drain ip-10-0-2-34 --ignore-daemonsets --delete-emptydir-data

3. Reconcile PVC storage class

Standardize on a single storage class that works across clouds, e.g., aws-efs with the same provisioner parameters.

Before (incorrect):

storageClassName: efs-gp2

After (correct):

storageClassName: aws-efs

Apply the change by updating the volumeClaimTemplates in the StatefulSet YAML and performing a kubectl rollout restart (requires PVC recreation; see ReclaimPolicy for data migration strategy).

4. Force a clean rollout after the fixes

kubectl rollout restart statefulset vllm -n vllm
kubectl rollout status statefulset vllm -n vllm --watch

The controller should now create pods on nodes that satisfy the selector, mount the PVC successfully, and pass the readiness probe.

Verification – Confirm that the StatefulSet is healthy

Pod phase check:

kubectl get pods -l app=vllm -n vllm -o wide

Expected output:

NAME      READY   STATUS    RESTARTS   AGE   IP           NODE
vllm-0    1/1     Running   0          5m    10.0.1.12    ip-10-0-1-12
vllm-1    1/1     Running   0          5m    10.0.2.34    ip-10-0-2-34
vllm-2    1/1     Running   0          5m    10.0.3.56    ip-10-0-3-56

Readiness probe validation:

curl -s http://vllm-0.vllm-headless.vllm.svc.cluster.local:8080/healthz

Should return OK or a JSON payload indicating model load success.

Ingress health check (if using NGINX Ingress):

curl -s http://my-vllm.example.com/healthz

Should also return OK.

Metrics sanity (Prometheus scrape):

curl -s http://vllm-0:9090/metrics | grep vllm_up

Value should be 1 for each replica.

Prevention – Operational guardrails for future updates

Area	Guardrail	Implementation
Node selector drift	Enforce consistent labels across all node pools	Use a GitOps policy that validates `nodeSelector` against a central label inventory (OPA/Kubernetes Admission Controller)
Kernel/runtime compatibility	Standardize node OS version across hybrid clouds	Maintain a base AMI/OS image version in the CI pipeline; run `kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kernelVersion}'` in a nightly job
PVC storage class consistency	Single source of truth for `storageClassName`	Define a Helm value `global.storageClass` and reference it in all charts; add a Helm test that checks the generated YAML
Readiness probe reliability	Make health endpoint independent of storage mount timing	Implement a two‑stage probe: first check container liveness, then verify model load via a separate endpoint (e.g., `/model_ready`)
Rollout safety	PodDisruptionBudget with appropriate `minAvailable`	Set `minAvailable: 2` for a 3‑replica StatefulSet and monitor `pdb` violations during upgrades

FAQ – Common follow‑up questions

Why does the StatefulSet update succeed on AWS nodes but fail on on‑prem nodes?
Because the AWS node pool already carries the accelerator=nvidia label and runs a compatible kernel. The on‑prem nodes lack the label and have an older kernel that triggers the node.kubernetes.io/not-compatible taint, preventing scheduling.
Can I change the node selector of an existing StatefulSet without recreating pods?
No. The StatefulSet controller treats spec.template.spec.nodeSelector as immutable for existing pods. You must either add matching labels to the existing nodes or delete and recreate the StatefulSet (or use kubectl rollout restart after fixing the selector).
How do I verify which storage class a PVC is bound to after a Helm upgrade?
Run kubectl get pvc -n vllm -o jsonpath='{.items[*].spec.storageClassName}'. The output should match the value defined in your Helm values.yaml. A mismatch indicates a drift that can cause FailedMount errors.
What is the best way to test readiness probes after a model file resize?
Perform a manual curl against the pod’s /healthz endpoint before the rollout, and use kubectl exec to list the model directory inside the container. Ensure the probe includes a retry‑backoff that tolerates the time needed for the PVC resize to propagate.
Is there a way to automatically reconcile node labels when a new node pool is added?
Yes. Deploy an Admission Controller (e.g., OPA Gatekeeper) that rejects node creation lacking required labels, or use a DaemonSet that applies the missing labels on node join events.

Related Topic Hub: Model Serving Troubleshooting Hub