Kafka StatefulSet Rolling Update Stalls After PVC Conflict In Kubernetes

Q: Problem

A Kubernetes StatefulSet that runs a Kafka cluster is stuck during a rolling update. The update was triggered to apply a new container image and a small configuration change. After the first pod (kafka-0) is updated successfully, the next pod (kafka-1) never becomes Ready. The rollout remains in Progressing state indefinitely, causing the API gateway that streams request logs to Kafka to experience back‑pressure and increased latency.

Q: Root Cause

The rolling update stalls because the StatefulSet controller attempts to delete the old pod (kafka-1) before the new pod’s PVC is fully detached from the previous pod. In this deployment the PVCs are defined with volumeClaimTemplates and persistentVolumeReclaimPolicy: Retain. When the old pod is terminated, the underlying PersistentVolume (PV) remains bound to the PVC, but the nodeAffinity of the PV forces it to stay on the original node. The new pod is scheduled on a different node (due to node‑level resource pressure), and the controller tries to mount the same PVC on the new node, which Kubernetes rejects with the conflict error:

Q: 1. Remove node‑affinity from PVs

Update the StorageClass to use allowVolumeExpansion: true and volumeBindingMode: WaitForFirstConsumer. This defers PV provisioning until the pod is scheduled, eliminating static node affinity.

Q: 2. Change StatefulSet update strategy

Set podManagementPolicy: Parallel and updateStrategy.type: RollingUpdate with partition control to force a graceful termination of the old pod before the new one is created.

Q: 3. Perform a controlled rollout

# Apply storage class changes kubectl apply -f storageclass.yaml # Delete existing PVCs (if safe) or migrate them kubectl delete pvc -l app=kafka -n streaming # Apply updated StatefulSet kubectl apply -f statefulset.yaml # Trigger rollout (optional if image tag changed) kubectl rollout restart sts/kafka -n streaming Verify 1. Rollout status kubectl rollout status sts/kafka -n streaming # Expected output statefulset "kafka" successfully rolled out 2. Pod readiness kubectl get pods -l app=kafka -n streaming # Expected NAME READY STATUS RESTARTS AGE kafka-0 1/1 Running 0 5m kafka-1 1/1 Running 0 4m kafka-2 1/1 Running 0 3m 3. Broker logs – no lock errors kubectl logs kafka-1 -n streaming -c kafka | grep "Failed to acquire lock" # No output 4. API gateway throughput # Assuming Prometheus metric kafka_producer_records_per_sec curl -s http://prometheus.example.com/api/v1/query?query=kafka_producer_records_per_sec | jq . # Values should return to baseline (~10k msgs/s) Prevent Use WaitForFirstConsumer binding mode for any storage class that backs StatefulSets. This prevents premature node binding. Set podManagementPolicy: Parallel for Kafka clusters where pods can be safely stopped one at a time. Enable graceful shutdown by configuring terminationGracePeriodSeconds ≥ 120 s and ensuring log.dirs are cleaned up on exit. Monitor PVC attachment events – create an alert on KubePodFailedMount or volume_attach_failed_total metrics. Run a pre‑upgrade dry‑run using kubectl rollout status --watch on a staging namespace to catch affinity conflicts before production. Related Topic Hub: Data Infrastructure Troubleshooting Hub

Problem

A Kubernetes StatefulSet that runs a Kafka cluster is stuck during a rolling update. The update was triggered to apply a new container image and a small configuration change. After the first pod (kafka-0) is updated successfully, the next pod (kafka-1) never becomes Ready. The rollout remains in Progressing state indefinitely, causing the API gateway that streams request logs to Kafka to experience back‑pressure and increased latency.

Observed symptoms

kubectl rollout status reports waiting for pod kafka-1 to become ready
Readiness probe failures for the affected pod
Pod logs show “Failed to acquire lock on /var/lib/kafka/data”
Events contain PersistentVolumeClaim is already bound to a different pod
Message ingestion rate drops by ~80% during the stall

Root Cause

The rolling update stalls because the StatefulSet controller attempts to delete the old pod (kafka-1) before the new pod’s PVC is fully detached from the previous pod. In this deployment the PVCs are defined with volumeClaimTemplates and persistentVolumeReclaimPolicy: Retain. When the old pod is terminated, the underlying PersistentVolume (PV) remains bound to the PVC, but the nodeAffinity of the PV forces it to stay on the original node. The new pod is scheduled on a different node (due to node‑level resource pressure), and the controller tries to mount the same PVC on the new node, which Kubernetes rejects with the conflict error:

Warning  FailedMount  kubelet, node-2  AttachVolume.Attach failed for volume "data-kafka-1" : 
volume is already attached to node "node-1"

Because the readiness probe checks the Kafka listener on port 9092, the probe times out while the broker cannot start without its data directory, leading to the observed readiness failures.

Debug

1. Examine rollout status

kubectl rollout status sts/kafka -n streaming
# Output
statefulset "kafka" rolling update in progress: 1 out of 3 new pods have been updated...
Waiting for pod kafka-1 to become ready

2. Inspect pod events and describe PVC

kubectl describe pod kafka-1 -n streaming
# Relevant excerpt
Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Pulling         2m    kubelet, node-2    Pulling image "confluentinc/cp-kafka:7.2.1"
  Warning FailedMount     1m30s kubelet, node-2    AttachVolume.Attach failed for volume "data-kafka-1": volume is already attached to node "node-1"

kubectl get pvc data-kafka-1 -n streaming -o yaml
# Excerpt
spec:
  volumeName: pv-kafka-1
  storageClassName: fast-ssd
  accessModes:
  - ReadWriteOnce
status:
  phase: Bound
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 500Gi
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node-1

3. Verify PV node affinity

kubectl get pv pv-kafka-1 -o yaml | grep -A5 nodeAffinity
# Output
nodeAffinity:
  required:
    nodeSelectorTerms:
    - matchExpressions:
      - key: kubernetes.io/hostname
        operator: In
        values:
        - node-1

4. Check readiness probe definition

kubectl get sts kafka -n streaming -o yaml | grep -A5 readinessProbe
# Output
readinessProbe:
  exec:
    command:
    - /bin/bash
    - -c
    - |
      kafka-broker-api-versions.sh --bootstrap-server localhost:9092
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 6

5. Capture broker logs

kubectl logs kafka-1 -n streaming -c kafka
# Sample log
[2024-10-12 14:03:21,123] WARN [LogManager] Failed to acquire lock on /var/lib/kafka/data (java.io.IOException: Device or resource busy)
[2024-10-12 14:03:21,124] INFO  [LogManager] Shutting down LogManager

Solution

The fix consists of two parts: (1) make PVCs node‑agnostic during rolling updates, and (2) adjust the StatefulSet update strategy to ensure the old pod is fully terminated before the new pod starts.

1. Remove node‑affinity from PVs

Update the StorageClass to use allowVolumeExpansion: true and volumeBindingMode: WaitForFirstConsumer. This defers PV provisioning until the pod is scheduled, eliminating static node affinity.

# Before (storageclass.yaml)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
reclaimPolicy: Retain
volumeBindingMode: Immediate

# After (storageclass.yaml)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Apply the updated class and recreate the PVCs (or migrate them using kubectl patch if data must be preserved).

2. Change StatefulSet update strategy

Set podManagementPolicy: Parallel and updateStrategy.type: RollingUpdate with partition control to force a graceful termination of the old pod before the new one is created.

# Before (statefulset.yaml)
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  serviceName: kafka
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
      - name: kafka
        image: confluentinc/cp-kafka:7.2.1
        readinessProbe: ...
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      storageClassName: fast-ssd
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 500Gi

# After (statefulset.yaml)
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  serviceName: kafka
  replicas: 3
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      terminationGracePeriodSeconds: 120
      containers:
      - name: kafka
        image: confluentinc/cp-kafka:7.3.0   # new version
        readinessProbe: ...
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      storageClassName: fast-ssd
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 500Gi

Key changes:

podManagementPolicy: Parallel – allows the controller to delete the old pod before creating the replacement, avoiding simultaneous mounts.
terminationGracePeriodSeconds – gives Kafka enough time to shut down cleanly and release the data directory lock.
Upgrade the image in the same manifest to keep the rollout atomic.

3. Perform a controlled rollout

# Apply storage class changes
kubectl apply -f storageclass.yaml

# Delete existing PVCs (if safe) or migrate them
kubectl delete pvc -l app=kafka -n streaming

# Apply updated StatefulSet
kubectl apply -f statefulset.yaml

# Trigger rollout (optional if image tag changed)
kubectl rollout restart sts/kafka -n streaming

Verify

1. Rollout status

kubectl rollout status sts/kafka -n streaming
# Expected output
statefulset "kafka" successfully rolled out

2. Pod readiness

kubectl get pods -l app=kafka -n streaming
# Expected
NAME       READY   STATUS    RESTARTS   AGE
kafka-0    1/1     Running   0          5m
kafka-1    1/1     Running   0          4m
kafka-2    1/1     Running   0          3m

3. Broker logs – no lock errors

kubectl logs kafka-1 -n streaming -c kafka | grep "Failed to acquire lock"
# No output

4. API gateway throughput

# Assuming Prometheus metric kafka_producer_records_per_sec
curl -s http://prometheus.example.com/api/v1/query?query=kafka_producer_records_per_sec | jq .
# Values should return to baseline (~10k msgs/s)

Prevent

Use WaitForFirstConsumer binding mode for any storage class that backs StatefulSets. This prevents premature node binding.
Set podManagementPolicy: Parallel for Kafka clusters where pods can be safely stopped one at a time.
Enable graceful shutdown by configuring terminationGracePeriodSeconds ≥ 120 s and ensuring log.dirs are cleaned up on exit.
Monitor PVC attachment events – create an alert on KubePodFailedMount or volume_attach_failed_total metrics.
Run a pre‑upgrade dry‑run using kubectl rollout status --watch on a staging namespace to catch affinity conflicts before production.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does the rollout succeed on the first pod but fail on the second?
The first pod (kafka-0) is scheduled on the node that already hosts its PV, so the mount succeeds. The second pod is scheduled on a different node due to resource pressure, causing the PV’s node affinity to clash.
Can I keep reclaimPolicy: Retain and still avoid the conflict?
Yes, but you must delete the old pod and explicitly detach the volume (e.g., kubectl patch pv … -p '{"spec":{"nodeAffinity":null}}') before the new pod starts. Using WaitForFirstConsumer is simpler and safer.
Is podManagementPolicy: Parallel safe for Kafka?
Kafka tolerates a single broker down as long as the replication factor is ≥ 2. Parallel policy only changes the order of pod termination; it does not start multiple new brokers simultaneously.
What if I cannot change the storage class because other workloads depend on it?
Create a dedicated storage class for Kafka with the desired binding mode, and reference it in the volumeClaimTemplates. Existing PVCs can be migrated using kubectl cp or snapshot/restore mechanisms.
How can I detect similar PVC‑node affinity issues early?
Add a Prometheus rule that fires on kubelet_volume_attached{status!="true"} or on the KubePodFailedMount alert. Include the pod name and PVC in the alert payload for quick triage.