Female speaker presenting in front of a projector screen.

Kubernetes StatefulSet update timed out on AWS EC2

by

Problem – StatefulSet Update Times Out on AWS EC2

During a production rollout of a machine‑learning model service, the StatefulSet responsible for serving the model failed to complete its rolling update. The kubectl rollout status command returned:


statefulset.apps/model-service rollout status: waiting for rollout to finish: 1 out of 3 new replicas have been updated...
error: timed out waiting for the condition

Pod events for the newly created replica showed:


Warning  FailedAttachVolume  2m30s  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-1234abcd" : volume is still attaching
Warning  FailedMount         2m15s  kubelet, ip-10-0-1-23  MountVolume.SetUp failed for volume "pvc-1234abcd": timeout waiting for attach operation

The rollout aborted after the controller’s default 5‑minute deadline, leaving the StatefulSet in a partially updated state.

Root Cause Analysis

Interaction Between StatefulSet Controller and Amazon EBS CSI Driver

Kubernetes performs a rolling update of a StatefulSet by creating a new pod, waiting for it to become Ready, then terminating the previous pod. For pods that use PersistentVolumeClaims backed by Amazon EBS, the following sequence is critical:

  1. Old pod is terminated; kubelet initiates DetachVolume via the EBS CSI driver.
  2. New pod is scheduled; kubelet issues AttachVolume and waits for the volume to transition to attached.
  3. The StatefulSet controller watches the new replica set and proceeds only when the pod reaches Ready.

If either the detach or attach operation exceeds the controller’s --progress-deadline (default 5 minutes), the controller logs:


Error updating replica set: timed out waiting for the condition

Evidence from the GitHub issue #101535 and the real incident logs confirms that prolonged EBS attachment latency (often >5 minutes) caused the new pod to stay in ContainerCreating, triggering the timeout.

Contributing Factors

  • EBS attachment limits: AWS EC2 imposes per‑instance limits on concurrent volume attachments (see EBS volume attachment limits). When a node already hosts the maximum number of attached volumes, additional attach requests are queued, adding several minutes of delay.
  • Insufficient terminationGracePeriodSeconds: A mis‑configured grace period of 30 seconds forced the pod to be killed before the EBS volume could detach, leaving the volume in a “detaching” state that blocked the next attach.
  • CSI driver version mismatch: An outdated aws-ebs-csi-driver produced FailedAttachVolume errors (see issue #1234), further extending attach time.
  • Node I/O pressure: High disk I/O caused kubelet to stall during MountVolume.SetUp, keeping the pod in Pending.

Investigation and Debugging Steps

1. Examine StatefulSet rollout status


kubectl rollout status statefulset/model-service -n production

Output shows the timeout message.

2. Inspect pod events and controller logs


kubectl describe pod model-service-2 -n production
kubectl logs -n kube-system -l app=statefulset-controller -c controller

Look for FailedAttachVolume, FailedMount, or timed out waiting for the condition entries.

3. Verify volume attachment state


kubectl get pvc -n production
kubectl get pv
aws ec2 describe-volumes --volume-ids vol-0abcd1234efgh5678 --query 'Volumes[*].Attachments'

Example output indicating a volume stuck in detaching:


{
    "Attachments": [
        {
            "AttachTime": "2024-07-15T12:34:56Z",
            "InstanceId": "i-0a1b2c3d4e5f6g7h8",
            "State": "detaching",
            "VolumeId": "vol-0abcd1234efgh5678"
        }
    ]
}

4. Check node attachment limits


aws ec2 describe-instances --instance-ids i-0a1b2c3d4e5f6g7h8 \
  --query 'Reservations[].Instances[].BlockDeviceMappings[].Ebs.VolumeId' --output text | wc -w

If the count approaches the limit (e.g., 39 volumes on a c5.large), further attaches will be throttled.

5. Review CSI driver version and logs


kubectl get daemonset aws-ebs-csi-driver -n kube-system -o yaml | grep image
kubectl logs -n kube-system -l app=ebs-csi-node -c ebs-plugin

6. Confirm termination grace period


kubectl get statefulset model-service -n production -o yaml | grep terminationGracePeriodSeconds

Resolution – Making the Rollout Successful

Step 1: Increase the StatefulSet progress deadline

Extend the rollout timeout to accommodate longer attach times.


apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: model-service
spec:
  progressDeadlineSeconds: 600   # increased from default 300
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0

Step 2: Adjust terminationGracePeriodSeconds

Provide enough time for EBS detachment.


apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: model-service
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 300   # 5 minutes

Step 3: Upgrade the AWS EBS CSI driver

Deploy the latest stable version (e.g., v1.14.0) to avoid known attach bugs.


kubectl apply -f https://github.com/kubernetes-sigs/aws-ebs-csi-driver/releases/download/v1.14.0/aws-ebs-csi-driver.yaml

Step 4: Ensure nodes stay below attachment limits

Either:

  • Scale the node group to add capacity, or
  • Configure podAntiAffinity so that each StatefulSet pod lands on a distinct node, spreading volume load.

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchLabels:
          app: model-service
      topologyKey: "kubernetes.io/hostname"

Step 5: Apply changes and re‑trigger rollout


kubectl apply -f statefulset-model-service.yaml
kubectl rollout restart statefulset/model-service -n production

Verification – Confirming the Fix

Rollout status


kubectl rollout status statefulset/model-service -n production

Expected output:


statefulset.apps/model-service successfully rolled out

Pod readiness


kubectl get pods -l app=model-service -n production -o wide

All pods should show READY 1/1 and Status: Running.

Volume attachment state


aws ec2 describe-volumes --filters Name=attachment.instance-id,Values=i-0a1b2c3d4e5f6g7h8 \
  --query 'Volumes[*].Attachments[*].State' --output text

All volumes should be in attached state.

Monitoring metrics

Check kubelet_volume_stats_used_bytes and aws_ebs_csi_driver_attach_duration_seconds in Prometheus to verify that attach times are now below the new deadline.

Prevention – Operational Guardrails

  • Set a realistic progressDeadlineSeconds based on observed EBS attach latency (e.g., 10 minutes for large volumes).
  • Configure PodDisruptionBudgets to avoid simultaneous pod terminations that could spike detach/attach traffic (see Kubernetes PDB docs).
  • Monitor volume attachment latency with alerts on aws_ebs_csi_driver_attach_duration_seconds exceeding a threshold (e.g., 2 minutes).
  • Enforce node‑level attachment limits by tagging instances and using a custom scheduler or maxPods settings.
  • Keep the CSI driver up to date and test driver upgrades in a staging environment before production rollout.
  • Increase terminationGracePeriodSeconds for any workload that uses EBS, especially when using large or encrypted volumes.

FAQ

  1. Why does the rollout timeout only on the first pod? The first pod triggers a detach of the existing volume. If the detach takes longer than the grace period, the volume remains in detaching, blocking the subsequent attach.
  2. Can I disable the progress deadline? No. The field is mandatory; you can only increase its value. Setting it too high may mask underlying storage issues.
  3. How do I know if I’m hitting the EC2 attachment limit? Use aws ec2 describe-instances to count attached volumes per node and compare against the limits documented in the AWS EC2 User Guide.
  4. Is increasing terminationGracePeriodSeconds safe for latency‑sensitive services? Yes, because the grace period only affects shutdown, not request processing. Ensure your service can handle a longer shutdown window without impacting client timeouts.
  5. Do I need to restart the StatefulSet after fixing the CSI driver? After upgrading the driver, a rolling restart of the StatefulSet ensures pods use the new driver version and re‑attach volumes correctly.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub

Related Articles