Kubernetes StatefulSet rollout error after ONNX Runtime multi-GPU update

Problem

A Kubernetes StatefulSet that runs ONNX Runtime for distributed multi‑GPU training fails during a rollout after upgrading the ONNX Runtime container to a version that adds multi‑GPU support. The controller logs contain:

controller-runtime: failed to update StatefulSet "onnx-trainer": the object has been modified; please try again later

New pods remain in Pending state with the following events:

Warning  FailedCreate  12s (x3)  statefulset-controller  create Pod onnx-trainer-2 failed: pod "onnx-trainer-2" is forbidden: spec: Forbidden: updates to fields other than
"spec.replicas", "spec.template", "spec.updateStrategy", "spec.volumeClaimTemplates" are not allowed

Result: the rollout aborts, the StatefulSet never reaches the desired replica count, and the training job cannot start.

Root Cause

The multi‑GPU update introduced two immutable changes to the StatefulSet spec:

• Volume claim template name change: The new ONNX Runtime image requires a gpu-pvc that is bound to nvidia.com/gpu. The previous manifest used data-pvc. Changing the metadata.name field of volumeClaimTemplates is immutable for a StatefulSet.
• Pod security context: To expose all GPUs the updated manifest added securityContext.runAsUser: 0 and securityContext.capabilities.add: ["IPC_LOCK"]. These fields are part of the pod template, but the controller treats them as immutable because they affect the underlying Pod identity (UID and volume mounts) that StatefulSet guarantees to keep stable.

During a rolling update Kubernetes attempts to apply the new spec.template. Because the change touches immutable fields, the API server rejects the update, and the controller repeatedly retries, eventually logging the “failed to update StatefulSet” error.

Debug

Step‑by‑step investigation that reproduced the failure:

1. Inspect the StatefulSet events

kubectl describe sts onnx-trainer | grep -A5 Events

Output shows the Forbidden error above.

2. Compare the old and new manifests

# Old manifest (v1.2.0)
cat statefulset-old.yaml
# New manifest (v1.3.0)
cat statefulset-new.yaml

3. Check the controller manager logs for retry details

kubectl -n kube-system logs -l component=kube-controller-manager \
  -c kube-controller-manager | grep onnx-trainer

4. Validate immutability rules

kubectl explain statefulset.spec.volumeClaimTemplates
kubectl explain pod.spec.securityContext

Both commands confirm that metadata.name of a volumeClaimTemplate and certain securityContext fields cannot be changed on an existing StatefulSet.

Solution

The fix consists of two parts: (1) keep immutable fields unchanged, and (2) apply the required GPU‑specific changes via a new StatefulSet version or by using the OnDelete update strategy.

Option A – Create a new StatefulSet (recommended for major version bumps)

Rename the StatefulSet and its associated Headless Service. This avoids the immutable‑field conflict entirely.

Before (failing manifest)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: onnx-trainer
spec:
  serviceName: onnx-trainer-svc
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: data-pvc          # <-- immutable name
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 100Gi
  template:
    metadata:
      labels:
        app: onnx-trainer
    spec:
      containers:
      - name: onnx
        image: onnxruntime/multi-gpu:1.3.0
        resources:
          limits:
            nvidia.com/gpu: 2
        # No securityContext

After (working manifest)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: onnx-trainer-v2          # new name
spec:
  serviceName: onnx-trainer-svc
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: gpu-pvc               # new PVC name, allowed because resource is new
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 100Gi
  template:
    metadata:
      labels:
        app: onnx-trainer
    spec:
      securityContext:
        runAsUser: 0
        capabilities:
          add: ["IPC_LOCK"]
      containers:
      - name: onnx
        image: onnxruntime/multi-gpu:1.3.0
        resources:
          limits:
            nvidia.com/gpu: 2

Apply the new manifest and delete the old StatefulSet (the PVCs are retained because they are bound to the new volumeClaimTemplates name):

kubectl delete sts onnx-trainer
kubectl apply -f statefulset-v2.yaml

Option B – Use OnDelete strategy for in‑place migration

If renaming is not feasible, switch the update strategy to OnDelete, manually delete each pod, and let Kubernetes recreate them with the new immutable fields.

# Patch the StatefulSet
kubectl patch sts onnx-trainer -p '{"spec":{"updateStrategy":{"type":"OnDelete"}}}'

# Delete pods one by one
for i in $(seq 0 2); do
  kubectl delete pod onnx-trainer-$i
  # Wait for the replacement pod to become Ready
  kubectl wait --for=condition=Ready pod/onnx-trainer-$i --timeout=180s
done

After all pods are recreated, you can switch the strategy back to RollingUpdate if desired.

Verify

Confirm that the rollout completes and the training job starts:

kubectl get sts onnx-trainer* -o wide
kubectl get pods -l app=onnx-trainer -o wide

Expected output:

NAME               READY   STATUS    RESTARTS   AGE
onnx-trainer-0     1/1     Running   0          3m
onnx-trainer-1     1/1     Running   0          3m
onnx-trainer-2     1/1     Running   0          3m

Check that each pod has access to the required GPUs:

kubectl exec onnx-trainer-0 -- nvidia-smi

Sample nvidia-smi output should list two GPUs per pod.

Prevent

• Immutable‑field audit: Before applying a new manifest, run kubectl diff and compare metadata.name of volumeClaimTemplates and any securityContext fields against the live object.
• Versioned StatefulSet names: Adopt a naming convention that includes a version suffix (e.g., onnx-trainer-v1, onnx-trainer-v2) for breaking changes.
• Separate GPU configuration: Store GPU‑specific flags in a ConfigMap or an init container that can be changed without touching immutable fields.
• Automated rollout validation: Add a post‑deployment job that queries kubectl get pods -l app=onnx-trainer -o jsonpath='{.items[*].status.phase}' and fails the CI/CD pipeline if any pod stays Pending for more than a threshold.
• Documentation lockstep: Keep the ONNX Runtime release notes and the Kubernetes manifest versioned together; any change that adds new PVCs or security contexts must be reflected in a new StatefulSet name.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

1. Why does the rollout succeed when I only change the container image tag?
   Changing the image does not modify immutable fields; Kubernetes treats it as a harmless template update. The failure occurs only when fields that are part of the pod identity (PVC name, securityContext) are altered.
2. Can I use kubectl rollout restart to force the update?
   rollout restart triggers a pod template hash change, but the underlying immutable field conflict still prevents the API server from accepting the new spec, so the command will fail with the same “Forbidden” error.
3. Is it safe to delete the old PVCs after creating the new StatefulSet?
   Only if the data stored in the old PVCs is no longer needed. The new PVCs are independent resources; deleting the old ones will free storage but also permanently remove any persisted training checkpoints.
4. How can I detect immutable‑field violations before a CI/CD pipeline pushes the manifest?
   Add a lint step that runs kubectl get sts <name> -o json | jq -r '.spec.volumeClaimTemplates[].metadata.name' and compares it to the manifest. Any mismatch should abort the pipeline.
5. Will using the OnDelete strategy cause downtime for the training job?
   Yes. Deleting a pod removes its local GPU memory and any in‑flight training step. For long‑running training jobs, prefer the “new StatefulSet name” approach to achieve a zero‑downtime rollout.