Mistral AI StatefulSet rolling update stuck after PVC lock contention

Problem

The Mistral AI deployment is managed by a StatefulSet that mounts a PersistentVolumeClaim (PVC) for model storage. During a rolling update to a new model version, the update stalls: the old pod never terminates, the new pod remains in Pending, and the Readiness probe repeatedly times out. The kubectl describe sts mistral-ai output shows a RollingUpdate strategy with partition: 0, but no progress is made.

Observed Symptoms

  • Old pod stays in Terminating for >10 minutes.
  • New pod stuck in Pending with event “waiting for volume to be attached”.
  • Controller logs contain error acquiring lock on PVC messages.
  • Readiness probe logs: Readiness probe failed: Get http://localhost:8080/health: dial tcp 127.0.0.1:8080: connect: connection refused.

Root Cause

The Mistral AI container uses a file‑based lock (/data/model.lock) to protect concurrent writes to the model directory. When the StatefulSet updates, the old pod holds the lock while it is shutting down. The new pod, started on the same PVC (because StatefulSet reuses the same volume), attempts to acquire the same lock during its initialization script. The lock acquisition blocks indefinitely, preventing the container from starting and causing the readiness probe to fail. Because the old pod cannot finish termination until the lock is released, the controller cannot delete it, leading to a deadlock.

Contributing factors:

  • Lock file is stored on the PVC rather than an in‑memory or side‑car mechanism.
  • TerminationGracePeriodSeconds is set to 30 seconds, insufficient for the lock release sequence.
  • Readiness probe starts before the lock‑release script finishes, causing immediate failures.

Debug

Step‑by‑step investigation that reproduced the failure:

  1. Inspect the StatefulSet and PVC status:

    kubectl -n production get sts mistral-ai -o yaml

    Key excerpt:

    spec:
      updateStrategy:
        type: RollingUpdate
        rollingUpdate:
          partition: 0
      template:
        spec:
          terminationGracePeriodSeconds: 30
          containers:
          - name: mistral
            image: registry.example.com/mistral:2.4.1
            volumeMounts:
            - name: model-data
              mountPath: /data
          volumes:
          - name: model-data
            persistentVolumeClaim:
              claimName: mistral-model-pvc
    
  2. Check events for the new pod:

    kubectl -n production describe pod mistral-ai-2

    Relevant events:

    Events:
      Type    Reason            Age   From               Message
      ----    ------            ----  ----               -------
      Normal  Scheduled         2m    default-scheduler  Successfully assigned production/mistral-ai-2 to node-12
      Normal  Pulling           2m    kubelet            Pulling image "registry.example.com/mistral:2.4.1"
      Normal  Pulled            2m    kubelet            Successfully pulled image
      Normal  Created           2m    kubelet            Created container mistral
      Warning FailedMount       2m    kubelet            error acquiring lock on PVC: lock file /data/model.lock is held by PID 1234
    
  3. Examine the lock file on the PVC from the node:

    # ssh node-12
    # ls -l /var/lib/kubelet/pods/$(docker inspect -f '{{.Id}}' mistral-ai-2)/volumes/kubernetes.io~csi/pvc-xxxxxx/data/model.lock
    -rw-r--r-- 1 1000 1000 0 Sep 12 10:15 /data/model.lock
    # cat /data/model.lock
    PID=1234
    

    The PID belongs to the terminating container.

  4. Review the container entrypoint script (simplified):

    #!/usr/bin/env bash
    LOCK_FILE=/data/model.lock
    if [ -f "$LOCK_FILE" ]; then
      echo "Lock present, waiting..."
      while kill -0 $(cat $LOCK_FILE) 2>/dev/null; do
        sleep 1
      done
      echo "Lock released"
    fi
    exec mistral-server --model-dir /data/models
    

    The script blocks until the lock file disappears, but the termination signal (SIGTERM) does not delete the file.

Solution

Two complementary changes resolve the deadlock:

1. Move the lock to a side‑car container with a shared emptyDir

By storing the lock in an emptyDir volume, the lock is scoped to the pod lifecycle and disappears when the pod terminates.

Before:

volumeMounts:
- name: model-data
  mountPath: /data
volumes:
- name: model-data
  persistentVolumeClaim:
    claimName: mistral-model-pvc

After:

volumeMounts:
- name: model-data
  mountPath: /data
- name: lock-dir
  mountPath: /lock
volumes:
- name: model-data
  persistentVolumeClaim:
    claimName: mistral-model-pvc
- name: lock-dir
  emptyDir: {}

Update the entrypoint to use /lock/model.lock instead of /data/model.lock. The lock file is now removed automatically when the pod ends.

2. Extend terminationGracePeriodSeconds and make the readiness probe start later

Allow the container enough time to clean up the lock and shut down gracefully.

Before:

terminationGracePeriodSeconds: 30
readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

After:

terminationGracePeriodSeconds: 120
readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10

3. Ensure the lock file is removed on SIGTERM

Add a trap in the entrypoint script:

#!/usr/bin/env bash
LOCK_FILE=/lock/model.lock
cleanup() {
  rm -f "$LOCK_FILE"
}
trap cleanup TERM INT

# Acquire lock
echo $$ > "$LOCK_FILE"
# Rest of startup...
exec mistral-server --model-dir /data/models

4. Apply the updated manifest

kubectl -n production apply -f mistral-statefulset.yaml

Kubernetes will now perform a rolling update without deadlocking.

Verify

Confirm that the update proceeds and the new model version is running:

  1. Watch the rollout:
  2. kubectl -n production rollout status sts/mistral-ai --watch

    Expected output:

    statefulset rolling update complete
    
  3. Check pod phases:
  4. kubectl -n production get pods -l app=mistral-ai -o wide

    All pods should be Running with READY 1/1.

  5. Validate the lock file is absent:
  6. # ssh node-12
    # find /var/lib/kubelet/pods -name model.lock
    # (no output)
    
  7. Run a functional test against the new model endpoint:
  8. curl -s http://mistral-ai.production.svc.cluster.local/v1/predict -d '{"text":"Hello"}' | jq .

    The response should contain the expected inference from the new model version.

Prevent

  • Use pod‑scoped temporary storage for coordination artifacts. Locks, sockets, or PID files that need to disappear with the pod should live in emptyDir or tmpfs, not on a PVC.
  • Graceful shutdown handling. Always trap SIGTERM and clean up resources (files, sockets, DB connections) before exiting.
  • Readiness probe timing. Delay the probe until after any one‑time initialization that may block (e.g., lock acquisition, model download).
  • Set a generous terminationGracePeriodSeconds for stateful workloads. The default 30 s is often too short for I/O‑bound cleanup.
  • Monitor lock contention. Export a custom metric (e.g., mistral_lock_wait_seconds) and alert if the wait time exceeds a threshold.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  1. Why does the pod stay in Terminating even after I delete it? The container holds a file lock on the PVC. Kubernetes sends SIGTERM, but the lock file is not removed, so the pod cannot be garbage‑collected until the lock disappears.
  2. Can I keep the lock on the PVC and still have rolling updates work? Only if you implement a lock‑release protocol that runs during SIGTERM (e.g., a side‑car that watches the main container’s PID) or use a lease object in the API server instead of a file lock.
  3. Is increasing terminationGracePeriodSeconds alone sufficient? No. The lock file still blocks the new pod from starting because it resides on the same PVC. The root issue is the lock’s storage location.
  4. What if I need the lock to survive pod restarts (e.g., for a long‑running model download)? Store the lock in a separate ConfigMap or use a Kubernetes Lease resource, which is not tied to pod lifecycle and can be safely reacquired.
  5. How do I detect similar lock‑contention issues in other stateful workloads? Look for patterns: pods stuck in Terminating, events mentioning “FailedMount” or “error acquiring lock”, and readiness probe failures immediately after a rollout.