Problem
The Mistral AI deployment is managed by a StatefulSet that mounts a PersistentVolumeClaim (PVC) for model storage. During a rolling update to a new model version, the update stalls: the old pod never terminates, the new pod remains in Pending, and the Readiness probe repeatedly times out. The kubectl describe sts mistral-ai output shows a RollingUpdate strategy with partition: 0, but no progress is made.
Observed Symptoms
- Old pod stays in
Terminatingfor >10 minutes. - New pod stuck in
Pendingwith event “waiting for volume to be attached”. - Controller logs contain
error acquiring lock on PVCmessages. - Readiness probe logs:
Readiness probe failed: Get http://localhost:8080/health: dial tcp 127.0.0.1:8080: connect: connection refused.
Root Cause
The Mistral AI container uses a file‑based lock (/data/model.lock) to protect concurrent writes to the model directory. When the StatefulSet updates, the old pod holds the lock while it is shutting down. The new pod, started on the same PVC (because StatefulSet reuses the same volume), attempts to acquire the same lock during its initialization script. The lock acquisition blocks indefinitely, preventing the container from starting and causing the readiness probe to fail. Because the old pod cannot finish termination until the lock is released, the controller cannot delete it, leading to a deadlock.
Contributing factors:
- Lock file is stored on the PVC rather than an in‑memory or side‑car mechanism.
- TerminationGracePeriodSeconds is set to
30seconds, insufficient for the lock release sequence. - Readiness probe starts before the lock‑release script finishes, causing immediate failures.
Debug
Step‑by‑step investigation that reproduced the failure:
-
Inspect the StatefulSet and PVC status:
kubectl -n production get sts mistral-ai -o yamlKey excerpt:
spec: updateStrategy: type: RollingUpdate rollingUpdate: partition: 0 template: spec: terminationGracePeriodSeconds: 30 containers: - name: mistral image: registry.example.com/mistral:2.4.1 volumeMounts: - name: model-data mountPath: /data volumes: - name: model-data persistentVolumeClaim: claimName: mistral-model-pvc -
Check events for the new pod:
kubectl -n production describe pod mistral-ai-2Relevant events:
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m default-scheduler Successfully assigned production/mistral-ai-2 to node-12 Normal Pulling 2m kubelet Pulling image "registry.example.com/mistral:2.4.1" Normal Pulled 2m kubelet Successfully pulled image Normal Created 2m kubelet Created container mistral Warning FailedMount 2m kubelet error acquiring lock on PVC: lock file /data/model.lock is held by PID 1234 -
Examine the lock file on the PVC from the node:
# ssh node-12 # ls -l /var/lib/kubelet/pods/$(docker inspect -f '{{.Id}}' mistral-ai-2)/volumes/kubernetes.io~csi/pvc-xxxxxx/data/model.lock -rw-r--r-- 1 1000 1000 0 Sep 12 10:15 /data/model.lock # cat /data/model.lock PID=1234The PID belongs to the terminating container.
-
Review the container entrypoint script (simplified):
#!/usr/bin/env bash LOCK_FILE=/data/model.lock if [ -f "$LOCK_FILE" ]; then echo "Lock present, waiting..." while kill -0 $(cat $LOCK_FILE) 2>/dev/null; do sleep 1 done echo "Lock released" fi exec mistral-server --model-dir /data/modelsThe script blocks until the lock file disappears, but the termination signal (SIGTERM) does not delete the file.
Solution
Two complementary changes resolve the deadlock:
1. Move the lock to a side‑car container with a shared emptyDir
By storing the lock in an emptyDir volume, the lock is scoped to the pod lifecycle and disappears when the pod terminates.
Before:
volumeMounts:
- name: model-data
mountPath: /data
volumes:
- name: model-data
persistentVolumeClaim:
claimName: mistral-model-pvc
After:
volumeMounts:
- name: model-data
mountPath: /data
- name: lock-dir
mountPath: /lock
volumes:
- name: model-data
persistentVolumeClaim:
claimName: mistral-model-pvc
- name: lock-dir
emptyDir: {}
Update the entrypoint to use /lock/model.lock instead of /data/model.lock. The lock file is now removed automatically when the pod ends.
2. Extend terminationGracePeriodSeconds and make the readiness probe start later
Allow the container enough time to clean up the lock and shut down gracefully.
Before:
terminationGracePeriodSeconds: 30
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
After:
terminationGracePeriodSeconds: 120
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
3. Ensure the lock file is removed on SIGTERM
Add a trap in the entrypoint script:
#!/usr/bin/env bash
LOCK_FILE=/lock/model.lock
cleanup() {
rm -f "$LOCK_FILE"
}
trap cleanup TERM INT
# Acquire lock
echo $$ > "$LOCK_FILE"
# Rest of startup...
exec mistral-server --model-dir /data/models
4. Apply the updated manifest
kubectl -n production apply -f mistral-statefulset.yaml
Kubernetes will now perform a rolling update without deadlocking.
Verify
Confirm that the update proceeds and the new model version is running:
- Watch the rollout:
- Check pod phases:
- Validate the lock file is absent:
- Run a functional test against the new model endpoint:
kubectl -n production rollout status sts/mistral-ai --watch
Expected output:
statefulset rolling update complete
kubectl -n production get pods -l app=mistral-ai -o wide
All pods should be Running with READY 1/1.
# ssh node-12
# find /var/lib/kubelet/pods -name model.lock
# (no output)
curl -s http://mistral-ai.production.svc.cluster.local/v1/predict -d '{"text":"Hello"}' | jq .
The response should contain the expected inference from the new model version.
Prevent
- Use pod‑scoped temporary storage for coordination artifacts. Locks, sockets, or PID files that need to disappear with the pod should live in
emptyDirortmpfs, not on a PVC. - Graceful shutdown handling. Always trap
SIGTERMand clean up resources (files, sockets, DB connections) before exiting. - Readiness probe timing. Delay the probe until after any one‑time initialization that may block (e.g., lock acquisition, model download).
- Set a generous
terminationGracePeriodSecondsfor stateful workloads. The default 30 s is often too short for I/O‑bound cleanup. - Monitor lock contention. Export a custom metric (e.g.,
mistral_lock_wait_seconds) and alert if the wait time exceeds a threshold.
Related Topic Hub: LLM Systems Troubleshooting Hub
FAQ
- Why does the pod stay in
Terminatingeven after I delete it? The container holds a file lock on the PVC. Kubernetes sends SIGTERM, but the lock file is not removed, so the pod cannot be garbage‑collected until the lock disappears. - Can I keep the lock on the PVC and still have rolling updates work? Only if you implement a lock‑release protocol that runs during SIGTERM (e.g., a side‑car that watches the main container’s PID) or use a lease object in the API server instead of a file lock.
- Is increasing
terminationGracePeriodSecondsalone sufficient? No. The lock file still blocks the new pod from starting because it resides on the same PVC. The root issue is the lock’s storage location. - What if I need the lock to survive pod restarts (e.g., for a long‑running model download)? Store the lock in a separate
ConfigMapor use a KubernetesLeaseresource, which is not tied to pod lifecycle and can be safely reacquired. - How do I detect similar lock‑contention issues in other stateful workloads? Look for patterns: pods stuck in
Terminating, events mentioning “FailedMount” or “error acquiring lock”, and readiness probe failures immediately after a rollout.