LlamaIndex StatefulSet Update Fails With Pod Termination Delay

Q: Problem Description

A production LlamaIndex service deployed as a StatefulSet began failing during rolling updates. Pods entered the Terminating phase for more than ten minutes, PVC unmounts stalled, and the vector‑store index became corrupted. The symptoms manifested as:

Q: Investigation and Debugging Steps 1. Observe pod termination state

kubectl get pods -n prod -l app=llama-index -w NAME READY STATUS RESTARTS AGE llama-index-0 1/1 Running 0 12d llama-index-1 1/1 Terminating 0 12d Notice the Terminating pod never transitions to Completed.

Q: 2. Inspect events and kubelet logs

kubectl describe pod llama-index-1 -n prod Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Killing 2m kubelet Killing container with id docker://llama-index:Need to kill pod Warning FailedDeletePod 1m kubelet Error syncing pod, failed to delete pod: context deadline exceeded 3. Check LlamaIndex container logs for lock activity kubectl logs llama-index-1 -n prod 2025-02-10T14:22:31.112Z INFO LlamaIndex: Starting ingestion workers 2025-02-10T14:22:31.115Z INFO LlamaIndex: Acquired lock on /data/index.lock 2025-02-10T14:25:00.001Z WARN LlamaIndex: Shutdown signal received, waiting for workers... 2025-02-10T14:25:30.000Z ERROR LlamaIndex: Timeout waiting for workers to finish 2025-02-10T14:25:30.001Z INFO LlamaIndex: Exiting without releasing lock 4. Verify PVC attachment state kubectl get pvc -n prod -l app=llama-index NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE llama-index-data Bound pvc-1234abcd-5678-efgh-ijkl-9mnopqrstuv 50Gi RWO fast-ssd 12d If the volume remains Bound to the terminating pod, the next pod cannot mount it.

Q: 5. Capture a short packet trace (optional)

kubectl exec -it llama-index-0 -n prod -- tcpdump -i any -nn port 8080 -c 10 6. Correlate with known incidents Fintech Q3 2024 incident – pod stuck >10 min, index corruption. Media streaming Mar 2025 – ordered rollout caused volume attachment timeout. Resolution The fix consists of three coordinated changes:

Q: 2. Increase terminationGracePeriodSeconds

Set to 120 seconds to allow ingestion workers to finish and the lock file to be removed.

Q: 5. Apply the updated manifest

kubectl apply -f llama-index-statefulset.yaml kubectl rollout status statefulset/llama-index -n prod Verification Rollout completes without stuck pods kubectl get pods -n prod -l app=llama-index NAME READY STATUS RESTARTS AGE llama-index-0 1/1 Running 0 12d llama-index-1 1/1 Running 0 5m Health endpoint returns 200 curl -s -o /dev/null -w "%{http_code}" http://llama-index.prod.svc.cluster.local/healthz 200 No lock‑related errors on startup kubectl logs llama-index-1 -n prod | grep lock # No output, indicating lock was released Index integrity check passes kubectl exec -it llama-index-1 -n prod -- llama-index-cli verify --path /data Index verification succeeded: 0 corrupted segments Ingestion pipeline resumes kubectl logs ingestion-job-7d9f9 -n prod | tail -n 5 2025-03-02T08:15:12.345Z INFO Ingestion completed for doc-id 12345 Prevention and Best Practices Graceful shutdown policy – Always set terminationGracePeriodSeconds to at least twice the longest expected ingestion batch. PreStop hook – Use the official LlamaIndex CLI to flush and delete lock files. The pattern is documented in Getting Started with Persistent Storage. Parallel pod management – For services that do not require strict ordering, set podManagementPolicy: Parallel to avoid volume‑attachment deadlocks. Monitoring – Add alerts on: Pod status = Terminating for >30 s. PVC detach failures (“device or resource busy”). LlamaIndex log pattern “Failed to acquire lock”. Backup strategy – Periodically snapshot the PVC (e.g., using CSI snapshot) to recover from rare corruption events. Related Topic Hub: RAG Systems Troubleshooting Hub

In this article

Table of Contents

Problem Description

A production LlamaIndex service deployed as a StatefulSet began failing during rolling updates. Pods entered the Terminating phase for more than ten minutes, PVC unmounts stalled, and the vector‑store index became corrupted. The symptoms manifested as:

Ingress returned 503 Service Unavailable – LlamaIndex API not ready.
Ingestion pipelines timed out with Failed to acquire lock on /data/index.lock.
Container logs showed Index corruption detected: Unexpected EOF while reading index file on restart.
Kubernetes events contained "Error syncing pod, failed to delete pod: context deadline exceeded" and "Failed to unmount volume /data: device or resource busy".

The issue broke the SLA for high‑concurrency RAG queries and caused a cascade of failures across downstream services.

Root Cause Analysis

The failure stems from an interaction of three Kubernetes mechanisms and LlamaIndex’s file‑locking semantics:

Graceful shutdown timing – The default terminationGracePeriodSeconds (30 s) is insufficient for LlamaIndex to flush pending writes and release the lock on the vector store. When the grace period expires, the kubelet sends SIGKILL, leaving the index file partially written.
Missing preStop hook – Without an explicit hook to invoke llama_index.flush() (or the equivalent CLI command), the process does not guarantee that all in‑flight ingestion threads have completed. This was highlighted in GitHub Issue #1195.
PodManagementPolicy = OrderedReady – The default ordered rollout forces the next pod to wait for the previous pod’s PVC to detach. If the previous pod hangs on unmount, the whole rollout stalls, as observed in the fintech incident (Q3 2024).

Combined, these factors cause the PVC to remain bound, the index lock file to persist, and new pods to start while the old lock is still held, leading to the “index lock already held” error and index corruption.

Investigation and Debugging Steps

1. Observe pod termination state

kubectl get pods -n prod -l app=llama-index -w
NAME                READY   STATUS        RESTARTS   AGE
llama-index-0       1/1     Running       0          12d
llama-index-1       1/1     Terminating   0          12d

Notice the Terminating pod never transitions to Completed.

2. Inspect events and kubelet logs

kubectl describe pod llama-index-1 -n prod
Events:
  Type    Reason                     Age   From               Message
  ----    ------                     ----  ----               -------
  Normal  Killing                    2m    kubelet            Killing container with id docker://llama-index:Need to kill pod
  Warning FailedDeletePod            1m    kubelet            Error syncing pod, failed to delete pod: context deadline exceeded

3. Check LlamaIndex container logs for lock activity

kubectl logs llama-index-1 -n prod
2025-02-10T14:22:31.112Z INFO  LlamaIndex: Starting ingestion workers
2025-02-10T14:22:31.115Z INFO  LlamaIndex: Acquired lock on /data/index.lock
2025-02-10T14:25:00.001Z WARN  LlamaIndex: Shutdown signal received, waiting for workers...
2025-02-10T14:25:30.000Z ERROR LlamaIndex: Timeout waiting for workers to finish
2025-02-10T14:25:30.001Z INFO  LlamaIndex: Exiting without releasing lock

4. Verify PVC attachment state

kubectl get pvc -n prod -l app=llama-index
NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
llama-index-data   Bound    pvc-1234abcd-5678-efgh-ijkl-9mnopqrstuv   50Gi       RWO            fast-ssd       12d

If the volume remains Bound to the terminating pod, the next pod cannot mount it.

5. Capture a short packet trace (optional)

kubectl exec -it llama-index-0 -n prod -- tcpdump -i any -nn port 8080 -c 10

6. Correlate with known incidents

Fintech Q3 2024 incident – pod stuck >10 min, index corruption.
Media streaming Mar 2025 – ordered rollout caused volume attachment timeout.

Resolution

The fix consists of three coordinated changes:

1. Add a robust `preStop` hook

Invoke the LlamaIndex CLI to flush and release the lock before the container receives SIGTERM.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: llama-index
spec:
  serviceName: llama-index
  podManagementPolicy: Parallel   # change discussed below
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0
  template:
    metadata:
      labels:
        app: llama-index
    spec:
      terminationGracePeriodSeconds: 120
      containers:
      - name: llama-index
        image: myrepo/llama-index:2.3.1
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: data
          mountPath: /data
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "llama-index-cli flush --path /data && rm -f /data/index.lock"]
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: llama-index-data

2. Increase `terminationGracePeriodSeconds`

Set to 120 seconds to allow ingestion workers to finish and the lock file to be removed.

3. Switch `podManagementPolicy` to `Parallel`

This prevents the next pod from waiting for the previous pod’s PVC to detach, eliminating the volume‑attachment bottleneck observed in the media streaming outage.

4. Ensure correct `fsGroup` for file‑system permissions

Adding a security context guarantees the LlamaIndex process can delete index.lock after shutdown.

securityContext:
  fsGroup: 2000

5. Apply the updated manifest

kubectl apply -f llama-index-statefulset.yaml
kubectl rollout status statefulset/llama-index -n prod

Verification

Rollout completes without stuck pods

kubectl get pods -n prod -l app=llama-index
NAME                READY   STATUS    RESTARTS   AGE
llama-index-0       1/1     Running   0          12d
llama-index-1       1/1     Running   0          5m

Health endpoint returns 200

curl -s -o /dev/null -w "%{http_code}" http://llama-index.prod.svc.cluster.local/healthz
200

No lock‑related errors on startup

kubectl logs llama-index-1 -n prod | grep lock
# No output, indicating lock was released

Index integrity check passes

kubectl exec -it llama-index-1 -n prod -- llama-index-cli verify --path /data
Index verification succeeded: 0 corrupted segments

Ingestion pipeline resumes

kubectl logs ingestion-job-7d9f9 -n prod | tail -n 5
2025-03-02T08:15:12.345Z INFO  Ingestion completed for doc-id 12345

Prevention and Best Practices

Graceful shutdown policy – Always set terminationGracePeriodSeconds to at least twice the longest expected ingestion batch.
PreStop hook – Use the official LlamaIndex CLI to flush and delete lock files. The pattern is documented in Getting Started with Persistent Storage.
Parallel pod management – For services that do not require strict ordering, set podManagementPolicy: Parallel to avoid volume‑attachment deadlocks.
Monitoring – Add alerts on:
- Pod status = Terminating for >30 s.
- PVC detach failures (“device or resource busy”).
- LlamaIndex log pattern “Failed to acquire lock”.
Backup strategy – Periodically snapshot the PVC (e.g., using CSI snapshot) to recover from rare corruption events.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

Why does the pod stay in Terminating even after SIGTERM?
Because LlamaIndex holds a file lock on the vector store. Without a preStop hook, the lock is not released, causing the kubelet to wait until the grace period expires, then force‑kill the process.
Can I keep the default OrderedReady policy?
You can, but you must ensure the previous pod releases its PVC within the grace period. In practice, Parallel is safer for rolling updates of LlamaIndex.
Is increasing terminationGracePeriodSeconds alone sufficient?
No. The process must still be instructed to flush and release the lock. The preStop hook is the critical piece.
How do I verify that the index is not corrupted after a rollout?
Run llama-index-cli verify --path /data inside a running pod. A successful exit code and “0 corrupted segments” indicate integrity.
What monitoring metric should I watch to detect a stuck rollout early?
Watch the kube_pod_status_phase{phase="Terminating"} metric and set an alert for any pod remaining in that state longer than 30 seconds.