Problem – Vector Index Corruption During Rolling Updates
During a rolling update of a GPT‑4 retrieval‑augmented generation (RAG) service, the similarity index stored on each node becomes inconsistent. Queries that rely on nearest‑neighbor search return missing or incorrect results, and downstream generation may produce unrelated or malformed text. The issue appears only when the update spans multiple nodes; a single‑node restart does not exhibit the problem.
Root Cause – Non‑Atomic Index Replacement and Stale Cache
The RAG service loads a faiss (or hnswlib) index from a shared NFS mount. The rolling update script performs the following steps on each node:
#!/bin/bash
systemctl stop gpt4-rag
cp /tmp/new_index.faiss /data/index.faiss
systemctl start gpt4-rag
Two independent failures combine to corrupt the index:
- Partial file copy: The
cpcommand is not atomic. If a node is terminated while the copy is in progress, the target file is left truncated. Subsequent nodes read the truncated file because the NFS cache does not invalidate the stale inode. - In‑memory cache reuse: The service keeps the index object in memory across restarts when the
index.faisspath does not change. After a failed copy, the process reloads the corrupted file but silently falls back to the previously cached (now mismatched) vectors, causing inconsistent nearest‑neighbor results.
The combination of an incomplete on‑disk index and a stale in‑memory representation leads to missing vectors, wrong distances, and occasional segmentation faults that surface as “index corruption” in the logs.
Debug – Investigation Steps
1. Identify corrupted nodes via logs
2026-11-18 03:12:45.123 INFO [index_loader] Loading index from /data/index.faiss
2026-11-18 03:12:45.130 ERROR [faiss] Failed to read header: file too short
2026-11-18 03:12:45.131 WARN [service] Falling back to in‑memory cache (size=1.2M vectors)
The error file too short is a strong indicator of a truncated copy.
2. Verify file size and checksum on each node
# On node A (suspected corrupted)
ls -l /data/index.faiss
# -rw-r--r-- 1 gpt4 gpt4 8423456 Nov 18 03:12 index.faiss
md5sum /data/index.faiss
# d41d8cd98f00b204e9800998ecf8427e /data/index.faiss (empty file)
# On node B (healthy)
md5sum /data/index.faiss
# 5f4dcc3b5aa765d61d8327deb882cf99 /data/index.faiss
The empty‑file checksum confirms truncation.
3. Inspect NFS cache state
# Show cached inode attributes
nfsstat -c
# Attributes: 0 stale, 3 valid, 1 invalidated
Stale cache entries explain why other nodes continued to read the corrupted file.
4. Reproduce the race condition locally
Run the copy in a background job while killing the process after a random delay:
#!/bin/bash
cp /tmp/new_index.faiss /data/index.faiss &
PID=$!
sleep $((RANDOM % 2))
kill -9 $PID # Simulate node termination during copy
Subsequent reads produce the same “file too short” error.
Solution – Atomic Index Deployment with Versioned Files
1. Use a write‑ahead temporary file and atomic rename
# Deploy script (deploy_index.sh)
#!/bin/bash
set -euo pipefail
TMP="/data/index.faiss.tmp.$(date +%s%N)"
FINAL="/data/index.faiss"
# Copy to a temporary name
cp /tmp/new_index.faiss "$TMP"
# Verify checksum before promotion
EXPECTED="5f4dcc3b5aa765d61d8327deb882cf99"
ACTUAL=$(md5sum "$TMP" | awk '{print $1}')
if [[ "$ACTUAL" != "$EXPECTED" ]]; then
echo "Checksum mismatch: $ACTUAL != $EXPECTED" >&2
exit 1
fi
# Atomic rename replaces the old index
mv "$TMP" "$FINAL"
sync # Ensure metadata is flushed to NFS
The mv operation on the same filesystem is atomic, guaranteeing that readers either see the old or the new complete index, never a half‑written file.
2. Invalidate in‑memory cache on restart
# systemd service override (gpt4-rag.service)
[Service]
Environment=INDEX_PATH=/data/index.faiss
ExecStartPre=/usr/bin/rm -f /tmp/index_cache.lock
ExecStart=/usr/local/bin/gpt4-rag --index $INDEX_PATH --no-cache-reuse
The --no-cache-reuse flag forces a fresh load, eliminating stale objects.
3. Coordinate rolling updates with a lock file
# orchestrator snippet (Kubernetes Job)
apiVersion: batch/v1
kind: Job
metadata:
name: index-deploy
spec:
template:
spec:
containers:
- name: deploy
image: alpine
command: ["sh", "-c", "
while ! curl -sf http://gpt4-rag-leader/health; do sleep 2; done
./deploy_index.sh
"]
restartPolicy: OnFailure
The job waits for the leader node to report health before proceeding, ensuring that at least one node runs with a valid index before any others are updated.
Verify – Confirming Index Integrity After Update
1. Check file checksum on all nodes
for NODE in node-{1..5}; do
ssh $NODE "md5sum /data/index.faiss"
done
# All outputs should match the expected checksum.
2. Run a deterministic nearest‑neighbor query
# Query script (test_query.py)
import numpy as np, faiss, requests, json
vec = np.random.rand(1, 768).astype('float32')
resp = requests.post(
"http://gpt4-rag-node-1/query",
json={"vector": vec.tolist(), "k": 5}
)
print(json.dumps(resp.json(), indent=2))
Execute the script against each node; the returned IDs and distances must be identical.
3. Monitor service health endpoint
curl -s http://gpt4-rag-node-1/health | jq .
{
"status": "ok",
"index_checksum": "5f4dcc3b5aa765d61d8327deb882cf99",
"uptime_seconds": 342
}
The index_checksum field should match the deployed checksum.
Prevent – Operational Guardrails
- Atomic deployment pipeline: Always copy to a temporary filename and rename.
- Checksum verification: Store the expected SHA‑256/MD5 hash as a ConfigMap or secret and validate before promotion.
- Cache busting: Disable index object reuse across restarts or implement a version‑tagged cache key.
- Rolling‑update coordination: Use leader election (e.g., etcd lock) to ensure at least one healthy node runs before others are updated.
- Monitoring: Export
index_checksumandindex_load_errorsas Prometheus metrics; alert on non‑zero error counters. - Backup & rollback: Keep the previous index file (e.g.,
index.faiss.bak) and provide a quickmv index.faiss.bak index.faissrollback path.
Related Topic Hub: LLM Systems Troubleshooting Hub
FAQ
- Why does the index appear fine on some nodes but not others?
Because NFS caching can serve a stale inode. Nodes that accessed the file before the corrupted copy retain the old (valid) data, while nodes that read after the truncation see the incomplete file. - Can I avoid the atomic rename by using rsync?
Rsync with--inplacesuffers the same partial‑write problem. Use rsync without--inplace(it writes to a temporary file then renames) or rely on the explicit temporary‑file pattern shown above. - Is disabling the in‑memory cache safe for performance?
The cache only stores the deserialized index object, not the raw vectors. Reloading a 2 GB index takes ~200 ms on modern hardware, which is acceptable during a rolling restart. For high‑throughput services, keep the cache but add a version check to invalidate it when the on‑disk checksum changes. - What if the NFS server crashes during the rename?
The rename operation is atomic at the client side; if the server crashes, the client will retry the operation. Ensure the NFS export usessyncorno_root_squashto guarantee metadata persistence. - How do I detect silent corruption that does not trigger log errors?
Periodically compute and compare the index checksum via a health endpoint or a scheduled job. Mismatched checksums indicate that at least one replica holds a different file, prompting an automated redeploy.