Prometheus vector index corruption in a Kubernetes cluster

Problem – Symptoms and impact

Operators of a distributed monitoring stack observed the following after a routine Helm upgrade and a subsequent node‑drain event:

• Queries return empty results for recent time ranges (e.g., last 30 minutes) while older data is still available.
• Prometheus logs repeatedly contain errors such as:

tsdb: error loading block "/prometheus/blocks/01D8K5Z7M2G5F6J8V0H9L2R3": corrupt block meta

failed to open WAL: unexpected EOF while reading entry

out of order sample detected (timestamp: 1683024000, last timestamp: 1683023995)

• Increased query latency (often >10 s) and frequent 500 Internal Server Error responses from the /api/v1/query_range endpoint.
• Alertmanager receives “no data” alerts for critical services that were previously healthy.

The incident affected three Prometheus instances running in HA mode, each backed by a PersistentVolumeClaim (PVC) provisioned via the official Helm chart. The root cause was a corrupted vector index in the TSDB.

Root cause analysis

Prometheus stores time‑series data in a write‑ahead log (WAL) and periodically compacts the WAL into immutable blocks. Each block contains a meta.json file and a set of index files that map series identifiers to their location on disk. The vector index (the on‑disk representation of the series label set) is built during compaction and is critical for query planning.

According to the Prometheus TSDB documentation, the index is only safe to read when the block is fully written and its meta.json checksum matches. Corruption typically occurs when:

1. Unclean shutdown – the pod is terminated while the WAL is still being flushed, leading to a truncated WAL (failed to open WAL).
2. Concurrent writes to the same PVC – the HA guide warns that multiple Prometheus instances must not share a storage volume; otherwise, compaction can interleave, producing corrupt block meta and “out of order sample” errors (see GitHub issue #10112).
3. PVC migration without data migration – changing the storage class during a Helm upgrade without copying the existing block directory results in an empty blocks/ directory and subsequent index mismatches (real incident #2).
4. Snapshot restore to an older state – restoring a PVC snapshot that predates the latest head block leaves the TSDB without the most recent index files, causing “tsdb: error loading block” (mailing list thread April 2023).

In the reported incident, the primary trigger was a node‑drain that forced the PVC to detach from the original node and re‑attach on a different node while the Prometheus process was still writing to the WAL. The kubelet forced an unmount due to disk pressure, and the subsequent pod restart loaded a partially written WAL, corrupting the vector index of the most recent block.

Investigation and debugging steps

Below is a reproducible debugging workflow that helped isolate the corruption:

1. Inspect pod status and recent events

kubectl get pod -n monitoring -l app=prometheus -o wide
kubectl describe pod -n monitoring prometheus-0
# Look for events such as "Killing container with id ... due to termination signal"

Check Prometheus logs for index‑related errors

kubectl logs -n monitoring prometheus-0 | grep -E "tsdb|WAL|corrupt"

Validate the integrity of the TSDB blocks

# Exec into the pod (or mount the PVC on a debug pod)
kubectl exec -it -n monitoring prometheus-0 -- sh
cd /prometheus
promtool tsdb analyze blocks/ | grep -i error
# Sample output:
# block 01D8K5Z7M2G5F6J8V0H9L2R3: corrupt block meta

Identify the last successfully compacted block

ls -1 /prometheus/blocks | sort
# The most recent healthy block is usually the highest timestamp.

Confirm that only one Prometheus instance is attached to the PVC

kubectl get pvc -n monitoring prometheus-k8s-db -o yaml | grep -i claimRef
# Verify that the nodeName is unique across replicas.

Examine the WAL for truncation

promtool tsdb wal analyze /prometheus/wal
# Look for "unexpected EOF" messages.

Check recent Helm upgrade diff

helm diff upgrade prometheus prometheus-community/kube-prometheus-stack \
  -f values.yaml --namespace monitoring
# Ensure that storageSpec.volumeClaimTemplate was not altered unintentionally.

Resolution – Fixing the corrupted index

The remediation consisted of three coordinated actions: safe shutdown, block cleanup, and PVC re‑attachment with WAL recovery.

Step 1 – Graceful shutdown of all Prometheus replicas

# Scale down the StatefulSet to zero to stop writes
kubectl scale statefulset prometheus -n monitoring --replicas=0
# Wait for pods to terminate
kubectl wait --for=delete pod -l app=prometheus -n monitoring --timeout=120s

Scaling to zero guarantees that no process holds the WAL lock, preventing further corruption.

Step 2 – Remove the corrupted block and replay the WAL

# Exec into a temporary debug pod that mounts the same PVC
kubectl run -i --tty debug --image=busybox --restart=Never \
  -n monitoring --overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"prometheus-k8s-db"}}],"containers":[{"name":"debug","volumeMounts":[{"mountPath":"/data","name":"data"}]}]}}' -- sh
cd /data

# Identify corrupted block (example: 01D8K5Z7M2G5F6J8V0H9L2R3)
rm -rf blocks/01D8K5Z7M2G5F6J8V0H9L2R3

# Verify WAL can be replayed
promtool tsdb wal replay wal/
# Expected output: "WAL replay completed successfully"

If promtool tsdb wal replay reports errors, use the --repair flag (available in recent Prometheus versions) to truncate the offending segment.

Step 3 – Restart the StatefulSet

# Scale the StatefulSet back to the desired replica count
kubectl scale statefulset prometheus -n monitoring --replicas=3
# Verify pods become Ready
kubectl get pods -n monitoring -l app=prometheus -w

Before / After configuration comparison

Aspect	Before (corrupted)	After (fixed)
Replica count	3 (running concurrently during PVC detach)	3 (scaled down to 0 before PVC re‑attach)
Storage class	standard (unchanged during upgrade)	standard (no change)
Helm values – storageSpec.volumeClaimTemplate.spec.accessModes	ReadWriteOnce (default)	ReadWriteOnce (unchanged)
Pod termination grace period	30 s (insufficient for WAL flush)	120 s (set via terminationGracePeriodSeconds: 120)

Verification – Confirming the fix

After the pods are Ready, perform the following checks:

1. Query recent data

curl -g 'http://prometheus-0.monitoring.svc:9090/api/v1/query_range?query=up&start=$(date -d "5 minutes ago" +%s)&end=$(date +%s)&step=15s' | jq .data.result

Expected: non‑empty result set for the last 5 minutes.

2. Inspect Prometheus health endpoint

curl http://prometheus-0.monitoring.svc:9090/-/healthy
# Should return HTTP 200

3. Run promtool tsdb analyze on the block directory

promtool tsdb analyze /prometheus/blocks/
# No errors reported.

4. Check Alertmanager alerts

kubectl get alerts -n monitoring
# Previously “NoData” alerts should be cleared.

Prevention – Best practices to avoid future vector index corruption

• Enforce single‑writer semantics: Ensure that each PVC is bound to exactly one Prometheus replica. Use the PodDisruptionBudget and affinity rules to keep replicas on distinct nodes.
• Graceful termination: Set terminationGracePeriodSeconds to at least 120 seconds and enable preStop hooks that call /api/v1/quit to flush the WAL before SIGTERM.
• Separate storage for HA replicas: Deploy each Prometheus instance with its own PVC (the default in the kube-prometheus-stack chart) rather than sharing a volume.
• Regular snapshots and WAL backups: Use promtool tsdb snapshot on a schedule and store snapshots on a different storage class. Verify snapshot integrity before a restore.
• Monitor TSDB health: Create alerts on log patterns such as “corrupt block meta”, “failed to open WAL”, and on metrics like prometheus_tsdb_wal_truncate_failures_total.
• Avoid PVC class changes without data migration: If a storage class must change, perform a rsync or kubectl cp of the blocks/ and wal/ directories to a new PVC before updating the Helm release.

Related Topic Hub: Observability Troubleshooting Hub

FAQ

1. Why does the corruption only appear after a node‑drain?
   During a drain, the kubelet may force‑unmount the PVC before Prometheus finishes flushing the WAL. The abrupt loss of the WAL lock leads to a partially written block, which manifests as index corruption.
2. Can I safely run two Prometheus replicas with the same PVC for HA?
   No. The Prometheus HA guide explicitly states that concurrent writes to the same TSDB cause compaction races and “corrupt block meta” errors (GitHub issue #10112). Use separate PVCs and configure remote_write for cross‑replica replication instead.
3. How do I know if a block is corrupted without restarting Prometheus?
   Run promtool tsdb analyze /prometheus/blocks/. It will report errors such as “corrupt block meta” or missing index files. You can also grep the Prometheus logs for “tsdb: error loading block”.
4. Is there a way to recover a corrupted block automatically?
   Prometheus does not automatically repair corrupted blocks. The recommended approach is to delete the offending block directory after confirming that the data it contains is either duplicated in earlier blocks or can be re‑ingested, then let Prometheus replay the WAL.
5. What alert should I set up to catch index corruption early?
   Create a PrometheusRule that fires on log matches:

alert: PrometheusTSDBCorruption
  expr: sum by (instance) (rate(prometheus_tsdb_wal_truncate_failures_total[5m])) > 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "TSDB WAL truncate failures on {{ $labels.instance }}"
    description: "Prometheus instance {{ $labels.instance }} reported WAL truncate failures, indicating possible index corruption."