Black cables and wires connected to the back of modular LED display panels

etcd cluster inconsistency causing Elasticsearch index metadata divergence in staging

by

Problem

In the staging environment the Elasticsearch service reports index_not_found_exception and vector‑search queries return empty results. Simultaneously the Elasticsearch logs contain messages such as:


[2024-10-15T14:22:31.842Z] [WARN ][o.e.b.IndexMetaData] [es-node-1] failed to load index metadata for index [products] from cluster state: index not found
[2024-10-15T14:22:32.017Z] [ERROR][o.e.c.s.MasterService] [es-node-2] master not discovered, aborting request

Investigation revealed that the underlying coordination data stored in etcd diverged between members, causing Elasticsearch nodes to read different index routing tables. The symptom cascade is:

  • Stale or missing index metadata in Elasticsearch cluster state.
  • Vector‑search queries hit empty shards.
  • Routing information (shard allocation) differs across nodes.
  • Health checks report yellow or red status despite all pods being running.

Root Cause

The staging deployment uses a three‑member etcd cluster (two control‑plane pods and one side‑car) to store Elasticsearch index coordination documents. A transient network partition during a rolling update caused the following sequence:

  1. Member etcd-2 lost connectivity to the quorum for ~45 seconds.
  2. etcd’s --auto-compaction-retention was set to 1h, so the isolated member continued to accept writes locally, creating a divergent revision series.
  3. When connectivity was restored, etcd’s default --max-snapshots policy kept the divergent snapshots, and the leader elected the majority (etcd‑0 + etcd‑1) without reconciling the divergent log entries from etcd‑2.
  4. Elasticsearch’s etcd-watcher client, running on each node, subscribed to the /es/indices/ prefix. Nodes that connected to the majority saw the newer revision, while the node that re‑joined via etcd‑2 read the older revision, resulting in inconsistent index metadata across the Elasticsearch cluster.

In short, the inconsistency originated from an etcd quorum loss combined with aggressive auto‑compaction and insufficient snapshot retention, which prevented automatic reconciliation of divergent histories.

Debug

The following step‑by‑step debugging process was used to confirm the inconsistency and isolate the offending etcd member.

1. Verify etcd cluster health


# Check health of each member
for m in etcd-0 etcd-1 etcd-2; do
  kubectl exec -n staging $m -- etcdctl endpoint health --cluster
done

Expected output (all members healthy):


etcd-0: https://10.1.2.3:2379 is healthy: successfully pinged etcd cluster
etcd-1: https://10.1.2.4:2379 is healthy: successfully pinged etcd cluster
etcd-2: https://10.1.2.5:2379 is healthy: successfully pinged etcd cluster

Observed output showed a connection timeout for etcd-2 during the incident window.

2. Compare revision numbers


kubectl exec -n staging etcd-0 -- etcdctl endpoint status -w json
kubectl exec -n staging etcd-2 -- etcdctl endpoint status -w json

Sample JSON excerpts:

Member Revision Raft Term
etcd-0 124578 42
etcd-2 119342 41

The mismatch confirmed divergent logs.

3. Inspect Elasticsearch index metadata stored in etcd


# Dump the /es/indices/ prefix from each member
kubectl exec -n staging etcd-0 -- etcdctl get /es/indices/ --prefix -w json > etcd0.json
kubectl exec -n staging etcd-2 -- etcdctl get /es/indices/ --prefix -w json > etcd2.json

Diffing the two JSON files revealed missing shard routing entries for the products index on etcd-2.

4. Correlate with Elasticsearch logs


kubectl logs -n staging es-node-1 | grep "failed to load index metadata"

Log excerpt from the node that connected to etcd-2:


[2024-10-15T14:22:31.842Z] [WARN ][o.e.b.IndexMetaData] [es-node-1] failed to load index metadata for index [products] from cluster state: index not found

5. Verify network partition timing


kubectl get events -n staging --field-selector involvedObject.kind=Pod,involvedObject.name=etcd-2 -w

Events showed a NetworkPolicyDenied event exactly 45 seconds before the inconsistency appeared.

Solution

The remediation consisted of two parts: (1) force a consistent snapshot restore across the etcd members, and (2) adjust etcd configuration to avoid future divergent histories.

1. Force a cluster‑wide snapshot restore

We elected the healthy majority (etcd-0 + etcd-1) as the source of truth and performed a manual snapshot restore on the out‑of‑sync member.

Before (etcd-2 configuration):


apiVersion: v1
kind: ConfigMap
metadata:
  name: etcd-config
data:
  ETCD_AUTO_COMPACTION_RETENTION: "1"
  ETCD_MAX_SNAPSHOTS: "5"
  ETCD_INITIAL_CLUSTER_STATE: "new"

After (etcd-2 configuration):


apiVersion: v1
kind: ConfigMap
metadata:
  name: etcd-config
data:
  ETCD_AUTO_COMPACTION_RETENTION: "7200"   # 2 hours
  ETCD_MAX_SNAPSHOTS: "30"
  ETCD_INITIAL_CLUSTER_STATE: "existing"

Restore steps:


# 1. Stop the out‑of‑sync pod
kubectl rollout pause statefulset/etcd -n staging
kubectl delete pod etcd-2 -n staging

# 2. Copy the latest snapshot from a healthy member
kubectl exec -n staging etcd-0 -- \
  etcdctl snapshot save /tmp/etcd-snap.db

# 3. Restore on the new pod (executed in the init container)
kubectl exec -n staging etcd-2 -- \
  etcdctl snapshot restore /tmp/etcd-snap.db \
    --name etcd-2 \
    --initial-cluster etcd-0=https://etcd-0.etcd:2380,etcd-1=https://etcd-1.etcd:2380,etcd-2=https://etcd-2.etcd:2380 \
    --initial-cluster-token etcd-cluster-2024 \
    --initial-advertise-peer-urls https://etcd-2.etcd:2380

# 4. Resume the StatefulSet
kubectl rollout resume statefulset/etcd -n staging

After the pod restarted, etcdctl endpoint status showed identical revision numbers across all members.

2. Update Elasticsearch side‑car to re‑watch on leader change

We added a watch‑restart hook to the side‑car container so that any leader election triggers a full re‑load of the /es/indices/ prefix.


# sidecar entrypoint snippet (bash)
while true; do
  etcdctl watch /es/indices/ --prefix --rev=0 &
  WATCH_PID=$!
  # Wait for leader change event
  if etcdctl endpoint status | grep -q "leader"; then
    kill $WATCH_PID
    # Force full reload in Elasticsearch
    curl -XPOST http://localhost:9200/_cluster/settings -d '{"persistent":{"cluster.routing.rebalance.enable":"all"}}'
  fi
  sleep 30
done

Verify

Validation was performed at three layers: etcd consistency, Elasticsearch cluster state, and functional vector‑search queries.

Etcd consistency check


kubectl exec -n staging etcd-0 -- etcdctl endpoint status -w json | jq '.[] | {Member:.Endpoint, Revision:.Revision}'

All members reported the same revision (e.g., 124578).

Elasticsearch cluster health


curl -s http://elasticsearch-staging:9200/_cluster/health?pretty

Expected output:


{
  "cluster_name" : "staging-es",
  "status" : "green",
  "number_of_nodes" : 3,
  "active_shards_percent_as_number" : 100.0,
  ...
}

Index metadata sanity


curl -s http://elasticsearch-staging:9200/_cat/indices?v

All expected indices, including products, appear with green status.

Vector‑search functional test


curl -s -XPOST http://elasticsearch-staging:9200/products/_search -H 'Content-Type: application/json' -d '{
  "knn": {
    "field": "embedding_vector",
    "query_vector": [0.12,0.34,0.56],
    "k": 5,
    "num_candidates": 10
  }
}'

Response contains hits.total > 0, confirming that routing and metadata are now consistent.

Prevent

To avoid recurrence, the following guardrails were put in place:

  • Etcd quorum monitoring: Prometheus alerts on etcd_server_has_leader == 0 and on member health failures lasting >30 seconds.
  • Extended snapshot retention: Set ETCD_MAX_SNAPSHOTS=30 and ETCD_AUTO_COMPACTION_RETENTION=7200 (2 h) to give enough overlap for recovery.
  • NetworkPolicy strictness: Enforced policyTypes: [Ingress, Egress] with explicit allow rules for etcd peer ports (2380) and client ports (2379) across all namespaces.
  • Rolling update safety: Added a pre‑hook that checks etcdctl endpoint health on all members before allowing a pod restart.
  • Elasticsearch watch resilience: Side‑car now restarts its watch on any leader change event, guaranteeing that all nodes reload the latest index metadata.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

  1. Why did only one Elasticsearch node see the stale index metadata?
    Because that node’s side‑car connected to the out‑of‑sync etcd member (etcd-2) which held an older revision. The other nodes were connected to the majority leader and received the newer revision.
  2. Can I rely on etcd’s automatic compaction to prevent divergence?
    Automatic compaction only removes old revisions; it does not reconcile divergent logs caused by a quorum loss. Retaining enough snapshots and ensuring a stable network are required.
  3. What is the safest way to perform a rolling update of the etcd StatefulSet?
    Pause the StatefulSet, verify that etcdctl endpoint health returns healthy for all members, then update one pod at a time while monitoring the etcd_server_has_leader metric.
  4. How do I know if my Elasticsearch side‑car is watching the correct etcd revision?
    Inspect the side‑car logs for the line “watch started at revision X”. Compare X with the output of etcdctl endpoint status. Mismatched revisions indicate a possible split‑brain.
  5. Is increasing ETCD_MAX_SNAPSHOTS enough to avoid this issue?
    It helps, but you must also increase ETCD_AUTO_COMPACTION_RETENTION and ensure network stability. Snapshots alone cannot heal a member that has accepted writes while isolated.