etcd state divergence after rolling upgrade leads to stale Redis config

Problem Description

During a rolling upgrade of an on‑premises Kubernetes‑managed etcd cluster (v3.5.0 → v3.5.7) the AI model‑serving stack began loading outdated model parameters. The model‑config microservice reads configuration from etcd, writes the current config_version to a Redis key (model:config), and caches the full parameter payload with a 12‑hour TTL. After the upgrade the following symptoms were observed:

• Log entry from the model service: etcdclient: connection error: request timed out
• Redis GET model:config returned a version number that did not match the etcd revision (e.g., config_version=42 vs. etcd revision=57)
• Model pods continued to serve weights from an older checkpoint for several hours
• Metrics showed a spike in model_loader_errors_total and a drop in request latency due to fallback to cached config

Root Cause Analysis

The upgrade introduced a temporary loss of quorum, causing a split‑brain situation. The sequence of events, corroborated by the GitHub issue and the etcd‑io discussion, is summarised below:

According to the Redis Persistence guide, a long TTL combined with no explicit invalidation means the stale entry persists until expiration or manual deletion. The etcd client library does not automatically purge downstream caches on revision mismatch, which is why the inconsistency propagated.

Investigation and Debugging

1. Verify etcd health and revision numbers

   kubectl -n kube-system exec etcd-0 -- etcdctl endpoint status --write-out=table

   Expected output (post‑upgrade):

   +----------------+------------------+------------+------------+--------+
   |   ENDPOINT     |   CLIENT URL     | IS LEADER  | VERSION    | REV    |
   +----------------+------------------+------------+------------+--------+
   | https://10.0.0.1:2379 | https://10.0.0.1:2379 | true       | 3.5.7 | 57 |
   | https://10.0.0.2:2379 | https://10.0.0.2:2379 | false      | 3.5.7 | 57 |
   +----------------+------------------+------------+------------+--------+

   If any member reports a lower REV, a split‑brain persisted.

2. Inspect etcd logs for leader changes

   journalctl -u etcd -f | grep "leader changed"

   Sample log:

   2024-08-12T03:14:27.123Z etcdserver: leader changed, term=5, leader=10.0.0.2:2379

3. Check Redis key state and TTL

   redis-cli GET model:config

   Output:

   "config_version=42"

   redis-cli TTL model:config

   Output:

   43200  # seconds (12 h)

4. Correlate with model‑service logs

   kubectl logs -l app=model-config -c service | grep "etcdclient"

   Sample snippet:

   2024-08-12T03:15:02Z etcdclient: request timed out (etcdserver: request timed out)

5. Confirm that the stale version is being used downstream

   kubectl exec -ti $(kubectl get pod -l app=model-loader -o jsonpath="{.items[0].metadata.name}") -- cat /var/log/model-loader.log | grep "using cached config"

   Sample output:

   2024-08-12T03:20:45Z Cache miss for version 42, using fallback

Resolution

The fix consists of two parts: restoring etcd consistency and ensuring Redis cache invalidation on revision change.

1. Re‑establish etcd quorum and purge divergent members

Remove the out‑of‑sync member, let the cluster heal, then re‑add it from a fresh snapshot.

# Drain the problematic member
kubectl -n kube-system delete pod etcd-2

# Wait for the StatefulSet to recreate the pod with the latest snapshot
kubectl -n kube-system wait --for=condition=Ready pod -l app=etcd

# Verify all members report the same revision
kubectl -n kube-system exec etcd-0 -- etcdctl endpoint status --write-out=table

2. Add explicit cache invalidation in the model‑config service

Update the code to compare the etcd revision with the cached version and delete the Redis key when they diverge.

// Pseudo‑code snippet (Go)
rev, err := etcdClient.GetRevision(ctx, "model/config")
if err != nil {
    log.Error(err)
    // fallback to cache
}
cachedVer, _ := redisClient.Get(ctx, "model:config").Int()
if cachedVer != rev {
    // Invalidate stale entry
    redisClient.Del(ctx, "model:config")
    // Repopulate with fresh config
    cfg, _ := etcdClient.Get(ctx, "model/config")
    redisClient.Set(ctx, "model:config", rev, 12*time.Hour)
}

Before the change, the service blindly wrote the version it read, even if it came from a minority partition.

// Old logic
ver, _ := etcdClient.Get(ctx, "model/config")
redisClient.Set(ctx, "model:config", ver, 12h)

// New logic with revision check
rev, _ := etcdClient.GetRevision(ctx, "model/config")
cached, _ := redisClient.Get(ctx, "model:config")
if cached != rev {
    redisClient.Del(ctx, "model:config")
    redisClient.Set(ctx, "model:config", rev, 12h)
}

3. Reduce Redis TTL for configuration keys

Set a shorter TTL (e.g., 15 minutes) and rely on explicit invalidation for longer‑lived entries.

# Example Redis configuration (via ConfigMap)
maxmemory-policy allkeys-lru

Validation

1. Confirm etcd cluster health:

   kubectl -n kube-system exec etcd-0 -- etcdctl endpoint health

   Expected output:

   https://10.0.0.1:2379 is healthy

2. Verify that Redis no longer holds stale data:

   redis-cli GET model:config

   Output should match the current etcd revision (e.g., "config_version=57").

3. Check model‑loader logs for absence of fallback messages:

   kubectl logs -l app=model-loader | grep "using fallback"

   No lines should be returned.

4. Run a functional test that triggers a config reload:

   curl -s http://model-service.local/v1/reload-config | jq .revision

   Response should equal the etcd revision and the Redis TTL should reset to 15 minutes.

Prevention and Best Practices

• Enable etcd health alerts – monitor etcd_server_has_leader and etcd_server_leader_changes_seen_total as described in the Redis Monitoring guide and the etcd metrics documentation.
• Use short TTLs for configuration caches and always perform revision‑based invalidation.
• Perform rolling upgrades with quorum checks – pause the upgrade if any member reports etcdserver: request timed out or a reduced revision.
• Automate snapshot consistency checks – after each upgrade, run a script that compares etcdctl endpoint status revision across all members.
• Deploy a sidecar that watches etcd revision changes and pushes invalidation events to Redis via Pub/Sub, eliminating reliance on TTL alone.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

1. Why does the model service fall back to Redis only after an etcd upgrade?
   Because the service treats any etcd read error as a cache‑hit scenario. During the upgrade, leader step‑downs generate etcdserver: request timed out, triggering the fallback path.
2. Can I rely on Redis replication to purge stale entries automatically?
   No. Replication only copies data; it does not reconcile version skew. Explicit invalidation based on etcd revision is required.
3. What metric should I watch to detect etcd split‑brain early?
   Watch etcd_server_leader_changes_seen_total and the etcd_debugging_mvcc_index_compaction_pause_duration_seconds for sudden spikes, which indicate frequent leader changes.
4. Is reducing the Redis TTL enough to avoid stale configs?
   A shorter TTL mitigates the window of exposure but does not eliminate it. The root cause—etcd revision mismatch—must still be addressed.
5. How do I handle version skew after a manual etcd restore?
   After restoring from a snapshot, run a purge script that deletes all downstream cache keys (e.g., redis-cli KEYS model:* → DEL) before bringing services back online.