Etcd Cluster State Divergence Causing AMD GPU Node Scheduling Errors In Production

Q: Debug – Investigating the Divergence 1. Verify etcd member health

$ etcdctl endpoint health --cluster 127.0.0.1:2379 is healthy: successfully committed proposal: took = 3.456ms 127.0.0.2:2379 is unhealthy: request timed out 127.0.0.3:2379 is healthy: successfully committed proposal: took = 2.987ms Notice the timeout on 127.0.0.2. This matches the “member ID mismatch” error in the etcd logs.

Q: 2. Inspect the member list for duplicates

$ etcdctl member list ID Peer URLs Client URLs Status 0x12345678 https://etcd-0:2380 https://etcd-0:2379 started 0x9abcdef0 https://etcd-1:2380 https://etcd-1:2379 started 0x12345678 https://etcd-2:2380 https://etcd-2:2379 started (unhealthy) The duplicate ID 0x12345678 indicates that a node rejoined with a stale member record.

Q: 3. Check Kubernetes node objects

$ kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.status.conditions[?(@.type=="Ready")].status} {.spec.taints}{"\n"}{end}' gpu-node-01 NotReady [{node.kubernetes.io/not-ready:NoSchedule}] gpu-node-02 Ready [] Only gpu-node-01 is tainted, even though the ROCm driver reports the device as healthy.

Q: 4. Correlate with driver logs

Sep 06 12:34:56 hostname amdgpu[1234]: Failed to allocate resources for device: stale node entry This log appears when the kubelet does not publish node status, confirming the stale API object.

Q: 5. Review recent etcd snapshots

$ ls -l /var/lib/etcd/snapshots/ total 8 -rw------- 1 root root 32M Sep 5 23:10 snapshot-20240905-2310.db -rw------- 1 root root 32M Sep 6 00:05 snapshot-20240906-0005.db If a snapshot was restored without proper member re‑join steps (see Kubernetes etcd upgrade docs), the cluster may retain divergent membership.

Q: Solution – Restoring Consistent etcd State and Re‑registering GPU Nodes Step 1 – Remove duplicate/unhealthy members

# Identify the stale member ID (example: 0x12345678 on etcd-2) $ etcdctl member remove 0x12345678 Member 0x12345678 removed Step 2 – Re‑add the removed member with a fresh peer URL $ etcdctl member add etcd-2 --peer-urls=https://etcd-2:2380 Member added with ID: 0xdeadbeef Step 3 – Restart the etcd service on the re‑joined node # systemd on each etcd node $ systemctl restart etcd $ systemctl status etcd ● etcd.service - etcd key-value store Loaded: loaded (/etc/systemd/system/etcd.service; enabled) Active: active (running) since Mon 2024-09-06 12:45:00 UTC; 2min ago Step 4 – Verify cluster health and quorum $ etcdctl endpoint health --cluster 127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.123ms 127.0.0.2:2379 is healthy: successfully committed proposal: took = 2.987ms 127.0.0.3:2379 is healthy: successfully committed proposal: took = 2.456ms Step 5 – Clean up stale node objects in Kubernetes # Delete the NotReady node object; kubelet will recreate it $ kubectl delete node gpu-node-01 node "gpu-node-01" deleted # Wait for kubelet to re‑register $ kubectl get nodes -w NAME STATUS ROLES AGE VERSION gpu-node-01 Ready 10s v1.28.2 Step 6 – Ensure AMD GPU device plugin reports readiness $ kubectl logs -n kube-system daemonset/amd-gpu-device-plugin -c device-plugin 2024-09-06T12:50:12Z INFO Device plugin started, 4 AMD GPUs detected on gpu-node-01 After the node re‑appears as Ready, GPU‑specific pods schedule successfully.

In this article

Table of Contents

Problem – etcd Cluster State Divergence Triggering AMD GPU Node Scheduling Errors

In a production AI training platform that relies on Kubernetes to schedule AMD GPU‑accelerated workloads, operators observed a sudden increase in pod pending states and job failures. The symptoms were traced back to GPU worker nodes being marked NotReady by the kubelet, despite the underlying ROCm driver stack reporting no hardware faults.

Typical error messages seen in the logs:


Kube-scheduler log:
node(s) had taint {node.kubernetes.io/not-ready:} that the pod didn't tolerate

kubelet log:
Failed to get node info from API server: etcdserver: leader changed

etcdserver: request timed out – member ID mismatch in cluster status logs

These errors manifested after a brief network partition affecting the etcd quorum. The result was a divergent key‑value store where node objects in the API server no longer reflected the actual health of AMD GPU workers, causing the scheduler to back‑off on GPU‑specific pods.

Root Cause – Inconsistent etcd Membership and Stale Node Objects

etcd guarantees linearizable reads only when a quorum is present (etcd official docs). During the network glitch, one etcd member lost connectivity and entered a candidate state. The remaining members elected a new leader, but the isolated member continued to accept writes once the partition healed, leading to revision drift (see GitHub issue #12345).

Consequences:

Duplicate or unknown member IDs appeared in etcdctl member list, with some entries marked unhealthy.
Kubernetes API server persisted node objects based on stale revisions, retaining NotReady taints for AMD GPU nodes.
GPU driver logs (ROCm docs) showed “Failed to allocate resources for device: stale node entry” because the kubelet refused to report node status.

The root cause is therefore a cluster state divergence in etcd, which corrupts the source of truth for node health and taint information used by the scheduler.

Debug – Investigating the Divergence

1. Verify etcd member health


$ etcdctl endpoint health --cluster
127.0.0.1:2379 is healthy: successfully committed proposal: took = 3.456ms
127.0.0.2:2379 is unhealthy: request timed out
127.0.0.3:2379 is healthy: successfully committed proposal: took = 2.987ms

Notice the timeout on 127.0.0.2. This matches the “member ID mismatch” error in the etcd logs.

2. Inspect the member list for duplicates


$ etcdctl member list
ID          Peer URLs                     Client URLs                   Status
0x12345678  https://etcd-0:2380           https://etcd-0:2379           started
0x9abcdef0  https://etcd-1:2380           https://etcd-1:2379           started
0x12345678  https://etcd-2:2380           https://etcd-2:2379           started (unhealthy)

The duplicate ID 0x12345678 indicates that a node rejoined with a stale member record.

3. Check Kubernetes node objects


$ kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.status.conditions[?(@.type=="Ready")].status} {.spec.taints}{"\n"}{end}'
gpu-node-01 NotReady [{node.kubernetes.io/not-ready:NoSchedule}]
gpu-node-02 Ready []

Only gpu-node-01 is tainted, even though the ROCm driver reports the device as healthy.

4. Correlate with driver logs


Sep 06 12:34:56 hostname amdgpu[1234]: Failed to allocate resources for device: stale node entry

This log appears when the kubelet does not publish node status, confirming the stale API object.

5. Review recent etcd snapshots


$ ls -l /var/lib/etcd/snapshots/
total 8
-rw------- 1 root root  32M Sep  5 23:10 snapshot-20240905-2310.db
-rw------- 1 root root  32M Sep  6 00:05 snapshot-20240906-0005.db

If a snapshot was restored without proper member re‑join steps (see Kubernetes etcd upgrade docs), the cluster may retain divergent membership.

Solution – Restoring Consistent etcd State and Re‑registering GPU Nodes

Step 1 – Remove duplicate/unhealthy members


# Identify the stale member ID (example: 0x12345678 on etcd-2)
$ etcdctl member remove 0x12345678
Member 0x12345678 removed

Step 2 – Re‑add the removed member with a fresh peer URL


$ etcdctl member add etcd-2 --peer-urls=https://etcd-2:2380
Member added with ID: 0xdeadbeef

Step 3 – Restart the etcd service on the re‑joined node


# systemd on each etcd node
$ systemctl restart etcd
$ systemctl status etcd
● etcd.service - etcd key-value store
   Loaded: loaded (/etc/systemd/system/etcd.service; enabled)
   Active: active (running) since Mon 2024-09-06 12:45:00 UTC; 2min ago

Step 4 – Verify cluster health and quorum


$ etcdctl endpoint health --cluster
127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.123ms
127.0.0.2:2379 is healthy: successfully committed proposal: took = 2.987ms
127.0.0.3:2379 is healthy: successfully committed proposal: took = 2.456ms

Step 5 – Clean up stale node objects in Kubernetes


# Delete the NotReady node object; kubelet will recreate it
$ kubectl delete node gpu-node-01
node "gpu-node-01" deleted

# Wait for kubelet to re‑register
$ kubectl get nodes -w
NAME          STATUS   ROLES    AGE   VERSION
gpu-node-01   Ready       10s   v1.28.2

Step 6 – Ensure AMD GPU device plugin reports readiness


$ kubectl logs -n kube-system daemonset/amd-gpu-device-plugin -c device-plugin
2024-09-06T12:50:12Z INFO Device plugin started, 4 AMD GPUs detected on gpu-node-01

After the node re‑appears as Ready, GPU‑specific pods schedule successfully.

Verify – Confirming Full Recovery

Node health: kubectl get nodes shows all GPU nodes in Ready state without NotReady taints.
Scheduler behavior: Observe that new AMD GPU pods transition from Pending to Running within seconds.
etcd consistency: Run etcdctl endpoint status --write-out=table and verify that DBSize, Revision, and RaftTerm are identical across members.
GPU driver logs: No “stale node entry” messages appear after kubelet re‑registration.

Prevent – Guardrails to Avoid Future Divergence

Area	Recommendation	Implementation
Network reliability	Deploy redundant network paths and enable TCP keep‑alive for etcd peer traffic.	Set `net.ipv4.tcp_keepalive_time=60` on all etcd hosts.
etcd monitoring	Alert on `etcd_server_has_leader` and member health failures.	Prometheus rule: `etcd_server_has_leader == 0` → critical alert.
Snapshot restore process	Always run `etcdctl member add` after a restore and verify `etcdctl member list` for duplicates.	Automate with a post‑restore script that checks for `unhealthy` members.
Kubernetes node lifecycle	Enable the `NodeLifecycleController` to evict nodes with stale etcd entries after a configurable grace period.	Set `--node-monitor-grace-period=40s` in the controller‑manager.
GPU device plugin health	Expose a custom metric (`amd_gpu_node_ready`) and alert when it drops to zero.	Instrument the device plugin with Prometheus client library.

FAQ – Common Follow‑Up Questions

Why does the issue appear only on AMD GPU nodes and not on CPU nodes?
GPU nodes carry an additional node.kubernetes.io/not-ready taint that is set when the kubelet cannot publish node status. CPU nodes lack this taint, so they remain schedulable even if the underlying etcd entry is stale.
Can I prevent etcd leader changes during a rolling upgrade?
Use the --skip-verify-etcd flag cautiously and ensure that the upgrade window includes a full etcdctl endpoint health check after each master node restart. Maintaining a stable quorum (odd number of members) reduces unnecessary elections.
How do I detect duplicate member IDs before they cause divergence?
Run a periodic etcdctl member list and compare the ID column across members. A simple script can raise an alert if the same ID appears more than once.
Is restoring from an older snapshot safe if the cluster experienced a partition?
Only if you first remove all members that were part of the partition and then re‑add them using the snapshot’s member list. Otherwise, the restored data will coexist with divergent revisions.
What metric should I monitor to catch stale node entries early?
Watch apiserver_storage_objects_total{resource="nodes"} for sudden drops, and correlate with etcd_server_has_leader. A mismatch often indicates that the API server is serving from an out‑of‑date revision.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub