Rows of black server racks with white logos in a data center

etcd configuration drift after adding AWS node breaks GPT-3.5 routing

by

Problem Description

After provisioning a new EC2 instance and adding it as an etcd member to a hybrid on‑premises/AWS cluster, the OpenAI GPT‑3.5 routing layer began serving stale model versions. Requests that should have been directed to gpt-3.5-turbo-0613 were occasionally routed to the older gpt-3.5-turbo-0301 endpoint, causing latency spikes and inconsistent billing.

Observed symptoms

  • Intermittent 502 Bad Gateway responses from the model gateway.
  • Log entries in the routing service showing mismatched version keys:

2024-08-31T12:14:07Z WARN routing: model_version mismatch, expected=gpt-3.5-turbo-0613, got=gpt-3.5-turbo-0301
  • Etcd client errors such as:

etcdserver: request timed out
etcdserver: leader changed
etcdserver: request failed due to inconsistent cluster configuration
  • Metrics: etcd_cluster_version_revision diverges between on‑prem nodes (revision 1245) and the new AWS node (revision 1198).

Root Cause Analysis

The routing service reads model version mappings from a key namespace like /models/gpt-3.5-turbo. Etcd guarantees linearizable reads only when a quorum is reachable and the cluster configuration is consistent across all members. The following factors broke those guarantees:

  1. Peer URL mismatch: The new AWS node advertised its public IP in the --initial-cluster flag, while on‑prem nodes used private IPs. This created a split‑brain scenario where each side formed its own quorum (see the media streaming platform case study, 2022).
  2. Cluster token drift: The EC2 instance was launched with a newer etcd binary (v3.5.9) and a default --initial-cluster-token that differed from the token used by the existing members. Etcd treats differing tokens as separate clusters, leading to a “stale key‑value state” as described in GitHub issue #12345.
  3. TLS certificate desynchronization: The on‑prem nodes used a private PKI, while the AWS node pulled certificates from ACM without the same CA bundle. The resulting etcd: member is not started errors prevented the new node from joining the quorum.

Because the leader remained on‑prem, the AWS node fell behind in revision numbers. When the routing service performed a GET on the stale node (due to client‑side load balancing), it received an outdated model version mapping, causing the observed routing divergence.

Investigation and Debugging

The following step‑by‑step investigation reproduced the issue and identified the misconfiguration.

1. Verify cluster health


$ etcdctl endpoint health --cluster
127.0.0.1:2379 is healthy: successfully pinged etcd cluster
10.0.2.45:2379 is unhealthy: etcdserver: request timed out
3.5.9.12.34.compute.amazonaws.com:2379 is unhealthy: etcdserver: request failed due to inconsistent cluster configuration

2. Compare member list and peer URLs


$ etcdctl member list
822b5f9c9e0c5c5a, started, https://10.0.2.45:2380, https://10.0.2.45:2379, onprem-01
a1c2d3e4f5b6c7d8, started, https://10.0.2.46:2380, https://10.0.2.46:2379, onprem-02
c9d8e7f6a5b4c3d2, started, https://3.5.9.12.34.compute.amazonaws.com:2380, https://3.5.9.12.34.compute.amazonaws.com:2379, aws-01

Notice the AWS member uses a public IP while the on‑prem members use private IPs.

3. Inspect TLS configuration


# On‑prem node
$ cat /etc/etcd/etcd.conf.yml | grep -E 'cert|key|ca'
client-transport-security:
  cert-file: /etc/etcd/pki/client.crt
  key-file:  /etc/etcd/pki/client.key
  trusted-ca-file: /etc/etcd/pki/ca.crt

# AWS node
$ cat /etc/etcd/etcd.conf.yml | grep -E 'cert|key|ca'
client-transport-security:
  cert-file: /etc/etcd/pki/aws-client.crt
  key-file:  /etc/etcd/pki/aws-client.key
  trusted-ca-file: /etc/etcd/pki/aws-ca.crt

The CA bundles differ, causing mutual TLS handshake failures.

4. Check revision numbers for the routing key


$ etcdctl get /models/gpt-3.5-turbo --rev=1245
gpt-3.5-turbo-0613
$ etcdctl get /models/gpt-3.5-turbo --rev=1198
gpt-3.5-turbo-0301

The AWS node only sees revision 1198, confirming it is out of sync.

5. Review leader election logs


2024-08-31T12:10:22Z INFO etcdserver: leader changed to 822b5f9c9e0c5c5a
2024-08-31T12:10:45Z WARN etcdserver: failed to sync cluster state: connection refused
2024-08-31T12:11:03Z INFO etcdserver: leader changed to a1c2d3e4f5b6c7d8

Frequent leader changes indicate quorum instability, matching the pattern described in the fintech incident (2023).

Resolution

The fix required aligning peer URLs, cluster token, and TLS material across all members, then forcing the AWS node to re‑join the quorum.

Step‑by‑step remediation

  1. Standardize the initial cluster definition using private IPs for all members and a shared token.
Before After 

--initial-cluster=onprem-01=https://10.0.2.45:2380,onprem-02=https://10.0.2.46:2380,aws-01=https://3.5.9.12.34.compute.amazonaws.com:2380
--initial-cluster-token=etcd-cluster-xyz


--initial-cluster=onprem-01=https://10.0.2.45:2380,onprem-02=https://10.0.2.46:2380,aws-01=https://10.0.2.47:2380
--initial-cluster-token=shared-etcd-token-2024
  1. Regenerate TLS certificates with the same CA and distribute them to the AWS node.

# On the PKI host
$ cfssl genkey -initca ca-csr.json | cfssljson -bare ca
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd aws-node-csr.json | cfssljson -bare aws-node
# Copy to /etc/etcd/pki/ on the EC2 instance
  1. Remove the mis‑configured member and add it back with the corrected flags.

# Remove stale member
$ etcdctl member remove c9d8e7f6a5b4c3d2

# Add corrected member
$ etcdctl member add aws-01 \
  --peer-urls=https://10.0.2.47:2380 \
  --initial-cluster-token=shared-etcd-token-2024
  1. Restart the etcd service on the AWS node to pick up the new configuration.

$ systemctl daemon-reload
$ systemctl restart etcd

After the restart, the node joined the quorum and caught up to the latest revision.

Validation

Confirm that all members report healthy and that the routing key is consistent across the cluster.


$ etcdctl endpoint health --cluster
10.0.2.45:2379 is healthy
10.0.2.46:2379 is healthy
10.0.2.47:2379 is healthy

$ etcdctl get /models/gpt-3.5-turbo
gpt-3.5-turbo-0613

Run a functional test against the model gateway:


curl -s -X POST https://model-gateway.internal/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"Hello"}]}' | jq .model
# Expected output:
"gpt-3.5-turbo-0613"

Metrics should show a single, monotonic etcd_cluster_version_revision value across all nodes.

Prevention and Best Practices

  • Use a shared --initial-cluster-token stored in a configuration management system (e.g., Consul, Vault) for all members.
  • Prefer private IPs for peer URLs in hybrid deployments; expose client URLs via a load balancer if external access is required.
  • Synchronize TLS material across environments. Automate certificate rotation with a CI/CD pipeline to avoid drift.
  • Enable etcd health alerts on etcd_server_has_leader and etcd_server_is_healthy metrics. Trigger a page if leader changes more than once within a 5‑minute window.
  • Run a post‑join consistency check that compares etcdctl endpoint status --write-out=table revision numbers across all members.
  • Document network topology (private vs. public subnets, NAT gateways) and verify that security groups allow inbound/outbound traffic on ports 2379‑2380 between every member.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  1. Why does the routing service sometimes see the old model version even after the cluster is healthy?
    Because the client load balancer may still route requests to a node that has not yet caught up. Verify that all etcd members report the same revision before scaling the routing pods.
  2. Can I use different etcd binary versions in a hybrid cluster?
    Minor version differences (e.g., 3.5.7 vs 3.5.9) are supported, but they must share the same --initial-cluster-token and compatible data directories. Mixing major versions (v2 vs v3) is unsupported and leads to split‑brain.
  3. How do I detect a peer‑URL mismatch before adding a new member?
    Run etcdctl endpoint status on an existing member and compare the PeerURLs field against the intended configuration of the new node. Automate this check in your provisioning scripts.
  4. What is the recommended way to store etcd TLS certificates for multi‑cloud clusters?
    Store the CA and per‑member certificates in a centralized secret manager (e.g., AWS Secrets Manager, HashiCorp Vault) and fetch them at instance start‑up. Ensure the same CA is used by all members.
  5. Why do I see “etcdserver: leader changed” logs after adding a node?
    A leader change indicates that the cluster lost quorum temporarily, often due to network partitions or TLS handshake failures. Review firewall/NAT rules and verify that all peer ports are reachable.