Elasticsearch Shard Rebalancing Fails During Blue-green Deployment

Q: 3. Align node attributes and master‑eligibility settings

Ensure the node.attr.* definitions are identical across blue and green nodes. For example:

In this article

Table of Contents

Problem – Shard Rebalancing Fails During a Blue‑Green Deployment

During a zero‑downtime blue‑green swap, the primary cluster (blue) is drained while the secondary cluster (green) is brought online. Operators observed the following symptoms after traffic cut‑over:

Repeated log entries such as:


[2026-06-27T14:02:13,456][WARN ][cluster.routing.allocation.decider] [node2] failed to allocate shards
reason: cluster_state_version mismatch (expected=12345, actual=12340)

Cluster health API reports RED with unassigned primary shards:


GET /_cluster/health?pretty
{
  "cluster_name" : "green",
  "status" : "red",
  "unassigned_shards" : 3,
  ...
}

Allocation explanation shows the decider rejecting moves:


GET /_cluster/allocation/explain?pretty
{
  "explain" : false,
  "reason" : "cluster_state_version mismatch",
  "deciders" : [
    {
      "name" : "cluster_state_version",
      "explanation" : "node [node2] has a stale cluster state version"
    }
  ]
}

Traffic experiences a brief outage (≈0.5 s) while replicas cannot be promoted.

Root Cause – Divergent Cluster State and Allocation Decider Mismatch

The Elasticsearch routing and allocation subsystem relies on a single, consistent cluster state. During a blue‑green deployment the following conditions commonly arise:

Cluster state version drift: The blue and green clusters run independent master‑eligible nodes. If the green cluster joins the blue cluster after a partial rolling restart, the master on green may still hold an older cluster_state_version. Allocation deciders reject shard moves until the versions converge, producing the “cluster_state_version mismatch” error seen in the logs.
Evidence: GitHub issue #60387 reports the same mismatch during traffic cut‑over.
Node attribute divergence: Even with identical hardware, subtle differences (e.g., JVM heap size, custom attributes) cause the allocation_decider to treat a node as unsuitable. The media streaming incident highlighted this problem.
Misaligned master‑eligibility settings: A mismatched minimum_master_nodes (or the newer discovery.type=single-node) between clusters can prevent the new master from accepting the latest state, leading to “cluster state is not recovered” messages.
Allocation filtering left enabled: The deployment script may disable shard movement with cluster.routing.allocation.enable: none and forget to re‑enable it, effectively blocking any rebalance.

Combined, these factors make the green cluster view the primary shards as stale, preventing promotion and causing allocation failures.

Debug – Step‑by‑Step Investigation

The following checklist reproduces the diagnostic workflow used in the fintech incident:

Check cluster health and shard allocation status:

curl -s -XGET 'http://green:9200/_cluster/health?pretty'

Inspect the current cluster state version on both clusters:

curl -s -XGET 'http://blue:9200/_cluster/state?filter_path=metadata.cluster_uuid,master_node,version' | jq .
curl -s -XGET 'http://green:9200/_cluster/state?filter_path=metadata.cluster_uuid,master_node,version' | jq .

Look for a version difference greater than 0.

Retrieve allocation explanation for an unassigned shard:

curl -s -XPOST 'http://green:9200/_cluster/allocation/explain' -H 'Content-Type: application/json' -d '{
  "index": "orders",
  "shard": 0,
  "primary": true
}' | jq .

Verify node attributes:

curl -s -XGET 'http://green:9200/_nodes?filter_path=nodes.*.attributes' | jq .

Confirm allocation settings:

curl -s -XGET 'http://green:9200/_cluster/settings?include_defaults=true' | jq .persistent.cluster.routing.allocation

Check master election logs (on each master‑eligible node):
```
journalctl -u elasticsearch | grep -i 'master'
```

Solution – Align Cluster State and Enable Controlled Rebalance

The fix consists of three coordinated actions:

1. Force a cluster state sync

Trigger a full cluster state publication from the blue master and ensure the green master acknowledges it.

# On the blue master
curl -XPOST 'http://blue:9200/_cluster/voting_config_exclusions?node_names=green_master' -s
# Wait for the exclusion to propagate, then re‑add the node
curl -XDELETE 'http://blue:9200/_cluster/voting_config_exclusions?node_names=green_master' -s

This forces a new election and a fresh cluster_state_version that both sides will share.

2. Re‑enable allocation with explicit filtering

Temporarily disable allocation, then re‑enable it using index‑level filters to avoid a rebalance storm.

# Disable globally (if previously set)
curl -XPUT 'http://green:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "none"
  }
}'

# Re‑enable with include/exclude to target only green nodes
curl -XPUT 'http://green:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "all",
    "cluster.routing.allocation.include._name": "green-node-*"
  }
}'

3. Align node attributes and master‑eligibility settings

Ensure the node.attr.* definitions are identical across blue and green nodes. For example:

Setting	Before	After
JVM heap size	`-Xms2g -Xmx2g`	`-Xms4g -Xmx4g` (matched to green)
Discovery master quorum	`minimum_master_nodes: 2`	`minimum_master_nodes: 3` (same on both clusters)

Update elasticsearch.yml on all nodes and restart them in a rolling fashion:

# elasticsearch.yml snippet
node.attr.zone: green
cluster.routing.allocation.awareness.attributes: zone
discovery.seed_hosts: ["node1", "node2", "node3"]
cluster.initial_master_nodes: ["node1", "node2", "node3"]

4. Promote replicas manually if needed

When automatic promotion stalls, force it with the reroute API:

curl -XPOST 'http://green:9200/_cluster/reroute?pretty' -H 'Content-Type: application/json' -d '{
  "commands": [
    {
      "allocate_stale_primary": {
        "index": "orders",
        "shard": 0,
        "node": "green-node-1",
        "accept_data_loss": true
      }
    }
  ]
}'

Use accept_data_loss only after confirming the stale primary contains no newer data than the replica.

Verification – Confirming Successful Rebalance

Check cluster health:
```
curl -s -XGET 'http://green:9200/_cluster/health?pretty'
```
Expect "status":"green" and "unassigned_shards":0.

Validate allocation explanations for a sample shard:

curl -s -XPOST 'http://green:9200/_cluster/allocation/explain' -H 'Content-Type: application/json' -d '{
  "index": "orders",
  "shard": 0,
  "primary": true
}' | jq .explain

Expect true with no decider rejections.

Inspect node logs for “shard started” messages:

grep -i "started" /var/log/elasticsearch/elasticsearch.log | tail -n 10

Run a functional query against the migrated index to ensure data visibility:
```
curl -s -XGET 'http://green:9200/orders/_search?q=order_id:12345&pretty'
```

Prevention – Guardrails for Future Blue‑Green Swaps

Synchronize cluster state before cut‑over: Use the voting‑config‑exclusions trick or a dedicated _cluster/sync script to guarantee version parity.
Standardize node attributes: Store attribute definitions in a configuration management repository (Ansible, Chef) and validate with a pre‑deployment health check.

Leverage allocation filtering during deployments:

# Example: keep shards on blue nodes until green is ready
PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.exclude._name": "green-node-*"
  }
}

Automate health‑check pipelines that query /_cluster/health, /_cluster/allocation/explain, and node attribute consistency before traffic switch.
Document and version‑control the minimum_master_nodes (or use the newer cluster.initial_master_nodes) setting across environments.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does the “cluster_state_version mismatch” only appear after the green cluster joins?
The green master elected during the roll‑out still holds a stale state because it missed the latest master‑only publication. Until the masters reconcile, allocation deciders treat the green node as out‑of‑date, rejecting shard moves.
Can I safely use allocate_stale_primary without data loss?
Only if you have verified that the replica shard contains all acknowledged writes (e.g., via _cat/shards and write acknowledgment counts). Otherwise, accept the risk or perform a full reindex.
What role does cluster.routing.allocation.enable play in blue‑green deployments?
Disabling allocation (`none`) is a common safety net to prevent premature relocation. Forgetting to reset it leaves the cluster in a “no‑allocation” state, causing the exact symptoms described.
How do node attribute mismatches trigger allocation failures?
Allocation deciders compare required attributes (e.g., zone, disk_type) against node metadata. A missing or differing attribute causes the decider to reject the move, even if hardware is otherwise identical.
Is there a way to monitor cluster state version drift proactively?
Yes. Query /_cluster/state?filter_path=version from each master‑eligible node on a schedule and alert when the versions differ by more than 1.