Elasticsearch ReplicaSet Scaling Failure During Benchmarking

Q: 2. Index Lifecycle Management (ILM) Policies Not Updating Replica Count

ILM policies can set a fixed number_of_replicas on each phase. If a policy does not include a set_replica_count action, the replica count remains unchanged after index shrink or rollover, leading to excess replicas (official docs).

Q: 3. Allocation Filtering and Tier Preferences

In a multi‑tenant environment, aggressive cluster.routing.allocation.include/exclude filters were applied to enforce node tier separation. During the benchmark, these filters unintentionally excluded newly added nodes, causing replicas to stay UNASSIGNED (see GitHub issue #56321).

Q: Investigation and Debugging Steps Step 1 – Verify Cluster Health and Allocation Settings

curl -s -XGET 'http://localhost:9200/_cluster/health?pretty' Expected output (problematic case):

Q: Step 2 – Inspect Allocation Decider Settings

curl -s -XGET 'http://localhost:9200/_cluster/settings?include_defaults=true&pretty' Look for entries such as:

Q: Step 3 – Examine Unassigned Shard Reasons

curl -s -XGET 'http://localhost:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason' Typical output showing the root cause:

Q: Step 4 – Capture Allocation Decisions via Explain API

curl -s -XGET 'http://localhost:9200/my-index/_explain/0?pretty' Relevant snippet:

Q: Step 5 – Review ILM Policy for Replica Settings

curl -s -XGET 'http://localhost:9200/_ilm/policy/benchmark_policy?pretty' Ensure the set_replica_count action is present in the hot and warm phases.

In this article

Table of Contents

Problem: Replica Set Scaling Failure During Elasticsearch Benchmarking

During a performance benchmark on a multi‑node Elasticsearch cluster, replica shards do not scale up or down in response to the predefined desired replica count metrics. The symptoms observed include:

After adding two new data nodes, replica shards remain on a single node, causing CPU saturation on that node.
During a sustained bulk‑ingest test, replicas repeatedly fail allocation, producing “failed to allocate replica shard” errors and intermittent query latency spikes.
When an index is shrunk, the replica count does not decrease, leaving excess shards that increase heap usage and trigger GC pauses.

Typical log excerpts:

[2026-07-09T14:32:11,123][WARN ][o.e.cluster.routing.allocation.decider.AllocationDecider] [node-1] failed to allocate replica shard [my-index][0], reason: [no valid shard copy]

[2026-07-09T14:35:47,987][WARN ][o.e.cluster.routing.allocation.decider.DiskThresholdDecider] [node-2] cluster.routing.allocation.disk.watermark.flood_stage exceeded

These failures lead to a yellow cluster health status despite apparent resource availability.

Root Cause Analysis

1. Shard Allocation Settings Interfering with Dynamic Scaling

Elasticsearch’s allocation deciders evaluate several cluster routing settings before moving replicas:

Setting	Effect
`cluster.routing.allocation.enable`	Controls whether allocation, rebalancing, or both are allowed. If set to `none`, replicas will not be allocated or moved (official docs).
`cluster.routing.rebalance.enable`	When set to `none`, the cluster will not rebalance existing shards, which matches the log entry “shard allocation failed: [NO_REBALANCE]”.
`cluster.routing.allocation.disk.watermark.*`	Disk watermarks trigger `NO_REBALANCE` or `FLOOD_STAGE` states, preventing replica allocation under high ingestion load (official docs).

In the benchmark environment, the following misconfigurations were identified:

cluster.routing.allocation.enable: primaries was applied to reduce allocation chatter during load, unintentionally blocking replica allocation.
Disk watermarks were left at the default 85%/90%/95%, but bulk ingestion temporarily pushed node disks past the flood_stage threshold, causing allocation loops (see real incident “disk watermarks triggered shard relocation loops”).

2. Index Lifecycle Management (ILM) Policies Not Updating Replica Count

ILM policies can set a fixed number_of_replicas on each phase. If a policy does not include a set_replica_count action, the replica count remains unchanged after index shrink or rollover, leading to excess replicas (official docs).

3. Allocation Filtering and Tier Preferences

In a multi‑tenant environment, aggressive cluster.routing.allocation.include/exclude filters were applied to enforce node tier separation. During the benchmark, these filters unintentionally excluded newly added nodes, causing replicas to stay UNASSIGNED (see GitHub issue #56321).

Investigation and Debugging Steps

Step 1 – Verify Cluster Health and Allocation Settings

curl -s -XGET 'http://localhost:9200/_cluster/health?pretty'

Expected output (problematic case):

{
  "cluster_name" : "benchmark-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 7,
  "number_of_data_nodes" : 5,
  "active_primary_shards" : 120,
  "active_shards" : 180,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 30,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 12,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 66.7
}

Step 2 – Inspect Allocation Decider Settings

curl -s -XGET 'http://localhost:9200/_cluster/settings?include_defaults=true&pretty'

Look for entries such as:

"cluster.routing.allocation.enable" : "primaries",
"cluster.routing.rebalance.enable" : "none",
"cluster.routing.allocation.disk.watermark.low" : "85%",
"cluster.routing.allocation.disk.watermark.high" : "90%",
"cluster.routing.allocation.disk.watermark.flood_stage" : "95%"

Step 3 – Examine Unassigned Shard Reasons

curl -s -XGET 'http://localhost:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason'

Typical output showing the root cause:

my-index 0 r UNASSIGNED NO_VALID_SHARD_COPY
my-index 1 r UNASSIGNED NODE_LEFT
my-index 2 r UNASSIGNED DISK_WATERMARK

Step 4 – Capture Allocation Decisions via Explain API

curl -s -XGET 'http://localhost:9200/my-index/_explain/0?pretty'

Relevant snippet:

"allocation" : {
  "deciders" : [
    {
      "name" : "disk_threshold",
      "explanation" : "cannot allocate because the node is above the flood stage watermark"
    },
    {
      "name" : "rebalance",
      "explanation" : "rebalance disabled (cluster.routing.rebalance.enable=none)"
    }
  ]
}

Step 5 – Review ILM Policy for Replica Settings

curl -s -XGET 'http://localhost:9200/_ilm/policy/benchmark_policy?pretty'

Ensure the set_replica_count action is present in the hot and warm phases.

Resolution

1. Adjust Cluster Routing Allocation Settings

Enable full allocation and rebalance during the benchmark:

# Before
PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries",
    "cluster.routing.rebalance.enable": "none"
  }
}

# After
PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "all",
    "cluster.routing.rebalance.enable": "all"
  }
}

Explanation: Setting allocation.enable to all allows both primary and replica allocation. Restoring rebalance enables the cluster to redistribute replicas across newly added nodes.

2. Tune Disk Watermark Thresholds for Benchmark Loads

Raise the high and flood‑stage watermarks temporarily to avoid throttling during bulk ingestion:

# Before
PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%"
  }
}

# After (benchmark window)
PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "98%"
  }
}

After the benchmark, revert to production‑grade values to protect node stability.

3. Update ILM Policy to Dynamically Adjust Replica Count

Add a set_replica_count step in the hot phase and a set_priority in the warm phase to shrink replicas after index shrink:

# Before (missing replica step)
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_age": "7d" }
        }
      }
    }
  }
}

# After (adds replica handling)
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_age": "7d" },
          "set_replica_count": { "number_of_replicas": 2 }
        }
      },
      "warm": {
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "set_replica_count": { "number_of_replicas": 1 }
        }
      }
    }
  }
}

4. Remove Over‑Restrictive Allocation Filters

If cluster.routing.allocation.include/exclude filters are used, ensure they encompass the new nodes:

# Example of a restrictive filter (problematic)
PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.exclude._ip": "10.0.0.5,10.0.0.6"
  }
}

# Adjusted filter (allow new nodes)
PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.exclude._ip": ""
  }
}

Validation

Check Cluster Health

curl -s -XGET 'http://localhost:9200/_cluster/health?pretty'

Expected output after fixes:

{
  "status" : "green",
  "active_shards_percent_as_number" : 100.0,
  ...
}

Verify Replica Distribution Across Nodes

curl -s -XGET 'http://localhost:9200/_cat/shards?v&h=index,shard,prirep,node'

Confirm that replicas are spread roughly evenly across all data nodes.

Monitor Disk Watermark Metrics

GET /_nodes/stats/fs?filter_path=nodes.*.fs.total

Ensure no node reports disk.used_percent exceeding the configured high watermark.

Run a Mini‑Benchmark

Execute a short bulk‑ingest job (e.g., 10 GB) and observe that replica allocation stabilizes without “failed to allocate replica shard” warnings in elasticsearch.log.

Prevention and Best Practices

Separate Benchmark and Production Settings: Keep a dedicated benchmark.yml that temporarily relaxes allocation and disk watermarks, and automate re‑application of production settings after the run.
Automated Allocation Health Checks: Use a watchdog script that queries /_cluster/health and /_cat/shards every minute; trigger an alert if unassigned_shards > 0 for > 2 minutes.
ILM‑Driven Replica Management: Include set_replica_count actions in every ILM phase that changes index size or tier.
Avoid Over‑Filtering: When using allocation filters, prefer tier‑based attributes (node.attr.box_type) over IP‑based excludes to reduce accidental node exclusion.
Monitor Disk Watermarks in Real‑Time: Export cluster.routing.allocation.disk.watermark.* metrics to a monitoring system (Prometheus, Grafana) and set alerts before hitting flood_stage.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why do replicas stay on a single node after adding new data nodes?
Because cluster.routing.rebalance.enable was set to none (or cluster.routing.allocation.enable limited to primaries), preventing the rebalance decider from moving replicas onto the new nodes.
What does the “NO_REBALANCE” message indicate?
It is logged when the rebalance decider is disabled. The cluster will not relocate any shard, including replicas, even if resources are available (official docs).
How can I tell which watermark triggered a replica allocation failure?
Inspect the unassigned.reason column from /_cat/shards or use the Explain API. The reason will be DISK_WATERMARK or FLOOD_STAGE with the specific watermark name in the log message.
Do ILM policies automatically adjust replica counts after an index shrink?
Only if the policy includes a set_replica_count action in the phase that performs the shrink. Without it, the existing replica count persists, leading to excess shards.
Can allocation filters cause replicas to remain UNASSIGNED even when nodes have capacity?
Yes. Filters such as cluster.routing.allocation.exclude._ip or include._tier_preference can unintentionally exclude newly added nodes, as seen in GitHub issue #56321. Verify filter values before benchmark runs.