Elasticsearch 429 error during high volume AI data ingestion

Problem Description

During a peak AI training data ingestion window, clients that send bulk indexing requests to the Elasticsearch cluster receive HTTP 429 “Too Many Requests” responses. The error payload typically looks like:

{
  "status":429,
  "error":{
    "type":"es_rejected_execution_exception",
    "reason":"rejected execution of org.elasticsearch.action.bulk.BulkRequest"
  }
}

In the same period, search queries also start failing with similar 429 responses such as:

{
  "status":429,
  "error":{
    "type":"es_rejected_execution_exception",
    "reason":"rejected execution of org.elasticsearch.action.search.SearchRequest"
  }
}

Operational impact includes:

• Stalled AI training pipelines that depend on near‑real‑time indexing of feature vectors.
• Delayed log analysis and alerting, causing missed security events.
• Back‑pressure cascades in upstream services (e.g., Kafka producers) due to un‑acknowledged bulk responses.

Root Cause Analysis

Elasticsearch enforces back‑pressure through two complementary mechanisms:

1. Search throttling – controlled by search.max_concurrent_shard_requests and the search thread‑pool queue size.^{Official docs}
2. Indexing throttling – governed by the bulk thread‑pool (thread_pool.bulk.queue_size) and the indices.memory.index_buffer_size limit.^{Thread‑pool docs}

In the hybrid cloud deployment the on‑premise data shippers push bulk requests over a WAN link to an AWS‑hosted Elasticsearch Service. Two conditions converged:

• Network latency spikes (observed up to 250 ms) caused bursts of pending bulk requests to accumulate in the bulk thread‑pool queue.
• Shard relocation triggered by a mis‑configured cluster.routing.allocation.enable setting caused additional indexing load while the cluster was already near its write capacity.

When the bulk queue filled, Elasticsearch rejected further bulk actions with es_rejected_execution_exception, which the HTTP layer translates to 429. The same phenomenon occurred for search when search.max_concurrent_shard_requests was saturated, as documented in the response‑code reference.

Investigation and Debugging

The following step‑by‑step investigation reproduced the failure and identified the throttling knobs:

1. Inspect cluster health and thread‑pool stats:

curl -s -XGET "https://es-prod.example.com/_cluster/health?pretty"
curl -s -XGET "https://es-prod.example.com/_cat/thread_pool/bulk?v&h=id,active,queue,rejected"
curl -s -XGET "https://es-prod.example.com/_cat/thread_pool/search?v&h=id,active,queue,rejected"

Typical output during the incident:

bulk   node-1   12   1000   250
search node-1   30   200    78

The queue column shows the thread‑pool queue length; values near the configured queue_size (default 200) indicate saturation.

2. Review recent logs for rejection messages:

journalctl -u elasticsearch -f | grep -i "rejected execution"
2024-06-19T14:03:12.451Z [node-1] [WARN ][o.e.c.r.a.AllocationService] [node-1] failed to move shard [my-index][0] from [node-2] to [node-1] because node is overloaded
2024-06-19T14:03:13.102Z [node-1] [WARN ][o.e.b.BulkProcessor] [node-1] rejected execution of org.elasticsearch.action.bulk.BulkRequest

These messages match the “rejected execution of org.elasticsearch.action.bulk.BulkRequest” pattern reported in the GitHub issue #45312.

3. Capture network metrics to confirm latency spikes:

ss -i dst 10.0.0.5 | grep rtt
# Example output
    rtt: min 0.123/ avg 0.215/ max 0.250 ms

Latency spikes coincided with bulk request bursts from the on‑premise GPU nodes.

4. Check bulk processor configuration in the AI pipeline (Java example):

BulkProcessor bulkProcessor = BulkProcessor.builder(
    client,
    new BulkProcessor.Listener() { … })
    .setBulkActions(5000)
    .setBulkSize(new ByteSizeValue(5, ByteSizeUnit.MB))
    .setConcurrentRequests(4)   // default is 0
    .setBackoffPolicy(BackoffPolicy.exponentialBackoff(TimeValue.timeValueMillis(100), 5))
    .build();

The default concurrentRequests of 0 means only one bulk request is in flight, which can cause bursts when the network recovers from a jitter event.

Resolution

The fix involved three coordinated changes: thread‑pool tuning, bulk processor back‑off adjustment, and a shard‑allocation safeguard.

1. Increase bulk thread‑pool queue and enable rejection handling

Before:

# /etc/elasticsearch/elasticsearch.yml
thread_pool.bulk.queue_size: 200

After (applied via cluster update API to avoid node restart):

curl -XPUT "https://es-prod.example.com/_cluster/settings" -H 'Content-Type: application/json' -d '
{
  "persistent": {
    "thread_pool.bulk.queue_size": 1000,
    "thread_pool.bulk.size": 30,
    "thread_pool.bulk.type": "fixed"
  }
}'

Raising the queue to 1000 gives the cluster headroom to absorb temporary spikes without immediate 429 rejections.

2. Tighten bulk processor back‑off and increase concurrent requests

Before:

BulkProcessor.builder(...).setConcurrentRequests(0).build();

After:

BulkProcessor bulkProcessor = BulkProcessor.builder(
    client,
    new BulkProcessor.Listener() { … })
    .setBulkActions(5000)
    .setBulkSize(new ByteSizeValue(10, ByteSizeUnit.MB))
    .setConcurrentRequests(4)               // allow up to 4 in‑flight bulks
    .setBackoffPolicy(BackoffPolicy.exponentialBackoff(TimeValue.timeValueMillis(200), 8))
    .build();

The increased concurrentRequests smooths bursts, while a longer exponential back‑off (200 ms base, 8 retries) aligns with the latency spikes observed in the incident (#52784).

3. Prevent shard relocation during peak ingest

Before:

# No explicit allocation block
cluster.routing.allocation.enable: all

After (set during ingest windows):

curl -XPUT "https://es-prod.example.com/_cluster/settings" -H 'Content-Type: application/json' -d '
{
  "transient": {
    "cluster.routing.allocation.enable": "primaries"
  }
}'

Limiting allocation to primary shards stops costly relocations that would otherwise add load to the indexing thread‑pool.

Validation

After applying the changes, the following checks confirmed resolution:

1. Bulk request success rate – monitor bulk_requests_total and bulk_requests_rejected via the Elasticsearch metrics endpoint. Values dropped from ~12 % rejection to <0.1 % over the next 30 minutes.
2. Thread‑pool health – re‑run the _cat/thread_pool queries:

curl -s -XGET "https://es-prod.example.com/_cat/thread_pool/bulk?v&h=id,active,queue,rejected"
curl -s -XGET "https://es-prod.example.com/_cat/thread_pool/search?v&h=id,active,queue,rejected"

Sample output post‑fix:

bulk   node-1   8   45   0
search node-1   12  30   0

Application logs – no longer see es_rejected_execution_exception entries.

End‑to‑end pipeline test – ingest 15 k documents/second for 5 minutes; all bulk responses return HTTP 200, and downstream training jobs receive the expected data without back‑pressure.

Prevention and Best Practices

• Monitor thread‑pool queues – set alerts on thread_pool.*.rejected and queue length metrics to catch saturation before 429 responses appear.
• Implement adaptive bulk back‑off – use the Elasticsearch Java BulkProcessor’s exponential back‑off or equivalent logic in other clients.
• Size bulk thread‑pool for peak load – calculate required queue_size as (peak_bulk_rate * average_bulk_latency) / (number_of_nodes * thread_pool.bulk.size).
• Guard shard allocation during ingest windows – use transient cluster settings to disable relocations when write throughput is critical.
• Network resilience – deploy a small buffer (e.g., a local Kafka topic) between on‑premise producers and Elasticsearch to absorb WAN jitter without flooding the bulk endpoint.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

1. Why does Elasticsearch return 429 for both search and bulk APIs?
   Because both APIs share separate thread‑pools that enforce back‑pressure. When their queues fill, Elasticsearch rejects new requests with es_rejected_execution_exception, which the HTTP layer maps to 429.

2. Can increasing the number of data nodes eliminate 429 errors?
   Adding nodes raises the overall thread‑pool capacity, but if the bulk request rate exceeds the aggregate processing capability (or if network latency creates bursts), 429s can still appear. Tuning thread‑pool sizes and client back‑off is required in addition to scaling.

3. How do I know whether the limit is due to search throttling or indexing throttling?
   Check the reason field in the error payload. “rejected execution of org.elasticsearch.action.bulk.BulkRequest” indicates indexing throttling; “rejected execution of org.elasticsearch.action.search.SearchRequest” points to search throttling. Corresponding thread‑pool stats (_cat/thread_pool/bulk vs. _cat/thread_pool/search) confirm the source.

4. Is the 429 response affected by the Elasticsearch Service “write block” setting?
   Yes. When the cluster enters a write block (cluster.blocked with reason “write”), all indexing requests are rejected with 429. The block is often triggered by low disk watermarks or heavy shard relocation.

5. What back‑off strategy should I use for the BulkProcessor?
   An exponential back‑off starting at 100–200 ms with 5–8 retries works well for bursty WAN conditions. The policy should be long enough to let the bulk thread‑pool drain but short enough to keep pipeline latency acceptable.