Elasticsearch bulk API batch size exceeded during CI/CD data ingestion

Elasticsearch Bulk API Batch Size Exceeded During CI/CD Data Ingestion

Problem Description

During automated model‑training pipelines, a bulk indexing step fails with HTTP 413 or a BulkIndexError. Typical log excerpts include:

[2026-06-30T12:14:02,871][ERROR][o.e.b.BulkProcessor] [node-1] request size exceeds the configured limit
org.elasticsearch.ElasticsearchException: request size exceeds the configured limit
    at org.elasticsearch.http.netty4.Netty4HttpChannelHandler.handleRequest(Netty4HttpChannelHandler.java:215)
    ...
Caused by: java.lang.IllegalArgumentException: request size exceeds the configured limit

Other observed symptoms:

• CI job aborts after the first bulk call (e.g., GitLab CI, Jenkins, Azure DevOps).
• Metrics show a spike in thread_pool.bulk.rejected and http.total_requests counters.
• Only the largest batch fails; smaller batches succeed.

Root Cause Analysis

Elasticsearch enforces a hard limit on the HTTP request body size via the http.max_content_length setting (default 100mb). The Bulk API documentation states that any bulk payload larger than this limit is rejected with a 413 response (Bulk API docs).

In CI/CD pipelines the ingestion step often aggregates a full training dataset (hundreds of megabytes) into a single bulk request to reduce network round‑trips. When the aggregated size crosses the http.max_content_length threshold, Elasticsearch returns:

HTTP/1.1 413 Request Entity Too Large
{
  "error": {
    "type": "illegal_argument_exception",
    "reason": "request size exceeds the configured limit"
  },
  "status": 413
}

Two additional constraints can amplify the failure:

• Thread‑pool bulk queue saturation: thread_pool.bulk.queue_size defaults to 200. Parallel CI agents can flood the bulk thread pool, leading to 429 “Too many requests” errors (Cluster Settings).
• Client‑side bulk helpers (e.g., elasticsearch-py.helpers.bulk) often buffer a configurable number of actions before sending. If the chunk_size or max_chunk_bytes is set too high, a single request can exceed the server limit (Bulk Processor Best Practices).

Investigation and Debugging Steps

1. Confirm the exact error from Elasticsearch logs.

   
   journalctl -u elasticsearch | grep "request size exceeds"
   

   Expected output:

   
   Oct 12 03:22:14 node-1 elasticsearch[12345]: [WARN ][o.e.b.BulkProcessor] [node-1] request size exceeds the configured limit
   

2. Measure the bulk payload size generated by the CI job.

   
   # Assuming the CI job writes the bulk payload to a temporary file
   du -h bulk_payload.json
   # Example output:
   150M    bulk_payload.json
   

3. Inspect the client configuration. For Python:

   
   from elasticsearch import Elasticsearch, helpers
   
   es = Elasticsearch(
       ["https://es-prod.example.com:9200"],
       http_auth=("elastic", "****"),
       maxsize=25,
       timeout=30,
   )
   
   # Bulk helper defaults
   helpers.bulk(es, actions, chunk_size=5000)  # <-- may produce >100 MB chunks
   

4. Check server‑side limits.

   
   curl -s -XGET "https://es-prod.example.com:9200/_cluster/settings?include_defaults=true" | jq '.defaults.http.max_content_length'
   # Example output:
   "100mb"
   

5. Validate thread‑pool saturation.

   
   curl -s "https://es-prod.example.com:9200/_cat/thread_pool/bulk?v"
   # Output columns: active, queue, rejected, completed
   

   If queue frequently hits the queue_size limit (default 200), consider reducing parallelism.

Solution

The fix consists of two parts: adjust the bulk request size on the client side and, if necessary, tune the Elasticsearch server limits.

Client‑Side Adjustments

Split the payload into smaller chunks that stay comfortably below the http.max_content_length threshold (e.g., 50 MB). The elasticsearch‑py bulk helper supports max_chunk_bytes.

Before

helpers.bulk(es, actions, chunk_size=5000)  # may create >100 MB chunks

After

helpers.bulk(
    es,
    actions,
    chunk_size=2000,               # fewer docs per chunk
    max_chunk_bytes=50 * 1024 * 1024  # 50 MB limit per request
)

For Java or Logstash pipelines, set bulk_actions or bulk_size accordingly:

# Logstash output configuration
output {
  elasticsearch {
    hosts => ["es-prod.example.com:9200"]
    bulk_actions => 2000          # default 500
    bulk_size => "50mb"           # default 20mb
    flush_interval => "5s"
  }
}

Server‑Side Tuning (Optional)

If the ingestion volume cannot be reduced (e.g., a one‑shot data lake load), increase the limit safely:

# Increase to 200 MB (requires a rolling restart)
PUT /_cluster/settings
{
  "persistent": {
    "http.max_content_length": "200mb"
  }
}

Note: Raising the limit consumes more heap and network buffers; monitor jvm.heap_used_percent and network.inbound.bytes after the change.

For thread‑pool saturation, reduce parallel bulk workers in the CI job:

# Example: limit to 4 concurrent bulk processes
export BULK_CONCURRENCY=4
python ingest.py --workers $BULK_CONCURRENCY

Verification

1. Re‑run the CI job with the updated client settings. Expected log snippet:

   
   [2026-06-30T12:18:45,102][INFO ][o.e.b.BulkProcessor] [node-1] successfully indexed 2000 documents (45.2mb) in 1.2s
   

2. Confirm no 413 responses:

   
   curl -s -o /dev/null -w "%{http_code}" -XPOST "https://es-prod.example.com:9200/_bulk" --data-binary @small_chunk.json
   # Should output 200
   

3. Check bulk thread‑pool metrics:

   
   curl -s "https://es-prod.example.com:9200/_cat/thread_pool/bulk?v"
   # queue column should stay well below the configured limit
   

4. Validate data integrity:

   
   curl -s "https://es-prod.example.com:9200/my-index/_count"
   # Count should match the number of source documents
   

Prevention and Best Practices

• Adopt a safe bulk size range: 5 MB – 50 MB per request is recommended for most clusters (Bulk Processor docs).
• Make bulk size configurable per environment (e.g., CI vs. production) rather than hard‑coding.
• Monitor http.total_requests and thread_pool.bulk.rejected in your observability stack; trigger alerts when thresholds exceed 80 % of the configured limits.
• Implement exponential back‑off and retry for bulk calls that receive 429 or 413 responses.
• Keep http.max_content_length at a sane default (100 mb) and only increase after load‑testing with realistic data volumes.

FAQ

1. Why does the bulk request succeed locally but fail in the CI pipeline?

   Local runs often use a smaller dataset or a higher http.max_content_length on a dev cluster. CI pipelines typically aggregate the full training set, producing a payload that exceeds the production limit.

2. Can I increase http.max_content_length without restarting the cluster?

   No. Changing this setting requires a full node restart because it is an HTTP server parameter.

3. What is the relationship between thread_pool.bulk.queue_size and bulk size errors?

   When many parallel bulk requests are queued, Elasticsearch may reject new requests with 429. This is separate from the 413 size limit but can appear together if the CI job spawns many workers.

4. How do I calculate an appropriate max_chunk_bytes value?

   Measure the average document size (e.g., curl .../_search?size=0&filter_path=hits.total) and multiply by the desired number of docs per chunk. Keep the total under 80 % of http.max_content_length to allow overhead.

5. Is there a way to let Elasticsearch automatically split oversized bulk payloads?

   The server does not split requests. The client must perform chunking. Use the official bulk helper libraries or implement custom chunking logic.

Related Topic Hub: Data Infrastructure Troubleshooting Hub