Elasticsearch Node OOM During Concurrent Vector Search And Embedding Ingestion

Q: 2. Lower the request circuit‑breaker limit for vector operations

Reduce the request breaker to 30 % of the heap and increase the fielddata breaker to 60 % to give vector searches a dedicated headroom.

Q: 3. Tune bulk indexing size and refresh interval

Limit bulk request size to 2 MB and set refresh_interval to -1 during ingestion bursts.

Q: 5. Enable the knn plugin’s native memory circuit breaker (if using the plugin)

When using the knn plugin, set its own breaker to 25 % of the heap.

In this article

Table of Contents

Problem – OOM and CPU Saturation Under Concurrent Vector Search & Embedding Ingestion

In a production Retrieval‑Augmented Generation (RAG) service the Elasticsearch cluster receives >15 k requests / second. Each request either:

Executes a k‑nn similarity search on a dense_vector field.
Streams newly generated embeddings via bulk indexing.

During peak traffic the following symptoms appear on the hot data nodes:

JVM heap usage climbs to 95 % and stays there.
Frequent “stop‑the‑world” GC pauses (e.g. [gc][0] GC(1234) Pause Young (Normal) (G1 Evacuation Pause) 12.5ms).
CPU load average > 12 on 8‑core machines, with [node][I] high CPU usage detected, load average: 12.5 in logs.
Search thread‑pool rejections: [thread_pool][search][rejected] rejected execution.
CircuitBreaker exceptions: CircuitBreakingException: request is too large, data too large.
Eventually the node crashes with java.lang.OutOfMemoryError: Java heap space.

Root Cause – Memory‑Intensive Vector Operations + Unchecked Bulk Ingestion

The combination of two resource‑heavy patterns exhausts the JVM heap:

Vector field memory pressure: Each dense_vector (or knn_vector) is stored in the fielddata cache and also participates in the request circuit breaker. When many concurrent k‑nn searches run, the per‑search memory allocation spikes, quickly hitting the default request breaker limit (60 % of the heap).
Bulk embedding ingestion: The ingestion pipeline streams embeddings in large bulk requests (often >10 MB). Without throttling, each bulk request allocates temporary buffers for source parsing, Lucene segment merges, and vector field indexing. The indexing performance guide warns that bulk sizes >5 MB can cause heap churn, especially with vector fields.
Thread‑pool saturation: The default search thread pool queue length (1000) and size (calculated as #cores * 3) are insufficient for 15 k QPS. Requests pile up, causing longer GC windows and more concurrent memory allocations.
Mis‑aligned heap sizing: The JVM heap was set to 30 GB on a 64 GB machine, which is above the recommended 50 % of physical RAM. This leaves the OS page cache starved, increasing swap pressure and further degrading GC.

In short, the node runs out of heap because vector search and bulk indexing both allocate large, short‑lived objects, and the default circuit‑breaker and thread‑pool limits do not protect the heap under this load.

Debug – Investigation Steps

Collect heap and GC metrics:
```
curl -s http://localhost:9200/_nodes/stats/jvm?pretty
```
Look for heap_used_percent consistently > 90 % and long young_gc_time values.

Search for circuit‑breaker logs:

grep -i "CircuitBreakingException" /var/log/elasticsearch/elasticsearch.log

Typical entry:

2024-07-31T12:45:02,123 [node-1] [search] [I] [circuit_breaker] [request] request is too large, data too large; 
limit: 6.2gb usage: 6.5gb

Inspect thread‑pool rejections:
```
curl -s http://localhost:9200/_cat/thread_pool/search?v
```
High rejected count confirms saturation.
Capture a short packet trace of a bulk request (optional):
```
tcpdump -i eth0 -s 0 -w bulk.pcap port 9200 and src host gateway_ip
```
Verify payload size (>10 MB) and request rate.
Review index mapping for vector fields:
```
curl -s http://localhost:9200/my-index/_mapping?pretty
```
Confirm use of knn_vector with dimension set correctly and index_options not overly aggressive.
Check heap size configuration:
```
cat /etc/elasticsearch/jvm.options | grep -iXms
```
Typical line: -Xms30g and -Xmx30g.

Solution – Tuning Heap, Circuit Breakers, Thread Pools, and Bulk Ingestion

The fix consists of four coordinated changes.

1. Resize JVM heap to 50 % of physical RAM

On a 64 GB host set the heap to 32 GB (still within the 50 % guideline) and leave the rest for the OS page cache.

# /etc/elasticsearch/jvm.options
-Xms32g
-Xmx32g

2. Lower the request circuit‑breaker limit for vector operations

Reduce the request breaker to 30 % of the heap and increase the fielddata breaker to 60 % to give vector searches a dedicated headroom.

# /etc/elasticsearch/elasticsearch.yml
indices.breaker.total.limit: 70%          # default
indices.breaker.request.limit: 30%        # new
indices.breaker.fielddata.limit: 60%      # new

3. Tune bulk indexing size and refresh interval

Limit bulk request size to 2 MB and set refresh_interval to -1 during ingestion bursts.

# Example bulk client configuration (Java High‑Level REST Client)
BulkProcessor bulkProcessor = BulkProcessor.builder(
    client::bulkAsync,
    (request, bulkListener) -> {})
    .setBulkActions(5000)               // ~2 MB per bulk
    .setConcurrentRequests(4)
    .setFlushInterval(TimeValue.timeValueSeconds(5))
    .setBulkSize(new ByteSizeValue(2, ByteSizeUnit.MB))
    .build();

# Index settings update
PUT /my-index/_settings
{
  "index": {
    "refresh_interval": "-1",
    "translog.flush_threshold_size": "512mb"
  }
}

4. Expand the search thread pool and reduce queue size

Allocate a larger pool (8 × cores) and a shorter queue to force back‑pressure earlier.

# /etc/elasticsearch/elasticsearch.yml
thread_pool.search.size: 64          # 8 * 8‑core node
thread_pool.search.queue_size: 200   # lower than default 1000

5. Enable the `knn` plugin’s native memory circuit breaker (if using the plugin)

When using the knn plugin, set its own breaker to 25 % of the heap.

# /etc/elasticsearch/elasticsearch.yml
knn.memory.circuit_breaker.limit: 25%

After applying the changes, restart the node:

systemctl restart elasticsearch

Verify – Confirming the Fix

Heap health:

curl -s http://localhost:9200/_nodes/stats/jvm?pretty | jq '.nodes[].jvm.mem.heap_used_percent'

Expected: steady 55‑70 % under load.

GC pause duration (via node.stats or jstat):
```
jstat -gcutil $(pgrep -f elasticsearch) 1000 5
```
Young GC < 5 ms, no Full GC spikes.

No circuit‑breaker violations:

grep -i "CircuitBreakingException" /var/log/elasticsearch/elasticsearch.log || echo "none"

Search thread‑pool rejections remain at zero:

curl -s http://localhost:9200/_cat/thread_pool/search?v | awk '{print $5}'

Throughput:
Run a synthetic load test (e.g., vegeta) targeting 15 k QPS and observe latency < 100 ms and no OOM.

Prevent – Operational Guardrails and Ongoing Monitoring

Set up a watcher alert on heap_used_percent > 80 % and on indices.breaker.request.tripped count.
Enable search.throttled metrics and alert when thread_pool.search.rejected > 0.
Cap bulk request size at 2 MB in all ingestion services; enforce via CI linting of client configs.
Periodically run GET /_nodes/stats/indices?filter_path=indices.segments.memory_in_bytes to track vector field memory growth.
Consider off‑loading vector search to a dedicated k‑NN node pool with larger heap and isolated circuit‑breaker settings.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does the request circuit breaker trigger only during peak ingestion?
Because each bulk request creates temporary in‑heap buffers for the vector field; the cumulative size exceeds the default 60 % limit only when many bulk requests run concurrently.
Can I keep the heap larger than 50 % of RAM without OOM?
Larger heaps increase GC pause times. With vector‑heavy workloads the GC cost outweighs the extra memory, so staying near the 50 % guideline is recommended.
Is reducing the search queue size safe?
A smaller queue forces the client to back‑off earlier, preventing unbounded memory growth. The client should implement retry with exponential back‑off.
Do I need to change the knn index settings (e.g., ef_search)?
Adjusting ef_search can reduce per‑search memory, but the primary OOM driver is bulk ingestion. Tune ef_search only after stabilizing heap usage.
How do I monitor vector field memory specifically?
Use the indices.breaker.fielddata and indices.breaker.request stats, or query GET _nodes/stats/indices?filter_path=indices.segments.memory_in_bytes for the knn_vector field.