Haystack Query Latency Spikes After Bulk Document Indexing

Q: 1. Examine Elasticsearch segment count

curl -s -XGET 'http://es-prod:9200/my-index/_segments?human' | jq '.indices["my-index"].total' Sample output before bulk indexing:

Q: 2. Check fielddata cache pressure

curl -s -XGET 'http://es-prod:9200/_nodes/stats/indices/fielddata?human' | \ jq '.nodes[].indices.fielddata.memory_size_in_bytes' Typical values:

Q: 3. Correlate with Haystack logs

2024-09-12 14:03:21,874 DEBUG haystack.query Query took 8.342 seconds: SELECT ... FROM ... WHERE ... 4. Verify refresh activity curl -s -XGET 'http://es-prod:9200/my-index/_stats?human' | jq '.indices["my-index"].total.refresh.total' Refresh count jumps from 120 to 1 200 during bulk indexing.

Q: 1. Adjust refresh_interval for bulk loads

Set a higher refresh interval (e.g., 30s) before the bulk job and restore it afterward.

Q: 2. Trigger a forced merge

Run a FORCE_MERGE with max_num_segments=1 once the bulk load finishes. This consolidates the many tiny segments into a few large ones.

Q: 3. Limit fielddata cache and use doc values

Change the mapping for fields that are only used for sorting/aggregations to use doc_values instead of loading fielddata into heap.

In this article

Table of Contents

Problem

After a bulk indexing run of several hundred thousand documents, the Haystack SearchQuerySet latency spikes dramatically. The 95th‑percentile response time, which normally sits around 0.8 s, climbs to 5 s–12 s for simple term queries. The issue appears only after the indexing job finishes; during indexing the query latency remains low.

Typical symptoms observed in production:

API gateway logs show GET /search?q=... taking > 5 s.
Haystack logs contain repeated DEBUG haystack.query: Query took X.XXX seconds entries.
CPU usage on the Elasticsearch node stays around 30 % while memory usage spikes to 85 %.
Prometheus metrics for http_request_duration_seconds{handler="search"} show a right‑skewed distribution after each bulk index.

Root Cause

The latency spike is caused by Elasticsearch segment merging and fielddata cache pressure triggered by the bulk indexing operation:

Segment proliferation – The bulk API creates many small Lucene segments. Until a merge runs, each query must search across thousands of segments, increasing per‑shard query time.
Fielddata cache exhaustion – Haystack’s default text fields are mapped as keyword for aggregations. After indexing, the fielddata cache fills with the newly added terms, causing eviction and GC pauses.
Refresh interval – The index’s refresh_interval is set to 1s. During bulk indexing the refreshes happen after every 5 k documents, forcing Elasticsearch to open a new segment and write a new translog, which further inflates segment count.

Combined, these factors make the query path traverse a large number of tiny segments and contend for fielddata memory, resulting in the observed latency spikes.

Debug

Below is the step‑by‑step investigation that confirmed the root cause.

1. Examine Elasticsearch segment count

curl -s -XGET 'http://es-prod:9200/my-index/_segments?human' | jq '.indices["my-index"].total'

Sample output before bulk indexing:

{
  "count": 12,
  "memory_in_bytes": 10485760,
  "terms_memory_in_bytes": 2097152,
  "stored_fields_memory_in_bytes": 5242880,
  "term_vectors_memory_in_bytes": 0,
  "norms_memory_in_bytes": 0,
  "points_memory_in_bytes": 0,
  "doc_values_memory_in_bytes": 0,
  "index_writer_memory_in_bytes": 0,
  "version_map_memory_in_bytes": 0,
  "fixed_bit_set_memory_in_bytes": 0,
  "max_unsafe_auto_id_timestamp": -1,
  "file_sizes": {}
}

After bulk indexing (immediately before a merge):

{
  "count": 842,
  "memory_in_bytes": 524288000,
  "terms_memory_in_bytes": 104857600,
  ...
}

2. Check fielddata cache pressure

curl -s -XGET 'http://es-prod:9200/_nodes/stats/indices/fielddata?human' | \
jq '.nodes[].indices.fielddata.memory_size_in_bytes'

Typical values:

Before indexing: 150 MB
After indexing: 1.8 GB (near the fielddata.cache.size limit of 2 GB)

3. Correlate with Haystack logs

2024-09-12 14:03:21,874 DEBUG haystack.query Query took 8.342 seconds: SELECT ... FROM ... WHERE ...

4. Verify refresh activity

curl -s -XGET 'http://es-prod:9200/my-index/_stats?human' | jq '.indices["my-index"].total.refresh.total'

Refresh count jumps from 120 to 1 200 during bulk indexing.

Solution

The fix consists of three coordinated changes: tune the refresh interval, force a controlled merge after indexing, and limit fielddata usage.

1. Adjust `refresh_interval` for bulk loads

Set a higher refresh interval (e.g., 30s) before the bulk job and restore it afterward.

# Before bulk indexing
curl -XPUT 'http://es-prod:9200/my-index/_settings' -H 'Content-Type: application/json' -d '
{
  "index": {
    "refresh_interval": "30s",
    "translog.flush_threshold_size": "512mb"
  }
}'

After the bulk job completes, reset to the production value (1s).

# After bulk indexing
curl -XPUT 'http://es-prod:9200/my-index/_settings' -H 'Content-Type: application/json' -d '
{
  "index": {
    "refresh_interval": "1s"
  }
}'

2. Trigger a forced merge

Run a FORCE_MERGE with max_num_segments=1 once the bulk load finishes. This consolidates the many tiny segments into a few large ones.

curl -XPOST 'http://es-prod:9200/my-index/_forcemerge?max_num_segments=1&flush=true'

Monitor the _cat/segments API to confirm the segment count drops back to <20.

3. Limit fielddata cache and use doc values

Change the mapping for fields that are only used for sorting/aggregations to use doc_values instead of loading fielddata into heap.

# Before (problematic)
PUT my-index/_mapping
{
  "properties": {
    "category": { "type": "text" }
  }
}

After applying keyword with doc_values:

# After (fixed)
PUT my-index/_mapping
{
  "properties": {
    "category": {
      "type": "keyword",
      "doc_values": true
    }
  }
}

If the index already exists, reindex into a new index with the corrected mapping, then alias the new index to the original name.

4. Update Haystack query configuration (optional)

Disable unnecessary highlight or facets on high‑traffic endpoints to reduce fielddata usage.

# settings.py
HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch5_backend.Elasticsearch5SearchEngine',
        'URL': 'http://es-prod:9200/',
        'INDEX_NAME': 'my-index',
        'KWARGS': {
            'settings': {
                'highlight': False,
                'max_result_window': 10000
            }
        }
    },
}

Verify

After applying the changes, perform the following checks:

Segment count – Should be <20 per shard.

curl -s 'http://es-prod:9200/_cat/segments/my-index?v' | wc -l

Fielddata usage – Should stay below 25 % of the configured cache.

curl -s 'http://es-prod:9200/_nodes/stats/indices/fielddata?human' | \
jq '.nodes[].indices.fielddata.memory_size_in_bytes'

Query latency – Run a representative query set and capture percentiles.
```
ab -n 500 -c 20 "http://api-prod/search?q=machine+learning"
```
Expected 95th percentile <2 s.

Refresh interval restored – Verify the setting is back to 1s.

curl -s 'http://es-prod:9200/my-index/_settings' | jq '.["my-index"].settings.index.refresh_interval'

Prevent

To avoid recurrence, embed the following guardrails into the CI/CD pipeline and operational runbooks:

Guardrail	Implementation
Automated segment health check	Run a nightly `_cat/segments` query; alert if total segments per shard > 100.
Fielddata cache monitoring	Prometheus rule: `es_fielddata_memory_bytes{job="es"} > 0.75 * es_fielddata_cache_size_bytes`.
Refresh interval tuning during bulk loads	Wrap bulk jobs with a pre‑step that sets `refresh_interval` to `30s` and a post‑step that restores it and triggers `FORCE_MERGE`.
Mapping hygiene	Enforce `keyword` + `doc_values` for aggregatable fields via schema validation scripts.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

Why does query latency improve after a manual FORCE_MERGE but revert after the next bulk load?
The bulk load creates many small segments again. Without resetting refresh_interval and merging, the segment count grows, reproducing the latency issue.
Can I keep refresh_interval high permanently to avoid this problem?
No. A high refresh interval delays document visibility for end users. The recommended pattern is to increase it only for the duration of the bulk operation and restore the low‑latency setting afterward.
My fielddata cache still spikes even after changing mappings to keyword. What else can I do?
Disable doc_values loading for fields not used in aggregations, and set indices.breaker.fielddata.limit to a safe percentage (e.g., 40 %). Also, review any custom Haystack SearchQuerySet that requests .facet(...) on large text fields.
Is there a way to automate the merge step without impacting query availability?
Yes. Use the max_num_segments parameter with a value of 5 and schedule the merge during low‑traffic windows. Elasticsearch will perform the merge in the background without blocking reads.
How can I detect that a refresh is causing excessive segment creation before it impacts latency?
Monitor indices.refresh.total and indices.refresh.time_in_millis. A sudden increase in refresh.total combined with a rising segments.count metric is an early indicator.