Elasticsearch RAG retrieval component not mapping documents to answers

Problem – Retrieval‑Augmented Generation (RAG) Returns Empty or Incomplete Answers in Batch Jobs

In a production batch pipeline, dozens to thousands of queries are sent to Elasticsearch via the _msearch API. The retrieval component successfully runs, but the generation stage receives either an empty context field or a truncated list of documents. Consequently the final answers are missing critical information or are completely blank.

Typical symptoms observed in logs:

2026-05-30T12:14:07.842Z ERROR [rag-orchestrator] - No documents found for query id 7f3a9c12‑d4b1‑4e8a‑a9f2‑c5e6b7f9d3e1, skipping generation step
2026-05-30T12:14:07.845Z WARN  [batch-worker] - ElasticsearchException: search_phase_execution_exception: all shards failed
2026-05-30T12:14:07.846Z WARN  [batch-worker] - RejectedExecutionException: thread pool [search] is full
2026-05-30T12:14:07.850Z INFO  [rag-orchestrator] - Generated answer: "" (empty string)

These messages appear despite the index containing relevant knowledge‑base documents and the msearch request returning hits when executed individually.

Root Cause Analysis

1. Query Parameter Omission in Batch Requests

The official RAG guide (Elastic Documentation – RAG guide) shows a retrieval query that explicitly sets size (e.g., "size": 5) to limit the number of context documents. In several batch jobs the size field was omitted, causing Elasticsearch to fall back to the default size=10. When combined with a from offset inherited from a previous request, the effective window exceeded max_result_window, triggering the error:

Result window is too large, request [from + size] exceeds max_result_window

Because the error is thrown per‑shard, the orchestrator logs “all shards failed” and treats the response as a successful empty hit set.

2. Thread‑Pool Saturation Under High Concurrency

Batch pipelines often run msearch with a concurrency level that exceeds the default search thread pool size (defaults to node.max_local_storage_nodes * 3). When the pool is exhausted, Elasticsearch returns RejectedExecutionException: thread pool [search] is full. The RAG layer interprets the partial success as “no documents” for the affected queries.

3. Index Refresh Lag

Knowledge‑base updates are indexed just before the batch starts. With the default refresh interval of 1s, newly indexed documents may not be searchable during the first few msearch calls. This timing issue is documented in the ingest pipeline guide (Ingest pipelines) and explains why some queries receive zero hits while later ones succeed.

4. Scripted Scoring Misconfiguration

Some implementations use a script_score query to boost relevance based on custom metadata. A subtle syntax error (e.g., missing closing brace) in the search template causes the script to evaluate to null, which Elasticsearch treats as a filter that discards all documents. The RAG orchestrator logs “No documents found for query id …” even though the request syntax is otherwise valid.

Investigation and Debugging Steps

1. Reproduce with a single query. Run the same query that fails in the batch using curl or POST /index/_search. Verify that hits are returned.
2. Inspect the _msearch payload. Capture the raw HTTP body sent by the batch worker.
3. Check thread‑pool health. Use the Cat API:

curl -XGET 'http://es-node:9200/_cat/thread_pool/search?v&h=id,active,rejected,largest'

Look for non‑zero rejected counts.

4. Validate index refresh timing. Query the _refresh API before the batch starts:

curl -XPOST 'http://es-node:9200/_refresh'

Review search template syntax. Retrieve the stored template used for RAG retrieval:

curl -XGET 'http://es-node:9200/_scripts/rag_template?pretty'

Look for JSON parsing errors such as “Failed to parse query: Unexpected token”.

Examine Elasticsearch logs for shard failures. Example snippet from es.log:

2026-05-30 12:14:07,842 [search] [node-1] [WARN ][o.e.search.SearchPhaseController] [node-1] search_phase_execution_exception: all shards failed

Resolution – Making Retrieval Reliable in Batch RAG Pipelines

1. Explicitly Set size and Enforce max_result_window

Update the search template to always include a safe size (e.g., 5) and guard against large from values.

Before (template snippet):
{
  "size": "{{size}}",
  "query": { ... }
}

After (template snippet):
{
  "size": 5,
  "query": { ... }
}

2. Increase Search Thread‑Pool Capacity

Adjust the node settings in elasticsearch.yml or via the Cluster Update Settings API:

# elasticsearch.yml
thread_pool.search.size: 30
thread_pool.search.queue_size: 200

# API call
curl -XPUT 'http://es-node:9200/_cluster/settings' -H 'Content-Type: application/json' -d '
{
  "persistent": {
    "thread_pool.search.size": "30",
    "thread_pool.search.queue_size": "200"
  }
}'

Raising the pool prevents RejectedExecutionException under high concurrency.

3. Force Index Refresh Before Batch Starts

Integrate an explicit refresh step in the pipeline orchestration script:

# Bash snippet
curl -s -XPOST "http://es-node:9200/_refresh" && echo "Index refreshed"

Alternatively, set refresh_interval to -1 during bulk indexing and manually refresh once the bulk load completes.

4. Harden Search Template Validation

Enable strict JSON validation and add a fallback clause that logs the rendered query before execution.

# Example of a wrapper script in Python
import json, logging, requests

def render_template(template_id, params):
    resp = requests.get(f"http://es-node:9200/_scripts/{template_id}")
    tmpl = resp.json()['_source']['script']['source']
    rendered = tmpl.format(**params)
    try:
        json.loads(rendered)  # validates JSON
    except json.JSONDecodeError as e:
        logging.error(f"Template rendering failed: {e}")
        raise
    return rendered

Verification – Confirming the Fix

• Run a representative batch of _msearch requests and verify that each response contains a hits.total.value > 0 field.
• Check orchestrator logs for the absence of “No documents found for query id” messages.
• Monitor the search thread‑pool metrics; rejected count should stay at zero.
• Execute a downstream generation call and confirm that the answer includes the expected context snippets.

Sample successful log excerpt:

2026-05-31T03:02:15.123Z INFO  [rag-orchestrator] - Retrieved 4 documents for query id 9b2e1f4a‑c7d3‑4f8b‑a1e5‑d9f6c2b8e7a0
2026-05-31T03:02:15.456Z INFO  [generation] - Generated answer: "The user can reset their password by clicking the 'Forgot password' link on the login page..."

Prevention – Operational Guardrails

Guardrail	Implementation
Enforce query size limits	Include size in every search template; add CI lint that rejects templates missing the field.
Thread‑pool capacity monitoring	Set up Kibana alerts on node.search.rejected > 0 over a 5‑minute window.
Refresh awareness	Ingest pipeline emits a custom event after bulk indexing; downstream batch job subscribes and waits for the event before starting.
Template syntax validation	Run a pre‑deployment test that renders each stored script with a dummy payload and parses the result.
Result window safety	Configure index.max_result_window to a conservative value (e.g., 1000) and add a request‑validation step that aborts if from + size exceeds it.

FAQ – Common Follow‑Up Questions

1. Why does the retrieval work for a single query but fail in _msearch? In batch mode the request payload may omit required parameters (like size) or inherit pagination offsets from previous sub‑requests, leading to result‑window violations.
2. How can I tell which documents were actually sent to the generation model? Enable _source in the retrieval query and log the hits.hits._source array before passing it downstream. The RAG guide recommends using a script_fields processor to format the context.
3. What is the safest concurrency level for _msearch? Start with a concurrency of node.max_local_storage_nodes * 2 and monitor node.search.rejected. Adjust upward only after confirming that the thread‑pool queue remains low.
4. Can I avoid explicit refreshes and still see newly indexed docs? Set refresh_interval to -1 during bulk load, then call POST /_refresh once. This guarantees visibility without incurring the overhead of per‑request refreshes.
5. Is there a way to automatically retry queries that hit RejectedExecutionException? Wrap the _msearch call in a retry loop with exponential back‑off. Elasticsearch client libraries (e.g., the Java High Level REST Client) expose RetryableException that can be caught and retried.

Related Topic Hub: Data Infrastructure Troubleshooting Hub