Haystack DocumentRetriever context window overflow with long documents

Debug – Investigation Steps

Follow these reproducible steps to isolate the overflow source:

1. Inspect retrieved document sizes in logs.

INFO haystack.retriever.elasticsearch - Retrieved document id: 7f9c3a - length: 12457 tokens

If the length exceeds the configured max_seq_len, the chunking step is faulty.

2. Check the index mapping for the text field.

curl -X GET "localhost:9200/my-index/_mapping?pretty"

Confirm that the text field is stored as text (not keyword) and that no ignore_above limit is truncating data.

3. Validate the splitter configuration.

from haystack.nodes import PreProcessor

preprocessor = PreProcessor(
    split_by="sentence",
    split_length=500,          # tokens, not characters
    split_respect_sentence_boundary=True,
    max_seq_len=4096
)

Print a sample chunk size:

chunks = preprocessor.process(documents=[my_doc])
print(f"Chunk 0 token count: {len(chunks[0].content.split())}")
# Expected: <= 4096

4. Measure the effective prompt size after retrieval.

from haystack.pipelines import ExtractiveQAPipeline
pipeline = ExtractiveQAPipeline(reader, retriever)

result = pipeline.run(query="What is the main conclusion?", params={"Retriever": {"top_k": 5}})
prompt = pipeline.get_prompt()
print(f"Prompt token count: {len(prompt.split())}")

If the prompt exceeds the model limit, reduce top_k or enable truncation.

Solution – Fixing the Overflow

1. Enforce Correct Chunk Size at Ingestion

Update the preprocessing step to respect the model’s max_seq_len. Example for a 4 k‑token model:

# Before (problematic)
preprocessor = PreProcessor(split_by="sentence", split_length=8000)

# After (fixed)
preprocessor = PreProcessor(
    split_by="sentence",
    split_length=3500,            # leave headroom for prompt tokens
    split_respect_sentence_boundary=True,
    max_seq_len=4096
)

Re‑index the documents after applying the new splitter. This guarantees each stored chunk is ≤ 4 096 tokens.

2. Configure max_seq_len in the DocumentStore

Set the limit explicitly so the retriever can warn or truncate oversized entries:

from haystack.document_stores import ElasticsearchDocumentStore

document_store = ElasticsearchDocumentStore(
    host="localhost",
    index="reports",
    embedding_dim=768,
    max_seq_len=4096,          # matches LLM context window
    similarity="cosine"
)

If existing documents exceed this limit, the store will raise ElasticsearchDocumentStoreError. Use the clean_documents utility to delete or re‑chunk them.

3. Limit Aggregated Prompt Size

Adjust the retriever’s top_k and enable on‑the‑fly truncation:

retriever = ElasticsearchRetriever(
    document_store=document_store,
    top_k=3,                       # fewer documents
    return_embedding=False,
    max_seq_len=4096,
    truncate=True                  # cut excess tokens per document
)

When truncate=True, Haystack will slice each retrieved document to the configured max_seq_len before concatenation, preventing the final prompt from exceeding the model limit.

4. Optional: Use a “Hybrid” Retriever for Large Corpora

Combine sparse BM25 with dense embeddings and set a lower top_k for the dense part. This reduces the chance of pulling a single massive chunk.

from haystack.nodes import BM25Retriever, EmbeddingRetriever

bm25 = BM25Retriever(document_store=document_store, top_k=5)
dense = EmbeddingRetriever(document_store=document_store, top_k=2, embedding_model="sentence-transformers/all-MiniLM-L6-v2")

# In pipeline, chain both retrievers and merge results.

Verification – Confirming the Fix

1. Run a test query and inspect the logged token counts.

INFO haystack.retriever.elasticsearch - Retrieved document length: 3820 tokens, within limit 4096
INFO haystack.pipeline - Prompt token count: 3995 (OK for 4096‑token model)

response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[{"role": "user", "content": prompt}]
)
print(response.choices[0].message.content)   # succeeds

Prevention – Operational Guardrails

• Static analysis of chunk size: Add a CI step that loads a sample document, runs the pre‑processor, and asserts len(chunk.tokens) <= max_seq_len.
• Monitoring alerts: Emit a custom metric (haystack.retriever.oversized_chunks) whenever a retrieved document length exceeds a threshold; alert on a rate > 0.
• Automated re‑chunking job: Periodically scan the Elasticsearch index for documents where _source.text_token_count > max_seq_len and re‑process them with the correct splitter.
• Configuration as code: Store max_seq_len, split_length, and top_k in a version‑controlled YAML file; enforce consistency across dev, staging, and prod.
• Document size logging: Include the token count in every retrieval log line (as shown above) to quickly spot regressions.

FAQ – Common Follow‑Up Questions

1. Why does the error only appear after the pipeline was updated to a newer LLM?
   The new model has a smaller context window (e.g., 4 096 vs. 8 192 tokens). Existing chunks that were previously acceptable now exceed the limit. Update max_seq_len and re‑chunk accordingly.
2. Can I keep large documents intact and let the retriever truncate them at query time?
   Yes, set truncate=True on the ElasticsearchRetriever. The retriever will slice each document to max_seq_len before concatenation, but note that truncation may cut off relevant information; explicit chunking is preferred for deterministic results.
3. How do I know the exact token count of a stored chunk?
   Haystack stores the token count in the meta field if you enable preprocessor.tokenizer. You can query it directly:

GET reports/_search
{
  "_source": ["text", "meta.token_count"],
  "query": { "match_all": {} }
}

Related Topic Hub: RAG Systems Troubleshooting Hub

Related Articles

• Elasticsearch context window overflow with large document indices