img IX mining rig inside white and gray room

LlamaIndex context window overflow after recursive retrieval

by

Problem Statement

In a high‑throughput Retrieval‑Augmented Generation (RAG) service built on LlamaIndex, recursive retrieval combined with hybrid search (BM25 + embeddings) caused the assembled prompt to exceed the target LLM’s context window. The overflow manifested as:

  • OpenAI API error: 400 Bad Request - This model's maximum context length is 8192 tokens
  • LlamaIndexError: Prompt exceeds max token limit (requested: 10500, limit: 8192)
  • Truncated responses and occasional “Prompt exceeds max token length” warnings in the application logs.

The failure occurs sporadically under load (200 + QPS) when the RecursiveRetriever returns dozens of document chunks and metadata fields are concatenated without any token‑budget check.

Technical Background

LlamaIndex builds a prompt by concatenating:

  1. System and user messages.
  2. Retrieved Node objects (text + optional metadata).
  3. Optional summaries generated by SummaryIndex.

The PromptBuilder respects the max_input_size parameter (see LlamaIndex Documentation – Retrieval & Query Pipelines). If the total token count exceeds the model’s max context size, the OpenAI endpoint rejects the request with a 400 error.

Recursive retrieval walks the graph of documents, fetching child nodes for each parent node. Without a token‑aware guard, each hop adds a variable number of tokens, and metadata expansion (e.g., source, timestamps, scores) can inflate the payload by ~40% as observed in production logs.

Root Cause Analysis

The overflow originates from three interacting issues:

Component Assumption Failure Mode
RecursiveRetriever Each hop returns a bounded number of nodes. Top‑k is static (e.g., top_k=10) but each node may spawn k children, leading to exponential growth.
Hybrid Search BM25 and embedding scores produce a fixed set of chunks. Combined results are merged without deduplication, yielding 30+ chunks per query (see real incident “Hybrid BM25 + embedding retrieval returned 30+ document chunks”).
Metadata Concatenation Metadata is optional and small. When metadata_mode='all' is enabled, each node’s metadata string adds ~15 tokens, inflating the prompt by ~40% (see batch inference incident).

Because the PromptBuilder does not perform a pre‑flight token count when max_input_size is omitted, the assembled prompt can reach 12 k tokens, exceeding the 8192‑token limit of gpt‑3.5‑turbo (OpenAI API Reference). The overflow triggers the 400 Bad Request errors observed in logs.

Investigation and Debugging Steps

  1. Reproduce the error locally with a deterministic query that triggers recursion.
  2. Enable LlamaIndex debug logging to capture token counts.
    3. import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger("llama_index")
logger.setLevel(logging.DEBUG)
  3. Inspect the assembled prompt length before the API call.
    4. from llama_index.prompts.base import PromptBuilder

builder = PromptBuilder(...)
prompt = builder.build_prompt(...)
token_count = builder.llm.get_num_tokens(prompt)
logger.debug(f"Prompt token count: {token_count}")
  4. Capture a sample prompt that caused the failure.
    5. --- Prompt start ---
System: You are an assistant...
User: ...
--- Retrieved nodes (12) ---
[Node 1 text] ... Metadata: source=doc1, score=0.92
[Node 2 text] ... Metadata: source=doc2, score=0.87
...
--- Prompt end ---
  5. Measure token contribution of each part using tiktoken or the LLM’s get_num_tokens method.
    6. import tiktoken
enc = tiktoken.encoding_for_model("gpt-3.5-turbo")
def tokens(text): return len(enc.encode(text))

text_tokens = tokens(node.text)
meta_tokens = tokens(str(node.metadata))
print(f"text: {text_tokens}, meta: {meta_tokens}")
  6. Check recursive depth and top‑k values.
    7. print(f"Retriever depth: {retriever.max_depth}")
print(f"Top‑k per hop: {retriever.top_k}")
  7. Confirm model limits from the OpenAI documentation.
    8. # gpt-3.5-turbo: 8192 tokens
# gpt-4: 8192 tokens (or 32768 for 32k variant)

Resolution – Token‑Aware Retrieval Pipeline

The fix consists of three coordinated changes:

1. Dynamic Top‑K based on token budget

Compute the remaining token budget after accounting for system prompts and then set retriever.top_k accordingly.

# Desired maximum tokens for the model
MAX_TOKENS = 8192
# Reserve tokens for system/user messages and response buffer
RESERVED = 1024

def compute_budget(prompt_base):
    base_tokens = llm.get_num_tokens(prompt_base)
    return MAX_TOKENS - RESERVED - base_tokens

def adjust_top_k(retriever, budget, avg_node_tokens=150):
    # Conservative estimate: each node ≈ avg_node_tokens tokens
    max_nodes = budget // avg_node_tokens
    retriever.top_k = max(1, min(retriever.top_k, max_nodes))

2. Token‑aware truncation with TokenTextSplitter

Replace raw node concatenation with a splitter that respects the remaining budget.

from llama_index.text_splitter import TokenTextSplitter

splitter = TokenTextSplitter(
    chunk_size=300,   # tokens per chunk
    chunk_overlap=20,
    tokenizer=llm.get_tokenizer()
)

def truncate_nodes(nodes, budget):
    truncated = []
    used = 0
    for node in nodes:
        node_tokens = llm.get_num_tokens(node.text)
        if used + node_tokens > budget:
            # Split the node to fit the budget
            parts = splitter.split_text(node.text)
            for part in parts:
                part_tokens = llm.get_num_tokens(part)
                if used + part_tokens > budget:
                    break
                truncated.append(part)
                used += part_tokens
            break
        else:
            truncated.append(node.text)
            used += node_tokens
    return truncated

3. Metadata pruning

Only include essential metadata fields; drop verbose entries.

# Before
metadata = {"source": doc_id, "timestamp": ts, "score": score, "full_text": raw_text}

# After (pruned)
metadata = {"source": doc_id, "score": round(score, 2)}

4. Set max_input_size explicitly

Enforce the limit at the PromptBuilder level.

builder = PromptBuilder(
    llm=llm,
    max_input_size=MAX_TOKENS - RESERVED,
    tokenizer=llm.get_tokenizer()
)

5. Guard against recursion explosion

Limit recursion depth and enforce a hard node count cap.

retriever = RecursiveRetriever(
    index=vector_index,
    max_depth=2,          # stop after two hops
    node_limit=15        # absolute cap across all hops
)

Validation – Confirming the Fix

  1. Run a load test (e.g., hey -n 1000 -c 200) against the RAG endpoint.
  2. Monitor logs for the absence of “Prompt exceeds max token limit” messages.
  3. Check the final token count per request.
    4. INFO Prompt token count: 7850
INFO Request succeeded (status=200)
  4. Verify that the LLM responses are no longer truncated and that latency remains within SLA (< 500 ms for prompt assembly).

Operational Experience & Lessons Learned

  • Misleading symptom: The 400 Bad Request error initially appeared as a network issue; only after enabling LlamaIndex debug logs did the token overflow surface.
  • Common incorrect assumption: “Setting a static top_k guarantees a bounded prompt.” In recursive pipelines the effective node count multiplies.
  • Production edge case: During peak load, the hybrid retriever returned 30+ chunks because BM25 and embedding results were merged without deduplication. Adding a set‑based dedup step reduced chunks by ~40%.
  • Lesson: Always compute the token budget **after** adding system prompts and before any recursive expansion.

Best Practices & Prevention

  • Define a MAX_TOKENS constant per model and reserve a safety margin (≈10‑15%).
  • Use TokenTextSplitter or PromptOptimizer (LlamaIndex “Prompt Optimizer” guide) for any variable‑size content.
  • Enable metadata_mode='compact' unless full metadata is required.
  • Instrument metrics:
    • prompt_token_count
    • retrieved_node_count
    • recursion_depth
  • Set alerts on spikes of Prompt exceeds max token limit errors.
  • Automate a pre‑flight token estimation step in the query handler.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

  1. Why does the overflow only happen under high QPS?
    Because the hybrid retriever returns more chunks when the index cache is warm, and the static top_k is applied per request without token budgeting, causing occasional bursts that exceed the limit.
  2. Can I keep full metadata and still stay within the token budget?
    Yes, by moving metadata to a separate lookup (e.g., a key‑value store) and referencing it with a short identifier in the prompt instead of embedding the full string.
  3. Is max_input_size sufficient on its own?
    It prevents the PromptBuilder from emitting an oversized prompt, but you still need to adjust retrieval parameters (top‑k, depth) to keep the number of nodes within the budget.
  4. How does TokenTextSplitter differ from the regular TextSplitter?
    TokenTextSplitter counts tokens using the LLM’s tokenizer, ensuring that each chunk respects the model’s token limits, whereas TextSplitter works on character length only.
  5. What should I monitor to catch this issue early?
    Track prompt_token_count and alert when it exceeds 90% of the model’s limit; also monitor the rate of 400 Bad Request responses from the LLM endpoint.