LlamaIndex Token Limit Exceeded During Model Evaluation

Q: Investigation & Debugging Steps 1. Reproduce the error with a minimal query

python evaluate.py --query "Explain the warranty policy for product X" Observe the traceback; note the token count reported in the exception.

Q: 2. Inspect the built prompt

# Example snippet inside evaluate.py prompt = query_engine._build_prompt(query) print("=== PROMPT START ===") print(prompt) print("=== PROMPT END ===") print(f"Prompt token count: {len(tokenizer.encode(prompt))}") Typical output (truncated):

Q: 3. Verify retriever settings

print(query_engine.retriever.similarity_top_k) # e.g., 20 print(query_engine.retriever.max_chunks_per_query) # default None 4. Check model limits import openai model = "gpt-3.5-turbo" print(openai.Model.retrieve(model).max_context_length) # 4096 5. Review community reports GitHub issue #2124 highlights adjusting max_input_size and chunk_size. Issue #1987 recommends RecursiveCharacterTextSplitter to enforce hierarchical chunking. Stack Overflow answer (78543219) suggests lowering chunk_size and applying metadata_filter to prune unused fields. Solution – Reducing Prompt Size Below Model Limits 1. Configure hierarchical chunking Split large documents into a two‑level hierarchy: first coarse chunks (~10 k tokens) then fine‑grained chunks (~2 k tokens). LlamaIndex provides RecursiveCharacterTextSplitter for this pattern.

Q: 2. Limit retrieved chunks per query

# before: unlimited retrieval retriever = index.as_retriever(similarity_top_k=20) # after: cap at 5 chunks and enforce token budget retriever = index.as_retriever( similarity_top_k=20, max_chunks_per_query=5, # optional: filter out metadata fields that are not needed metadata_filter=lambda md: {"title": md.get("title")} ) 3. Trim metadata in the prompt # before: default node prompt includes full metadata dict def node_prompt(node): return f"[Document: {node.metadata}] {node.text}" # after: keep only essential fields def node_prompt(node): title = node.metadata.get("title", "Untitled") return f"[{title}] {node.text[:1500]}..." # truncate long text if needed 4. Set max_input_size on the LLM chain # before: default (no limit) llm_chain = LLMChain(llm=OpenAI(model="gpt-3.5-turbo")) # after: enforce model token ceiling llm_chain = LLMChain( llm=OpenAI(model="gpt-3.5-turbo"), max_input_size=3800, # leave headroom for response tokens ) 5. Enable fallback truncation If the prompt still exceeds the limit, LlamaIndex can automatically drop the lowest‑scoring chunks.

In this article

Table of Contents

Problem – Token Limit Exceeded During Model Evaluation with LlamaIndex

When running automated evaluation of a large document corpus through LlamaIndex, the evaluation script aborts with errors such as:

openai.error.InvalidRequestError: This model's maximum context length is 4096 tokens
ValueError: Prompt exceeds max_input_size (got 5273 tokens, max allowed 4096)
RuntimeError: Token limit exceeded while building prompt – consider reducing chunk size or number of retrieved nodes

The failure occurs during the RetrieverQueryEngine / LLMChain step where the retrieved chunks are concatenated with the user prompt. The result is an incomplete or missing answer, causing evaluation metrics (e.g., recall, exact‑match) to be skewed.

Root Cause Analysis

LlamaIndex builds a prompt by stitching together:

the original query string
metadata (source IDs, titles, etc.)
the text of each retrieved node (chunk)

Each component consumes tokens from the LLM’s context window. The following factors commonly push the total token count over the model limit:

Chunk size too large – default chunk_size (e.g., 512 characters) can still translate to ~800 tokens for dense text, especially for scientific or code documents.
Unbounded number of retrieved chunks – the retriever may return dozens of nodes for a single query if similarity_top_k is high or max_chunks_per_query is unset.
Metadata bloat – including full file paths, timestamps, and raw JSON in the prompt inflates token usage.
Model context limit mismatch – using a model with a 4 k token ceiling (e.g., gpt‑3.5‑turbo) while the prompt builds to >5 k tokens.

These observations align with the official LlamaIndex documentation on Chunking & Token Management and the Prompt Optimizations guide, which both warn that “the total prompt length must stay below the model’s max token limit.”

Investigation & Debugging Steps

1. Reproduce the error with a minimal query

python evaluate.py --query "Explain the warranty policy for product X"

Observe the traceback; note the token count reported in the exception.

2. Inspect the built prompt

# Example snippet inside evaluate.py
prompt = query_engine._build_prompt(query)
print("=== PROMPT START ===")
print(prompt)
print("=== PROMPT END ===")
print(f"Prompt token count: {len(tokenizer.encode(prompt))}")

Typical output (truncated):

=== PROMPT START ===
You are a helpful assistant...

[Document: manual_001.txt] Warranty starts on purchase date...
[Document: manual_045.txt] Warranty covers...
...
=== PROMPT END ===
Prompt token count: 5273

3. Verify retriever settings

print(query_engine.retriever.similarity_top_k)   # e.g., 20
print(query_engine.retriever.max_chunks_per_query)  # default None

4. Check model limits

import openai
model = "gpt-3.5-turbo"
print(openai.Model.retrieve(model).max_context_length)  # 4096

5. Review community reports

GitHub issue #2124 highlights adjusting max_input_size and chunk_size.
Issue #1987 recommends RecursiveCharacterTextSplitter to enforce hierarchical chunking.
Stack Overflow answer (78543219) suggests lowering chunk_size and applying metadata_filter to prune unused fields.

Solution – Reducing Prompt Size Below Model Limits

1. Configure hierarchical chunking

Split large documents into a two‑level hierarchy: first coarse chunks (~10 k tokens) then fine‑grained chunks (~2 k tokens). LlamaIndex provides RecursiveCharacterTextSplitter for this pattern.

# before: default SimpleNodeParser
from llama_index import SimpleNodeParser
parser = SimpleNodeParser(chunk_size=512)

# after: hierarchical splitter
from llama_index.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
    chunk_sizes=[10000, 2000],   # first level 10k, second level 2k
    chunk_overlap=200,
)
parser = SimpleNodeParser(text_splitter=splitter)

2. Limit retrieved chunks per query

# before: unlimited retrieval
retriever = index.as_retriever(similarity_top_k=20)

# after: cap at 5 chunks and enforce token budget
retriever = index.as_retriever(
    similarity_top_k=20,
    max_chunks_per_query=5,
    # optional: filter out metadata fields that are not needed
    metadata_filter=lambda md: {"title": md.get("title")}
)

3. Trim metadata in the prompt

# before: default node prompt includes full metadata dict
def node_prompt(node):
    return f"[Document: {node.metadata}] {node.text}"

# after: keep only essential fields
def node_prompt(node):
    title = node.metadata.get("title", "Untitled")
    return f"[{title}] {node.text[:1500]}..."   # truncate long text if needed

4. Set `max_input_size` on the LLM chain

# before: default (no limit)
llm_chain = LLMChain(llm=OpenAI(model="gpt-3.5-turbo"))

# after: enforce model token ceiling
llm_chain = LLMChain(
    llm=OpenAI(model="gpt-3.5-turbo"),
    max_input_size=3800,   # leave headroom for response tokens
)

5. Enable fallback truncation

If the prompt still exceeds the limit, LlamaIndex can automatically drop the lowest‑scoring chunks.

query_engine = RetrieverQueryEngine.from_args(
    retriever=retriever,
    llm=OpenAI(model="gpt-3.5-turbo"),
    max_input_size=3800,
    truncate_prompt=True,   # drops excess chunks
)

Verification – Confirming the Fix

Run a single evaluation query and capture the prompt token count.
Expect Prompt token count < 3800 (or the model’s limit minus reserved response tokens).
Check that the response is complete (no truncation warnings in logs).
Run the full evaluation suite and confirm that no InvalidRequestError or Token limit exceeded exceptions appear.

Sample successful log excerpt:

2026-06-02 14:32:11 INFO  evaluate.py: Query executed successfully
Prompt token count: 3421
Response token count: 512
Evaluation metric: exact_match=0.84, recall=0.91

Prevention – Operational Guardrails

Monitor prompt length – expose a metric llamaindex_prompt_tokens and alert when it approaches 90 % of the model limit.
Enforce a global max_input_size in all LLMChain/QueryEngine instances; treat it as a required configuration parameter.
Standardize chunking policy across projects – store the chosen chunk_sizes in a shared config file.
Strip non‑essential metadata before indexing; keep only fields required for downstream prompts.
Automated smoke test – run a quick “prompt size sanity check” after any index rebuild.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

Why does the error appear only after a certain number of queries?
Because the retriever caches node scores; as more queries run, higher‑scoring nodes accumulate and the default max_chunks_per_query (None) lets the engine pull an ever‑larger set of chunks, eventually exceeding the token ceiling.
Can I increase the model’s context window instead of shrinking the prompt?
Only if you switch to a model with a larger limit (e.g., gpt‑4‑32k). The same token‑budget logic still applies; you must adjust max_input_size accordingly.
Is there a way to automatically prune metadata fields?
Yes. Provide a metadata_filter callable to the retriever or customize the node prompt builder to select only needed keys.
What is the recommended chunk_size for dense technical documents?
Empirically, a two‑level split of 10 k → 2 k tokens works well for 4 k‑token models, as demonstrated in the enterprise QA pipeline incident.
How do I know the exact token count of a prompt before sending it?
Use the same tokenizer the LLM uses (e.g., tiktoken.encoding_for_model("gpt-3.5-turbo")) and call len(encoding.encode(prompt)) in your code.