LangChain token limit exceeded during continuous real-time streaming

Problem

When streaming user messages into a LangChain chain that generates live responses, the application intermittently throws a TokenLimitExceededError. The exception originates from the underlying LLM provider (e.g., OpenAI, Anthropic) and aborts the current generation, causing a visible pause or failure in the UI.

Typical log excerpt:

2024-10-05 14:23:11,842 - app.chain - ERROR - TokenLimitExceededError: Prompt exceeds max token limit (8192) for model gpt-4o-mini
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/langchain/chains/base.py", line 215, in _call
    result = self.run(**inputs)
  File "/usr/local/lib/python3.11/site-packages/langchain/chains/conversation/base.py", line 102, in run
    response = self.llm_chain.run(**combined_inputs)
  File "/usr/local/lib/python3.11/site-packages/langchain/chains/llm.py", line 78, in run
    return self.llm(**full_prompt)
  File "/usr/local/lib/python3.11/site-packages/langchain/llms/openai.py", line 312, in __call__
    raise TokenLimitExceededError(message)
langchain.exceptions.TokenLimitExceededError: Prompt exceeds max token limit (8192) for model gpt-4o-mini

Root Cause

The chain uses a ConversationBufferMemory (or similar) that appends every incoming user message and every generated assistant reply to the prompt. In a continuous streaming scenario, the memory grows without bound. Each new turn adds the raw text plus the system prompt and any chain‑level instructions, eventually pushing the total token count past the model’s context window.

Key contributing factors:

• Unbounded memory retention: No truncation strategy is applied, so the prompt accumulates all historic turns.
• Token mis‑estimation: The code relies on len(text) (character count) to decide when to prune, which underestimates token usage for multi‑byte characters and tokenizers.
• Streaming latency: The application buffers several partial user messages before sending them to the LLM, inflating the prompt size further.

Debug

Step‑by‑step investigation that reproduced the failure:

1. Inspect memory size after each turn.

def log_memory_size(memory):
    prompt = memory.load_memory_variables({})["history"]
    token_count = len(memory.llm.get_tokenizer().encode(prompt))
    logger.info(f"Memory token count: {token_count}")

Sample output before failure:

2024-10-05 14:22:58,101 - app.chain - INFO - Memory token count: 7850
2024-10-05 14:23:04,317 - app.chain - INFO - Memory token count: 8023
2024-10-05 14:23:10,532 - app.chain - INFO - Memory token count: 8195

The third log line shows the count exceeding the 8192‑token limit.

2. Validate token counting method by comparing naive len(text.split()) against the tokenizer.

# Naive count (incorrect)
naive = len(prompt.split())
# Correct count (OpenAI tokenizer)
from tiktoken import encoding_for_model
enc = encoding_for_model("gpt-4o-mini")
correct = len(enc.encode(prompt))
logger.info(f"Naive={naive}, Correct={correct}")

Result:

Naive=7800, Correct=8195

3. Capture a packet trace to ensure the request payload size matches the token count.

# Using tcpdump on the outbound HTTPS traffic
sudo tcpdump -i any -s 0 -w llm_req.pcap port 443 and host api.openai.com

Inspecting the capture confirmed a payload of ~65 KB, consistent with a >8k token prompt.

Solution

The fix consists of three coordinated changes:

1. Switch to a sliding‑window memory implementation

Replace ConversationBufferMemory with ConversationSummaryBufferMemory (or a custom SlidingWindowMemory) that retains only the most recent N turns or a token budget.

# Before (unbounded buffer)
from langchain.memory import ConversationBufferMemory
memory = ConversationBufferMemory(return_messages=True)

# After (bounded sliding window)
from langchain.memory import ConversationSummaryBufferMemory
memory = ConversationSummaryBufferMemory(
    llm=ChatOpenAI(model="gpt-4o-mini"),
    max_token_limit=6000,   # leave headroom for new user input + system prompt
    return_messages=True,
)

2. Use the LLM’s tokenizer to enforce the limit

Implement a helper that trims the oldest turns until the token count is below a safe threshold.

def trim_memory(memory, llm, max_total_tokens=7500):
    enc = llm.get_tokenizer()
    history = memory.load_memory_variables({})["history"]
    tokens = enc.encode(history)
    while len(tokens) > max_total_tokens:
        # Remove the oldest user‑assistant pair
        memory.chat_memory.messages.pop(0)   # user message
        memory.chat_memory.messages.pop(0)   # assistant reply
        history = memory.load_memory_variables({})["history"]
        tokens = enc.encode(history)

3. Guard the streaming pipeline

Before sending a new chunk to the chain, invoke the trimming logic and log the resulting token count.

def stream_handler(chunk):
    # Append chunk to the conversation
    memory.chat_memory.add_user_message(chunk)
    # Ensure we stay under the limit
    trim_memory(memory, llm)
    # Run the chain
    response = chain.run()
    return response

Verify

After deploying the changes, repeat the earlier logging steps:

2024-10-05 15:01:12,004 - app.chain - INFO - Memory token count: 6423
2024-10-05 15:01:18,219 - app.chain - INFO - Memory token count: 6580
2024-10-05 15:01:24,433 - app.chain - INFO - Memory token count: 6725

The token count stays comfortably below the 8192 limit even after dozens of turns.

Additional verification:

• Run an end‑to‑end load test that streams 200 messages at 1 msg/s; monitor for any TokenLimitExceededError in the logs.
• Inspect the HTTP request size with tcpdump; payload should remain under ~50 KB.
• Execute a functional test that asserts the assistant’s reply is returned for the 200th message without error.

Prevent

Operational safeguards to avoid recurrence:

• Metric collection: Export memory_token_count to Prometheus and set an alert at 7500 tokens.
• Health check endpoint: Return OK only if the current token count is below the safe threshold.
• Configuration flag: Make max_token_limit an environment variable so it can be tuned per deployment.
• Automated tests: Include a unit test that simulates 100 streaming turns and asserts the token count never exceeds the limit.
• Documentation: Clearly note that any custom memory implementation must respect the LLM’s context window.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

1. Why does the error appear only after several minutes of streaming?
   Because the memory accumulates each turn. The first few dozen messages stay under the limit; once the cumulative token count crosses the model’s context window, the LLM rejects the request.
2. Can I increase the model’s context window instead of trimming?
   Some providers offer larger windows (e.g., 32 k tokens). Switching models can raise the ceiling, but the underlying problem—unbounded memory—remains and will eventually hit the new limit.
3. Is ConversationSummaryBufferMemory always safe?
   It summarizes older turns into a compact representation, which reduces token usage dramatically. However, the summarizer itself consumes tokens, so you still need a hard token budget and trimming logic for edge cases.
4. How do I calculate the token budget for a new model?
   Use the model’s documented max_tokens (e.g., 8192) and reserve at least 15‑20 % for the incoming user message, system prompt, and response. Set max_token_limit = model_max_tokens * 0.8 as a safe starting point.
5. My logs still show occasional spikes above the threshold—what should I do?
   Enable a secondary guard that aborts the request early if len(enc.encode(prompt)) > model_max_tokens - safety_margin. Return a graceful error to the client and trigger a retry after trimming.