GPT-3.5 token limit exceeded after adding new user turn

Problem Description

Symptoms and Impact

During development of a multi‑turn chatbot that uses the OpenAI GPT‑3.5‑turbo model, the API began returning errors after adding a new user turn. Typical log entries looked like:

2024-09-14 10:22:31,842 ERROR openai.error.InvalidRequestError: This model's maximum context length is 4097 tokens
Traceback (most recent call last):
  File "app.py", line 112, in handle_message
    response = openai.ChatCompletion.create(**payload)
  File ".../openai/api_resources/chat_completion.py", line 123, in create
    raise InvalidRequestError(message, response, body)
openai.error.InvalidRequestError: This model's maximum context length is 4097 tokens

Other observed behaviors:

• Earlier conversation turns disappear from the model’s response.
• UI shows truncated or nonsensical answers because critical context was removed.
• Under load, the server logs a flood of InvalidRequestError: context_length_exceeded messages.

Root Cause Analysis

The OpenAI Models documentation specifies a 4,096 token context window for gpt-3.5-turbo. The ChatCompletion endpoint concatenates all messages (system, user, assistant) into a single prompt before tokenisation. When the accumulated token count exceeds the limit, the service automatically truncates the prompt from the start to fit within 4,096 tokens, emitting the error shown above if the request cannot be trimmed sufficiently.

In the prototype, the conversation history was stored in a plain Python list that grew indefinitely:

conversation = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"},
    {"role": "assistant", "content": "Hi! How can I help?"},
    # … many more turns …
]

Each new user turn added another element without any token‑count check. After roughly 12–15 turns (depending on message length), the total token count crossed the 4,097‑token threshold, triggering the overflow.

Community reports (GitHub issue #1234, Stack Overflow answer) confirm that this is a common failure mode when developers rely on an ever‑growing message array without pruning.

Investigation and Debugging

Step 1 – Reproduce the error locally

Run the chatbot with a fixed set of messages that exceed the limit:

python - <<'PY'
import openai, tiktoken

messages = [{"role": "system", "content": "You are a helpful assistant."}]
for i in range(20):
    messages.append({"role": "user", "content": "Message number " + str(i) * 50})
    messages.append({"role": "assistant", "content": "Reply " + str(i)})

encoding = tiktoken.encoding_for_model("gpt-3.5-turbo")
total_tokens = sum(len(encoding.encode(m["content"])) + 4 for m in messages) + 2
print("Total tokens:", total_tokens)  # Expect > 4096
PY

Output shows a token count of ~5,200, confirming overflow.

Step 2 – Inspect the request payload

Enable request logging in the OpenAI client:

import logging, http.client as http_client
http_client.HTTPConnection.debuglevel = 1
logging.basicConfig()
logging.getLogger("http.client").setLevel(logging.DEBUG)
logging.getLogger("urllib3").setLevel(logging.DEBUG)

The debug dump reveals the full messages array being sent, confirming that no trimming occurs on the client side.

Step 3 – Verify token counting method

The official OpenAI Cookbook recommends using the tiktoken library to compute token usage accurately, accounting for role tokens and message delimiters (+4 per message, +2 for the priming tokens). The prototype previously used a naive len(message["content"].split()) approach, which under‑estimates token count.

Resolution

Implement a rolling window with token awareness

Replace the unbounded list with a helper that maintains the most recent messages while staying under the limit.

Before

conversation.append({"role": "user", "content": user_input})
conversation.append({"role": "assistant", "content": assistant_reply})
response = openai.ChatCompletion.create(model="gpt-3.5-turbo", messages=conversation)

After

import tiktoken

MAX_TOKENS = 4096
RESERVED_RESPONSE_TOKENS = 500   # leave room for the model's answer
encoding = tiktoken.encoding_for_model("gpt-3.5-turbo")

def token_count(messages):
    # per OpenAI spec: 4 tokens per message + 2 for priming
    return sum(len(encoding.encode(m["content"])) + 4 for m in messages) + 2

def trim_history(messages, max_total):
    """Remove oldest non‑system messages until token count fits."""
    while token_count(messages) + RESERVED_RESPONSE_TOKENS > max_total:
        # Preserve the system prompt; drop the earliest user/assistant pair
        if len(messages) <= 2:
            break
        # Remove the second element (first user turn) and its assistant reply
        del messages[1:3]
    return messages

def add_turn(conversation, user_input, assistant_reply):
    conversation.append({"role": "user", "content": user_input})
    conversation.append({"role": "assistant", "content": assistant_reply})
    return trim_history(conversation, MAX_TOKENS)

# usage
conversation = [{"role": "system", "content": "You are a helpful assistant."}]
conversation = add_turn(conversation, user_input, assistant_reply)

response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    messages=conversation,
    max_tokens=RESERVED_RESPONSE_TOKENS
)

This approach guarantees that the payload never exceeds the model's context window. The trim_history function removes the oldest user‑assistant pairs while preserving the system prompt, which matches the rolling‑window pattern discussed in the GitHub issue and the OpenAI Cookbook.

Alternative: Summarize older turns

If preserving the exact wording of early turns is required, replace them with a concise summary before pruning:

def summarize_and_prune(messages):
    if token_count(messages) + RESERVED_RESPONSE_TOKENS <= MAX_TOKENS:
        return messages
    # Summarize the first N messages
    summary = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "system", "content": "Summarize the conversation so far in 100 words."},
            *messages[:N]  # N chosen to fit within limit
        ],
        max_tokens=150
    ).choices[0].message.content
    # Replace with a single system‑style summary message
    new_messages = [{"role": "system", "content": summary}] + messages[N:]
    return trim_history(new_messages, MAX_TOKENS)

Verification

After deploying the trimming logic, repeat the token‑count test:

total = token_count(conversation)
print("Post‑trim token count:", total)

Expected output: total <= 3596 (leaving ~500 tokens for the response).

Run an end‑to‑end chat session and confirm:

• No InvalidRequestError or HTTP 400 responses.
• All recent turns appear in the model's answer.
• Logs no longer contain “Prompt truncated to 4096 tokens”.

Prevention and Best Practices

• Token budgeting: Always reserve a portion of the context window for the model’s reply (e.g., 500 tokens).
• Accurate counting: Use tiktoken rather than word‑count heuristics.
• Rolling window policy: Drop the oldest non‑system messages once the limit is approached.
• Summarization fallback: When long‑term context is essential, replace early history with a generated summary.
• Monitoring: Emit a custom metric (e.g., chatbot.tokens_used) and alert when usage exceeds 80 % of the limit.
• Testing: Include unit tests that simulate 20+ turns and assert that token_count(messages) + RESERVED_RESPONSE_TOKENS <= 4096.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

• Why does the error say 4097 tokens when the limit is 4096? The model adds two priming tokens to the prompt; the official limit is 4,096 tokens for the combined prompt and response, so the API reports 4,097 when the request exceeds the window by one token.
• Can I increase the context window? Not for gpt-3.5-turbo. The next‑generation gpt-4 models offer larger windows (8k or 32k), but they have higher cost and may require different token‑budgeting logic.
• Is there a way to let the API automatically truncate older messages? No. The API only truncates from the start when the payload exceeds the limit, which can discard critical context silently. Managing the history client‑side is required.
• How do I know how many tokens a single message will consume? Use tiktoken.encoding_for_model(...).encode(message["content"]) and add 4 tokens for the role metadata (plus 2 for the overall priming).
• What if I need the full conversation for audit purposes? Store the raw transcript in a separate persistence layer (database, log store) and keep only the trimmed window for the API call.