Token Limit Exceeded Error In Nginx During Real-time LLM Streaming

Q: 4. Adjust rate‑limit zone for token bursts

If limit_req is used on the streaming endpoint, raise the burst:

In this article

Table of Contents

Problem Description

A real‑time LLM streaming service proxies requests through Nginx (HTTP/2). When the model generates responses that exceed a few kilobytes of token data, the client receives a truncated stream and the Nginx error log contains entries such as:


2026/09/13 12:45:27 [error] 12#12: *12345 client intended to send too large body, request body size exceeds client_max_body_size (8k) while reading client request header, client: 10.12.34.56, server: ai.example.com, request: "POST /chat HTTP/2.0"
2026/09/13 12:45:27 [error] 12#12: *12345 http2 error code=PROTOCOL_ERROR, stream was reset, client: 10.12.34.56, server: ai.example.com
2026/09/13 12:45:27 [error] 12#12: *12345 upstream sent unexpected EOF while reading response header from upstream, client: 10.12.34.56, server: ai.example.com, request: "POST /chat HTTP/2.0"

Impact:

Clients see HTTP 502 or abrupt stream termination.
Conversation state is lost, requiring retries.
Service‑level objectives for latency and availability are violated.

Root Cause Analysis

The failure originates from several Nginx limits that were tuned for typical REST payloads but are insufficient for high‑throughput token streams:

client_max_body_size – limits the size of the request body. After a model update the average prompt size grew beyond the default 1 MiB, triggering the “client intended to send too large body” error (see NGINX Docs – “client_max_body_size”).
proxy_buffer_size / proxy_buffers – define how much of the upstream response Nginx buffers before forwarding. The default 4 KB buffer is far smaller than the ~8 KB JSON payloads observed in production (see NGINX Docs – “proxy_buffer_size” and the real incident where 8 KB responses caused 502 errors).
HTTP/2 flow‑control window – Nginx’s default window (64 KB) can be exhausted by a burst of tokens if buffering is disabled, leading to http2 error code=PROTOCOL_ERROR (see NGINX HTTP/2 Module documentation and Stack Overflow 73584212).
Request rate limiting (limit_req) – aggressive limit zones can treat a rapid token stream as a burst of requests, emitting “request exceeded the configured limit”.

In combination, these limits cause Nginx to abort the connection before the full LLM response reaches the client.

Investigation and Debugging

Step‑by‑step diagnostics that proved the above hypotheses:

Inspect error logs for the exact messages (shown above). The presence of both “client intended to send too large body” and “http2 error code=PROTOCOL_ERROR” indicates multiple limits being hit.
Capture a failing request with curl to reproduce the error locally:

curl -v -X POST https://ai.example.com/chat \
  -H "Content-Type: application/json" \
  --http2 \
  -d '@large_prompt.json' 2> curl.log

Typical excerpt from curl.log:

* HTTP/2 502 
* OpenSSL SSL_read: error:1408A0C1:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0

Check Nginx configuration values:

nginx -T | grep -E 'client_max_body_size|proxy_buffer|http2|limit_req'

Result shows defaults:

# client_max_body_size 1m;
proxy_buffer_size 4k;
proxy_buffers 8 4k;
http2_max_field_size 16k;
limit_req_zone $binary_remote_addr zone=one:10m rate=5r/s;

Monitor runtime metrics with nginx -s reload and curl -I to verify that the upstream (LLM) is sending the full payload (no 502 from upstream itself).
Packet capture (tcpdump) on the Nginx‑to‑upstream interface confirms that the upstream sends >64 KB of data in a single HTTP/2 frame, which exceeds the default flow‑control window.

Resolution

The fix consists of three adjustments:

1. Increase request body limit

Before:

# /etc/nginx/conf.d/ai.conf
server {
    listen 443 ssl http2;
    client_max_body_size 1m;   # default
    ...
}

After:

# /etc/nginx/conf.d/ai.conf
server {
    listen 443 ssl http2;
    client_max_body_size 10m;   # accommodate larger prompts
    ...
}

2. Disable response buffering for streaming endpoints

Before:

location /chat {
    proxy_pass http://llm_backend;
    proxy_buffering on;          # default
    proxy_buffer_size 4k;
    proxy_buffers 8 4k;
}

After (streaming mode):

location /chat {
    proxy_pass http://llm_backend;
    proxy_buffering off;         # turn off buffering
    proxy_request_buffering off; # avoid buffering request body
    proxy_http_version 1.1;      # keep‑alive for streaming
}

3. Raise HTTP/2 flow‑control window

Added to the http block:

http {
    ...
    # Increase the initial flow‑control window for each stream
    http2_max_window_size 256k;
    # Optional: raise header size limits if JSON payloads contain large fields
    large_client_header_buffers 4 16k;
}

4. Adjust rate‑limit zone for token bursts

If limit_req is used on the streaming endpoint, raise the burst:

limit_req_zone $binary_remote_addr zone=chat:10m rate=10r/s;
...
location /chat {
    limit_req zone=chat burst=20 nodelay;
    ...
}

After applying the changes, reload Nginx:

nginx -t && systemctl reload nginx

Why it works:

Increasing client_max_body_size prevents the request from being rejected before reaching the LLM.
Disabling buffering lets Nginx pipe the upstream response directly to the client, eliminating the 4 KB buffer overflow that caused “upstream sent unexpected EOF”.
Raising http2_max_window_size gives the client enough credit to accept large token bursts without triggering a PROTOCOL_ERROR.
Adjusting limit_req removes false positives on high‑throughput streams.

Validation

Confirm the fix with the following steps:

Run a curl request that previously failed:

curl -v -X POST https://ai.example.com/chat \
  -H "Content-Type: application/json" \
  --http2 \
  -d '@large_prompt.json' -o response.json

Check that the HTTP status is 200 and the response file contains the full JSON payload (e.g., >64 KB).
Inspect Nginx error log for the absence of the previous messages:

grep -iE "client intended|protocol_error|upstream sent" /var/log/nginx/error.log

Verify HTTP/2 flow‑control windows via nghttp -v or nghttp2 -c to ensure the new http2_max_window_size is advertised.
Run a load test (e.g., hey or wrk2) that streams 100 concurrent requests, each generating ~200 KB of tokens, and confirm no 502/504 errors appear.

Operational Experience

During the investigation we observed a few misleading symptoms:

The initial “client intended to send too large body” log suggested a request‑size problem, but the real blocker was the response buffering.
Disabling proxy_buffering introduced a subtle increase in upstream connection count; monitoring showed a 15 % rise in open sockets, which is acceptable after adjusting worker_connections from 1024 to 4096.
Only one of the three Nginx worker processes exhibited the HTTP/2 PROTOCOL_ERROR because the token burst timing aligned with its flow‑control window exhaustion. Raising the window globally solved the asymmetric behavior.

Best Practices and Prevention

Set client_max_body_size based on the maximum expected prompt size plus a safety margin.
For streaming endpoints, always use proxy_buffering off and proxy_request_buffering off to avoid hidden buffer limits.
Configure http2_max_window_size to at least 256 KB for LLM services that emit large token bursts.
Monitor Nginx metrics: nginx_http_requests_total, nginx_http_upstream_response_time_seconds, and HTTP/2 flow‑control window usage via custom exporter.
Implement alerting on log patterns such as “client intended to send too large body” or “http2 error code=PROTOCOL_ERROR”.
Periodically test with a payload that exceeds the current limits to verify that configuration changes remain effective after upgrades.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

Why does the error appear only after a model update?
The new model generates longer responses (more tokens). The existing Nginx buffers and HTTP/2 windows were sized for the previous output length, so the increased payload now exceeds those limits.
Can I keep proxy_buffering on and still stream?
Yes, but you must increase proxy_buffer_size and proxy_buffers to accommodate the maximum response size, and ensure proxy_busy_buffers_size is also raised. This adds latency and memory pressure, so disabling buffering is preferred for real‑time streams.
What is the relationship between client_max_body_size and proxy_request_buffering?
client_max_body_size limits the size of the request body accepted by Nginx. When proxy_request_buffering off, Nginx streams the request body directly to the upstream without buffering, but the size still cannot exceed client_max_body_size.
How do I know the correct http2_max_window_size value?
Measure the peak size of a single token burst (including JSON framing). Set the window to at least twice that value to provide headroom. A common starting point for LLM streams is 256 KB.
Do these changes affect other services behind the same Nginx instance?
Only the locations where the directives are overridden (e.g., /chat) are affected. Global settings like http2_max_window_size apply to all HTTP/2 streams, but the increase from the default 64 KB to 256 KB is safe for typical web traffic.