GPT-4 streaming response truncates after disaster recovery failover

Problem – Intermittent Truncation of GPT‑4 Streaming Responses During DR Failover

During a planned disaster‑recovery (DR) exercise the OpenAI GPT‑4 API was accessed via the secondary cloud region. Clients that used the stream=true endpoint began receiving truncated payloads: the HTTP connection closed after a few tokens, leaving the response incomplete. The issue was intermittent, appearing only when traffic was routed through the failover path and latency spiked above 300 ms.

Root Cause – Latency‑Induced HTTP/2 Flow‑Control Exhaustion in the Failover Proxy

The OpenAI service front‑ends are exposed behind an internal HTTP/2 reverse‑proxy that enforces per‑stream flow‑control windows. In the primary region the proxy is co‑located with the model inference nodes, keeping round‑trip latency < 100 ms and allowing the default flow‑control window (64 KB) to be replenished quickly.

When traffic is rerouted to the DR region the proxy sits in a different availability zone, adding ~250 ms of network latency. The increased round‑trip time delays WINDOW_UPDATE frames, causing the client’s flow‑control window to be exhausted after a few hundred bytes of token data. The proxy then aborts the stream with RST_STREAM (ENHANCE_YOUR_CALM), which the client interprets as a normal connection close, resulting in a truncated response.

Debug – Step‑by‑Step Investigation

1. Reproduce the symptom with a minimal client

import requests, json, time

def stream_chat(prompt):
    url = "https://api.openai.com/v1/chat/completions"
    headers = {"Authorization": "Bearer $OPENAI_API_KEY"}
    payload = {
        "model": "gpt-4",
        "messages": [{"role": "user", "content": prompt}],
        "stream": True
    }
    with requests.post(url, headers=headers, json=payload, stream=True, timeout=30) as r:
        for line in r.iter_lines():
            if line:
                data = json.loads(line.decode())
                print(data["choices"][0]["delta"].get("content", ""), end="", flush=True)

stream_chat("Explain the difference between TCP and UDP.")

Running this client against the primary endpoint (api.openai.com) returns the full answer. When the DNS entry is overridden to the DR endpoint (api-dr.openai.com) the stream stops after ~5 tokens.

2. Capture HTTP/2 frames with nghttp2

# Install nghttp2 client
sudo apt-get install nghttp2-client

# Record a stream against the DR endpoint
nghttp -v -H "Authorization: Bearer $OPENAI_API_KEY" \
    -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}],"stream":true}' \
    https://api-dr.openai.com/v1/chat/completions > dr_trace.log 2>&1

Key excerpt from dr_trace.log:

[  0.000] C: SETTINGS (0x4) ack=0
[  0.001] S: SETTINGS (0x4) ack=0
[  0.002] C: HEADERS: :method: POST
[  0.002] C: HEADERS: :path: /v1/chat/completions
...
[  0.312] S: DATA (len=65536)   # first window of tokens
[  0.313] C: WINDOW_UPDATE (increment=65536)
[  0.620] S: DATA (len=65536)   # second window
[  0.621] C: WINDOW_UPDATE (increment=65536)
[  0.928] S: RST_STREAM (error=0x0b ENHANCE_YOUR_CALM)

The RST_STREAM appears after ~130 KB of data, matching the exhaustion of the client’s flow‑control window.

3. Verify latency impact with ping and traceroute

# Ping the DR endpoint
ping -c 5 api-dr.openai.com
# Approx. 260 ms avg RTT

# Traceroute to see extra hops
traceroute api-dr.openai.com

4. Check proxy metrics (if accessible)

# Assuming Prometheus endpoint is exposed
curl -s http://proxy-dr.internal/metrics | grep http2_flow_control_window
# Output:
http2_flow_control_window{region="dr"} 65536

Solution – Increase HTTP/2 Flow‑Control Window and Adjust Timeout Settings

The fix consists of two coordinated changes:

  1. Proxy configuration: raise the per‑stream flow‑control window from 64 KB to 256 KB for the DR region.
  2. Client library: enable automatic window replenishment and increase the read timeout to accommodate higher latency.

Proxy configuration change (before / after)

# Before (proxy.conf)
http2 {
    stream_window_size 65536;   # 64 KB
    max_concurrent_streams 1000;
}
# After (proxy.conf) – applied only to DR region
http2 {
    stream_window_size 262144;  # 256 KB
    max_concurrent_streams 2000;
}

After reloading the proxy (nginx -s reload or equivalent), the flow‑control window is large enough to survive the additional RTT without requiring immediate WINDOW_UPDATE frames.

Client‑side adjustment (before / after)

# Before – default requests timeout
response = requests.post(url, headers=headers, json=payload, stream=True, timeout=30)
# After – extended timeout and explicit stream handling
session = requests.Session()
adapter = requests.adapters.HTTPAdapter(pool_maxsize=10, max_retries=3)
session.mount('https://', adapter)

response = session.post(
    url,
    headers=headers,
    json=payload,
    stream=True,
    timeout=(10, 120)   # (connect, read) seconds
)

# Ensure the client reads and acknowledges WINDOW_UPDATE
for chunk in response.iter_content(chunk_size=8192):
    # Process chunk...
    pass

Verify – Confirm Full Streaming Responses After the Fix

Functional verification

python verify_stream.py
# Expected output: complete answer without premature termination

Sample successful run (truncated output omitted for brevity):

... The main differences between TCP and UDP are:
- TCP provides reliable, ordered delivery...
- UDP is connectionless and does not guarantee delivery...

Metric validation

# Query proxy metrics again
curl -s http://proxy-dr.internal/metrics | grep http2_flow_control_window
# Output:
http2_flow_control_window{region="dr"} 262144

Additionally, monitor the http2_stream_errors_total counter; it should stay at zero for the DR region.

Prevent – Operational Guardrails to Avoid Recurrence

  • Dynamic flow‑control tuning: implement a controller that adjusts stream_window_size based on observed RTT (e.g., window = 4 * RTT * bandwidth_estimate).
  • Latency‑aware health checks: configure the load balancer to route streaming traffic to a region only if average RTT < 200 ms, otherwise fall back to non‑streaming mode.
  • Client timeout policy: enforce a minimum read timeout of 2 × expected RTT for all streaming calls.
  • Observability: add alerts on http2_stream_errors_total and on sudden spikes in client_window_exhausted metrics.
  • Testing: include a DR‑region streaming test in the CI pipeline that simulates high latency using tc qdisc add dev eth0 root netem delay 250ms.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  1. Why does the stream work in the primary region but fail in DR?
    The primary proxy is co‑located with the inference nodes, keeping RTT low enough for the default 64 KB flow‑control window. In DR the added latency prevents timely WINDOW_UPDATE frames, exhausting the window.
  2. Can I disable HTTP/2 to avoid flow‑control issues?
    Disabling HTTP/2 forces HTTP/1.1, which does not use per‑stream flow‑control, but it incurs higher overhead and removes multiplexing benefits. Adjusting the window size is the preferred solution.
  3. Is there a risk of increasing the window size too much?
    Very large windows can increase memory pressure on the proxy and client. 256 KB is a safe compromise for typical token payloads; larger values should be tested under load.
  4. How do I know if my client library respects WINDOW_UPDATE?
    Most modern HTTP/2 libraries (e.g., hyper, nghttp2, requests with urllib3) handle it automatically. Verify by inspecting library source or enabling debug logging (export HTTP2_DEBUG=1).
  5. What if the DR region experiences even higher latency (e.g., >500 ms)?
    Consider falling back to non‑streaming mode for that region or increasing the window further (up to 1 MB). Additionally, implement a latency‑based routing rule to avoid streaming over excessively delayed paths.