Hugging Face streamer connection timeout during token generation

Problem – Intermittent Streamer Time‑outs During Token Generation

When a FastAPI gateway proxies a request to a GPU‑hosted TextGenerationPipeline with streamer=True, the client receives a socket hang‑up or ReadTimeoutError after the first token (or sometimes before any token is emitted). The failure is reproducible under concurrent load and manifests as:

ReadTimeoutError: Server timed out while streaming tokens (client‑side timeout exceeded)
socket hang up
HTTPError 504: Gateway Timeout
StreamingGeneratorError: Timeout while waiting for next token chunk

Typical latency targets are sub‑500 ms time‑to‑first‑token (TTFT) and continuous <200 ms intervals between token chunks for an interactive UI.

Root Cause Analysis

The timeout originates from a chain of latency contributors that exceed the client‑side deadline:

• GPU warm‑up & memory pressure: When multiple requests arrive, the CUDA context may be re‑initialized or experience fragmentation, adding 200‑400 ms before the first token is generated (see the “GPU memory fragmentation” incident).
• Streamer internal queue: Streamer uses a bounded queue.Queue with a default timeout of 30 s. Under load, the generation loop can block on torch.cuda.synchronize(), causing the queue to stay empty longer than the client’s 400 ms deadline (GitHub issue #28045).
• Uvicorn/Starlette keep‑alive: The default keep‑alive is 30 s; however, the FastAPI StreamingResponse aborts if the generator does not yield within the server’s timeout (GitHub issue #30612).
• Ingress timeout: An NGINX ingress with proxy_read_timeout 500ms closes the SSE connection before the model can emit the next token under peak load (Kubernetes incident).
• Client‑side EventSource timeout: Browsers typically drop an EventSource after 30 s of inactivity, but custom JavaScript libraries may enforce a stricter 400 ms “read timeout”.

Collectively, these factors cause the streamer to miss the client‑side threshold, leading to the observed errors.

Investigation and Debugging Steps

1. Collect server logs around the failure.

   2026-08-05 12:14:32,874 INFO  uvicorn.access: 200 POST /generate 0.842ms
   2026-08-05 12:14:33,112 WARNING streaming: Streamer queue empty for 0.42s
   2026-08-05 12:14:33,514 ERROR uvicorn.error: StreamingGeneratorError: Timeout while waiting for next token chunk
   

2. Enable detailed transformer generation profiling.

   import logging, time
   logging.basicConfig(level=logging.DEBUG)
   start = time.time()
   outputs = pipeline(prompt, streamer=True, max_new_tokens=50)
   print("TTFT:", time.time() - start)
   

3. Inspect GPU utilization and CUDA synchronization latency.

   nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv -l 1
   

4. Capture the HTTP stream with tcpdump or ss to verify packet gaps.

   # Capture first 5 seconds of the SSE stream
   tcpdump -i eth0 -s 0 -w stream.pcap port 8000 and '(tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) = 0)'
   

5. Check FastAPI/Uvicorn timeout settings.

   # uvicorn command line
   uvicorn app:app --host 0.0.0.0 --port 8000 --timeout-keep-alive 5
   

6. Verify ingress timeout configuration.

   # NGINX snippet
   proxy_read_timeout 2s;
   proxy_send_timeout 2s;
   

7. Reproduce with a single request to isolate load‑related factors. If the issue disappears, the root cause is likely contention or timeout mis‑configuration.

Resolution – Making the Stream Robust Under Load

The fix consists of three layers: pipeline tuning, server‑side timeout adjustments, and ingress configuration.

1. Pipeline Tuning

Reduce warm‑up latency and guarantee a token is emitted quickly.

# Before
pipeline = pipeline(
    "text-generation",
    model="bigscience/bloom-560m",
    device=0,
    streamer=True,
    max_new_tokens=100,
)

# After – enable pre‑warm and adjust generation config
from transformers import TextGenerationPipeline, GenerationConfig, TextStreamer

generation_cfg = GenerationConfig(
    max_new_tokens=100,
    do_sample=False,
    temperature=0.0,
    pad_token_id=50256,
    eos_token_id=50256,
    # Reduce internal waiting time
    early_stopping=True,
)

streamer = TextStreamer(
    tokenizer=pipeline.tokenizer,
    skip_prompt=True,
    timeout=1.0,               # seconds – shorter than client deadline
)

pipeline = TextGenerationPipeline(
    model="bigscience/bloom-560m",
    device=0,
    generation_config=generation_cfg,
    streamer=streamer,
)

# Warm‑up once at startup
pipeline("warm up", max_new_tokens=1)

2. FastAPI / Uvicorn Timeout Adjustments

Increase the keep‑alive and generator timeout to accommodate occasional stalls.

# uvicorn launch (Docker entrypoint or systemd)
uvicorn app:app \
  --host 0.0.0.0 \
  --port 8000 \
  --timeout-keep-alive 10   # seconds, > client timeout
  --workers 4               # avoid single‑worker bottleneck

Update the StreamingResponse to set an explicit timeout on the generator:

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import asyncio

app = FastAPI()

async def token_stream(request: Request, prompt: str):
    generator = pipeline(prompt, streamer=True)
    try:
        async for chunk in generator:
            yield chunk
            # Reset client deadline after each token
            await asyncio.sleep(0)   # give control back to event loop
    except Exception as exc:
        # Log and re‑raise for visibility
        app.logger.error(f"Streaming error: {exc}")
        raise

@app.post("/generate")
async def generate(request: Request):
    data = await request.json()
    prompt = data["prompt"]
    return StreamingResponse(token_stream(request, prompt), media_type="text/event-stream")

3. Ingress / Load Balancer Timeout Alignment

Match the ingress read timeout to the maximum expected token interval plus a safety margin.

# NGINX Config (Ingress Controller)
location /generate {
    proxy_pass http://backend-service:8000;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_read_timeout 5s;   # > client 400 ms + worst‑case generation delay
    proxy_send_timeout 5s;
}

4. Optional: Use a Dedicated Streaming Worker

Run a separate process (e.g., Gunicorn with --worker-class uvicorn.workers.UvicornWorker) for streaming endpoints, isolating them from batch inference workers.

Verification – Confirming the Fix

1. Functional test with a simulated client.

   import requests, time
   
   def sse_client(url, timeout=1.0):
       with requests.get(url, stream=True, timeout=timeout) as r:
           for line in r.iter_lines():
               if line:
                   print("Chunk:", line.decode())
                   # reset per‑chunk timeout
                   r.raw._fp.fp._sock.settimeout(timeout)
   
   start = time.time()
   sse_client("http://localhost:8000/generate", timeout=0.5)
   print("Total elapsed:", time.time() - start)
   

   Expected output: first token arrives < 0.5 s, subsequent tokens every < 0.2 s, total request completes without ReadTimeoutError.

2. Load test with hey or wrk to simulate 30 concurrent streams.

   # Using wrk2 for constant request rate
   wrk -t4 -c30 -d30s -R20 http://localhost:8000/generate
   

   Monitor uvicorn.error logs for any “StreamingGeneratorError”. No such entries should appear.

3. Metrics validation. Add a Prometheus gauge for “streamer_token_latency_seconds” and verify the 95th percentile stays below 0.4 s after the fix.

Operational Experience – Lessons Learned

• Initial assumption that the default 30 s keep‑alive was sufficient proved wrong; the client‑side library enforced a much stricter per‑chunk deadline.
• NGINX ingress time‑outs are often overlooked; a 500 ms proxy_read_timeout silently killed streams during GPU memory pressure spikes.
• Enabling a warm‑up call at service start reduced the first‑request TTFT by ~250 ms, eliminating the “first token timeout” pattern seen in the logs.
• Using a dedicated streaming worker prevented the “Gunicorn timeout” observed when a batch inference worker blocked on torch.cuda.synchronize().

Best Practices and Prevention

Area	Recommendation
Model Warm‑up	Run a dummy generation at startup; keep the CUDA context alive.
Streamer Configuration	Set timeout on TextStreamer to slightly above the client per‑chunk deadline.
Server Time‑outs	Increase --timeout-keep-alive and use multiple Uvicorn workers.
Ingress Settings	Align proxy_read_timeout and proxy_send_timeout with the worst‑case token interval plus safety margin.
Monitoring	Expose TTFT and per‑token latency metrics; alert if 95th percentile exceeds 400 ms.
Load Isolation	Separate streaming endpoints from batch inference endpoints (different process or container).

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

1. Why does the stream succeed in development but fail in production?

   Production typically runs behind an ingress controller with aggressive proxy_read_timeout (e.g., 500 ms). Development bypasses the ingress, so the timeout is not triggered.

2. Can increasing max_new_tokens cause time‑outs?

   Yes. A larger max_new_tokens increases total generation time; if the model stalls on any token, the per‑chunk deadline is still enforced, leading to a timeout.

3. Is the Streamer timeout parameter the same as the client‑side timeout?

   No. The internal Streamer.timeout controls how long the generator waits for the queue to be consumed; it must be set lower than the client‑side deadline to avoid blocking the queue.

4. Do I need to modify the FastAPI StreamingResponse to handle time‑outs?

   Only if you want custom error handling. The default behavior propagates the exception to the client; adding a try/except around the generator lets you return a structured error payload.

5. How can I measure time‑to‑first‑token accurately?

   Wrap the pipeline call with a high‑resolution timer (e.g., time.perf_counter()) and log the delta before the first chunk is yielded. Correlate this with GPU utilization to spot warm‑up delays.