photo of computer cables

Weaviate streaming API incomplete results under high traffic

by

Weaviate Streaming API Incomplete Results Under High Traffic

Problem – Symptoms and Impact

During peak load the /v1/objects/{className}/{id}/stream endpoint returns truncated JSON payloads or the client receives a connection reset by peer error. Typical observations include:

  • Logs in Weaviate: 
    2024-05-12T14:23:07Z WARN streaming response incomplete: expected 12456 bytes, got 8421
  • API gateway (Kong) reports: 
    504 Gateway Timeout – upstream timed out after 30s
  • NGINX error log: 
    2024/05/12 14:23:08 [error] 112#112: *3456 upstream sent unexpected EOF while reading response header from upstream, client: 10.0.0.12, server: weaviate.example.com, request: "GET /v1/objects/Article/123/stream HTTP/2.0"
  • Client SDKs raise ReadTimeoutError: streaming response timed out.

The effect is partial search results, broken client UI components, and increased retry traffic, amplifying the load problem.

Root Cause Analysis

The streaming endpoint uses HTTP/2 server push with a long-lived response body. Under high concurrency the following chain of events can cause premature termination:

  1. API gateway timeout – Kong’s default proxy_read_timeout is 30 seconds (see GitHub issue #1589). When a vector search takes longer, Kong closes the upstream connection, emitting a GOAWAY frame. Weaviate logs “HTTP/2 GOAWAY received”.
  2. Proxy buffer limits – NGINX with default proxy_buffer_size 4k cannot hold the streaming payload during spikes (see Company Y incident). The buffer overflows, causing NGINX to truncate the response and log “upstream sent unexpected EOF”.
  3. Keep‑alive and idle timeout mismatches – Envoy’s idle_timeout 15s (Open‑source project Z) forces a GOAWAY after 15 seconds of inactivity, cutting off streams that are still emitting vectors.
  4. Weaviate connection limits – The default max_open_connections (see Weaviate Scalability guide) is 100. When 500+ concurrent streams are opened, excess connections are rejected, leading to “connection reset by peer”.

All these factors converge on the same symptom: the upstream proxy or Weaviate itself terminates the HTTP/2 stream before the full payload is delivered.

Investigation and Debugging Steps

Follow this checklist to isolate the component responsible:

  1. Collect end‑to‑end logs 
    # Weaviate logs (JSON format)
journalctl -u weaviate -f | grep streaming
# Example excerpt
{"ts":"2024-05-12T14:23:07Z","lvl":"WARN","msg":"streaming response incomplete: expected 12456 bytes, got 8421","class":"Article","id":"123"}
  2. Inspect API gateway metrics 
    # Kong admin API
curl -s http://localhost:8001/services/weaviate/metrics | grep proxy_read_timeout
# NGINX status
curl -s http://localhost/nginx_status
  3. Capture HTTP/2 frames (requires tcpdump and nghttp2 tools) 
    tcpdump -i eth0 -w weaviate.pcap port 443
nghttp -v -d weaviate.pcap

    Look for GOAWAY frames with error code NO_ERROR or ENHANCE_YOUR_CALM.

  4. Check proxy buffer statistics 
    # NGINX
cat /var/log/nginx/error.log | grep "proxy_buffer"
# Expected line when overflow occurs
2024/05/12 14:23:08 [error] 112#112: *3456 upstream sent unexpected EOF while reading response header from upstream
  5. Validate Weaviate connection limits 
    # Weaviate config dump
curl -s http://localhost:8080/v1/.well-known/openid-configuration | jq .max_open_connections
# Default is 100

Resolution – Configuration and Deployment Changes

Apply the following adjustments. Each block shows the configuration before and after the change.

1. Increase API gateway timeouts

Kong

# Before (default)
proxy_read_timeout = 30s

# After
proxy_read_timeout = 120s
proxy_send_timeout = 120s

Reason: Allows long‑running vector searches to finish without the gateway closing the upstream connection.

2. Raise NGINX proxy buffer sizes

# Before (default)
proxy_buffer_size 4k;
proxy_buffers 8 4k;

# After
proxy_buffer_size 64k;
proxy_buffers 16 64k;
proxy_busy_buffers_size 128k;

Reason: Prevents buffer overflow that truncates streaming payloads (see Company Y incident).

3. Align Envoy keep‑alive settings with Weaviate

# Before
idle_timeout: 15s

# After
idle_timeout: 300s
max_connection_duration: 0s   # disable forced termination

Reason: Avoids premature GOAWAY frames that cut off active streams.

4. Increase Weaviate max open connections

# weaviate.conf (YAML)
# Before
max_open_connections: 100

# After
max_open_connections: 1000

Reason: Accommodates the 500+ concurrent streams observed during load tests (FinTech startup incident).

5. Enable HTTP/2 flow control tuning (optional)

# weaviate.conf
http2:
  initial_window_size: 65535   # default 64KB
  max_frame_size: 16384        # increase if large vectors are streamed

Reason: Reduces the chance of flow‑control stalls under heavy load.

Verification – How to Confirm the Fix

  1. Run a load test with hey or k6 targeting the streaming endpoint, e.g.: 
    hey -c 500 -n 5000 "https://weaviate.example.com/v1/objects/Article/123/stream"

    Observe zero “connection reset by peer” or “streaming response incomplete” messages in Weaviate logs.

  2. Check gateway metrics for timeout counters; they should remain at zero during the test.
  3. Validate full payload size: 
    curl -s https://weaviate.example.com/v1/objects/Article/123/stream | wc -c
# Should match the expected byte count reported by Weaviate metadata endpoint
  4. Confirm no GOAWAY frames in a fresh packet capture: 
    nghttp -v -d weaviate.pcap | grep GOAWAY
# No output indicates streams remain open until completion

Prevention – Operational Best Practices

  • Monitor gateway timeout metrics (Kong proxy_read_timeout_total, NGINX proxy_timeout) and set alerts for spikes.
  • Track Weaviate connection usage via /metrics endpoint (weaviate_http_connections_open).
  • Configure health checks that verify a complete streaming response (e.g., a synthetic request that expects a known byte count).
  • Capacity plan for peak concurrency by running load tests that exceed expected traffic by 20‑30 % and adjusting max_open_connections accordingly.
  • Align keep‑alive and idle timeout values across all proxies (Kong, NGINX, Envoy) and Weaviate to avoid mismatched closures.

FAQ – Related Questions

  1. Why does the streaming API work in dev but fail in production?
    Production uses an API gateway (Kong/NGINX) with default timeout and buffer settings that are too low for long‑running vector searches, whereas the dev environment hits Weaviate directly.
  2. Can I disable HTTP/2 and fall back to HTTP/1.1 to avoid GOAWAY?
    Disabling HTTP/2 removes flow‑control benefits and may increase latency. The recommended approach is to tune timeouts and keep‑alive rather than disabling HTTP/2.
  3. What metric should I watch to detect upcoming streaming truncation?
    Watch weaviate_http_stream_errors_total and the gateway’s proxy_read_timeout_total. A rising trend indicates impending truncation.
  4. Is increasing max_open_connections enough for all workloads?
    Only if the underlying OS limits (ulimit nofile) are also raised. Verify with ulimit -n and adjust the systemd service file accordingly.
  5. Do client SDKs need any changes after fixing the gateway?
    No. The SDKs already handle HTTP/2 streams correctly; they only need the server side to keep the connection alive for the full duration of the response.

Related Topic Hub: Vector Databases Troubleshooting Hub