Weaviate Streaming Query Never Terminates After Stop Sequence Trigger Fails

Q: 3. Configuration mismatch after schema change

A schema migration inadvertently reset streaming.stopSequence to its default empty value. With no token configured, the handler never emits the termination frame, as observed in the production incident at the fintech firm. The server logs the warning “stopSequence not matched” but does not indicate that the token is undefined.

Q: Investigation and Debugging Steps Step 1 – Verify server configuration

weaviate-cli config get streaming # Expected output: streaming.maxChunkSize: 8192 streaming.stopSequence: "\n\n" If stopSequence is empty or missing, the issue is likely a config regression.

Q: Step 2 – Capture gRPC frames

Use grpcurl with the -proto flag and -vv to dump raw frames:

Q: Step 3 – Review metrics

Check the grpc.server.StreamsActive gauge and the streaming.handler.errors counter:

In this article

Table of Contents

Problem – Streaming queries never terminate after stop‑sequence trigger fails

In a high‑throughput real‑time pipeline that ingests and queries vectors via Weaviate’s gRPC API, clients observed that streaming GraphQL queries kept the connection open indefinitely. The expected stop sequence (e.g., \n\n) never caused the server to close the stream, resulting in:

Growing memory usage on the Weaviate nodes (OOM in a fintech fraud‑detection deployment).
Stalled grpc.server.StreamsActive metrics and back‑pressure on upstream services.
Client‑side errors such as rpc error: code = DeadlineExceeded desc = context deadline exceeded after the client‑side timeout.

Typical log excerpt from the affected nodes:


2024-09-10T12:34:56Z WARN streaming: stopSequence not matched, bytesRead=128
2024-09-10T12:34:56Z WARN streaming: client connection still open
2024-09-10T12:35:02Z ERROR streaming: unexpected EOF while waiting for stop token

Root Cause Analysis

1. How Weaviate processes stop sequences

According to the Weaviate GraphQL streaming guide, each streamed response is buffered until the server detects the configured streaming.stopSequence token. The detection logic works on a per‑chunk basis, using the streaming.maxChunkSize setting to accumulate bytes before scanning for the token.

2. Interaction with gRPC framing under load

The gRPC transport splits payloads into frames. Under high QPS (10 k QPS in the load‑test incident), the stop‑sequence bytes can be split across two frames. The server’s streaming_handler.go scans only the current chunk; if the token straddles a chunk boundary, the match fails and the stream never terminates. This behavior was discussed in GitHub issue #3124 (“gRPC streaming hangs under load – stop token not propagated”).

3. Configuration mismatch after schema change

A schema migration inadvertently reset streaming.stopSequence to its default empty value. With no token configured, the handler never emits the termination frame, as observed in the production incident at the fintech firm. The server logs the warning “stopSequence not matched” but does not indicate that the token is undefined.

Investigation and Debugging Steps

Step 1 – Verify server configuration


weaviate-cli config get streaming
# Expected output:
streaming.maxChunkSize: 8192
streaming.stopSequence: "\n\n"

If stopSequence is empty or missing, the issue is likely a config regression.

Step 2 – Capture gRPC frames

Use grpcurl with the -proto flag and -vv to dump raw frames:


grpcurl -vv -proto weaviate.proto \
  -d '{"query":"{ Get { Things { ... } } }"}' \
  localhost:8080 weaviate.v1.Weaviate/GraphQLStream

Inspect the output for the stop token split across frames:


... Frame 1 payload: "...data...\\n"
... Frame 2 payload: "\\n...more data..."

Step 3 – Review metrics

Check the grpc.server.StreamsActive gauge and the streaming.handler.errors counter:


curl -s http://localhost:2112/metrics | grep streaming
# Example output:
streaming_handler_errors_total{reason="stopSequenceNotMatched"} 42
grpc_server_streams_active 128

Step 4 – Reproduce with a minimal client

Run a single‑threaded Go client that forces the stop token to be sent in one chunk:


package main

import (
    "context"
    "io"
    "log"
    "time"

    pb "github.com/weaviate/weaviate/v1"
    "google.golang.org/grpc"
)

func main() {
    conn, err := grpc.Dial("localhost:8080", grpc.WithInsecure())
    if err != nil { log.Fatalf("dial: %v", err) }
    client := pb.NewWeaviateClient(conn)

    stream, err := client.GraphQLStream(context.Background(),
        &pb.GraphQLRequest{Query: "{ Get { Things { ... } } }"})
    if err != nil { log.Fatalf("stream init: %v", err) }

    // Force stop token in a single write
    _, _ = stream.Send(&pb.StreamChunk{Data: []byte("...payload...\n\n")})

    for {
        resp, err := stream.Recv()
        if err == io.EOF {
            log.Println("stream closed as expected")
            break
        }
        if err != nil {
            log.Fatalf("recv error: %v", err)
        }
        log.Printf("chunk: %s", resp.Data)
    }
}

If this client terminates correctly, the problem is not the server logic itself but the interaction with high‑throughput framing.

Resolution – Making stop‑sequence detection reliable

1. Adjust `streaming.maxChunkSize` to accommodate token fragmentation

Increase the chunk size so that the stop token is less likely to be split. The official docs recommend a size at least twice the token length.


# Before (default)
streaming.maxChunkSize: 8192
streaming.stopSequence: "\n\n"

# After
streaming.maxChunkSize: 16384
streaming.stopSequence: "\n\n"

2. Enable token‑spanning detection (available from v1.23.0)

Upgrade to Weaviate ≥ 1.23.0 where the streaming handler was patched to retain the last len(stopSequence)-1 bytes between chunks. This change is described in GitHub issue #2879.

3. Explicitly set the stop sequence after schema changes

Add the configuration to the weaviate.conf.yaml or via environment variables to avoid accidental resets:


# weaviate.conf.yaml
streaming:
  maxChunkSize: 16384
  stopSequence: "\n\n"

Or as env vars (Docker/K8s example):


WEAVIATE_STREAMING_MAXCHUNKSIZE=16384
WEAVIATE_STREAMING_STOPSEQUENCE="\n\n"

4. Client‑side buffering fix

For languages that buffer gRPC writes (e.g., Java gRPC), ensure the stop token is flushed immediately:


streamObserver.onNext(
    StreamChunk.newBuilder()
        .setData(ByteString.copyFromUtf8(payload + "\n\n"))
        .build());
streamObserver.onCompleted(); // forces flush

5. Restart the cluster after applying changes

Rolling restart each node to pick up the new configuration without downtime.

Validation – Confirming that streams now terminate

Run the same load‑test that previously hung (10 k QPS, 5‑node cluster).
Observe that grpc.server.StreamsActive returns to baseline (< 10) after each query.
Check logs for the new informational line:


2024-09-11T08:15:23Z INFO streaming: stopSequenceReceived, stream closed gracefully

Run a client‑side sanity check:


grpcurl -vv -proto weaviate.proto \
  -d '{"query":"{ Get { Things { ... } } }"}' \
  localhost:8080 weaviate.v1.Weaviate/GraphQLStream
# Expected: final frame contains "stopSequence" and connection closes with status OK.

Operational Experience – Lessons Learned

Misleading symptom: The server logs only warned about “stopSequence not matched” without indicating that the token was split across frames. Correlating this with high grpc.server.StreamsActive was key.
Common incorrect assumption: Setting a stop token once during initial deployment is sufficient. In practice, schema migrations or rolling upgrades can reset streaming defaults.
Production edge case: Network jitter on the client side caused the stop token bytes to be delivered in separate TCP packets, which gRPC then turned into separate frames. The server’s original implementation discarded the trailing bytes of a chunk, breaking detection.
Performance impact: Increasing maxChunkSize modestly (e.g., from 8 KB to 16 KB) added < 1 % latency while dramatically improving reliability under load.

Best Practices and Prevention

Always pin streaming.stopSequence in configuration files or environment variables; treat it as a required setting.
Monitor streaming_handler_errors_total{reason="stopSequenceNotMatched"} and set an alert threshold (e.g., > 5 per minute).
Enable the “token‑spanning” feature introduced in v1.23.0; verify the binary version with weaviate --version.
When deploying new client libraries, add integration tests that send the stop token in a fragmented manner to ensure the server handles it.
Keep streaming.maxChunkSize at least twice the length of the stop token and consider a higher value for high‑throughput workloads.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Why does the stop sequence work in dev but not in production? Production often runs with higher QPS, causing gRPC frames to split the token. Dev environments usually send the whole payload in a single frame, so the bug is hidden.
Can I use a custom stop token (e.g., “---END---”)? Yes. Set streaming.stopSequence to the desired string, ensure maxChunkSize is large enough, and verify that the token does not appear elsewhere in the data.
What error does the client see when the server never sends the termination frame? Typically rpc error: code = DeadlineExceeded desc = context deadline exceeded or a prolonged io.EOF after the client’s timeout.
Do I need to upgrade the client library to fix this? Not necessarily. The server‑side fix (token‑spanning detection) resolves the core issue, but some client libraries may need a flush call to ensure the stop token is sent in one write.
How can I test that the stop token is not being split? Capture the raw gRPC frames with grpcurl -vv or tcpdump -i any -s 0 -w capture.pcap port 8080 and inspect the payload boundaries for the token.