Rows of black server racks with white logos in a data center

Weaviate text generation ignores stop sequence on edge node

by

Problem: Weaviate text generation ignores the configured stop sequence on edge nodes

When running the /v1/generate endpoint on resource‑constrained edge devices (e.g., ARM64 Jetson Nano, Raspberry Pi, or IoT gateway VMs), the LLM backend continues producing tokens past the user‑defined stop sequence. The symptom is a runaway response that quickly exhausts memory and CPU, often ending with an OOM kill.

Typical observable behavior:

  • Generated text exceeds the expected length (e.g., >2 KB when a \n\n stop token should truncate after 300 bytes).
  • Log entry: WARN generation: stop token not found in generated text, continuing until max_tokens reached
  • API response error: { "error": "generation_failed", "message": "Stop sequence not triggered" }
  • Container OOM kill trace: Killed process 1234 (weaviate) total-vm:...

Root Cause Analysis

The generative AI module in Weaviate relies on the underlying transformer model’s tokenization pipeline to detect stop sequences. On edge nodes the following factors combine to break this detection:

  1. Tokenization mismatch: ARM64 builds of the sentencepiece tokenizer (used by GPT‑Neo) have a known bug where certain multi‑character stop strings (e.g., \n\n or END) are split into separate tokens that never appear together in the generated stream. This was reported in GitHub Issue #3124.
  2. Hardware‑specific floating‑point rounding: Low‑precision inference on devices without AVX2 can cause the model to skip the exact token that would match the stop sequence, especially when the stop token is rare.
  3. Resource throttling: Edge deployment guide recommends limiting max_tokens to avoid OOM, but when the stop token is missed the generation falls back to the hard limit, which is often set too high for the device.
  4. Configuration propagation bug: In versions prior to 1.22.0, the stop field from the API request was not correctly forwarded to the native inference engine on ARM64 builds, as documented in the Weaviate API reference.

Collectively, these issues cause the generation loop to never see the stop token, leading to the observed runaway output.

Investigation and Debugging Steps

Follow this checklist on the affected edge node:

  1. Confirm the stop sequence is sent correctly:
curl -X POST http://localhost:8080/v1/generate \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-neo-2.7b",
        "prompt": "Explain the safety protocol:",
        "max_tokens": 512,
        "stop": ["\n\n"]
      }'

Check the request payload in Weaviate logs (DEBUG generation: received request) to ensure the stop array is present.

  1. Inspect the tokenization of the stop sequence on the edge runtime:
# Python REPL on the edge node
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-2.7B")
>>> tokenizer.encode("\n\n", add_special_tokens=False)
[13, 13]   # Expected two newline tokens
>>> tokenizer.encode("END", add_special_tokens=False)
[50257]   # Single token on x86, but on ARM64 may split

If the token list differs from the x86 reference, the tokenizer is the culprit.

  1. Check the model binary and inference library version:
weaviate --version
# Expected: 1.22.0
cat /opt/weaviate/lib/torch_version.txt
# Expected: torch==2.0.1+cpu

Older versions (< 1.21) are known to miss the stop‑token propagation on ARM64.

  1. Capture the generation stream to see where it diverges:
weaviate-cli generate \
  --model gpt-neo-2.7b \
  --prompt "Test stop token:" \
  --stop "\n\n" \
  --max-tokens 256 \
  --log-level debug > generation.log 2>&1

# In generation.log look for:
#   "generated_token: ..."
#   "stop_token_match: false"
  1. Validate system resources (memory pressure can cause the inference engine to skip token checks):
free -h
# Ensure at least 2 GB free RAM for GPT‑Neo on Jetson Nano

cat /sys/fs/cgroup/memory/memory.limit_in_bytes
# Verify cgroup limit matches deployment spec

Resolution

The fix consists of three coordinated actions:

1. Upgrade to a patched Weaviate release (≥ 1.22.1)

Version 1.22.1 includes a back‑ported fix that forces the stop token list through the native inference bridge on ARM64.

# On the edge node
docker pull semitechnologies/weaviate:1.22.1-arm64
docker stop weaviate && docker rm weaviate
docker run -d --name weaviate \
  -p 8080:8080 \
  -v /data/weaviate:/var/lib/weaviate \
  semitechnologies/weaviate:1.22.1-arm64

2. Apply a custom token filter to enforce stop‑token detection

If upgrading is not possible, inject a post‑generation filter that scans the raw text for the stop sequence and truncates it before returning to the client.

# Python wrapper around the generate endpoint
import json, requests

def generate_with_stop(prompt, stop_seq):
    resp = requests.post(
        "http://localhost:8080/v1/generate",
        json={"model":"gpt-neo-2.7b","prompt":prompt,"max_tokens":512,"stop":[stop_seq]}
    )
    data = resp.json()
    text = data.get("generated_text","")
    # Manual stop‑token enforcement
    idx = text.find(stop_seq)
    if idx != -1:
        text = text[:idx]
    return text

print(generate_with_stop("Explain safety:", "\n\n"))

3. Adjust deployment configuration to avoid resource‑driven token skipping

Set a conservative max_tokens that leaves headroom for the stop‑token check, and enable the tokenizer_cache flag to reduce CPU pressure.

# weaviate.yaml (edge deployment)
generation:
  max_tokens: 256          # lower than default 512
  tokenizer_cache: true    # enable caching
  stop_sequences:
    - "\n\n"

Validation

After applying the fixes, verify the behavior with both automated tests and manual checks.

Automated test script

#!/usr/bin/env bash
set -euo pipefail

PROMPT="List safety steps:"
STOP_SEQ="\n\n"
EXPECTED="1."

for i in {1..5}; do
  OUT=$(curl -s -X POST http://localhost:8080/v1/generate \
    -H "Content-Type: application/json" \
    -d "{\"model\":\"gpt-neo-2.7b\",\"prompt\":\"$PROMPT\",\"max_tokens\":256,\"stop\":[\"$STOP_SEQ\"]}")
  TEXT=$(echo "$OUT" | jq -r .generated_text)
  if [[ "$TEXT" == *"$STOP_SEQ"* ]]; then
    echo "FAIL: Stop sequence still present"
    exit 1
  fi
  if [[ ${#TEXT} -gt 1024 ]]; then
    echo "FAIL: Output too large (${#TEXT} bytes)"
    exit 1
  fi
  echo "PASS $i: $(echo "$TEXT" | head -c 60)..."
done

Manual verification

# Expected log line after fix
INFO  generation: stop token matched, truncating output

Also monitor memory usage:

watch -n 1 "docker stats weaviate --format '{{.MemUsage}}'"
# Memory should stay below 1.2 GB on Jetson Nano

Operational Experience & Lessons Learned

  • Misleading symptom: The API still returns a 200 OK with a generated_text field, leading engineers to assume success while the stop token was silently ignored.
  • Common incorrect assumption: “The same configuration works on the cloud, so it must be fine on edge.” Tokenizer binaries differ between x86_64 and ARM64, so stop‑token handling must be validated on the target architecture.
  • Edge‑specific edge case: When the device runs under aggressive cgroup memory limits, the inference engine may drop the final token check to meet the deadline, effectively bypassing stop‑token logic.
  • Lesson: Pin the Weaviate version in edge CI pipelines and include a sanity test that asserts a known stop token truncates output.

Best Practices and Prevention

  • Always run weaviate --version and compare against the generative AI module docs for the minimum ARM64‑compatible release.
  • Include a health‑check endpoint that generates a short prompt with a unique stop token (e.g., "STOPTEST") and verifies truncation.
  • Set max_tokens to no more than 75 % of the device’s available VRAM/CPU budget.
  • Enable tokenizer_cache and pre‑warm the model on startup to avoid runtime tokenization glitches.
  • Instrument metrics: generation_stop_token_hits, generation_oom_events, and alert when generation_stop_token_hits drops below a threshold.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

  1. Why does the stop sequence work on my cloud instance but not on the Jetson Nano?
    The cloud instance uses an x86_64 build of sentencepiece where the stop token maps to a single token ID. The Jetson Nano’s ARM64 build splits the same string into multiple IDs, causing the stop‑token matcher to miss it.
  2. Can I use a multi‑character stop token like \n\n on edge devices?
    Yes, but you must ensure the tokenizer treats the sequence as a single token or enable a custom post‑generation filter. Upgrading to Weaviate ≥ 1.22.1 resolves most multi‑character cases.
  3. What is the recommended max_tokens setting for low‑memory edge nodes?
    Keep max_tokens ≤ 256 for models larger than 1 B parameters on devices with ≤ 2 GB RAM. Adjust downward if you observe frequent OOM events.
  4. How do I confirm that the stop token is being recognized by the inference engine?
    Enable debug logging (LOG_LEVEL=debug) and look for the line generation: stop token matched in the container logs.
  5. Is there a way to avoid the tokenizer bug without upgrading Weaviate?
    Implement a client‑side truncation filter (as shown in the solution) or replace the default tokenizer with a custom sentencepiece model that explicitly defines the stop token as a single piece.